From buildbot at python.org Sat Apr 1 00:30:36 2006 From: buildbot at python.org (buildbot at python.org) Date: Fri, 31 Mar 2006 22:30:36 +0000 Subject: [Python-checkins] buildbot warnings in g4 osx.4 trunk Message-ID: <20060331223036.6A3011E4003@bag.python.org> The Buildbot has detected a new failure of g4 osx.4 trunk. Full details are available at: http://www.python.org/dev/buildbot/all/g4%20osx.4%20trunk/builds/212 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: georg.brandl Build Had Warnings: warnings test sincerely, -The Buildbot From buildbot at python.org Sat Apr 1 00:35:38 2006 From: buildbot at python.org (buildbot at python.org) Date: Fri, 31 Mar 2006 22:35:38 +0000 Subject: [Python-checkins] buildbot warnings in alpha Tru64 5.1 2.4 Message-ID: <20060331223538.F24B51E4003@bag.python.org> The Buildbot has detected a new failure of alpha Tru64 5.1 2.4. Full details are available at: http://www.python.org/dev/buildbot/all/alpha%20Tru64%205.1%202.4/builds/10 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch branches/release24-maint] HEAD Blamelist: georg.brandl Build Had Warnings: warnings test sincerely, -The Buildbot From python-checkins at python.org Sat Apr 1 00:38:21 2006 From: python-checkins at python.org (thomas.wouters) Date: Sat, 1 Apr 2006 00:38:21 +0200 (CEST) Subject: [Python-checkins] r43510 - python/trunk/Modules/posixmodule.c Message-ID: <20060331223821.18D141E4003@bag.python.org> Author: thomas.wouters Date: Sat Apr 1 00:38:19 2006 New Revision: 43510 Modified: python/trunk/Modules/posixmodule.c Log: In the fdopen(fd, 'a') case on UNIX, don't try to set fd's flags to -1 if fcntl() and fdopen() both fail. Will backport. Modified: python/trunk/Modules/posixmodule.c ============================================================================== --- python/trunk/Modules/posixmodule.c (original) +++ python/trunk/Modules/posixmodule.c Sat Apr 1 00:38:19 2006 @@ -5777,7 +5777,7 @@ if (flags != -1) fcntl(fd, F_SETFL, flags | O_APPEND); fp = fdopen(fd, mode); - if (fp == NULL) + if (fp == NULL && flags != -1) /* restore old mode if fdopen failed */ fcntl(fd, F_SETFL, flags); } else { From neal at metaslash.com Sat Apr 1 00:32:10 2006 From: neal at metaslash.com (Neal Norwitz) Date: Fri, 31 Mar 2006 17:32:10 -0500 Subject: [Python-checkins] Python Regression Test Failures all (1) Message-ID: <20060331223210.GA7681@python.psfb.org> test_grammar test_opcodes test_operations test_builtin test_exceptions test_types test_MimeWriter test_StringIO test___all__ test___future__ test__locale test_aepack test_aepack skipped -- No module named aepack test_al test_al skipped -- No module named al test_anydbm test_applesingle test_applesingle skipped -- No module named macostools test_array test_ast test_asynchat test_atexit test_audioop test_augassign test_base64 test_bastion test_binascii test_binhex test_binop test_bisect test_bool test_bsddb test_bsddb185 test_bsddb185 skipped -- No module named bsddb185 test_bsddb3 Exception in thread reader 0: Traceback (most recent call last): File "/home/neal/python/trunk/Lib/threading.py", line 467, in __bootstrap self.run() File "/home/neal/python/trunk/Lib/threading.py", line 447, in run self.__target(*self.__args, **self.__kwargs) File "/home/neal/python/trunk/Lib/bsddb/test/test_thread.py", line 275, in readerThread rec = dbutils.DeadlockWrap(c.next, max_retries=10) File "/home/neal/python/trunk/Lib/bsddb/dbutils.py", line 62, in DeadlockWrap return function(*_args, **_kwargs) DBLockDeadlockError: (-30996, 'DB_LOCK_DEADLOCK: Locker killed to resolve a deadlock') Exception in thread reader 3: Traceback (most recent call last): File "/home/neal/python/trunk/Lib/threading.py", line 467, in __bootstrap self.run() File "/home/neal/python/trunk/Lib/threading.py", line 447, in run self.__target(*self.__args, **self.__kwargs) File "/home/neal/python/trunk/Lib/bsddb/test/test_thread.py", line 275, in readerThread rec = dbutils.DeadlockWrap(c.next, max_retries=10) File "/home/neal/python/trunk/Lib/bsddb/dbutils.py", line 62, in DeadlockWrap return function(*_args, **_kwargs) DBLockDeadlockError: (-30996, 'DB_LOCK_DEADLOCK: Locker killed to resolve a deadlock') test_bufio test_bz2 test_cProfile test_calendar test_call test_capi test_cd test_cd skipped -- No module named cd test_cfgparser test_cgi test_charmapcodec test_cl test_cl skipped -- No module named cl test_class test_cmath test_cmd_line test_code test_codeccallbacks test_codecencodings_cn test_codecencodings_hk test_codecencodings_jp test_codecencodings_kr test_codecencodings_tw test_codecmaps_cn test_codecmaps_hk test_codecmaps_jp test_codecmaps_kr test_codecmaps_tw test_codecs test_codeop test_coding test_coercion test_colorsys test_commands test_compare test_compile test_compiler test_complex test_contains test_contextlib test_cookie test_cookielib test_copy test_copy_reg test_cpickle test_crypt test_csv test_ctypes test_datetime test_dbm test_decimal test_decorators test_defaultdict test_deque test_descr test_descrtut test_dict test_difflib test_dircache test_dis test_distutils test_dl test_doctest test_doctest2 test_dumbdbm test_dummy_thread test_dummy_threading test_email test_email_codecs test_email_renamed test_enumerate test_eof test_errno test_exception_variations test_extcall test_fcntl test_file test_filecmp test_fileinput test_float test_fnmatch test_fork1 test_format test_fpformat test_frozen test_funcattrs test_functional test_future test_gc test_gdbm test_generators test_genexps test_getargs test_getargs2 test_getopt test_gettext test_gl test_gl skipped -- No module named gl test_glob test_global test_grp test_gzip test_hash test_hashlib test_heapq test_hexoct test_hmac test_hotshot test_htmllib test_htmlparser test_httplib test_imageop test_imaplib test_imgfile test_imgfile skipped -- No module named imgfile test_imp test_import test_importhooks test_index test_inspect test_ioctl test_ioctl skipped -- Unable to open /dev/tty test_isinstance test_iter test_iterlen test_itertools test_largefile test_list test_locale test_logging test_long test_long_future test_longexp test_macfs test_macfs skipped -- No module named macfs test_macostools test_macostools skipped -- No module named macostools test_macpath test_mailbox test_marshal test_math test_md5 test_mhlib test_mimetools test_mimetypes test_minidom test_mmap test_module test_multibytecodec test_multibytecodec_support test_multifile test_mutants test_netrc test_new test_nis test_nis skipped -- Local domain name not set test_normalization test_ntpath test_openpty test_operator test_optparse test_os test_parser test_peepholer test_pep247 test_pep263 test_pep277 test_pep277 skipped -- test works only on NT+ test_pep292 test_pep352 test_pickle test_pickletools test_pkg test_pkgimport test_platform test_plistlib test_plistlib skipped -- No module named plistlib test_poll test_popen [9898 refs] [9898 refs] [9898 refs] test_popen2 test_posix test_posixpath test_pow test_pprint test_profile test_profilehooks test_pty test_pwd test_pyclbr test_pyexpat test_queue test_quopri [10984 refs] [10984 refs] test_random test_re test_repr test_resource test_rfc822 test_rgbimg test_richcmp test_robotparser test_runpy test_sax test_scope test_scriptpackages test_scriptpackages skipped -- No module named aetools test_select test_set test_sets test_sgmllib test_sha test_shelve test_shlex test_shutil test_signal test_site test_slice test_socket test_socket_ssl test test_socket_ssl crashed -- : timed out test_socketserver test_softspace test_sort test_str test_strftime test_string test_stringprep test_strop test_strptime test_struct test_structseq test_subprocess [9893 refs] [9895 refs] [9893 refs] [9893 refs] [9893 refs] [9893 refs] [9893 refs] [9894 refs] [9894 refs] [9893 refs] [9894 refs] [9893 refs] [10110 refs] [9894 refs] [9894 refs] [9894 refs] [9894 refs] [9894 refs] [9894 refs] [9894 refs] this bit of output is from a test of stdout in a different process ... [9894 refs] [9893 refs] [10110 refs] test_sunaudiodev test_sunaudiodev skipped -- No module named sunaudiodev test_sundry test_symtable test_syntax test_sys [9893 refs] [9893 refs] test_tarfile test_tcl test_tcl skipped -- No module named _tkinter test_tempfile [9895 refs] test_textwrap test_thread test_threaded_import test_threadedtempfile test_threading test_threading_local test_threadsignals test_time test_timeout test_tokenize test_trace test_traceback test_transformer test_tuple test_ucn test_unary test_unicode test_unicode_file test_unicode_file skipped -- No Unicode filesystem semantics on this platform. test_unicodedata test_unittest test_univnewlines test_unpack test_urllib test_urllib2 test_urllib2net test_urllibnet test_urlparse test_userdict test_userlist test_userstring test_uu test_wait3 test_wait4 test_warnings test_wave test_weakref test_whichdb test_winreg test_winreg skipped -- No module named _winreg test_winsound test_winsound skipped -- No module named winsound test_with test_xdrlib test_xml_etree test_xml_etree_c test_xmllib test_xmlrpc test_xpickle test_xrange test_zipfile test_zipimport test_zlib 287 tests OK. 1 test failed: test_socket_ssl 20 tests skipped: test_aepack test_al test_applesingle test_bsddb185 test_cd test_cl test_gl test_imgfile test_ioctl test_macfs test_macostools test_nis test_pep277 test_plistlib test_scriptpackages test_sunaudiodev test_tcl test_unicode_file test_winreg test_winsound 1 skip unexpected on linux2: test_ioctl [415265 refs] From python-checkins at python.org Sat Apr 1 00:39:56 2006 From: python-checkins at python.org (thomas.wouters) Date: Sat, 1 Apr 2006 00:39:56 +0200 (CEST) Subject: [Python-checkins] r43511 - python/branches/release24-maint/Modules/posixmodule.c Message-ID: <20060331223956.0530E1E4003@bag.python.org> Author: thomas.wouters Date: Sat Apr 1 00:39:56 2006 New Revision: 43511 Modified: python/branches/release24-maint/Modules/posixmodule.c Log: Backport trunk's r43510: In the fdopen(fd, 'a') case on UNIX, don't try to set fd's flags to -1 if fcntl() and fdopen() both fail. Modified: python/branches/release24-maint/Modules/posixmodule.c ============================================================================== --- python/branches/release24-maint/Modules/posixmodule.c (original) +++ python/branches/release24-maint/Modules/posixmodule.c Sat Apr 1 00:39:56 2006 @@ -5467,7 +5467,7 @@ if (flags != -1) fcntl(fd, F_SETFL, flags | O_APPEND); fp = fdopen(fd, mode); - if (fp == NULL) + if (fp == NULL && flags != -1) /* restore old mode if fdopen failed */ fcntl(fd, F_SETFL, flags); } else { From buildbot at python.org Sat Apr 1 01:00:18 2006 From: buildbot at python.org (buildbot at python.org) Date: Fri, 31 Mar 2006 23:00:18 +0000 Subject: [Python-checkins] buildbot warnings in amd64 gentoo 2.4 Message-ID: <20060331230019.102671E4003@bag.python.org> The Buildbot has detected a new failure of amd64 gentoo 2.4. Full details are available at: http://www.python.org/dev/buildbot/all/amd64%20gentoo%202.4/builds/54 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch branches/release24-maint] HEAD Blamelist: thomas.wouters Build Had Warnings: warnings test sincerely, -The Buildbot From buildbot at python.org Sat Apr 1 01:02:02 2006 From: buildbot at python.org (buildbot at python.org) Date: Fri, 31 Mar 2006 23:02:02 +0000 Subject: [Python-checkins] buildbot warnings in amd64 gentoo trunk Message-ID: <20060331230203.64BB81E4003@bag.python.org> The Buildbot has detected a new failure of amd64 gentoo trunk. Full details are available at: http://www.python.org/dev/buildbot/all/amd64%20gentoo%20trunk/builds/242 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: thomas.wouters Build Had Warnings: warnings test sincerely, -The Buildbot From buildbot at python.org Sat Apr 1 01:28:45 2006 From: buildbot at python.org (buildbot at python.org) Date: Fri, 31 Mar 2006 23:28:45 +0000 Subject: [Python-checkins] buildbot warnings in x86 gentoo trunk Message-ID: <20060331232846.1BFCD1E4003@bag.python.org> The Buildbot has detected a new failure of x86 gentoo trunk. Full details are available at: http://www.python.org/dev/buildbot/all/x86%20gentoo%20trunk/builds/237 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: thomas.wouters Build Had Warnings: warnings test sincerely, -The Buildbot From buildbot at python.org Sat Apr 1 01:29:01 2006 From: buildbot at python.org (buildbot at python.org) Date: Fri, 31 Mar 2006 23:29:01 +0000 Subject: [Python-checkins] buildbot warnings in x86 gentoo 2.4 Message-ID: <20060331232901.66E711E4003@bag.python.org> The Buildbot has detected a new failure of x86 gentoo 2.4. Full details are available at: http://www.python.org/dev/buildbot/all/x86%20gentoo%202.4/builds/54 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch branches/release24-maint] HEAD Blamelist: thomas.wouters Build Had Warnings: warnings test sincerely, -The Buildbot From buildbot at python.org Sat Apr 1 01:38:25 2006 From: buildbot at python.org (buildbot at python.org) Date: Fri, 31 Mar 2006 23:38:25 +0000 Subject: [Python-checkins] buildbot warnings in g4 osx.4 2.4 Message-ID: <20060331233825.2C4EC1E4003@bag.python.org> The Buildbot has detected a new failure of g4 osx.4 2.4. Full details are available at: http://www.python.org/dev/buildbot/all/g4%20osx.4%202.4/builds/51 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch branches/release24-maint] HEAD Blamelist: thomas.wouters Build Had Warnings: warnings test sincerely, -The Buildbot From python-checkins at python.org Sat Apr 1 02:26:54 2006 From: python-checkins at python.org (tim.peters) Date: Sat, 1 Apr 2006 02:26:54 +0200 (CEST) Subject: [Python-checkins] r43512 - in python/trunk: Lib/random.py Lib/test/test_random.py Misc/NEWS Message-ID: <20060401002654.9CD1A1E4003@bag.python.org> Author: tim.peters Date: Sat Apr 1 02:26:53 2006 New Revision: 43512 Modified: python/trunk/Lib/random.py python/trunk/Lib/test/test_random.py python/trunk/Misc/NEWS Log: Another crack at bug #1460340: make random.sample(dict) work, this time by ugly brute force. Modified: python/trunk/Lib/random.py ============================================================================== --- python/trunk/Lib/random.py (original) +++ python/trunk/Lib/random.py Sat Apr 1 02:26:53 2006 @@ -285,6 +285,15 @@ large population: sample(xrange(10000000), 60) """ + # XXX Although the documentation says `population` is "a sequence", + # XXX attempts are made to cater to any iterable with a __len__ + # XXX method. This has had mixed success. Examples from both + # XXX sides: sets work fine, and should become officially supported; + # XXX dicts are much harder, and have failed in various subtle + # XXX ways across attempts. Support for mapping types should probably + # XXX be dropped (and users should pass mapping.keys() or .values() + # XXX explicitly). + # Sampling without replacement entails tracking either potential # selections (the pool) in a list or previous selections in a set. @@ -304,7 +313,9 @@ setsize = 21 # size of a small set minus size of an empty list if k > 5: setsize += 4 ** _ceil(_log(k * 3, 4)) # table size for big sets - if n <= setsize: # is an n-length list smaller than a k-length set + if n <= setsize or hasattr(population, "keys"): + # An n-length list is smaller than a k-length set, or this is a + # mapping type so the other algorithm wouldn't work. pool = list(population) for i in xrange(k): # invariant: non-selected at [0,n-i) j = _int(random() * (n-i)) @@ -320,10 +331,10 @@ j = _int(random() * n) selected_add(j) result[i] = population[j] - except (TypeError, KeyError): # handle sets and dictionaries + except (TypeError, KeyError): # handle (at least) sets if isinstance(population, list): raise - return self.sample(list(population), k) + return self.sample(tuple(population), k) return result ## -------------------- real-valued distributions ------------------- Modified: python/trunk/Lib/test/test_random.py ============================================================================== --- python/trunk/Lib/test/test_random.py (original) +++ python/trunk/Lib/test/test_random.py Sat Apr 1 02:26:53 2006 @@ -93,12 +93,28 @@ self.gen.sample(set(range(20)), 2) self.gen.sample(range(20), 2) self.gen.sample(xrange(20), 2) - self.gen.sample(dict.fromkeys('abcdefghijklmnopqrst'), 2) self.gen.sample(str('abcdefghijklmnopqrst'), 2) self.gen.sample(tuple('abcdefghijklmnopqrst'), 2) + + def test_sample_on_dicts(self): + self.gen.sample(dict.fromkeys('abcdefghijklmnopqrst'), 2) + # SF bug #1460340 -- random.sample can raise KeyError a = dict.fromkeys(range(10)+range(10,100,2)+range(100,110)) - self.gen.sample(a,3) + self.gen.sample(a, 3) + + # A followup to bug #1460340: sampling from a dict could return + # a subset of its keys or of its values, depending on the size of + # the subset requested. + N = 30 + d = dict((i, complex(i, i)) for i in xrange(N)) + for k in xrange(N+1): + samp = self.gen.sample(d, k) + # Verify that we got ints back (keys); the values are complex. + for x in samp: + self.assert_(type(x) is int) + samp.sort() + self.assertEqual(samp, range(N)) def test_gauss(self): # Ensure that the seed() method initializes all the hidden state. In Modified: python/trunk/Misc/NEWS ============================================================================== --- python/trunk/Misc/NEWS (original) +++ python/trunk/Misc/NEWS Sat Apr 1 02:26:53 2006 @@ -489,6 +489,11 @@ Library ------- +- Bug #1460340: ``random.sample(dict)`` failed in various ways. Dicts + aren't officially supported here, and trying to use them will probably + raise an exception some day. But dicts have been allowed, and "mostly + worked", so support for them won't go away without warning. + - Bug #1445068: getpass.getpass() can now be given an explicit stream argument to specify where to write the prompt. From python-checkins at python.org Sat Apr 1 02:41:11 2006 From: python-checkins at python.org (tim.peters) Date: Sat, 1 Apr 2006 02:41:11 +0200 (CEST) Subject: [Python-checkins] r43513 - in python/branches/release24-maint: Lib/random.py Lib/test/test_random.py Misc/NEWS Message-ID: <20060401004111.D78E01E4014@bag.python.org> Author: tim.peters Date: Sat Apr 1 02:41:10 2006 New Revision: 43513 Modified: python/branches/release24-maint/Lib/random.py python/branches/release24-maint/Lib/test/test_random.py python/branches/release24-maint/Misc/NEWS Log: Merge rev 43511 from the trunk. Another crack at bug #1460340: make random.sample(dict) work, this time by ugly brute force. Modified: python/branches/release24-maint/Lib/random.py ============================================================================== --- python/branches/release24-maint/Lib/random.py (original) +++ python/branches/release24-maint/Lib/random.py Sat Apr 1 02:41:10 2006 @@ -285,6 +285,15 @@ large population: sample(xrange(10000000), 60) """ + # XXX Although the documentation says `population` is "a sequence", + # XXX attempts are made to cater to any iterable with a __len__ + # XXX method. This has had mixed success. Examples from both + # XXX sides: sets work fine, and should become officially supported; + # XXX dicts are much harder, and have failed in various subtle + # XXX ways across attempts. Support for mapping types should probably + # XXX be dropped (and users should pass mapping.keys() or .values() + # XXX explicitly). + # Sampling without replacement entails tracking either potential # selections (the pool) in a list or previous selections in a # dictionary. @@ -302,7 +311,9 @@ random = self.random _int = int result = [None] * k - if n < 6 * k: # if n len list takes less space than a k len dict + if n < 6 * k or hasattr(population, "keys"): + # An n-length list is smaller than a k-length set, or this is a + # mapping type so the other algorithm wouldn't work. pool = list(population) for i in xrange(k): # invariant: non-selected at [0,n-i) j = _int(random() * (n-i)) @@ -316,10 +327,10 @@ while j in selected: j = _int(random() * n) result[i] = selected[j] = population[j] - except (TypeError, KeyError): # handle sets and dictionaries + except (TypeError, KeyError): # handle (at least) sets if isinstance(population, list): raise - return self.sample(list(population), k) + return self.sample(tuple(population), k) return result ## -------------------- real-valued distributions ------------------- Modified: python/branches/release24-maint/Lib/test/test_random.py ============================================================================== --- python/branches/release24-maint/Lib/test/test_random.py (original) +++ python/branches/release24-maint/Lib/test/test_random.py Sat Apr 1 02:41:10 2006 @@ -93,12 +93,28 @@ self.gen.sample(set(range(20)), 2) self.gen.sample(range(20), 2) self.gen.sample(xrange(20), 2) - self.gen.sample(dict.fromkeys('abcdefghijklmnopqrst'), 2) self.gen.sample(str('abcdefghijklmnopqrst'), 2) self.gen.sample(tuple('abcdefghijklmnopqrst'), 2) + + def test_sample_on_dicts(self): + self.gen.sample(dict.fromkeys('abcdefghijklmnopqrst'), 2) + # SF bug #1460340 -- random.sample can raise KeyError a = dict.fromkeys(range(10)+range(10,100,2)+range(100,110)) - self.gen.sample(a,3) + self.gen.sample(a, 3) + + # A followup to bug #1460340: sampling from a dict could return + # a subset of its keys or of its values, depending on the size of + # the subset requested. + N = 30 + d = dict((i, complex(i, i)) for i in xrange(N)) + for k in xrange(N+1): + samp = self.gen.sample(d, k) + # Verify that we got ints back (keys); the values are complex. + for x in samp: + self.assert_(type(x) is int) + samp.sort() + self.assertEqual(samp, range(N)) def test_gauss(self): # Ensure that the seed() method initializes all the hidden state. In Modified: python/branches/release24-maint/Misc/NEWS ============================================================================== --- python/branches/release24-maint/Misc/NEWS (original) +++ python/branches/release24-maint/Misc/NEWS Sat Apr 1 02:41:10 2006 @@ -4,6 +4,20 @@ (editors: check NEWS.help for information about editing NEWS using ReST.) +What's New in Python 2.4.4c1? +============================= + +*Release date: DD-MMM-2006* + +Library +------- + +- Bug #1460340: ``random.sample(dict)`` failed in various ways. Dicts + aren't officially supported here, and trying to use them will probably + raise an exception some day. But dicts have been allowed, and "mostly + worked", so support for them won't go away without warning. + + What's New in Python 2.4.3? =========================== @@ -14,10 +28,10 @@ - A few reference leaks were squished. -- A threading issue that caused random segfaults on some platforms from +- A threading issue that caused random segfaults on some platforms from the testsuite was fixed in test_capi. -- Reverted fix for Bug #1379994: Builtin unicode_escape and +- Reverted fix for Bug #1379994: Builtin unicode_escape and raw_unicode_escape codec now encodes backslash correctly. This caused another issue for unicode repr strings being double-escaped (SF Bug #1459029). Correct fix will be in 2.5, but is too risky for 2.4.3. From python-checkins at python.org Sat Apr 1 02:57:33 2006 From: python-checkins at python.org (anthony.baxter) Date: Sat, 1 Apr 2006 02:57:33 +0200 (CEST) Subject: [Python-checkins] r43514 - in python/trunk: Lib/sqlite3 Lib/sqlite3/test Lib/test/regrtest.py Lib/test/test_sqlite.py Misc/NEWS Modules/_sqlite setup.py Message-ID: <20060401005733.117151E4003@bag.python.org> Author: anthony.baxter Date: Sat Apr 1 02:57:31 2006 New Revision: 43514 Added: python/trunk/Lib/sqlite3/ (props changed) - copied from r43513, python/branches/sqlite-integration/Lib/sqlite3/ python/trunk/Lib/test/test_sqlite.py - copied unchanged from r43513, python/branches/sqlite-integration/Lib/test/test_sqlite.py python/trunk/Modules/_sqlite/ - copied from r43513, python/branches/sqlite-integration/Modules/_sqlite/ Modified: python/trunk/Lib/sqlite3/test/ (props changed) python/trunk/Lib/test/regrtest.py python/trunk/Misc/NEWS python/trunk/setup.py Log: merged the sqlite-integration branch. This is based on pysqlite2.1.3, and provides a DB-API interface in the standard library. You'll need sqlite 3.2.2 or later to build this - if you have an earlier version, the C extension module will not be built. Modified: python/trunk/Lib/test/regrtest.py ============================================================================== --- python/trunk/Lib/test/regrtest.py (original) +++ python/trunk/Lib/test/regrtest.py Sat Apr 1 02:57:31 2006 @@ -741,6 +741,7 @@ test_pwd test_resource test_signal + test_sqlite test_sunaudiodev test_threadsignals test_timing @@ -763,6 +764,7 @@ test_nis test_ntpath test_ossaudiodev + test_sqlite test_sunaudiodev """, 'mac': @@ -802,6 +804,7 @@ test_pwd test_resource test_signal + test_sqlite test_sunaudiodev test_sundry test_tarfile @@ -826,6 +829,7 @@ test_openpty test_pyexpat test_sax + test_sqlite test_sunaudiodev test_sundry """, @@ -848,6 +852,7 @@ test_openpty test_pyexpat test_sax + test_sqlite test_sunaudiodev test_sundry """, @@ -875,6 +880,7 @@ test_pyexpat test_queue test_sax + test_sqlite test_sunaudiodev test_sundry test_thread @@ -915,6 +921,7 @@ test_pty test_pwd test_strop + test_sqlite test_sunaudiodev test_sundry test_thread @@ -944,6 +951,7 @@ test_ntpath test_ossaudiodev test_poll + test_sqlite test_sunaudiodev """, 'sunos5': @@ -962,6 +970,7 @@ test_imgfile test_linuxaudiodev test_openpty + test_sqlite test_zipfile test_zlib """, @@ -988,6 +997,7 @@ test_openpty test_pyexpat test_sax + test_sqlite test_sunaudiodev test_zipfile test_zlib @@ -1013,6 +1023,7 @@ test_poll test_popen2 test_resource + test_sqlite test_sunaudiodev """, 'cygwin': @@ -1034,6 +1045,7 @@ test_nis test_ossaudiodev test_socketserver + test_sqlite test_sunaudiodev """, 'os2emx': @@ -1060,6 +1072,7 @@ test_pty test_resource test_signal + test_sqlite test_sunaudiodev """, 'freebsd4': @@ -1086,6 +1099,7 @@ test_scriptpackages test_socket_ssl test_socketserver + test_sqlite test_sunaudiodev test_tcl test_timeout @@ -1115,6 +1129,7 @@ test_macostools test_nis test_ossaudiodev + test_sqlite test_sunaudiodev test_tcl test_winreg @@ -1147,6 +1162,7 @@ test_plistlib test_scriptpackages test_tcl + test_sqlite test_sunaudiodev test_unicode_file test_winreg Modified: python/trunk/Misc/NEWS ============================================================================== --- python/trunk/Misc/NEWS (original) +++ python/trunk/Misc/NEWS Sat Apr 1 02:57:31 2006 @@ -489,6 +489,11 @@ Library ------- +- Added the sqlite3 package. This is based on pysqlite2.1.3, and provides + a DB-API interface in the standard library. You'll need sqlite 3.2.2 or + later to build this - if you have an earlier version, the C extension + module will not be built. + - Bug #1460340: ``random.sample(dict)`` failed in various ways. Dicts aren't officially supported here, and trying to use them will probably raise an exception some day. But dicts have been allowed, and "mostly Modified: python/trunk/setup.py ============================================================================== --- python/trunk/setup.py (original) +++ python/trunk/setup.py Sat Apr 1 02:57:31 2006 @@ -689,6 +689,81 @@ dblibs = [] dblib_dir = None + # The sqlite interface + + # We hunt for "#define SQLITE_VERSION_NUMBER nnnnn" + # We need to find a version >= 3002002 (> sqlite version 3.2.2) + sqlite_incdir = sqlite_libdir = None + sqlite_inc_paths = [ '/usr/include', + '/usr/include/sqlite', + '/usr/include/sqlite3', + '/usr/local/include', + '/usr/local/include/sqlite', + '/usr/local/include/sqlite3', + ] + MIN_SQLITE_VERSION = 3002002 + for d in sqlite_inc_paths + inc_dirs: + f = os.path.join(d, "sqlite3.h") + if os.path.exists(f): + if db_setup_debug: print "found %s"%f + f = open(f).read() + m = re.search(r"#define\WSQLITE_VERSION_NUMBER\W(\d+)", f) + if m: + sqlite_version = int(m.group(1)) + if sqlite_version >= MIN_SQLITE_VERSION: + # we win! + print "%s/sqlite3.h: version %s"%(d, sqlite_version) + sqlite_incdir = d + break + else: + if db_setup_debug: + print "%s: version %d is too old, need >= %s"%(d, + sqlite_version, MIN_SQLITE_VERSION) + + if sqlite_incdir: + sqlite_dirs_to_check = [ + os.path.join(sqlite_incdir, '..', 'lib64'), + os.path.join(sqlite_incdir, '..', 'lib'), + os.path.join(sqlite_incdir, '..', '..', 'lib64'), + os.path.join(sqlite_incdir, '..', '..', 'lib'), + ] + sqlite_libfile = self.compiler.find_library_file( + sqlite_dirs_to_check + lib_dirs, 'sqlite3') + sqlite_libdir = [os.path.abspath(os.path.dirname(sqlite_libfile))] + + if sqlite_incdir and sqlite_libdir: + sqlite_srcs = ['_sqlite/adapters.c', + '_sqlite/cache.c', + '_sqlite/connection.c', + '_sqlite/converters.c', + '_sqlite/cursor.c', + '_sqlite/microprotocols.c', + '_sqlite/module.c', + '_sqlite/prepare_protocol.c', + '_sqlite/row.c', + '_sqlite/statement.c', + '_sqlite/util.c', ] + + PYSQLITE_VERSION = "2.1.3" + sqlite_defines = [] + if sys.platform != "win32": + sqlite_defines.append(('PYSQLITE_VERSION', + '"%s"' % PYSQLITE_VERSION)) + else: + sqlite_defines.append(('PYSQLITE_VERSION', + '\\"'+PYSQLITE_VERSION+'\\"')) + sqlite_defines.append(('PY_MAJOR_VERSION', + str(sys.version_info[0]))) + sqlite_defines.append(('PY_MINOR_VERSION', + str(sys.version_info[1]))) + + exts.append(Extension('_sqlite3', sqlite_srcs, + define_macros=sqlite_defines, + include_dirs=["Modules/_sqlite", + sqlite_incdir], + library_dirs=sqlite_libdir, + runtime_library_dirs=sqlite_libdir, + libraries=["sqlite3",])) # Look for Berkeley db 1.85. Note that it is built as a different # module name so it can be included even when later versions are From python-checkins at python.org Sat Apr 1 03:08:30 2006 From: python-checkins at python.org (anthony.baxter) Date: Sat, 1 Apr 2006 03:08:30 +0200 (CEST) Subject: [Python-checkins] r43515 - python/trunk/Include/pyport.h Message-ID: <20060401010830.943831E4003@bag.python.org> Author: anthony.baxter Date: Sat Apr 1 03:08:29 2006 New Revision: 43515 Modified: python/trunk/Include/pyport.h Log: Add a clause to the PY_FORMAT_SIZE_T to stop warnings on 32 bit intel linux with gcc 4.0.2, after talking to Tim. But it won't break anything anywhere, so don't worry :-) Modified: python/trunk/Include/pyport.h ============================================================================== --- python/trunk/Include/pyport.h (original) +++ python/trunk/Include/pyport.h Sat Apr 1 03:08:29 2006 @@ -124,7 +124,9 @@ * Py_ssize_t on the platform. */ #ifndef PY_FORMAT_SIZE_T -# if SIZEOF_SIZE_T == SIZEOF_LONG +# if SIZEOF_SIZE_T == SIZEOF_INT +# define PY_FORMAT_SIZE_T "" +# elif SIZEOF_SIZE_T == SIZEOF_LONG # define PY_FORMAT_SIZE_T "l" # elif defined(MS_WINDOWS) # define PY_FORMAT_SIZE_T "I" From buildbot at python.org Sat Apr 1 03:25:27 2006 From: buildbot at python.org (buildbot at python.org) Date: Sat, 01 Apr 2006 01:25:27 +0000 Subject: [Python-checkins] buildbot warnings in alpha Tru64 5.1 2.4 Message-ID: <20060401012527.47AD21E4003@bag.python.org> The Buildbot has detected a new failure of alpha Tru64 5.1 2.4. Full details are available at: http://www.python.org/dev/buildbot/all/alpha%20Tru64%205.1%202.4/builds/12 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch branches/release24-maint] HEAD Blamelist: tim.peters Build Had Warnings: warnings test sincerely, -The Buildbot From python-checkins at python.org Sat Apr 1 03:29:05 2006 From: python-checkins at python.org (tim.peters) Date: Sat, 1 Apr 2006 03:29:05 +0200 (CEST) Subject: [Python-checkins] r43516 - python/trunk/Lib/test/test_socket_ssl.py Message-ID: <20060401012905.741EC1E4003@bag.python.org> Author: tim.peters Date: Sat Apr 1 03:28:51 2006 New Revision: 43516 Modified: python/trunk/Lib/test/test_socket_ssl.py Log: test_timeout(): Disable this new test on all platforms. The s.connect(("gmail.org", 995)) line has been timing out on all buildbot slaves for hours now, causing the test to fail. Modified: python/trunk/Lib/test/test_socket_ssl.py ============================================================================== --- python/trunk/Lib/test/test_socket_ssl.py (original) +++ python/trunk/Lib/test/test_socket_ssl.py Sat Apr 1 03:28:51 2006 @@ -27,23 +27,27 @@ buf = f.read() f.close() -if not sys.platform.startswith('win'): +# XXX Tim disabled this test on all platforms, for now, since the +# XXX s.connect(("gmail.org", 995)) +# XXX line starting timing out on all the builbot slaves. +if 0: not sys.platform.startswith('win'): def test_timeout(): test_support.requires('network') s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) s.settimeout(30.0) - # connect to service which issues an welcome banner (without need to write anything) + # connect to service which issues an welcome banner (without need to + # write anything) s.connect(("gmail.org", 995)) ss = socket.ssl(s) - # read part of return welcome banner twice,# read part of return welcome banner twice + # read part of return welcome banner twice ss.read(1) ss.read(1) s.close() else: def test_timeout(): pass - + def test_rude_shutdown(): try: import threading From python-checkins at python.org Sat Apr 1 03:31:18 2006 From: python-checkins at python.org (tim.peters) Date: Sat, 1 Apr 2006 03:31:18 +0200 (CEST) Subject: [Python-checkins] r43517 - python/branches/release24-maint/Lib/test/test_socket_ssl.py Message-ID: <20060401013118.D54BF1E4006@bag.python.org> Author: tim.peters Date: Sat Apr 1 03:31:04 2006 New Revision: 43517 Modified: python/branches/release24-maint/Lib/test/test_socket_ssl.py Log: Merge rev 43516 from trunk. test_timeout(): Disable this new test on all platforms. The s.connect(("gmail.org", 995)) line has been timing out on all buildbot slaves for hours now, causing the test to fail. Modified: python/branches/release24-maint/Lib/test/test_socket_ssl.py ============================================================================== --- python/branches/release24-maint/Lib/test/test_socket_ssl.py (original) +++ python/branches/release24-maint/Lib/test/test_socket_ssl.py Sat Apr 1 03:31:04 2006 @@ -28,23 +28,27 @@ buf = f.read() f.close() -if not sys.platform.startswith('win'): +# XXX Tim disabled this test on all platforms, for now, since the +# XXX s.connect(("gmail.org", 995)) +# XXX line starting timing out on all the builbot slaves. +if 0: not sys.platform.startswith('win'): def test_timeout(): test_support.requires('network') s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) s.settimeout(30.0) - # connect to service which issues an welcome banner (without need to write anything) + # connect to service which issues an welcome banner (without need to + # write anything) s.connect(("gmail.org", 995)) ss = socket.ssl(s) - # read part of return welcome banner twice,# read part of return welcome banner twice + # read part of return welcome banner twice ss.read(1) ss.read(1) s.close() else: def test_timeout(): pass - + def test_rude_shutdown(): try: import thread From python-checkins at python.org Sat Apr 1 03:32:19 2006 From: python-checkins at python.org (tim.peters) Date: Sat, 1 Apr 2006 03:32:19 +0200 (CEST) Subject: [Python-checkins] r43518 - python/trunk/Lib/test/test_socket_ssl.py Message-ID: <20060401013219.A193D1E4003@bag.python.org> Author: tim.peters Date: Sat Apr 1 03:32:13 2006 New Revision: 43518 Modified: python/trunk/Lib/test/test_socket_ssl.py Log: Fix stupid typo. Modified: python/trunk/Lib/test/test_socket_ssl.py ============================================================================== --- python/trunk/Lib/test/test_socket_ssl.py (original) +++ python/trunk/Lib/test/test_socket_ssl.py Sat Apr 1 03:32:13 2006 @@ -30,7 +30,7 @@ # XXX Tim disabled this test on all platforms, for now, since the # XXX s.connect(("gmail.org", 995)) # XXX line starting timing out on all the builbot slaves. -if 0: not sys.platform.startswith('win'): +if 0: #not sys.platform.startswith('win'): def test_timeout(): test_support.requires('network') From python-checkins at python.org Sat Apr 1 03:35:11 2006 From: python-checkins at python.org (tim.peters) Date: Sat, 1 Apr 2006 03:35:11 +0200 (CEST) Subject: [Python-checkins] r43519 - python/branches/release24-maint/Lib/test/test_socket_ssl.py Message-ID: <20060401013511.F30251E4003@bag.python.org> Author: tim.peters Date: Sat Apr 1 03:35:10 2006 New Revision: 43519 Modified: python/branches/release24-maint/Lib/test/test_socket_ssl.py Log: Merge rev 43518 from trunk. Fix stupid typo. Modified: python/branches/release24-maint/Lib/test/test_socket_ssl.py ============================================================================== --- python/branches/release24-maint/Lib/test/test_socket_ssl.py (original) +++ python/branches/release24-maint/Lib/test/test_socket_ssl.py Sat Apr 1 03:35:10 2006 @@ -31,7 +31,7 @@ # XXX Tim disabled this test on all platforms, for now, since the # XXX s.connect(("gmail.org", 995)) # XXX line starting timing out on all the builbot slaves. -if 0: not sys.platform.startswith('win'): +if 0: #not sys.platform.startswith('win'): def test_timeout(): test_support.requires('network') From python-checkins at python.org Sat Apr 1 08:11:08 2006 From: python-checkins at python.org (fred.drake) Date: Sat, 1 Apr 2006 08:11:08 +0200 (CEST) Subject: [Python-checkins] r43520 - in python/trunk: Doc/lib/liburlparse.tex Lib/urlparse.py Message-ID: <20060401061108.C98B61E4003@bag.python.org> Author: fred.drake Date: Sat Apr 1 08:11:07 2006 New Revision: 43520 Modified: python/trunk/Doc/lib/liburlparse.tex python/trunk/Lib/urlparse.py Log: add support for the sips: scheme (identical to sip: except for scheme name) Modified: python/trunk/Doc/lib/liburlparse.tex ============================================================================== --- python/trunk/Doc/lib/liburlparse.tex (original) +++ python/trunk/Doc/lib/liburlparse.tex Sat Apr 1 08:11:07 2006 @@ -23,9 +23,9 @@ \code{file}, \code{ftp}, \code{gopher}, \code{hdl}, \code{http}, \code{https}, \code{imap}, \code{mailto}, \code{mms}, \code{news}, \code{nntp}, \code{prospero}, \code{rsync}, \code{rtsp}, \code{rtspu}, -\code{sftp}, \code{shttp}, \code{sip}, \code{snews}, \code{svn}, +\code{sftp}, \code{shttp}, \code{sip}, \code{sips}, \code{snews}, \code{svn}, \code{svn+ssh}, \code{telnet}, \code{wais}. -\versionadded[Support for the \code{sftp} scheme]{2.5} +\versionadded[Support for the \code{sftp} and \code{sips} schemes]{2.5} The \module{urlparse} module defines the following functions: Modified: python/trunk/Lib/urlparse.py ============================================================================== --- python/trunk/Lib/urlparse.py (original) +++ python/trunk/Lib/urlparse.py Sat Apr 1 08:11:07 2006 @@ -16,12 +16,12 @@ 'snews', 'prospero', 'rtsp', 'rtspu', 'rsync', '', 'svn', 'svn+ssh', 'sftp'] non_hierarchical = ['gopher', 'hdl', 'mailto', 'news', - 'telnet', 'wais', 'imap', 'snews', 'sip'] + 'telnet', 'wais', 'imap', 'snews', 'sip', 'sips'] uses_params = ['ftp', 'hdl', 'prospero', 'http', 'imap', - 'https', 'shttp', 'rtsp', 'rtspu', 'sip', + 'https', 'shttp', 'rtsp', 'rtspu', 'sip', 'sips', 'mms', '', 'sftp'] uses_query = ['http', 'wais', 'imap', 'https', 'shttp', 'mms', - 'gopher', 'rtsp', 'rtspu', 'sip', ''] + 'gopher', 'rtsp', 'rtspu', 'sip', 'sips', ''] uses_fragment = ['ftp', 'hdl', 'http', 'gopher', 'news', 'nntp', 'wais', 'https', 'shttp', 'snews', 'file', 'prospero', ''] From python-checkins at python.org Sat Apr 1 09:23:09 2006 From: python-checkins at python.org (georg.brandl) Date: Sat, 1 Apr 2006 09:23:09 +0200 (CEST) Subject: [Python-checkins] r43521 - python/trunk/Doc/ref/ref3.tex Message-ID: <20060401072309.378A91E4003@bag.python.org> Author: georg.brandl Date: Sat Apr 1 09:23:08 2006 New Revision: 43521 Modified: python/trunk/Doc/ref/ref3.tex Log: bug #1462278: small fix in documentation of __op__ vs __rop__ methods Modified: python/trunk/Doc/ref/ref3.tex ============================================================================== --- python/trunk/Doc/ref/ref3.tex (original) +++ python/trunk/Doc/ref/ref3.tex Sat Apr 1 09:23:08 2006 @@ -2057,14 +2057,15 @@ \item Exception to the previous item: if the left operand is an instance of -a built-in type or a new-style class, and the right operand is an -instance of a proper subclass of that type or class, the right -operand's \method{__rop__()} method is tried \emph{before} the left -operand's \method{__op__()} method. This is done so that a subclass can -completely override binary operators. Otherwise, the left operand's -__op__ method would always accept the right operand: when an instance -of a given class is expected, an instance of a subclass of that class -is always acceptable. +a built-in type or a new-style class, and the right operand is an instance +of a proper subclass of that type or class and overrides the base's +\method{__rop__()} method, the right operand's \method{__rop__()} method +is tried \emph{before} the left operand's \method{__op__()} method. + +This is done so that a subclass can completely override binary operators. +Otherwise, the left operand's \method{__op__()} method would always +accept the right operand: when an instance of a given class is expected, +an instance of a subclass of that class is always acceptable. \item From python-checkins at python.org Sat Apr 1 09:23:12 2006 From: python-checkins at python.org (georg.brandl) Date: Sat, 1 Apr 2006 09:23:12 +0200 (CEST) Subject: [Python-checkins] r43522 - python/branches/release24-maint/Doc/ref/ref3.tex Message-ID: <20060401072312.95E981E4003@bag.python.org> Author: georg.brandl Date: Sat Apr 1 09:23:12 2006 New Revision: 43522 Modified: python/branches/release24-maint/Doc/ref/ref3.tex Log: bug #1462278: small fix in documentation of __op__ vs __rop__ methods (backport from rev. 43521) Modified: python/branches/release24-maint/Doc/ref/ref3.tex ============================================================================== --- python/branches/release24-maint/Doc/ref/ref3.tex (original) +++ python/branches/release24-maint/Doc/ref/ref3.tex Sat Apr 1 09:23:12 2006 @@ -2005,14 +2005,15 @@ \item Exception to the previous item: if the left operand is an instance of -a built-in type or a new-style class, and the right operand is an -instance of a proper subclass of that type or class, the right -operand's \method{__rop__()} method is tried \emph{before} the left -operand's \method{__op__()} method. This is done so that a subclass can -completely override binary operators. Otherwise, the left operand's -__op__ method would always accept the right operand: when an instance -of a given class is expected, an instance of a subclass of that class -is always acceptable. +a built-in type or a new-style class, and the right operand is an instance +of a proper subclass of that type or class and overrides the base's +\method{__rop__()} method, the right operand's \method{__rop__()} method +is tried \emph{before} the left operand's \method{__op__()} method. + +This is done so that a subclass can completely override binary operators. +Otherwise, the left operand's \method{__op__()} method would always +accept the right operand: when an instance of a given class is expected, +an instance of a subclass of that class is always acceptable. \item From python-checkins at python.org Sat Apr 1 09:33:18 2006 From: python-checkins at python.org (georg.brandl) Date: Sat, 1 Apr 2006 09:33:18 +0200 (CEST) Subject: [Python-checkins] r43523 - in python/trunk: Doc/lib/libsocket.tex Modules/socketmodule.c Message-ID: <20060401073318.5EAAF1E4003@bag.python.org> Author: georg.brandl Date: Sat Apr 1 09:33:08 2006 New Revision: 43523 Modified: python/trunk/Doc/lib/libsocket.tex python/trunk/Modules/socketmodule.c Log: Bug #1460564: document that socket.fromfd() duplicates the given file descriptor. Modified: python/trunk/Doc/lib/libsocket.tex ============================================================================== --- python/trunk/Doc/lib/libsocket.tex (original) +++ python/trunk/Doc/lib/libsocket.tex Sat Apr 1 09:33:08 2006 @@ -317,10 +317,11 @@ \end{funcdesc} \begin{funcdesc}{fromfd}{fd, family, type\optional{, proto}} -Build a socket object from an existing file descriptor (an integer as -returned by a file object's \method{fileno()} method). Address family, -socket type and protocol number are as for the \function{socket()} function -above. The file descriptor should refer to a socket, but this is not +Duplicate the file descriptor \var{fd} (an integer as returned by a file +object's \method{fileno()} method) and build a socket object from the +result. Address family, socket type and protocol number are as for the +\function{socket()} function above. +The file descriptor should refer to a socket, but this is not checked --- subsequent operations on the object may fail if the file descriptor is invalid. This function is rarely needed, but can be used to get or set socket options on a socket passed to a program as Modified: python/trunk/Modules/socketmodule.c ============================================================================== --- python/trunk/Modules/socketmodule.c (original) +++ python/trunk/Modules/socketmodule.c Sat Apr 1 09:33:08 2006 @@ -3168,7 +3168,8 @@ PyDoc_STRVAR(fromfd_doc, "fromfd(fd, family, type[, proto]) -> socket object\n\ \n\ -Create a socket object from the given file descriptor.\n\ +Create a socket object from a duplicate of the given\n\ +file descriptor.\n\ The remaining arguments are the same as for socket()."); #endif /* NO_DUP */ @@ -4052,7 +4053,7 @@ PyModule_AddIntConstant(m, "NETLINK_IP6_FW", NETLINK_IP6_FW); PyModule_AddIntConstant(m, "NETLINK_DNRTMSG", NETLINK_DNRTMSG); PyModule_AddIntConstant(m, "NETLINK_TAPBASE", NETLINK_TAPBASE); -#endif +#endif /* AF_NETLINK */ #ifdef AF_ROUTE /* Alias to emulate 4.4BSD */ PyModule_AddIntConstant(m, "AF_ROUTE", AF_ROUTE); From python-checkins at python.org Sat Apr 1 09:33:25 2006 From: python-checkins at python.org (georg.brandl) Date: Sat, 1 Apr 2006 09:33:25 +0200 (CEST) Subject: [Python-checkins] r43524 - in python/branches/release24-maint: Doc/lib/libsocket.tex Modules/socketmodule.c Message-ID: <20060401073325.209801E4003@bag.python.org> Author: georg.brandl Date: Sat Apr 1 09:33:22 2006 New Revision: 43524 Modified: python/branches/release24-maint/Doc/lib/libsocket.tex python/branches/release24-maint/Modules/socketmodule.c Log: Bug #1460564: document that socket.fromfd() duplicates the given file descriptor. (backport from rev. 43523) Modified: python/branches/release24-maint/Doc/lib/libsocket.tex ============================================================================== --- python/branches/release24-maint/Doc/lib/libsocket.tex (original) +++ python/branches/release24-maint/Doc/lib/libsocket.tex Sat Apr 1 09:33:22 2006 @@ -314,10 +314,11 @@ \end{funcdesc} \begin{funcdesc}{fromfd}{fd, family, type\optional{, proto}} -Build a socket object from an existing file descriptor (an integer as -returned by a file object's \method{fileno()} method). Address family, -socket type and protocol number are as for the \function{socket()} function -above. The file descriptor should refer to a socket, but this is not +Duplicate the file descriptor \var{fd} (an integer as returned by a file +object's \method{fileno()} method) and build a socket object from the +result. Address family, socket type and protocol number are as for the +\function{socket()} function above. +The file descriptor should refer to a socket, but this is not checked --- subsequent operations on the object may fail if the file descriptor is invalid. This function is rarely needed, but can be used to get or set socket options on a socket passed to a program as Modified: python/branches/release24-maint/Modules/socketmodule.c ============================================================================== --- python/branches/release24-maint/Modules/socketmodule.c (original) +++ python/branches/release24-maint/Modules/socketmodule.c Sat Apr 1 09:33:22 2006 @@ -3113,7 +3113,8 @@ PyDoc_STRVAR(fromfd_doc, "fromfd(fd, family, type[, proto]) -> socket object\n\ \n\ -Create a socket object from the given file descriptor.\n\ +Create a socket object from a duplicate of the given\n\ +file descriptor.\n\ The remaining arguments are the same as for socket()."); #endif /* NO_DUP */ From python-checkins at python.org Sat Apr 1 09:39:42 2006 From: python-checkins at python.org (georg.brandl) Date: Sat, 1 Apr 2006 09:39:42 +0200 (CEST) Subject: [Python-checkins] r43525 - in python/trunk: Doc/lib/libzlib.tex Modules/zlibmodule.c Message-ID: <20060401073942.71A361E4003@bag.python.org> Author: georg.brandl Date: Sat Apr 1 09:39:41 2006 New Revision: 43525 Modified: python/trunk/Doc/lib/libzlib.tex python/trunk/Modules/zlibmodule.c Log: Patch #1459631: documnent zlib.Decompress.flush() length parameter. Modified: python/trunk/Doc/lib/libzlib.tex ============================================================================== --- python/trunk/Doc/lib/libzlib.tex (original) +++ python/trunk/Doc/lib/libzlib.tex Sat Apr 1 09:39:41 2006 @@ -166,11 +166,14 @@ decompressed, and \member{unconsumed_tail} is an empty string. \end{methoddesc} -\begin{methoddesc}[Decompress]{flush}{} +\begin{methoddesc}[Decompress]{flush}{\optional{length}} All pending input is processed, and a string containing the remaining uncompressed output is returned. After calling \method{flush()}, the \method{decompress()} method cannot be called again; the only realistic action is to delete the object. + +The optional parameter \var{length} sets the initial size of the +output buffer. \end{methoddesc} \begin{seealso} Modified: python/trunk/Modules/zlibmodule.c ============================================================================== --- python/trunk/Modules/zlibmodule.c (original) +++ python/trunk/Modules/zlibmodule.c Sat Apr 1 09:39:41 2006 @@ -654,7 +654,9 @@ } PyDoc_STRVAR(decomp_flush__doc__, -"flush() -- Return a string containing any remaining decompressed data.\n" +"flush( [length] ) -- Return a string containing any remaining\n" +"decompressed data. length, if given, is the initial size of the\n" +"output buffer.\n" "\n" "The decompressor object can no longer be used after this call."); From python-checkins at python.org Sat Apr 1 09:39:45 2006 From: python-checkins at python.org (georg.brandl) Date: Sat, 1 Apr 2006 09:39:45 +0200 (CEST) Subject: [Python-checkins] r43526 - in python/branches/release24-maint: Doc/lib/libzlib.tex Modules/zlibmodule.c Message-ID: <20060401073945.C6DDA1E4003@bag.python.org> Author: georg.brandl Date: Sat Apr 1 09:39:45 2006 New Revision: 43526 Modified: python/branches/release24-maint/Doc/lib/libzlib.tex python/branches/release24-maint/Modules/zlibmodule.c Log: Patch #1459631: documnent zlib.Decompress.flush() length parameter. (backport from rev. 43525) Modified: python/branches/release24-maint/Doc/lib/libzlib.tex ============================================================================== --- python/branches/release24-maint/Doc/lib/libzlib.tex (original) +++ python/branches/release24-maint/Doc/lib/libzlib.tex Sat Apr 1 09:39:45 2006 @@ -162,11 +162,14 @@ decompressed, and \member{unconsumed_tail} is an empty string. \end{methoddesc} -\begin{methoddesc}[Decompress]{flush}{} +\begin{methoddesc}[Decompress]{flush}{\optional{length}} All pending input is processed, and a string containing the remaining uncompressed output is returned. After calling \method{flush()}, the \method{decompress()} method cannot be called again; the only realistic action is to delete the object. + +The optional parameter \var{length} sets the initial size of the +output buffer. \end{methoddesc} \begin{seealso} Modified: python/branches/release24-maint/Modules/zlibmodule.c ============================================================================== --- python/branches/release24-maint/Modules/zlibmodule.c (original) +++ python/branches/release24-maint/Modules/zlibmodule.c Sat Apr 1 09:39:45 2006 @@ -654,7 +654,9 @@ } PyDoc_STRVAR(decomp_flush__doc__, -"flush() -- Return a string containing any remaining decompressed data.\n" +"flush( [length] ) -- Return a string containing any remaining\n" +"decompressed data. length, if given, is the initial size of the\n" +"output buffer.\n" "\n" "The decompressor object can no longer be used after this call."); From python-checkins at python.org Sat Apr 1 09:42:42 2006 From: python-checkins at python.org (georg.brandl) Date: Sat, 1 Apr 2006 09:42:42 +0200 (CEST) Subject: [Python-checkins] r43527 - python/trunk/Doc/lib/libsignal.tex Message-ID: <20060401074242.04B7B1E4003@bag.python.org> Author: georg.brandl Date: Sat Apr 1 09:42:41 2006 New Revision: 43527 Modified: python/trunk/Doc/lib/libsignal.tex Log: Patch #1462496: typo in libsignal.tex Modified: python/trunk/Doc/lib/libsignal.tex ============================================================================== --- python/trunk/Doc/lib/libsignal.tex (original) +++ python/trunk/Doc/lib/libsignal.tex Sat Apr 1 09:42:41 2006 @@ -100,7 +100,7 @@ Any previously scheduled alarm is canceled (only one alarm can be scheduled at any time). The returned value is then the number of seconds before any previously set alarm was to have been delivered. - If \var{time} is zero, no alarm id scheduled, and any scheduled + If \var{time} is zero, no alarm is scheduled, and any scheduled alarm is canceled. The return value is the number of seconds remaining before a previously scheduled alarm. If the return value is zero, no alarm is currently scheduled. (See the \UNIX{} man page From python-checkins at python.org Sat Apr 1 09:42:44 2006 From: python-checkins at python.org (georg.brandl) Date: Sat, 1 Apr 2006 09:42:44 +0200 (CEST) Subject: [Python-checkins] r43528 - python/branches/release24-maint/Doc/lib/libsignal.tex Message-ID: <20060401074244.852C81E4010@bag.python.org> Author: georg.brandl Date: Sat Apr 1 09:42:44 2006 New Revision: 43528 Modified: python/branches/release24-maint/Doc/lib/libsignal.tex Log: Patch #1462496: typo in libsignal.tex (backport from rev. 43527) Modified: python/branches/release24-maint/Doc/lib/libsignal.tex ============================================================================== --- python/branches/release24-maint/Doc/lib/libsignal.tex (original) +++ python/branches/release24-maint/Doc/lib/libsignal.tex Sat Apr 1 09:42:44 2006 @@ -100,7 +100,7 @@ Any previously scheduled alarm is canceled (only one alarm can be scheduled at any time). The returned value is then the number of seconds before any previously set alarm was to have been delivered. - If \var{time} is zero, no alarm id scheduled, and any scheduled + If \var{time} is zero, no alarm is scheduled, and any scheduled alarm is canceled. The return value is the number of seconds remaining before a previously scheduled alarm. If the return value is zero, no alarm is currently scheduled. (See the \UNIX{} man page From python-checkins at python.org Sat Apr 1 09:46:54 2006 From: python-checkins at python.org (georg.brandl) Date: Sat, 1 Apr 2006 09:46:54 +0200 (CEST) Subject: [Python-checkins] r43529 - python/trunk/Lib/distutils/log.py Message-ID: <20060401074654.EDD1D1E4003@bag.python.org> Author: georg.brandl Date: Sat Apr 1 09:46:54 2006 New Revision: 43529 Modified: python/trunk/Lib/distutils/log.py Log: Bug #1458017: make distutils.Log._log more forgiving when passing in msg strings with '%', but without format args. Modified: python/trunk/Lib/distutils/log.py ============================================================================== --- python/trunk/Lib/distutils/log.py (original) +++ python/trunk/Lib/distutils/log.py Sat Apr 1 09:46:54 2006 @@ -20,7 +20,12 @@ def _log(self, level, msg, args): if level >= self.threshold: - print msg % args + if not args: + # msg may contain a '%'. If args is empty, + # don't even try to string-format + print msg + else: + print msg % args sys.stdout.flush() def log(self, level, msg, *args): From python-checkins at python.org Sat Apr 1 09:46:57 2006 From: python-checkins at python.org (georg.brandl) Date: Sat, 1 Apr 2006 09:46:57 +0200 (CEST) Subject: [Python-checkins] r43530 - python/branches/release24-maint/Lib/distutils/log.py Message-ID: <20060401074657.7F9AD1E4003@bag.python.org> Author: georg.brandl Date: Sat Apr 1 09:46:57 2006 New Revision: 43530 Modified: python/branches/release24-maint/Lib/distutils/log.py Log: Bug #1458017: make distutils.Log._log more forgiving when passing in msg strings with '%', but without format args. (backport from rev. 43529) Modified: python/branches/release24-maint/Lib/distutils/log.py ============================================================================== --- python/branches/release24-maint/Lib/distutils/log.py (original) +++ python/branches/release24-maint/Lib/distutils/log.py Sat Apr 1 09:46:57 2006 @@ -20,7 +20,12 @@ def _log(self, level, msg, args): if level >= self.threshold: - print msg % args + if not args: + # msg may contain a '%'. If args is empty, + # don't even try to string-format + print msg + else: + print msg % args sys.stdout.flush() def log(self, level, msg, *args): From python-checkins at python.org Sat Apr 1 09:57:02 2006 From: python-checkins at python.org (walter.doerwald) Date: Sat, 1 Apr 2006 09:57:02 +0200 (CEST) Subject: [Python-checkins] r43531 - in python/trunk: Doc/lib/libcalendar.tex Lib/calendar.py Misc/NEWS Message-ID: <20060401075702.1C28E1E4006@bag.python.org> Author: walter.doerwald Date: Sat Apr 1 09:57:00 2006 New Revision: 43531 Modified: python/trunk/Doc/lib/libcalendar.tex python/trunk/Lib/calendar.py python/trunk/Misc/NEWS Log: Bug #947906: Add classes LocaleTextCalendar and LocaleHTMLCalendar, that output localized month and weekday names and can cope with encodings. Modified: python/trunk/Doc/lib/libcalendar.tex ============================================================================== --- python/trunk/Doc/lib/libcalendar.tex (original) +++ python/trunk/Doc/lib/libcalendar.tex Sat Apr 1 09:57:00 2006 @@ -176,6 +176,24 @@ \end{methoddesc} +\begin{classdesc}{LocaleTextCalendar}{\optional{firstweekday\optional{, locale}}} +This subclass of \class{TextCalendar} can be passed a locale name in the +constructor and will return month and weekday names in the specified locale. +If this locale includes an encoding all strings containing month and weekday +names will be returned as unicode. +\versionadded{2.5} +\end{classdesc} + + +\begin{classdesc}{LocaleHTMLCalendar}{\optional{firstweekday\optional{, locale}}} +This subclass of \class{HTMLCalendar} can be passed a locale name in the +constructor and will return month and weekday names in the specified locale. +If this locale includes an encoding all strings containing month and weekday +names will be returned as unicode. +\versionadded{2.5} +\end{classdesc} + + For simple text calendars this module provides the following functions. \begin{funcdesc}{setfirstweekday}{weekday} Modified: python/trunk/Lib/calendar.py ============================================================================== --- python/trunk/Lib/calendar.py (original) +++ python/trunk/Lib/calendar.py Sat Apr 1 09:57:00 2006 @@ -5,7 +5,7 @@ Sunday as the last (the European convention). Use setfirstweekday() to set the first day of the week (0=Monday, 6=Sunday).""" -import sys, datetime +import sys, datetime, locale __all__ = ["IllegalMonthError", "IllegalWeekdayError", "setfirstweekday", "firstweekday", "isleap", "leapdays", "weekday", "monthrange", @@ -297,11 +297,13 @@ """ return ' '.join(self.formatweekday(i, width) for i in self.iterweekdays()) - def formatmonthname(self, theyear, themonth, width): + def formatmonthname(self, theyear, themonth, width, withyear=True): """ Return a formatted month name. """ - s = "%s %r" % (month_name[themonth], theyear) + s = month_name[themonth] + if withyear: + s = "%s %r" % (s, theyear) return s.center(width) def prmonth(self, theyear, themonth, w=0, l=0): @@ -343,9 +345,12 @@ # months in this row months = xrange(m*i+1, min(m*(i+1)+1, 13)) a('\n'*l) - a(formatstring((month_name[k] for k in months), colwidth, c).rstrip()) + names = (self.formatmonthname(theyear, k, colwidth, False) + for k in months) + a(formatstring(names, colwidth, c).rstrip()) a('\n'*l) - a(formatstring((header for k in months), colwidth, c).rstrip()) + headers = (header for k in months) + a(formatstring(headers, colwidth, c).rstrip()) a('\n'*l) # max number of weeks for this row height = max(len(cal) for cal in row) @@ -474,7 +479,92 @@ a(self.formatyear(theyear, width)) a('\n') a('\n') - return ''.join(v).encode(encoding) + return ''.join(v).encode(encoding, "xmlcharrefreplace") + + +class LocaleTextCalendar(TextCalendar): + """ + This class can be passed a locale name in the constructor and will return + month and weekday names in the specified locale. If this locale includes + an encoding all strings containing month and weekday names will be returned + as unicode. + """ + + def __init__(self, firstweekday=0, locale=None): + TextCalendar.__init__(self, firstweekday) + if locale is None: + locale = locale.getdefaultlocale() + self.locale = locale + + def formatweekday(self, day, width): + oldlocale = locale.setlocale(locale.LC_TIME, self.locale) + try: + encoding = locale.getlocale(locale.LC_TIME)[1] + if width >= 9: + names = day_name + else: + names = day_abbr + name = names[day] + if encoding is not None: + name = name.decode(encoding) + result = name[:width].center(width) + finally: + locale.setlocale(locale.LC_TIME, oldlocale) + return result + + def formatmonthname(self, theyear, themonth, width, withyear=True): + oldlocale = locale.setlocale(locale.LC_TIME, self.locale) + try: + encoding = locale.getlocale(locale.LC_TIME)[1] + s = month_name[themonth] + if encoding is not None: + s = s.decode(encoding) + if withyear: + s = "%s %r" % (s, theyear) + result = s.center(width) + finally: + locale.setlocale(locale.LC_TIME, oldlocale) + return result + + +class LocaleHTMLCalendar(HTMLCalendar): + """ + This class can be passed a locale name in the constructor and will return + month and weekday names in the specified locale. If this locale includes + an encoding all strings containing month and weekday names will be returned + as unicode. + """ + def __init__(self, firstweekday=0, locale=None): + HTMLCalendar.__init__(self, firstweekday) + if locale is None: + locale = locale.getdefaultlocale() + self.locale = locale + + def formatweekday(self, day): + oldlocale = locale.setlocale(locale.LC_TIME, self.locale) + try: + encoding = locale.getlocale(locale.LC_TIME)[1] + s = day_abbr[day] + if encoding is not None: + s = s.decode(encoding) + result = '%s' % (self.cssclasses[day], s) + finally: + locale.setlocale(locale.LC_TIME, oldlocale) + return result + + def formatmonthname(self, theyear, themonth, withyear=True): + oldlocale = locale.setlocale(locale.LC_TIME, self.locale) + try: + encoding = locale.getlocale(locale.LC_TIME)[1] + s = month_name[themonth] + if encoding is not None: + s = s.decode(encoding) + if withyear: + s = '%s %s' % (s, theyear) + result = '%s' % s + finally: + locale.setlocale(locale.LC_TIME, oldlocale) + return result # Support for old module level interface @@ -524,34 +614,60 @@ def main(args): import optparse - parser = optparse.OptionParser(usage="usage: %prog [options] [year] [month]") - parser.add_option("-w", "--width", - dest="width", type="int", default=2, - help="width of date column (default 2, text only)") - parser.add_option("-l", "--lines", - dest="lines", type="int", default=1, - help="number of lines for each week (default 1, text only)") - parser.add_option("-s", "--spacing", - dest="spacing", type="int", default=6, - help="spacing between months (default 6, text only)") - parser.add_option("-m", "--months", - dest="months", type="int", default=3, - help="months per row (default 3, text only)") - parser.add_option("-c", "--css", - dest="css", default="calendar.css", - help="CSS to use for page (html only)") - parser.add_option("-e", "--encoding", - dest="encoding", default=None, - help="Encoding to use for CSS output (html only)") - parser.add_option("-t", "--type", - dest="type", default="text", - choices=("text", "html"), - help="output type (text or html)") + parser = optparse.OptionParser(usage="usage: %prog [options] [year [month]]") + parser.add_option( + "-w", "--width", + dest="width", type="int", default=2, + help="width of date column (default 2, text only)" + ) + parser.add_option( + "-l", "--lines", + dest="lines", type="int", default=1, + help="number of lines for each week (default 1, text only)" + ) + parser.add_option( + "-s", "--spacing", + dest="spacing", type="int", default=6, + help="spacing between months (default 6, text only)" + ) + parser.add_option( + "-m", "--months", + dest="months", type="int", default=3, + help="months per row (default 3, text only)" + ) + parser.add_option( + "-c", "--css", + dest="css", default="calendar.css", + help="CSS to use for page (html only)" + ) + parser.add_option( + "-L", "--locale", + dest="locale", default=None, + help="locale to be used from month and weekday names" + ) + parser.add_option( + "-e", "--encoding", + dest="encoding", default=None, + help="Encoding to use for output" + ) + parser.add_option( + "-t", "--type", + dest="type", default="text", + choices=("text", "html"), + help="output type (text or html)" + ) (options, args) = parser.parse_args(args) + if options.locale and not options.encoding: + parser.error("if --locale is specified --encoding is required") + sys.exit(1) + if options.type == "html": - cal = HTMLCalendar() + if options.locale: + cal = LocaleHTMLCalendar(locale=options.locale) + else: + cal = HTMLCalendar() encoding = options.encoding if encoding is None: encoding = sys.getdefaultencoding() @@ -564,20 +680,26 @@ parser.error("incorrect number of arguments") sys.exit(1) else: - cal = TextCalendar() + if options.locale: + cal = LocaleTextCalendar(locale=options.locale) + else: + cal = TextCalendar() optdict = dict(w=options.width, l=options.lines) if len(args) != 3: optdict["c"] = options.spacing optdict["m"] = options.months if len(args) == 1: - print cal.formatyear(datetime.date.today().year, **optdict) + result = cal.formatyear(datetime.date.today().year, **optdict) elif len(args) == 2: - print cal.formatyear(int(args[1]), **optdict) + result = cal.formatyear(int(args[1]), **optdict) elif len(args) == 3: - print cal.formatmonth(int(args[1]), int(args[2]), **optdict) + result = cal.formatmonth(int(args[1]), int(args[2]), **optdict) else: parser.error("incorrect number of arguments") sys.exit(1) + if options.encoding: + result = result.encode(options.encoding) + print result if __name__ == "__main__": Modified: python/trunk/Misc/NEWS ============================================================================== --- python/trunk/Misc/NEWS (original) +++ python/trunk/Misc/NEWS Sat Apr 1 09:57:00 2006 @@ -920,7 +920,9 @@ - Bug #947906: An object oriented interface has been added to the calendar module. It's possible to generate HTML calendar now and the module can be - called as a script (e.g. via ``python -mcalendar``). + called as a script (e.g. via ``python -mcalendar``). Localized month and + weekday names can be ouput (even if an exotic encoding is used) using + special classes that use unicode. Build ----- From g.brandl at gmx.net Sat Apr 1 09:57:29 2006 From: g.brandl at gmx.net (Georg Brandl) Date: Sat, 01 Apr 2006 09:57:29 +0200 Subject: [Python-checkins] r43514 - in python/trunk: Lib/sqlite3 Lib/sqlite3/test Lib/test/regrtest.py Lib/test/test_sqlite.py Misc/NEWS Modules/_sqlite setup.py In-Reply-To: <20060401005733.117151E4003@bag.python.org> References: <20060401005733.117151E4003@bag.python.org> Message-ID: anthony.baxter wrote: > Author: anthony.baxter > Date: Sat Apr 1 02:57:31 2006 > New Revision: 43514 > > Added: > python/trunk/Lib/sqlite3/ (props changed) > - copied from r43513, python/branches/sqlite-integration/Lib/sqlite3/ > python/trunk/Lib/test/test_sqlite.py > - copied unchanged from r43513, python/branches/sqlite-integration/Lib/test/test_sqlite.py > python/trunk/Modules/_sqlite/ > - copied from r43513, python/branches/sqlite-integration/Modules/_sqlite/ > Modified: > python/trunk/Lib/sqlite3/test/ (props changed) > python/trunk/Lib/test/regrtest.py > python/trunk/Misc/NEWS > python/trunk/setup.py > Log: > merged the sqlite-integration branch. > This is based on pysqlite2.1.3, and provides a DB-API interface in > the standard library. You'll need sqlite 3.2.2 or later to build > this - if you have an earlier version, the C extension module will > not be built. One minor nit: the tp_name of sqlite3.Connection etc. is still "pysqlite2.dbapi2.Connection" etc. Do we care? Georg From python-checkins at python.org Sat Apr 1 10:35:20 2006 From: python-checkins at python.org (georg.brandl) Date: Sat, 1 Apr 2006 10:35:20 +0200 (CEST) Subject: [Python-checkins] r43532 - in python/trunk: Doc/lib/libsgmllib.tex Lib/sgmllib.py Lib/test/test_sgmllib.py Misc/NEWS Message-ID: <20060401083520.64E9A1E4003@bag.python.org> Author: georg.brandl Date: Sat Apr 1 10:35:18 2006 New Revision: 43532 Modified: python/trunk/Doc/lib/libsgmllib.tex python/trunk/Lib/sgmllib.py python/trunk/Lib/test/test_sgmllib.py python/trunk/Misc/NEWS Log: patch #1462498: handle entityrefs in attribute values. Modified: python/trunk/Doc/lib/libsgmllib.tex ============================================================================== --- python/trunk/Doc/lib/libsgmllib.tex (original) +++ python/trunk/Doc/lib/libsgmllib.tex Sat Apr 1 10:35:18 2006 @@ -95,12 +95,15 @@ should be used to support semantic interpretation of the start tag. The \var{attributes} argument is a list of \code{(\var{name}, \var{value})} pairs containing the attributes found inside the tag's -\code{<>} brackets. The \var{name} has been translated to lower case -and double quotes and backslashes in the \var{value} have been interpreted. +\code{<>} brackets. The \var{name} has been translated to lower case. +Double quotes and backslashes in the \var{value} have been interpreted, +as well as known entity and character references. For instance, for the tag \code{}, this method would be called as \samp{unknown_starttag('a', [('href', 'http://www.cwi.nl/')])}. The base implementation simply calls \var{method} with \var{attributes} as the only argument. +\versionadded[Handling of entity and character references within + attribute values]{2.5} \end{methoddesc} \begin{methoddesc}{handle_endtag}{tag, method} Modified: python/trunk/Lib/sgmllib.py ============================================================================== --- python/trunk/Lib/sgmllib.py (original) +++ python/trunk/Lib/sgmllib.py Sat Apr 1 10:35:18 2006 @@ -269,9 +269,37 @@ attrname, rest, attrvalue = match.group(1, 2, 3) if not rest: attrvalue = attrname - elif attrvalue[:1] == '\'' == attrvalue[-1:] or \ - attrvalue[:1] == '"' == attrvalue[-1:]: - attrvalue = attrvalue[1:-1] + else: + if (attrvalue[:1] == "'" == attrvalue[-1:] or + attrvalue[:1] == '"' == attrvalue[-1:]): + # strip quotes + attrvalue = attrvalue[1:-1] + l = 0 + new_attrvalue = '' + while l < len(attrvalue): + av_match = entityref.match(attrvalue, l) + if (av_match and av_match.group(1) in self.entitydefs and + attrvalue[av_match.end(1)] == ';'): + # only substitute entityrefs ending in ';' since + # otherwise we may break + # which is very common + new_attrvalue += self.entitydefs[av_match.group(1)] + l = av_match.end(0) + continue + ch_match = charref.match(attrvalue, l) + if ch_match: + try: + char = chr(int(ch_match.group(1))) + new_attrvalue += char + l = ch_match.end(0) + continue + except ValueError: + # invalid character reference, don't substitute + pass + # all other cases + new_attrvalue += attrvalue[l] + l += 1 + attrvalue = new_attrvalue attrs.append((attrname.lower(), attrvalue)) k = match.end(0) if rawdata[j] == '>': Modified: python/trunk/Lib/test/test_sgmllib.py ============================================================================== --- python/trunk/Lib/test/test_sgmllib.py (original) +++ python/trunk/Lib/test/test_sgmllib.py Sat Apr 1 10:35:18 2006 @@ -214,6 +214,20 @@ ("starttag", "e", [("a", "rgb(1,2,3)")]), ]) + def test_attr_values_entities(self): + """Substitution of entities and charrefs in attribute values""" + # SF bug #1452246 + self.check_events("""""", + [("starttag", "a", [("b", "<"), + ("c", "<>"), + ("d", "<->"), + ("e", "< "), + ("f", "&xxx;"), + ("g", " !"), + ("h", "Ǵ"), + ("i", "x?a=b&c=d;"), ])]) + def test_attr_funky_names(self): self.check_events("""""", [ ("starttag", "a", [("a.b", "v"), ("c:d", "v"), ("e-f", "v")]), Modified: python/trunk/Misc/NEWS ============================================================================== --- python/trunk/Misc/NEWS (original) +++ python/trunk/Misc/NEWS Sat Apr 1 10:35:18 2006 @@ -489,6 +489,9 @@ Library ------- +- Patch #1462498: sgmllib now handles entity and character references + in attribute values. + - Added the sqlite3 package. This is based on pysqlite2.1.3, and provides a DB-API interface in the standard library. You'll need sqlite 3.2.2 or later to build this - if you have an earlier version, the C extension From python-checkins at python.org Sat Apr 1 10:36:28 2006 From: python-checkins at python.org (anthony.baxter) Date: Sat, 1 Apr 2006 10:36:28 +0200 (CEST) Subject: [Python-checkins] r43533 - in python/trunk: Modules/_sqlite/cursor.c Modules/_sqlite/statement.c Modules/_sqlite/statement.h setup.py Message-ID: <20060401083628.EEBA11E4013@bag.python.org> Author: anthony.baxter Date: Sat Apr 1 10:36:27 2006 New Revision: 43533 Modified: python/trunk/Modules/_sqlite/cursor.c python/trunk/Modules/_sqlite/statement.c python/trunk/Modules/_sqlite/statement.h python/trunk/setup.py Log: backport r243 from the pysqlite2 svn repository - lowers the required version of SQLite3 from 3.2.2 to 3.0.8, by providing an alternative to sqlite3_transfer_bindings. setup.py also handles the common (in debian and ubuntu, at least) case of a buggy sqlite3.h SQLITE_VERSION_NUMBER. Modified: python/trunk/Modules/_sqlite/cursor.c ============================================================================== --- python/trunk/Modules/_sqlite/cursor.c (original) +++ python/trunk/Modules/_sqlite/cursor.c Sat Apr 1 10:36:27 2006 @@ -24,8 +24,6 @@ #include "cursor.h" #include "module.h" #include "util.h" -#include "microprotocols.h" -#include "prepare_protocol.h" /* used to decide wether to call PyInt_FromLong or PyLong_FromLongLong */ #define INT32_MIN (-2147483647 - 1) @@ -189,53 +187,6 @@ } } -int _bind_parameter(Cursor* self, int pos, PyObject* parameter) -{ - int rc = SQLITE_OK; - long longval; -#ifdef HAVE_LONG_LONG - PY_LONG_LONG longlongval; -#endif - const char* buffer; - char* string; - int buflen; - PyObject* stringval; - - if (parameter == Py_None) { - rc = sqlite3_bind_null(self->statement->st, pos); - } else if (PyInt_Check(parameter)) { - longval = PyInt_AsLong(parameter); - rc = sqlite3_bind_int64(self->statement->st, pos, (sqlite_int64)longval); -#ifdef HAVE_LONG_LONG - } else if (PyLong_Check(parameter)) { - longlongval = PyLong_AsLongLong(parameter); - /* in the overflow error case, longlongval is -1, and an exception is set */ - rc = sqlite3_bind_int64(self->statement->st, pos, (sqlite_int64)longlongval); -#endif - } else if (PyFloat_Check(parameter)) { - rc = sqlite3_bind_double(self->statement->st, pos, PyFloat_AsDouble(parameter)); - } else if (PyBuffer_Check(parameter)) { - if (PyObject_AsCharBuffer(parameter, &buffer, &buflen) == 0) { - rc = sqlite3_bind_blob(self->statement->st, pos, buffer, buflen, SQLITE_TRANSIENT); - } else { - PyErr_SetString(PyExc_ValueError, "could not convert BLOB to buffer"); - rc = -1; - } - } else if PyString_Check(parameter) { - string = PyString_AsString(parameter); - rc = sqlite3_bind_text(self->statement->st, pos, string, -1, SQLITE_TRANSIENT); - } else if PyUnicode_Check(parameter) { - stringval = PyUnicode_AsUTF8String(parameter); - string = PyString_AsString(stringval); - rc = sqlite3_bind_text(self->statement->st, pos, string, -1, SQLITE_TRANSIENT); - Py_DECREF(stringval); - } else { - rc = -1; - } - - return rc; -} - PyObject* _build_column_name(const char* colname) { const char* pos; @@ -394,7 +345,6 @@ PyObject* parameters_list = NULL; PyObject* parameters_iter = NULL; PyObject* parameters = NULL; - int num_params; int i; int rc; PyObject* func_args; @@ -403,11 +353,7 @@ PY_LONG_LONG lastrowid; int statement_type; PyObject* descriptor; - PyObject* current_param; - PyObject* adapted; PyObject* second_argument = NULL; - int num_params_needed; - const char* binding_name; long rowcount = 0; if (!check_thread(self->connection) || !check_connection(self->connection)) { @@ -557,10 +503,6 @@ statement_reset(self->statement); statement_mark_dirty(self->statement); - Py_BEGIN_ALLOW_THREADS - num_params_needed = sqlite3_bind_parameter_count(self->statement->st); - Py_END_ALLOW_THREADS - while (1) { parameters = PyIter_Next(parameters_iter); if (!parameters) { @@ -569,71 +511,9 @@ statement_mark_dirty(self->statement); - if (PyDict_Check(parameters)) { - /* parameters passed as dictionary */ - for (i = 1; i <= num_params_needed; i++) { - Py_BEGIN_ALLOW_THREADS - binding_name = sqlite3_bind_parameter_name(self->statement->st, i); - Py_END_ALLOW_THREADS - if (!binding_name) { - PyErr_Format(ProgrammingError, "Binding %d has no name, but you supplied a dictionary (which has only names).", i); - goto error; - } - - binding_name++; /* skip first char (the colon) */ - current_param = PyDict_GetItemString(parameters, binding_name); - if (!current_param) { - PyErr_Format(ProgrammingError, "You did not supply a value for binding %d.", i); - goto error; - } - - Py_INCREF(current_param); - adapted = microprotocols_adapt(current_param, (PyObject*)&SQLitePrepareProtocolType, NULL); - if (adapted) { - Py_DECREF(current_param); - } else { - PyErr_Clear(); - adapted = current_param; - } - - rc = _bind_parameter(self, i, adapted); - Py_DECREF(adapted); - - if (rc != SQLITE_OK) { - PyErr_Format(InterfaceError, "Error binding parameter :%s - probably unsupported type.", binding_name); - goto error; - } - } - } else { - /* parameters passed as sequence */ - num_params = PySequence_Length(parameters); - if (num_params != num_params_needed) { - PyErr_Format(ProgrammingError, "Incorrect number of bindings supplied. The current statement uses %d, and there are %d supplied.", - num_params_needed, num_params); - goto error; - } - for (i = 0; i < num_params; i++) { - current_param = PySequence_GetItem(parameters, i); - if (!current_param) { - goto error; - } - adapted = microprotocols_adapt(current_param, (PyObject*)&SQLitePrepareProtocolType, NULL); - - if (adapted) { - Py_DECREF(current_param); - } else { - PyErr_Clear(); - adapted = current_param; - } - - rc = _bind_parameter(self, i + 1, adapted); - Py_DECREF(adapted); - - if (rc != SQLITE_OK) { - PyErr_Format(InterfaceError, "Error binding parameter %d - probably unsupported type.", i); - goto error; - } - } + statement_bind_parameters(self->statement, parameters); + if (PyErr_Occurred()) { + goto error; } build_row_cast_map(self); @@ -642,7 +522,7 @@ if (rc != SQLITE_DONE && rc != SQLITE_ROW) { rc = statement_reset(self->statement); if (rc == SQLITE_SCHEMA) { - rc = statement_recompile(self->statement); + rc = statement_recompile(self->statement, parameters); if (rc == SQLITE_OK) { rc = _sqlite_step_with_busyhandler(self->statement->st, self->connection); } else { Modified: python/trunk/Modules/_sqlite/statement.c ============================================================================== --- python/trunk/Modules/_sqlite/statement.c (original) +++ python/trunk/Modules/_sqlite/statement.c Sat Apr 1 10:36:27 2006 @@ -22,7 +22,10 @@ */ #include "statement.h" +#include "cursor.h" #include "connection.h" +#include "microprotocols.h" +#include "prepare_protocol.h" /* prototypes */ int check_remaining_sql(const char* tail); @@ -82,7 +85,136 @@ return rc; } -int statement_recompile(Statement* self) +int statement_bind_parameter(Statement* self, int pos, PyObject* parameter) +{ + int rc = SQLITE_OK; + long longval; +#ifdef HAVE_LONG_LONG + PY_LONG_LONG longlongval; +#endif + const char* buffer; + char* string; + int buflen; + PyObject* stringval; + + if (parameter == Py_None) { + rc = sqlite3_bind_null(self->st, pos); + } else if (PyInt_Check(parameter)) { + longval = PyInt_AsLong(parameter); + rc = sqlite3_bind_int64(self->st, pos, (sqlite_int64)longval); +#ifdef HAVE_LONG_LONG + } else if (PyLong_Check(parameter)) { + longlongval = PyLong_AsLongLong(parameter); + /* in the overflow error case, longlongval is -1, and an exception is set */ + rc = sqlite3_bind_int64(self->st, pos, (sqlite_int64)longlongval); +#endif + } else if (PyFloat_Check(parameter)) { + rc = sqlite3_bind_double(self->st, pos, PyFloat_AsDouble(parameter)); + } else if (PyBuffer_Check(parameter)) { + if (PyObject_AsCharBuffer(parameter, &buffer, &buflen) == 0) { + rc = sqlite3_bind_blob(self->st, pos, buffer, buflen, SQLITE_TRANSIENT); + } else { + PyErr_SetString(PyExc_ValueError, "could not convert BLOB to buffer"); + rc = -1; + } + } else if PyString_Check(parameter) { + string = PyString_AsString(parameter); + rc = sqlite3_bind_text(self->st, pos, string, -1, SQLITE_TRANSIENT); + } else if PyUnicode_Check(parameter) { + stringval = PyUnicode_AsUTF8String(parameter); + string = PyString_AsString(stringval); + rc = sqlite3_bind_text(self->st, pos, string, -1, SQLITE_TRANSIENT); + Py_DECREF(stringval); + } else { + rc = -1; + } + + return rc; +} + +void statement_bind_parameters(Statement* self, PyObject* parameters) +{ + PyObject* current_param; + PyObject* adapted; + const char* binding_name; + int i; + int rc; + int num_params_needed; + int num_params; + + Py_BEGIN_ALLOW_THREADS + num_params_needed = sqlite3_bind_parameter_count(self->st); + Py_END_ALLOW_THREADS + + if (PyDict_Check(parameters)) { + /* parameters passed as dictionary */ + for (i = 1; i <= num_params_needed; i++) { + Py_BEGIN_ALLOW_THREADS + binding_name = sqlite3_bind_parameter_name(self->st, i); + Py_END_ALLOW_THREADS + if (!binding_name) { + PyErr_Format(ProgrammingError, "Binding %d has no name, but you supplied a dictionary (which has only names).", i); + return; + } + + binding_name++; /* skip first char (the colon) */ + current_param = PyDict_GetItemString(parameters, binding_name); + if (!current_param) { + PyErr_Format(ProgrammingError, "You did not supply a value for binding %d.", i); + return; + } + + Py_INCREF(current_param); + adapted = microprotocols_adapt(current_param, (PyObject*)&SQLitePrepareProtocolType, NULL); + if (adapted) { + Py_DECREF(current_param); + } else { + PyErr_Clear(); + adapted = current_param; + } + + rc = statement_bind_parameter(self, i, adapted); + Py_DECREF(adapted); + + if (rc != SQLITE_OK) { + PyErr_Format(InterfaceError, "Error binding parameter :%s - probably unsupported type.", binding_name); + return; + } + } + } else { + /* parameters passed as sequence */ + num_params = PySequence_Length(parameters); + if (num_params != num_params_needed) { + PyErr_Format(ProgrammingError, "Incorrect number of bindings supplied. The current statement uses %d, and there are %d supplied.", + num_params_needed, num_params); + return; + } + for (i = 0; i < num_params; i++) { + current_param = PySequence_GetItem(parameters, i); + if (!current_param) { + return; + } + adapted = microprotocols_adapt(current_param, (PyObject*)&SQLitePrepareProtocolType, NULL); + + if (adapted) { + Py_DECREF(current_param); + } else { + PyErr_Clear(); + adapted = current_param; + } + + rc = statement_bind_parameter(self, i + 1, adapted); + Py_DECREF(adapted); + + if (rc != SQLITE_OK) { + PyErr_Format(InterfaceError, "Error binding parameter %d - probably unsupported type.", i); + return; + } + } + } +} + +int statement_recompile(Statement* self, PyObject* params) { const char* tail; int rc; @@ -98,7 +230,17 @@ &tail); if (rc == SQLITE_OK) { + /* The efficient sqlite3_transfer_bindings is only available in SQLite + * version 3.2.2 or later. For older SQLite releases, that might not + * even define SQLITE_VERSION_NUMBER, we do it the manual way. + */ + #ifdef SQLITE_VERSION_NUMBER + #if SQLITE_VERSION_NUMBER >= 3002002 (void)sqlite3_transfer_bindings(self->st, new_st); + #endif + #else + statement_bind_parameters(self, params); + #endif (void)sqlite3_finalize(self->st); self->st = new_st; Modified: python/trunk/Modules/_sqlite/statement.h ============================================================================== --- python/trunk/Modules/_sqlite/statement.h (original) +++ python/trunk/Modules/_sqlite/statement.h Sat Apr 1 10:36:27 2006 @@ -45,7 +45,10 @@ int statement_create(Statement* self, Connection* connection, PyObject* sql); void statement_dealloc(Statement* self); -int statement_recompile(Statement* self); +int statement_bind_parameter(Statement* self, int pos, PyObject* parameter); +void statement_bind_parameters(Statement* self, PyObject* parameters); + +int statement_recompile(Statement* self, PyObject* parameters); int statement_finalize(Statement* self); int statement_reset(Statement* self); void statement_mark_dirty(Statement* self); Modified: python/trunk/setup.py ============================================================================== --- python/trunk/setup.py (original) +++ python/trunk/setup.py Sat Apr 1 10:36:27 2006 @@ -690,6 +690,7 @@ dblib_dir = None # The sqlite interface + sqlite_setup_debug = True # verbose debug prints from this script? # We hunt for "#define SQLITE_VERSION_NUMBER nnnnn" # We need to find a version >= 3002002 (> sqlite version 3.2.2) @@ -701,22 +702,37 @@ '/usr/local/include/sqlite', '/usr/local/include/sqlite3', ] - MIN_SQLITE_VERSION = 3002002 + MIN_SQLITE_VERSION_NUMBER = 3000008 + MIN_SQLITE_VERSION = "3.0.8" for d in sqlite_inc_paths + inc_dirs: f = os.path.join(d, "sqlite3.h") if os.path.exists(f): - if db_setup_debug: print "found %s"%f + if sqlite_setup_debug: print "sqlite: found %s"%f f = open(f).read() m = re.search(r"#define\WSQLITE_VERSION_NUMBER\W(\d+)", f) if m: sqlite_version = int(m.group(1)) - if sqlite_version >= MIN_SQLITE_VERSION: + if sqlite_version >= MIN_SQLITE_VERSION_NUMBER: # we win! print "%s/sqlite3.h: version %s"%(d, sqlite_version) sqlite_incdir = d break + elif sqlite_version == 3000000: + # Bug in a common version out there where + # SQLITE_VERSION_NUMBER was set incorrectly + if sqlite_setup_debug: + print "found buggy SQLITE_VERSION_NUMBER, checking" + m = re.search(r'#define\WSQLITE_VERSION\W+"([\.\d]+)"', + f) + if m: + sqlite_version = m.group(1) + if sqlite_version >= MIN_SQLITE_VERSION: + print "%s/sqlite3.h: version %s"%(d, + sqlite_version) + sqlite_incdir = d + break else: - if db_setup_debug: + if sqlite_setup_debug: print "%s: version %d is too old, need >= %s"%(d, sqlite_version, MIN_SQLITE_VERSION) From python-checkins at python.org Sat Apr 1 10:39:52 2006 From: python-checkins at python.org (georg.brandl) Date: Sat, 1 Apr 2006 10:39:52 +0200 (CEST) Subject: [Python-checkins] r43534 - python/trunk/Doc/lib/libsgmllib.tex Message-ID: <20060401083952.793AB1E4003@bag.python.org> Author: georg.brandl Date: Sat Apr 1 10:39:50 2006 New Revision: 43534 Modified: python/trunk/Doc/lib/libsgmllib.tex Log: clarify new docs on attr value entity expansion Modified: python/trunk/Doc/lib/libsgmllib.tex ============================================================================== --- python/trunk/Doc/lib/libsgmllib.tex (original) +++ python/trunk/Doc/lib/libsgmllib.tex Sat Apr 1 10:39:50 2006 @@ -95,9 +95,16 @@ should be used to support semantic interpretation of the start tag. The \var{attributes} argument is a list of \code{(\var{name}, \var{value})} pairs containing the attributes found inside the tag's -\code{<>} brackets. The \var{name} has been translated to lower case. +\code{<>} brackets. + +The \var{name} has been translated to lower case. Double quotes and backslashes in the \var{value} have been interpreted, -as well as known entity and character references. +as well as known character references and known entity references +terminated by a semicolon (normally, entity references can be terminated +by any non-alphanumerical character, but this would break the very +common case of \code{} when \code{eggs} +is a valid entity name). + For instance, for the tag \code{}, this method would be called as \samp{unknown_starttag('a', [('href', 'http://www.cwi.nl/')])}. The base implementation simply calls From python-checkins at python.org Sat Apr 1 10:50:37 2006 From: python-checkins at python.org (neal.norwitz) Date: Sat, 1 Apr 2006 10:50:37 +0200 (CEST) Subject: [Python-checkins] r43535 - in python/branches/release24-maint: Misc/NEWS configure configure.in Message-ID: <20060401085037.15AC91E401A@bag.python.org> Author: neal.norwitz Date: Sat Apr 1 10:50:35 2006 New Revision: 43535 Modified: python/branches/release24-maint/Misc/NEWS python/branches/release24-maint/configure python/branches/release24-maint/configure.in Log: Backport 43470: Get gcc to do strict IEEE math. This flag was already used for cc, just not gcc. Without this flag, doing pretty much anything with NaNs causes a Floating Point Exception signal. This causes the interpreter to quit. The failing tests this fixes are: test_long. This is somewhat equivalent to doing signal(SIGFPE, SIG_IGN). Modified: python/branches/release24-maint/Misc/NEWS ============================================================================== --- python/branches/release24-maint/Misc/NEWS (original) +++ python/branches/release24-maint/Misc/NEWS Sat Apr 1 10:50:35 2006 @@ -9,6 +9,12 @@ *Release date: DD-MMM-2006* +Core and builtins +----------------- + +Extension Modules +----------------- + Library ------- @@ -17,6 +23,14 @@ raise an exception some day. But dicts have been allowed, and "mostly worked", so support for them won't go away without warning. +Tools/Demos +----------- + +Build +----- + +- Fix test_long failure on Tru64 with gcc by using -mieee gcc option. + What's New in Python 2.4.3? =========================== Modified: python/branches/release24-maint/configure ============================================================================== --- python/branches/release24-maint/configure (original) +++ python/branches/release24-maint/configure Sat Apr 1 10:50:35 2006 @@ -1,5 +1,5 @@ #! /bin/sh -# From configure.in Revision: 41985 . +# From configure.in Revision: 43158 . # Guess values for system-dependent variables and create Makefiles. # Generated by GNU Autoconf 2.59 for python 2.4. # @@ -3839,6 +3839,9 @@ Darwin*) BASECFLAGS="$BASECFLAGS -Wno-long-double -no-cpp-precomp -mno-fused-madd" ;; + OSF*) + BASECFLAGS="$BASECFLAGS -mieee" + ;; esac ;; Modified: python/branches/release24-maint/configure.in ============================================================================== --- python/branches/release24-maint/configure.in (original) +++ python/branches/release24-maint/configure.in Sat Apr 1 10:50:35 2006 @@ -737,6 +737,9 @@ Darwin*) BASECFLAGS="$BASECFLAGS -Wno-long-double -no-cpp-precomp -mno-fused-madd" ;; + OSF*) + BASECFLAGS="$BASECFLAGS -mieee" + ;; esac ;; From python-checkins at python.org Sat Apr 1 10:51:26 2006 From: python-checkins at python.org (georg.brandl) Date: Sat, 1 Apr 2006 10:51:26 +0200 (CEST) Subject: [Python-checkins] r43536 - python/trunk/configure python/trunk/configure.in Message-ID: <20060401085126.90ECE1E4003@bag.python.org> Author: georg.brandl Date: Sat Apr 1 10:51:25 2006 New Revision: 43536 Modified: python/trunk/configure python/trunk/configure.in Log: patch #1416559: don't define _XOPEN_SOURCE on Mac OS Panther Modified: python/trunk/configure ============================================================================== --- python/trunk/configure (original) +++ python/trunk/configure Sat Apr 1 10:51:25 2006 @@ -1513,7 +1513,9 @@ ;; # On Mac OS X 10.4, defining _POSIX_C_SOURCE or _XOPEN_SOURCE # disables platform specific features beyond repair. - Darwin/8.*) + # On Mac OS X 10.3, defining _POSIX_C_SOURCE or _XOPEN_SOURCE + # has no effect, don't bother defineing them + Darwin/[78].*) define_xopen_source=no ;; Modified: python/trunk/configure.in ============================================================================== --- python/trunk/configure.in (original) +++ python/trunk/configure.in Sat Apr 1 10:51:25 2006 @@ -173,7 +173,9 @@ ;; # On Mac OS X 10.4, defining _POSIX_C_SOURCE or _XOPEN_SOURCE # disables platform specific features beyond repair. - Darwin/8.*) + # On Mac OS X 10.3, defining _POSIX_C_SOURCE or _XOPEN_SOURCE + # has no effect, don't bother defineing them + Darwin/[78].*) define_xopen_source=no ;; From python-checkins at python.org Sat Apr 1 10:51:30 2006 From: python-checkins at python.org (georg.brandl) Date: Sat, 1 Apr 2006 10:51:30 +0200 (CEST) Subject: [Python-checkins] r43537 - python/branches/release24-maint/configure python/branches/release24-maint/configure.in Message-ID: <20060401085130.05F971E4003@bag.python.org> Author: georg.brandl Date: Sat Apr 1 10:51:29 2006 New Revision: 43537 Modified: python/branches/release24-maint/configure python/branches/release24-maint/configure.in Log: patch #1416559: don't define _XOPEN_SOURCE on Mac OS Panther (backport from rev. 43536) Modified: python/branches/release24-maint/configure ============================================================================== --- python/branches/release24-maint/configure (original) +++ python/branches/release24-maint/configure Sat Apr 1 10:51:29 2006 @@ -1513,7 +1513,9 @@ ;; # On Mac OS X 10.4, defining _POSIX_C_SOURCE or _XOPEN_SOURCE # disables platform specific features beyond repair. - Darwin/8.*) + # On Mac OS X 10.3, defining _POSIX_C_SOURCE or _XOPEN_SOURCE + # has no effect, don't bother defineing them + Darwin/[78].*) define_xopen_source=no ;; Modified: python/branches/release24-maint/configure.in ============================================================================== --- python/branches/release24-maint/configure.in (original) +++ python/branches/release24-maint/configure.in Sat Apr 1 10:51:29 2006 @@ -173,7 +173,9 @@ ;; # On Mac OS X 10.4, defining _POSIX_C_SOURCE or _XOPEN_SOURCE # disables platform specific features beyond repair. - Darwin/8.*) + # On Mac OS X 10.3, defining _POSIX_C_SOURCE or _XOPEN_SOURCE + # has no effect, don't bother defineing them + Darwin/[78].*) define_xopen_source=no ;; From buildbot at python.org Sat Apr 1 10:55:38 2006 From: buildbot at python.org (buildbot at python.org) Date: Sat, 01 Apr 2006 08:55:38 +0000 Subject: [Python-checkins] buildbot warnings in amd64 gentoo trunk Message-ID: <20060401085538.24E2F1E4003@bag.python.org> The Buildbot has detected a new failure of amd64 gentoo trunk. Full details are available at: http://www.python.org/dev/buildbot/all/amd64%20gentoo%20trunk/builds/253 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: The web-page 'force build' button was pressed by 'anthony baxter': sqlite testing Build Source Stamp: [branch trunk] HEAD Blamelist: Build Had Warnings: warnings test sincerely, -The Buildbot From anthony at interlink.com.au Sat Apr 1 10:56:07 2006 From: anthony at interlink.com.au (Anthony Baxter) Date: Sat, 1 Apr 2006 19:56:07 +1100 Subject: [Python-checkins] r43514 - in python/trunk: Lib/sqlite3 Lib/sqlite3/test Lib/test/regrtest.py Lib/test/test_sqlite.py Misc/NEWS Modules/_sqlite setup.py In-Reply-To: References: <20060401005733.117151E4003@bag.python.org> Message-ID: <200604011956.09808.anthony@interlink.com.au> On Saturday 01 April 2006 18:57, Georg Brandl wrote: > One minor nit: the tp_name of sqlite3.Connection etc. is still > "pysqlite2.dbapi2.Connection" etc. Do we care? Should be fixed. I'll do that tomorrow if no-one beats me to it. Actually, that stuff should really be a #define, so that the pysqlite2 and Python sqlite modules don't diverge too far. From python-checkins at python.org Sat Apr 1 10:59:04 2006 From: python-checkins at python.org (georg.brandl) Date: Sat, 1 Apr 2006 10:59:04 +0200 (CEST) Subject: [Python-checkins] r43538 - python/trunk/Misc/NEWS Message-ID: <20060401085904.3DFB71E4003@bag.python.org> Author: georg.brandl Date: Sat Apr 1 10:59:03 2006 New Revision: 43538 Modified: python/trunk/Misc/NEWS Log: Update SQLite version requirement. Modified: python/trunk/Misc/NEWS ============================================================================== --- python/trunk/Misc/NEWS (original) +++ python/trunk/Misc/NEWS Sat Apr 1 10:59:03 2006 @@ -493,7 +493,7 @@ in attribute values. - Added the sqlite3 package. This is based on pysqlite2.1.3, and provides - a DB-API interface in the standard library. You'll need sqlite 3.2.2 or + a DB-API interface in the standard library. You'll need sqlite 3.0.8 or later to build this - if you have an earlier version, the C extension module will not be built. From python-checkins at python.org Sat Apr 1 11:08:06 2006 From: python-checkins at python.org (neal.norwitz) Date: Sat, 1 Apr 2006 11:08:06 +0200 (CEST) Subject: [Python-checkins] r43539 - python/trunk/Modules/_sqlite/connection.c python/trunk/Modules/_sqlite/cursor.c python/trunk/Modules/_sqlite/row.c python/trunk/Modules/_sqlite/statement.c Message-ID: <20060401090806.E4FEA1E4003@bag.python.org> Author: neal.norwitz Date: Sat Apr 1 11:08:06 2006 New Revision: 43539 Modified: python/trunk/Modules/_sqlite/connection.c python/trunk/Modules/_sqlite/cursor.c python/trunk/Modules/_sqlite/row.c python/trunk/Modules/_sqlite/statement.c Log: Make ssize_t clean Modified: python/trunk/Modules/_sqlite/connection.c ============================================================================== --- python/trunk/Modules/_sqlite/connection.c (original) +++ python/trunk/Modules/_sqlite/connection.c Sat Apr 1 11:08:06 2006 @@ -377,7 +377,7 @@ { long longval; const char* buffer; - int buflen; + Py_ssize_t buflen; PyObject* stringval; if (PyErr_Occurred()) { @@ -416,7 +416,7 @@ PyObject* cur_py_value; const char* val_str; PY_LONG_LONG val_int; - int buflen; + Py_ssize_t buflen; void* raw_buffer; args = PyTuple_New(argc); Modified: python/trunk/Modules/_sqlite/cursor.c ============================================================================== --- python/trunk/Modules/_sqlite/cursor.c (original) +++ python/trunk/Modules/_sqlite/cursor.c Sat Apr 1 11:08:06 2006 @@ -239,7 +239,7 @@ PY_LONG_LONG intval; PyObject* converter; PyObject* converted; - int nbytes; + Py_ssize_t nbytes; PyObject* buffer; void* raw_buffer; const char* val_str; Modified: python/trunk/Modules/_sqlite/row.c ============================================================================== --- python/trunk/Modules/_sqlite/row.c (original) +++ python/trunk/Modules/_sqlite/row.c Sat Apr 1 11:08:06 2006 @@ -126,7 +126,7 @@ } } -int row_length(Row* self, PyObject* args, PyObject* kwargs) +Py_ssize_t row_length(Row* self, PyObject* args, PyObject* kwargs) { return PyTuple_GET_SIZE(self->data); } @@ -138,7 +138,7 @@ PyMappingMethods row_as_mapping = { - /* mp_length */ (inquiry)row_length, + /* mp_length */ (lenfunc)row_length, /* mp_subscript */ (binaryfunc)row_subscript, /* mp_ass_subscript */ (objobjargproc)0, }; Modified: python/trunk/Modules/_sqlite/statement.c ============================================================================== --- python/trunk/Modules/_sqlite/statement.c (original) +++ python/trunk/Modules/_sqlite/statement.c Sat Apr 1 11:08:06 2006 @@ -94,7 +94,7 @@ #endif const char* buffer; char* string; - int buflen; + Py_ssize_t buflen; PyObject* stringval; if (parameter == Py_None) { From buildbot at python.org Sat Apr 1 11:39:55 2006 From: buildbot at python.org (buildbot at python.org) Date: Sat, 01 Apr 2006 09:39:55 +0000 Subject: [Python-checkins] buildbot warnings in x86 gentoo 2.4 Message-ID: <20060401093955.3D8FF1E4003@bag.python.org> The Buildbot has detected a new failure of x86 gentoo 2.4. Full details are available at: http://www.python.org/dev/buildbot/all/x86%20gentoo%202.4/builds/60 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch branches/release24-maint] HEAD Blamelist: georg.brandl,neal.norwitz Build Had Warnings: warnings test sincerely, -The Buildbot From python-checkins at python.org Sat Apr 1 12:32:14 2006 From: python-checkins at python.org (andrew.kuchling) Date: Sat, 1 Apr 2006 12:32:14 +0200 (CEST) Subject: [Python-checkins] r43540 - python/trunk/Doc/whatsnew/whatsnew24.tex Message-ID: <20060401103214.06B891E4003@bag.python.org> Author: andrew.kuchling Date: Sat Apr 1 12:32:13 2006 New Revision: 43540 Modified: python/trunk/Doc/whatsnew/whatsnew24.tex Log: Update status of this PEP Modified: python/trunk/Doc/whatsnew/whatsnew24.tex ============================================================================== --- python/trunk/Doc/whatsnew/whatsnew24.tex (original) +++ python/trunk/Doc/whatsnew/whatsnew24.tex Sat Apr 1 12:32:13 2006 @@ -803,8 +803,8 @@ The PEP also proposes that all \keyword{import} statements be absolute imports, with a leading \samp{.} character to indicate a relative -import. This part of the PEP is not yet implemented, and will have to -wait for Python 2.5 or some other future version. +import. This part of the PEP was not implemented for Python 2.4, +but was completed for Python 2.5. \begin{seealso} \seepep{328}{Imports: Multi-Line and Absolute/Relative} From neal at metaslash.com Sat Apr 1 12:36:50 2006 From: neal at metaslash.com (Neal Norwitz) Date: Sat, 1 Apr 2006 05:36:50 -0500 Subject: [Python-checkins] Python Regression Test Failures doc (1) Message-ID: <20060401103650.GA832@python.psfb.org> TEXINPUTS=/home/neal/python/trunk/Doc/commontex: python /home/neal/python/trunk/Doc/tools/mkhowto --html --about html/stdabout.dat --iconserver ../icons --favicon ../icons/pyfav.png --address "See About this document... for information on suggesting changes." --up-link ../index.html --up-title "Python Documentation Index" --global-module-index "../modindex.html" --dvips-safe --dir html/lib lib/lib.tex *** Session transcript and error messages are in /home/neal/python/trunk/Doc/html/lib/lib.how. *** Exited with status 1. The relevant lines from the transcript are: ------------------------------------------------------------------------ +++ latex lib This is TeX, Version 3.14159 (Web2C 7.4.5) (/home/neal/python/trunk/Doc/lib/lib.tex LaTeX2e <2001/06/01> Babel and hyphenation patterns for american, french, german, ngerman, n ohyphenation, loaded. (/home/neal/python/trunk/Doc/texinputs/manual.cls Document Class: manual 1998/03/03 Document class (Python manual) (/home/neal/python/trunk/Doc/texinputs/pypaper.sty (/usr/share/texmf/tex/latex/psnfss/times.sty) Using Times instead of Computer Modern. ) (/usr/share/texmf/tex/latex/misc/fancybox.sty Style option: `fancybox' v1.3 <2000/09/19> (tvz) ) (/usr/share/texmf/tex/latex/base/report.cls Document Class: report 2001/04/21 v1.4e Standard LaTeX document class (/usr/share/texmf/tex/latex/base/size10.clo)) (/home/neal/python/trunk/Doc/texinputs/fancyhdr.sty) Using fancier footers than usual. (/home/neal/python/trunk/Doc/texinputs/fncychap.sty) Using fancy chapter headings. (/home/neal/python/trunk/Doc/texinputs/python.sty (/usr/share/texmf/tex/latex/tools/longtable.sty) (/home/neal/python/trunk/Doc/texinputs/underscore.sty) (/usr/share/texmf/tex/latex/tools/verbatim.sty) (/usr/share/texmf/tex/latex/base/alltt.sty))) (/home/neal/python/trunk/Doc/commontex/boilerplate.tex (/home/neal/python/trunk/Doc/commontex/patchlevel.tex)) Writing index file lib.idx No file lib.aux. (/usr/share/texmf/tex/latex/psnfss/ot1ptm.fd) (/usr/share/texmf/tex/latex/psnfss/ot1phv.fd) [1] (/home/neal/python/trunk/Doc/commontex/copyright.tex (/usr/share/texmf/tex/latex/psnfss/omsptm.fd)) [2] Adding blank page after the abstract. [1] [2] No file lib.toc. Adding blank page after the table of contents. [1] [2] (/home/neal/python/trunk/Doc/lib/libintro.tex Chapter 1. (/usr/share/texmf/tex/latex/psnfss/ot1pcr.fd) LaTeX Warning: Reference `builtin' on page 1 undefined on input line 49. ) (/home/neal/python/trunk/Doc/lib/libobjs.tex [1] [2] Chapter 2. ) (/home/neal/python/trunk/Doc/lib/libfuncs.tex Underfull \hbox (badness 6188) in paragraph at lines 21--33 \OT1/pcr/m/n/10 _-import_-_-('spam.ham', globals(), locals(), ['eggs'])\OT1/ptm /m/n/10 '. Note that even though [3] [4] [5] [6] LaTeX Warning: Reference `bltin-file-objects' on page 7 undefined on input line 396. [7] Underfull \hbox (badness 10000) in paragraph at lines 466--470 []\OT1/ptm/m/n/10 Note that \OT1/pcr/m/n/10 filter(function, \OT1/ptm/m/it/10 l ist\OT1/pcr/m/n/10 ) \OT1/ptm/m/n/10 is equiv-a-lent to \OT1/pcr/m/n/10 [item f or item in \OT1/ptm/m/it/10 list \OT1/pcr/m/n/10 if [8] [9] [10] [11] [12] [13] [14] [15]) (/home/neal/python/trunk/Doc/lib/libstdtypes.tex [16] [17] [18] LaTeX Warning: Reference `built-in-funcs' on page 19 undefined on input line 29 6. [19] [20] [21] LaTeX Warning: Reference `codec-base-classes' on page 22 undefined on input lin e 600. [22] LaTeX Warning: Reference `codec-base-classes' on page 23 undefined on input lin e 613. LaTeX Warning: Reference `standard-encodings' on page 23 undefined on input lin e 614. [23] [24] [25] Overfull \hbox (8.48134pt too wide) in paragraph at lines 950--960 [] [26] [27] Overfull \hbox (21.88011pt too wide) in paragraph at lines 1057--1088 [] [28] [29] Overfull \hbox (98.60141pt too wide) in paragraph at lines 1268--1296 [] [30] [31] LaTeX Warning: Reference `built-in-funcs' on page 32 undefined on input line 14 83. LaTeX Warning: Reference `context-closing' on page 32 undefined on input line 1 535. [32] [33] [34] [35] [36]) (/home/neal/python/trunk/Doc/lib/libexcs.tex [37] Underfull \hbox (badness 10000) in paragraph at lines 94--98 \OT1/ptm/m/n/10 The base class for all built-in ex-cep-tions ex-cept \OT1/pcr/m /n/10 StopIteration\OT1/ptm/m/n/10 , \OT1/pcr/m/n/10 GeneratorExit\OT1/ptm/m/n/ 10 , [38] [39] [40] No file ../../Lib/test/exception_hierarchy.txt. ) (/home/neal/python/trunk/Doc/lib/libconsts.tex [41]) (/home/neal/python/trunk/Doc/lib/libstrings.tex [42] Chapter 3. LaTeX Warning: Reference `string-methods' on page 43 undefined on input line 10 . ) (/home/neal/python/trunk/Doc/lib/libstring.tex [43] [44] [45] LaTeX Warning: Reference `string-methods' on page 46 undefined on input line 23 7. [46] [47]) (/home/neal/python/trunk/Doc/lib/libre.tex [48] [49] [50] [51] [52] [53] [54] [55] [56] [57]) (/home/neal/python/trunk/Doc/lib/libstruct.tex [58] [59] LaTeX Warning: Reference `module-array' on page 60 undefined on input line 195. LaTeX Warning: Reference `module-xdrlib' on page 60 undefined on input line 196 . ) (/home/neal/python/trunk/Doc/lib/libdifflib.tex [60] [61] [62] [63] [64] [65] [66] [67]) (/home/neal/python/trunk/Doc/lib/libstringio.tex [68] LaTeX Warning: Reference `bltin-file-objects' on page 69 undefined on input lin e 11. [69]) (/home/neal/python/trunk/Doc/lib/libtextwrap.tex [70] [71]) (/home/neal/python/trunk/Doc/lib/libcodecs.tex Underfull \hbox (badness 5161) in paragraph at lines 52--56 []\OT1/ptm/m/n/10 The fac-tory func-tions must re-turn ob-jects pro-vid-ing the in-ter-faces de-fined by the base classes [72] [73] [74] Overfull \hbox (405.07822pt too wide) in paragraph at lines 288--300 [] [75] [76] [77] [78] [79] [80] Overfull \hbox (11.18082pt too wide) in alignment at lines 820--901 [] [] [] Overfull \hbox (11.18082pt too wide) in alignment at lines 901--981 [] [] [] Overfull \hbox (254.7505pt too wide) in alignment at lines 981--1062 [] [] [] [81] Overfull \hbox (254.7505pt too wide) in alignment at lines 1062--1142 [] [] [] Overfull \hbox (254.7505pt too wide) in alignment at lines 1142--1167 [] [] [] Package longtable Warning: Column widths have changed (longtable) in table 3.1 on input line 1167. [82] Overfull \hbox (465.03658pt too wide) in paragraph at lines 1179--1270 [] [83]) (/home/neal/python/trunk/Doc/lib/libunicodedata.tex [84]) (/home/neal/python/trunk/Doc/lib/libstringprep.tex [85] [86]) (/home/neal/python/trunk/Doc/lib/libfpformat.tex) Underfull \hbox (badness 10000) in paragraph at lines 50--98 (/home/neal/python/trunk/Doc/lib/datatypes.tex [87] [88] Chapter 4. ) (/home/neal/python/trunk/Doc/lib/libdatetime.tex LaTeX Warning: Reference `module-calendar' on page 89 undefined on input line 5 6. LaTeX Warning: Reference `module-time' on page 89 undefined on input line 57. [89] [90] Underfull \hbox (badness 10000) in paragraph at lines 185--187 \OT1/ptm/m/n/10 The small-est pos-si-ble dif-fer-ence be-tween non-equal \OT1/p cr/m/n/10 timedelta \OT1/ptm/m/n/10 ob-jects, (/usr/share/texmf/tex/latex/psnfss/omlptm.fd) Overfull \hbox (44.65097pt too wide) in paragraph at lines 204--235 [] [91] [92] Underfull \hbox (badness 10000) in paragraph at lines 430--439 \OT1/ptm/m/it/10 d\OT1/pcr/m/n/10 .month, \OT1/ptm/m/it/10 d\OT1/pcr/m/n/10 .da y, 0, 0, 0, \OT1/ptm/m/it/10 d\OT1/pcr/m/n/10 .weekday(), \OT1/ptm/m/it/10 d\OT 1/pcr/m/n/10 .toordinal() - date(\OT1/ptm/m/it/10 d\OT1/pcr/m/n/10 .year, 1, [93] Underfull \hbox (badness 10000) in paragraph at lines 464--467 []\OT1/ptm/m/n/10 The ISO cal-en-dar is a widely used vari-ant of the Gre-go-ri an cal-en-dar. See LaTeX Warning: Reference `strftime-behavior' on page 94 undefined on input line 507. [94] Underfull \hbox (badness 10000) in paragraph at lines 547--551 \OT1/ptm/m/n/10 Return the cur-rent lo-cal date-time, with \OT1/pcr/m/n/10 tzin fo None\OT1/ptm/m/n/10 . This is equiv-a-lent to Underfull \hbox (badness 10000) in paragraph at lines 561--566 []\OT1/ptm/m/n/10 Else \OT1/ptm/m/it/10 tz \OT1/ptm/m/n/10 must be an in-stance of a class \OT1/pcr/m/n/10 tzinfo \OT1/ptm/m/n/10 sub-class, and the cur-rent date Underfull \hbox (badness 10000) in paragraph at lines 561--566 \OT1/ptm/m/n/10 and time are con-verted to \OT1/ptm/m/it/10 tz\OT1/ptm/m/n/10 ' s time zone. In this case the re-sult is equiv-a-lent to Underfull \hbox (badness 10000) in paragraph at lines 582--586 []\OT1/ptm/m/n/10 Else \OT1/ptm/m/it/10 tz \OT1/ptm/m/n/10 must be an in-stance of a class \OT1/pcr/m/n/10 tzinfo \OT1/ptm/m/n/10 sub-class, and the times- Underfull \hbox (badness 10000) in paragraph at lines 582--586 \OT1/ptm/m/n/10 tamp is con-verted to \OT1/ptm/m/it/10 tz\OT1/ptm/m/n/10 's tim e zone. In this case the re-sult is equiv-a-lent to [95] Underfull \hbox (badness 5519) in paragraph at lines 646--648 \OT1/ptm/m/n/10 The lat-est rep-re-sentable \OT1/pcr/m/n/10 datetime\OT1/ptm/m/ n/10 , \OT1/pcr/m/n/10 datetime(MAXYEAR, 12, 31, 23, 59, 59, 999999, [96] [97] Underfull \hbox (badness 10000) in paragraph at lines 874--888 \OT1/ptm/m/it/10 d\OT1/pcr/m/n/10 .weekday(), \OT1/ptm/m/it/10 d\OT1/pcr/m/n/10 .toordinal() - date(\OT1/ptm/m/it/10 d\OT1/pcr/m/n/10 .year, 1, 1).toordinal() + 1, dst)) \OT1/ptm/m/n/10 The Underfull \hbox (badness 10000) in paragraph at lines 923--925 \OT1/ptm/m/n/10 Return a 3-tuple, (ISO year, ISO week num-ber, ISO week-day). T he same as [98] Underfull \hbox (badness 5064) in paragraph at lines 960--969 \OT1/ptm/m/n/10 Return a string rep-re-sent-ing the date and time, for ex-am-pl e \OT1/pcr/m/n/10 datetime(2002, 12, 4, 20, Underfull \hbox (badness 10000) in paragraph at lines 960--969 \OT1/pcr/m/n/10 30, 40).ctime() == 'Wed Dec 4 20:30:40 2002'\OT1/ptm/m/n/10 . \ OT1/pcr/m/n/10 d.ctime() \OT1/ptm/m/n/10 is equiv-a-lent to LaTeX Warning: Reference `strftime-behavior' on page 99 undefined on input line 973. [99] LaTeX Warning: Reference `strftime-behavior' on page 100 undefined on input lin e 1103. [100] [101] [102] [103] [104] [105]) (/home/neal/python/trunk/Doc/lib/libcalendar.tex [106] [107] [108] [109] LaTeX Warning: Reference `module-datetime' on page 110 undefined on input line 311. LaTeX Warning: Reference `module-time' on page 110 undefined on input line 312. ) (/home/neal/python/trunk/Doc/lib/libcollections.tex [110] [111] [112] [113] [114] Overfull \hbox (6.89723pt too wide) in paragraph at lines 337--337 []\OT1/pcr/m/n/9 >>> s = [('red', 1), ('blue', 2), ('red', 3), ('blue', 4), ('r ed', 1), ('blue', 4)][] ) (/home/neal/python/trunk/Doc/lib/libheapq.tex [115] [116] [117]) (/home/neal/python/trunk/Doc/lib/libbisect.tex) (/home/neal/python/trunk/Doc/lib/libarray.tex [118] [119] [120] LaTeX Warning: Reference `module-struct' on page 121 undefined on input line 23 0. LaTeX Warning: Reference `module-xdrlib' on page 121 undefined on input line 23 3. [121]) (/home/neal/python/trunk/Doc/lib/libsets.tex LaTeX Warning: Reference `immutable-transforms' on page 122 undefined on input line 49. LaTeX Warning: Reference `immutable-transforms' on page 122 undefined on input line 57. [122] Overfull \hbox (98.60141pt too wide) in paragraph at lines 135--163 [] [123] [124]) (/home/neal/python/trunk/Doc/lib/libsched.tex [125]) (/home/neal/python/trunk/Doc/lib/libmutex.tex [126]) (/home/neal/python/trunk/Doc/lib/libqueue.tex [127]) (/home/neal/python/trunk/Doc/lib/libweakref.tex [128] LaTeX Warning: Reference `weakref-extension' on page 129 undefined on input lin e 69. [129] [130] [131] [132]) (/home/neal/python/trunk/Doc/lib/libuserdict.tex [133] LaTeX Warning: Reference `typesmapping' on page 134 undefined on input line 40. LaTeX Warning: Reference `typesseq' on page 134 undefined on input line 97. [134] LaTeX Warning: Reference `string-methods' on page 135 undefined on input line 1 74. ) (/home/neal/python/trunk/Doc/lib/libtypes.tex [135] [136] [137]) (/home/neal/python/trunk/Doc/lib/libnew.tex) (/home/neal/python/trunk/Doc/lib/libcopy.tex [138] LaTeX Warning: Reference `module-pickle' on page 139 undefined on input line 96 . ) (/home/neal/python/trunk/Doc/lib/libpprint.tex [139] [140]) (/home/neal/python/trunk/Doc/lib/librepr.tex [141] [142]) Underfull \hbox (badness 10000) in paragraph at lines 120--121 (/home/neal/python/trunk/Doc/lib/numeric.tex [143] [144] Chapter 5. ) (/home/neal/python/trunk/Doc/lib/libmath.tex [145] [146] LaTeX Warning: Reference `module-cmath' on page 147 undefined on input line 208 . ) (/home/neal/python/trunk/Doc/lib/libcmath.tex [147]) (/home/neal/python/trunk/Doc/lib/libdecimal.tex [148] [149] [150] [151] [152] Overfull \hbox (21.09727pt too wide) in paragraph at lines 305--305 [] \OT1/pcr/m/n/9 digit ::= '0' | '1' | '2' | '3' | '4' | '5' | '6 ' | '7' | '8' | '9'[] [153] [154] [155] [156] [157] [158] [159] [160] [161] Overfull \vbox (959.57pt too high) has occurred while \output is active [162] [163] [164]) (/home/neal/python/trunk/Doc/lib/librandom.tex [165] Underfull \hbox (badness 6575) in paragraph at lines 119--124 \OT1/ptm/m/n/10 Return a ran-domly se-lected el-e-ment from \OT1/pcr/m/n/10 ran ge(\OT1/ptm/m/it/10 start\OT1/pcr/m/n/10 , \OT1/ptm/m/it/10 stop\OT1/pcr/m/n/10 , \OT1/ptm/m/it/10 step\OT1/pcr/m/n/10 )\OT1/ptm/m/n/10 . This is equiv-a-lent to [166] [167]) (/home/neal/python/trunk/Doc/lib/libitertools.tex [168] [169] [170] [171] [172] [173] [174]) (/home/neal/python/trunk/Doc/lib/libfunctional.tex [175] Overfull \vbox (255.57pt too high) has occurred while \output is active [176]) (/home/neal/python/trunk/Doc/lib/liboperator.tex [177] [178] [179] [180] [181] [182] [183]) (/home/neal/python/trunk/Doc/lib/netdata.tex [184] Chapter 6. ) (/home/neal/python/trunk/Doc/lib/email.tex LaTeX Warning: Reference `module-smtplib' on page 185 undefined on input line 5 8. LaTeX Warning: Reference `module-nntplib' on page 185 undefined on input line 5 9. (/home/neal/python/trunk/Doc/lib/emailmessage.tex [185] [186] [187] [188] [189] [190]) (/home/neal/python/trunk/Doc/lib/emailparser.tex [191] [192] [193]) (/home/neal/python/trunk/Doc/lib/emailgenerator.tex [194]) (/home/neal/python/trunk/Doc/lib/emailmimebase.tex [195] [196]) (/home/neal/python/trunk/Doc/lib/emailheaders.tex [197] [198]) (/home/neal/python/trunk/Doc/lib/emailcharsets.tex [199] [200]) (/home/neal/python/trunk/Doc/lib/emailencoders.tex [201]) (/home/neal/python/trunk/Doc/lib/emailexc.tex [202]) (/home/neal/python/trunk/Doc/lib/emailutil.tex [203] [204]) (/home/neal/python/trunk/Doc/lib/emailiter.tex [205]) [206] [207] [208] [209] [210] [211] [212]) (/home/neal/python/trunk/Doc/lib/libmailcap.tex [213]) (/home/neal/python/trunk/Doc/lib/libmailbox.tex [214] [215]) (/home/neal/python/trunk/Doc/lib/libmhlib.tex [216] [217]) (/home/neal/python/trunk/Doc/lib/libmimetools.tex [218] LaTeX Warning: Reference `module-email' on page 219 undefined on input line 61. LaTeX Warning: Reference `module-rfc822' on page 219 undefined on input line 63 . LaTeX Warning: Reference `module-multifile' on page 219 undefined on input line 65. [219]) (/home/neal/python/trunk/Doc/lib/libmimetypes.tex [220] [221]) (/home/neal/python/trunk/Doc/lib/libmimewriter.tex [222]) (/home/neal/python/trunk/Doc/lib/libmimify.tex [223] LaTeX Warning: Reference `module-quopri' on page 224 undefined on input line 93 . ) (/home/neal/python/trunk/Doc/lib/libmultifile.tex LaTeX Warning: Reference `module-email' on page 224 undefined on input line 43. [224]) (/home/neal/python/trunk/Doc/lib/librfc822.tex [225] [226] LaTeX Warning: Reference `module-email' on page 227 undefined on input line 133 . LaTeX Warning: Reference `module-mailbox' on page 227 undefined on input line 1 35. [227] LaTeX Warning: Reference `module-mimetools' on page 228 undefined on input line 137. [228] Underfull \hbox (badness 7379) in paragraph at lines 254--270 []\OT1/pcr/m/n/10 Message \OT1/ptm/m/n/10 in-stances also sup-port a lim-ited m ap-ping in-ter-face. In par-tic-u-lar: \OT1/ptm/m/it/10 m\OT1/pcr/m/n/10 [name] \OT1/ptm/m/n/10 is like [229]) (/home/neal/python/trunk/Doc/lib/libbase64.tex [230] LaTeX Warning: Reference `module-binascii' on page 231 undefined on input line 151. ) (/home/neal/python/trunk/Doc/lib/libbinascii.tex [231] [232] LaTeX Warning: Reference `module-base64' on page 233 undefined on input line 13 9. LaTeX Warning: Reference `module-binhex' on page 233 undefined on input line 14 1. LaTeX Warning: Reference `module-uu' on page 233 undefined on input line 143. LaTeX Warning: Reference `module-quopri' on page 233 undefined on input line 14 5. ) (/home/neal/python/trunk/Doc/lib/libbinhex.tex LaTeX Warning: Reference `module-binascii' on page 233 undefined on input line 41. [233]) (/home/neal/python/trunk/Doc/lib/libquopri.tex LaTeX Warning: Reference `module-mimify' on page 234 undefined on input line 59 . LaTeX Warning: Reference `module-base64' on page 234 undefined on input line 60 . ) (/home/neal/python/trunk/Doc/lib/libuu.tex [234] LaTeX Warning: Reference `module-binascii' on page 235 undefined on input line 57. ) (/home/neal/python/trunk/Doc/lib/markup.tex [235] [236] Chapter 7. ) (/home/neal/python/trunk/Doc/lib/libhtmlparser.tex [237] [238]) (/home/neal/python/trunk/Doc/lib/libsgmllib.tex [239] ! Misplaced alignment tab character &. l.105 ...se of \code{} when \code{eggs} ? ! Emergency stop. l.105 ...se of \code{} when \code{eggs} Output written on lib.dvi (245 pages, 878844 bytes). Transcript written on lib.log. *** Session transcript and error messages are in /home/neal/python/trunk/Doc/html/lib/lib.how. *** Exited with status 1. +++ TEXINPUTS=/home/neal/python/trunk/Doc/lib:/home/neal/python/trunk/Doc/commontex:/home/neal/python/trunk/Doc/paper-letter:/home/neal/python/trunk/Doc/texinputs: +++ latex lib make: *** [html/lib/lib.html] Error 1 From python-checkins at python.org Sat Apr 1 12:49:14 2006 From: python-checkins at python.org (andrew.kuchling) Date: Sat, 1 Apr 2006 12:49:14 +0200 (CEST) Subject: [Python-checkins] r43541 - peps/trunk/pep-0357.txt Message-ID: <20060401104914.6837D1E4003@bag.python.org> Author: andrew.kuchling Date: Sat Apr 1 12:49:12 2006 New Revision: 43541 Modified: peps/trunk/pep-0357.txt Log: Typo fixes Modified: peps/trunk/pep-0357.txt ============================================================================== --- peps/trunk/pep-0357.txt (original) +++ peps/trunk/pep-0357.txt Sat Apr 1 12:49:12 2006 @@ -83,7 +83,7 @@ 2) Change the ISINT macro in ceval.c to ISINDEX and alter it to accomodate objects with the index slot defined. - 3) Change the _PyEval_SliceIndex function to accomodate objects + 3) Change the _PyEval_SliceIndex function to accommodate objects with the index slot defined. 4) Change all builtin objects (e.g. lists) that use the as_mapping @@ -149,4 +149,4 @@ Copyright - This document is placed in the public domain + This document is placed in the public domain. From python-checkins at python.org Sat Apr 1 12:50:08 2006 From: python-checkins at python.org (andrew.kuchling) Date: Sat, 1 Apr 2006 12:50:08 +0200 (CEST) Subject: [Python-checkins] r43542 - python/trunk/Misc/NEWS Message-ID: <20060401105008.BBE2B1E4003@bag.python.org> Author: andrew.kuchling Date: Sat Apr 1 12:50:08 2006 New Revision: 43542 Modified: python/trunk/Misc/NEWS Log: Some typo & grammar fixes Modified: python/trunk/Misc/NEWS ============================================================================== --- python/trunk/Misc/NEWS (original) +++ python/trunk/Misc/NEWS Sat Apr 1 12:50:08 2006 @@ -64,10 +64,10 @@ to "error" is triggered) when raising a warning for raising string exceptions. -- CO_GENERATOR_ALLOWED is no longer defined, this behavior is the default. +- CO_GENERATOR_ALLOWED is no longer defined. This behavior is the default. The name was removed from Include/code.h. -- PEP 308: conditional expressions were added (x if cond else y). +- PEP 308: conditional expressions were added: (x if cond else y). - Patch 1433928: - The copy module now "copies" function objects (as atomic objects). @@ -125,12 +125,12 @@ - Patch #1350409: Work around signal handling bug in Visual Studio 2005. -- Bug #1281408: Py_BuildValue now works correct even with unsigned longs +- Bug #1281408: Py_BuildValue now works correctly even with unsigned longs and long longs. - SF Bug #1350188, "setdlopenflags" leads to crash upon "import" - It was possible dlerror() returns a NULL pointer, use a default error - message in this case. + It was possible for dlerror() to return a NULL pointer, so + it will now use a default error message in this case. - Replaced most Unicode charmap codecs with new ones using the new Unicode translate string feature in the builtin charmap @@ -140,13 +140,13 @@ - Added a few more codecs for Mac OS encodings -- Speed up some Unicode operations. +- Sped up some Unicode operations. - A new AST parser implementation was completed. The abstract syntax tree is available for read-only (non-compile) access to Python code; an _ast module was added. -- SF bug #1167751: fix incorrect code being for generator expressions. +- SF bug #1167751: fix incorrect code being produced for generator expressions. The following code now raises a SyntaxError: foo(a = i for i in range(10)) - SF Bug #976608: fix SystemError when mtime of an imported file is -1. @@ -158,7 +158,7 @@ - SF bug #772896: unknown encoding results in MemoryError. -- All iterators now have a Boolean value of true. Formerly, some iterators +- All iterators now have a Boolean value of True. Formerly, some iterators supported a __len__() method which evaluated to False when the iterator was empty. @@ -198,7 +198,7 @@ - SF bug #1185883: Python's small-object memory allocator took over a block managed by the platform C library whenever a realloc specified a small new size. However, there's no portable way to know then how - much of the address space following the pointer is valid, so no + much of the address space following the pointer is valid, so there's no portable way to copy data from the C-managed block into Python's small-object space without risking a memory fault. Python's small-object realloc now leaves such blocks under the control of the platform C @@ -669,7 +669,7 @@ - Bug #1163178: Make IDNA return an empty string when the input is empty. - Patch #848017: Make Cookie more RFC-compliant. Use CRLF as default output - separator and do not output trailing semicola. + separator and do not output trailing semicolon. - Patch #1062060: urllib.urlretrieve() now raises a new exception, named ContentTooShortException, when the actually downloaded size does not @@ -707,7 +707,7 @@ - Bug #1177468: Don't cache the /dev/urandom file descriptor for os.urandom, as this can cause problems with apps closing all file descriptors. -- Bug #839151: Fix an attempt to access sys.argv in the warnings module +- Bug #839151: Fix an attempt to access sys.argv in the warnings module; it can be missing in embedded interpreters - Bug #1155638: Fix a bug which affected HTTP 0.9 responses in httplib. @@ -721,7 +721,7 @@ Bug #1224621. - The tokenize module has a new untokenize() function to support a full - roundtrip from lexed tokens back to Python sourcecode. In addition, + roundtrip from lexed tokens back to Python source code. In addition, the generate_tokens() function now accepts a callable argument that terminates by raising StopIteration. From thomas at python.org Sat Apr 1 16:43:23 2006 From: thomas at python.org (Thomas Wouters) Date: Sat, 1 Apr 2006 16:43:23 +0200 Subject: [Python-checkins] r43529 - python/trunk/Lib/distutils/log.py In-Reply-To: <20060401074654.EDD1D1E4003@bag.python.org> References: <20060401074654.EDD1D1E4003@bag.python.org> Message-ID: <9e804ac0604010643w19a5f196s6b98a94bc004e65@mail.gmail.com> On 4/1/06, georg.brandl wrote: > > Author: georg.brandl > Date: Sat Apr 1 09:46:54 2006 > New Revision: 43529 > > Modified: > python/trunk/Lib/distutils/log.py > Log: > Bug #1458017: make distutils.Log._log more forgiving when passing in > msg strings with '%', but without format args. I'm not sure if this is the right thing to do. It may be more forgiving, but anyone who passes a string to a function that expects a format shouldn't be looking for forgiveness. They should be made aware that %'s are special, or they will be bitten by it when they _do_ pass arguments. Modified: python/trunk/Lib/distutils/log.py > > ============================================================================== > --- python/trunk/Lib/distutils/log.py (original) > +++ python/trunk/Lib/distutils/log.py Sat Apr 1 09:46:54 2006 > @@ -20,7 +20,12 @@ > > def _log(self, level, msg, args): > if level >= self.threshold: > - print msg % args > + if not args: > + # msg may contain a '%'. If args is empty, > + # don't even try to string-format > + print msg > + else: > + print msg % args > sys.stdout.flush() > > def log(self, level, msg, *args): -- Thomas Wouters Hi! I'm a .signature virus! copy me into your .signature file to help me spread! -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-checkins/attachments/20060401/36e19ea5/attachment.htm From python-checkins at python.org Sat Apr 1 18:18:03 2006 From: python-checkins at python.org (jeremy.hylton) Date: Sat, 1 Apr 2006 18:18:03 +0200 (CEST) Subject: [Python-checkins] r43543 - python/trunk/Python/compile.c Message-ID: <20060401161803.3EB8F1E4003@bag.python.org> Author: jeremy.hylton Date: Sat Apr 1 18:18:02 2006 New Revision: 43543 Modified: python/trunk/Python/compile.c Log: Expand comments on line numbers and blocks. Reorder compiler_set_lineno() call for consistency. Modified: python/trunk/Python/compile.c ============================================================================== --- python/trunk/Python/compile.c (original) +++ python/trunk/Python/compile.c Sat Apr 1 18:18:02 2006 @@ -58,8 +58,9 @@ }; typedef struct basicblock_ { - /* next block in the list of blocks for a unit (don't confuse with - * b_next) */ + /* Each basicblock in a compilation unit is linked via b_list in the + reverse order that the block are allocated. b_list points to the next + block, not to be confused with b_next, which is next by control flow. */ struct basicblock_ *b_list; /* number of instructions used */ int b_iused; @@ -114,7 +115,9 @@ PyObject *u_private; /* for private name mangling */ int u_argcount; /* number of arguments for block */ - basicblock *u_blocks; /* pointer to list of blocks */ + /* Pointer to the most recently allocated block. By following b_list + members, you can reach all early allocated blocks. */ + basicblock *u_blocks; basicblock *u_curblock; /* pointer to current block */ int u_tmpname; /* temporary variables for list comps */ @@ -1194,7 +1197,7 @@ return NULL; } memset((void *)b, 0, sizeof(basicblock)); - assert (b->b_next == NULL); + /* Extend the singly linked list of blocks with new block. */ b->b_list = u->u_blocks; u->u_blocks = b; return b; @@ -1267,6 +1270,13 @@ return b->b_iused++; } +/* Set the i_lineno member of the instruction at offse off if the + line number for the current expression/statement (?) has not + already been set. If it has been set, the call has no effect. + + Every time a new node is b + */ + static void compiler_set_lineno(struct compiler *c, int off) { @@ -1609,7 +1619,6 @@ off = compiler_next_instr(c, c->u->u_curblock); if (off < 0) return 0; - compiler_set_lineno(c, off); i = &c->u->u_curblock->b_instr[off]; i->i_opcode = opcode; i->i_target = b; @@ -1618,6 +1627,7 @@ i->i_jabs = 1; else i->i_jrel = 1; + compiler_set_lineno(c, off); return 1; } @@ -2230,7 +2240,7 @@ ADDOP(c, POP_BLOCK); } compiler_pop_fblock(c, LOOP, loop); - if (orelse != NULL) + if (orelse != NULL) /* what if orelse is just pass? */ VISIT_SEQ(c, stmt, s->v.While.orelse); compiler_use_next_block(c, end); @@ -2610,8 +2620,10 @@ { int i, n; + /* Always assign a lineno to the next instruction for a stmt. */ c->u->u_lineno = s->lineno; c->u->u_lineno_set = false; + switch (s->kind) { case FunctionDef_kind: return compiler_function(c, s); @@ -3486,6 +3498,9 @@ { int i, n; + /* If expr e has a different line number than the last expr/stmt, + set a new line number for the next instruction. + */ if (e->lineno > c->u->u_lineno) { c->u->u_lineno = e->lineno; c->u->u_lineno_set = false; From dynkin at gmail.com Sat Apr 1 21:19:28 2006 From: dynkin at gmail.com (George Yoshida) Date: Sun, 2 Apr 2006 04:19:28 +0900 Subject: [Python-checkins] Python Regression Test Failures doc (1) In-Reply-To: <20060401103650.GA832@python.psfb.org> References: <20060401103650.GA832@python.psfb.org> Message-ID: <2f188ee80604011119i5dd372c1q6ccc230e25aeb597@mail.gmail.com> On 4/1/06, Neal Norwitz wrote: > TEXINPUTS=/home/neal/python/trunk/Doc/commontex: python /home/neal/python/trunk/Doc/tools/mkhowto --html --about html/stdabout.dat --iconserver ../icons --favicon ../icons/pyfav.png --address "See About this document... for information on suggesting changes." --up-link ../index.html --up-title "Python Documentation Index" --global-module-index "../modindex.html" --dvips-safe --dir html/lib lib/lib.tex > *** Session transcript and error messages are in /home/neal/python/trunk/Doc/html/lib/lib.how. > *** Exited with status 1. Maybe the culprit is this? > r43534 - python/trunk/Doc/lib/libsgmllib.tex Here's a diff against r43534. Index: Doc/lib/libsgmllib.tex =================================================================== --- Doc/lib/libsgmllib.tex (revision 43543) +++ Doc/lib/libsgmllib.tex (working copy) @@ -102,7 +102,7 @@ as well as known character references and known entity references terminated by a semicolon (normally, entity references can be terminated by any non-alphanumerical character, but this would break the very -common case of \code{} when \code{eggs} +common case of \code{} when \code{eggs} is a valid entity name). For instance, for the tag \code{}, this -- george From python-checkins at python.org Sat Apr 1 22:40:16 2006 From: python-checkins at python.org (georg.brandl) Date: Sat, 1 Apr 2006 22:40:16 +0200 (CEST) Subject: [Python-checkins] r43544 - python/trunk/Doc/lib/libsgmllib.tex Message-ID: <20060401204016.B995B1E4008@bag.python.org> Author: georg.brandl Date: Sat Apr 1 22:40:16 2006 New Revision: 43544 Modified: python/trunk/Doc/lib/libsgmllib.tex Log: Fix LaTeX oversight. Modified: python/trunk/Doc/lib/libsgmllib.tex ============================================================================== --- python/trunk/Doc/lib/libsgmllib.tex (original) +++ python/trunk/Doc/lib/libsgmllib.tex Sat Apr 1 22:40:16 2006 @@ -102,7 +102,7 @@ as well as known character references and known entity references terminated by a semicolon (normally, entity references can be terminated by any non-alphanumerical character, but this would break the very -common case of \code{} when \code{eggs} +common case of \code{} when \code{eggs} is a valid entity name). For instance, for the tag \code{}, this From python-checkins at python.org Sat Apr 1 22:40:23 2006 From: python-checkins at python.org (walter.doerwald) Date: Sat, 1 Apr 2006 22:40:23 +0200 (CEST) Subject: [Python-checkins] r43545 - in python/trunk: Doc/lib/libcalendar.tex Lib/calendar.py Message-ID: <20060401204023.804801E4006@bag.python.org> Author: walter.doerwald Date: Sat Apr 1 22:40:23 2006 New Revision: 43545 Modified: python/trunk/Doc/lib/libcalendar.tex python/trunk/Lib/calendar.py Log: Make firstweekday a simple attribute instead of hiding it behind a setter and a getter. Modified: python/trunk/Doc/lib/libcalendar.tex ============================================================================== --- python/trunk/Doc/lib/libcalendar.tex (original) +++ python/trunk/Doc/lib/libcalendar.tex Sat Apr 1 22:40:23 2006 @@ -36,15 +36,6 @@ \class{Calendar} instances have the following methods: -\begin{methoddesc}{firstweekday}{} -Return the first day of the week (as specified in the constructor -or changed via \method{setfirstweekday()}). -\end{methoddesc} - -\begin{methoddesc}{setfirstweekday}{weekday} -Set the first day of the week. -\end{methoddesc} - \begin{methoddesc}{iterweekdays}{weekday} Return an iterator for the week day numbers that will be used for one week. The first number from the iterator will be the Modified: python/trunk/Lib/calendar.py ============================================================================== --- python/trunk/Lib/calendar.py (original) +++ python/trunk/Lib/calendar.py Sat Apr 1 22:40:23 2006 @@ -128,25 +128,14 @@ """ def __init__(self, firstweekday=0): - self._firstweekday = firstweekday # 0 = Monday, 6 = Sunday - - def firstweekday(self): - return self._firstweekday - - def setfirstweekday(self, weekday): - """ - Set weekday (Monday=0, Sunday=6) to start each week. - """ - if not MONDAY <= weekday <= SUNDAY: - raise IllegalWeekdayError(weekday) - self._firstweekday = weekday + self.firstweekday = firstweekday # 0 = Monday, 6 = Sunday def iterweekdays(self): """ Return a iterator for one week of weekday numbers starting with the configured first one. """ - for i in xrange(self._firstweekday, self._firstweekday + 7): + for i in xrange(self.firstweekday, self.firstweekday + 7): yield i%7 def itermonthdates(self, year, month): @@ -157,13 +146,13 @@ """ date = datetime.date(year, month, 1) # Go back to the beginning of the week - days = (date.weekday() - self._firstweekday) % 7 + days = (date.weekday() - self.firstweekday) % 7 date -= datetime.timedelta(days=days) oneday = datetime.timedelta(days=1) while True: yield date date += oneday - if date.month != month and date.weekday() == self._firstweekday: + if date.month != month and date.weekday() == self.firstweekday: break def itermonthdays2(self, year, month): @@ -570,8 +559,14 @@ # Support for old module level interface c = TextCalendar() -firstweekday = c.firstweekday -setfirstweekday = c.setfirstweekday +def firstweekday(): + return c.firstweekday + +def setfirstweekday(firstweekday): + if not MONDAY <= firstweekday <= SUNDAY: + raise IllegalWeekdayError(firstweekday) + c.firstweekday = firstweekday + monthcalendar = c.monthdayscalendar prweek = c.prweek week = c.formatweek From g.brandl at gmx.net Sat Apr 1 22:40:27 2006 From: g.brandl at gmx.net (Georg Brandl) Date: Sat, 01 Apr 2006 22:40:27 +0200 Subject: [Python-checkins] Python Regression Test Failures doc (1) In-Reply-To: <2f188ee80604011119i5dd372c1q6ccc230e25aeb597@mail.gmail.com> References: <20060401103650.GA832@python.psfb.org> <2f188ee80604011119i5dd372c1q6ccc230e25aeb597@mail.gmail.com> Message-ID: George Yoshida wrote: > On 4/1/06, Neal Norwitz wrote: >> TEXINPUTS=/home/neal/python/trunk/Doc/commontex: python /home/neal/python/trunk/Doc/tools/mkhowto --html --about html/stdabout.dat --iconserver ../icons --favicon ../icons/pyfav.png --address "See About this document... for information on suggesting changes." --up-link ../index.html --up-title "Python Documentation Index" --global-module-index "../modindex.html" --dvips-safe --dir html/lib lib/lib.tex >> *** Session transcript and error messages are in /home/neal/python/trunk/Doc/html/lib/lib.how. >> *** Exited with status 1. > > > Maybe the culprit is this? >> r43534 - python/trunk/Doc/lib/libsgmllib.tex > Here's a diff against r43534. Thanks, fixed. Georg From buildbot at python.org Sat Apr 1 23:25:32 2006 From: buildbot at python.org (buildbot at python.org) Date: Sat, 01 Apr 2006 21:25:32 +0000 Subject: [Python-checkins] buildbot warnings in alpha Tru64 5.1 trunk Message-ID: <20060401212532.437261E4006@bag.python.org> The Buildbot has detected a new failure of alpha Tru64 5.1 trunk. Full details are available at: http://www.python.org/dev/buildbot/all/alpha%20Tru64%205.1%20trunk/builds/65 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: georg.brandl,walter.doerwald Build Had Warnings: warnings test sincerely, -The Buildbot From python-checkins at python.org Sun Apr 2 00:14:44 2006 From: python-checkins at python.org (fred.drake) Date: Sun, 2 Apr 2006 00:14:44 +0200 (CEST) Subject: [Python-checkins] r43546 - in python/trunk: Doc/lib/liburlparse.tex Lib/test/test_urlparse.py Lib/urlparse.py Misc/NEWS Message-ID: <20060401221444.8A1AA1E4006@bag.python.org> Author: fred.drake Date: Sun Apr 2 00:14:43 2006 New Revision: 43546 Modified: python/trunk/Doc/lib/liburlparse.tex python/trunk/Lib/test/test_urlparse.py python/trunk/Lib/urlparse.py python/trunk/Misc/NEWS Log: Patch #624325: urlparse.urlparse() and urlparse.urlsplit() results now sport attributes that provide access to the parts of the result. Modified: python/trunk/Doc/lib/liburlparse.tex ============================================================================== --- python/trunk/Doc/lib/liburlparse.tex (original) +++ python/trunk/Doc/lib/liburlparse.tex Sun Apr 2 00:14:43 2006 @@ -25,48 +25,74 @@ \code{nntp}, \code{prospero}, \code{rsync}, \code{rtsp}, \code{rtspu}, \code{sftp}, \code{shttp}, \code{sip}, \code{sips}, \code{snews}, \code{svn}, \code{svn+ssh}, \code{telnet}, \code{wais}. + \versionadded[Support for the \code{sftp} and \code{sips} schemes]{2.5} The \module{urlparse} module defines the following functions: -\begin{funcdesc}{urlparse}{urlstring\optional{, default_scheme\optional{, allow_fragments}}} -Parse a URL into 6 components, returning a 6-tuple: (addressing -scheme, network location, path, parameters, query, fragment -identifier). This corresponds to the general structure of a URL: +\begin{funcdesc}{urlparse}{urlstring\optional{, + default_scheme\optional{, allow_fragments}}} +Parse a URL into six components, returning a 6-tuple. This +corresponds to the general structure of a URL: \code{\var{scheme}://\var{netloc}/\var{path};\var{parameters}?\var{query}\#\var{fragment}}. Each tuple item is a string, possibly empty. -The components are not broken up in smaller parts (e.g. the network +The components are not broken up in smaller parts (for example, the network location is a single string), and \% escapes are not expanded. -The delimiters as shown above are not part of the tuple items, +The delimiters as shown above are not part of the result, except for a leading slash in the \var{path} component, which is -retained if present. - -Example: - -\begin{verbatim} -urlparse('http://www.cwi.nl:80/%7Eguido/Python.html') -\end{verbatim} - -yields the tuple +retained if present. For example: \begin{verbatim} +>>> from urlparse import urlparse +>>> o = urlparse('http://www.cwi.nl:80/%7Eguido/Python.html') +>>> o ('http', 'www.cwi.nl:80', '/%7Eguido/Python.html', '', '', '') +>>> o.scheme +'http' +>>> o.port +80 +>>> o.geturl() +'http://www.cwi.nl:80/%7Eguido/Python.html' \end{verbatim} If the \var{default_scheme} argument is specified, it gives the -default addressing scheme, to be used only if the URL string does not +default addressing scheme, to be used only if the URL does not specify one. The default value for this argument is the empty string. -If the \var{allow_fragments} argument is zero, fragment identifiers +If the \var{allow_fragments} argument is false, fragment identifiers are not allowed, even if the URL's addressing scheme normally does -support them. The default value for this argument is \code{1}. -\end{funcdesc} +support them. The default value for this argument is \constant{True}. -\begin{funcdesc}{urlunparse}{tuple} -Construct a URL string from a tuple as returned by \code{urlparse()}. +The return value is actually an instance of a subclass of +\pytype{tuple}. This class has the following additional read-only +convenience attributes: + +\begin{tableiv}{l|c|l|c}{member}{Attribute}{Index}{Value}{Value if not present} + \lineiv{scheme} {0} {URL scheme specifier} {empty string} + \lineiv{netloc} {1} {Network location part} {empty string} + \lineiv{path} {2} {Hierarchical path} {empty string} + \lineiv{params} {3} {Parameters for last path element} {empty string} + \lineiv{query} {4} {Query component} {empty string} + \lineiv{fragment}{5} {Fragment identifier} {empty string} + \lineiv{username}{ } {User name} {\constant{None}} + \lineiv{password}{ } {Password} {\constant{None}} + \lineiv{hostname}{ } {Host name (lower case)} {\constant{None}} + \lineiv{port} { } {Port number as integer, if present} {\constant{None}} +\end{tableiv} + +See section~\ref{urlparse-result-object}, ``Results of +\function{urlparse()} and \function{urlsplit()},'' for more +information on the result object. + +\versionchanged[Added attributes to return value]{2.5} +\end{funcdesc} + +\begin{funcdesc}{urlunparse}{parts} +Construct a URL from a tuple as returned by \code{urlparse()}. +The \var{parts} argument be any six-item iterable. This may result in a slightly different, but equivalent URL, if the -URL that was parsed originally had redundant delimiters, e.g. a ? with -an empty query (the draft states that these are equivalent). +URL that was parsed originally had unnecessary delimiters (for example, +a ? with an empty query; the RFC states that these are equivalent). \end{funcdesc} \begin{funcdesc}{urlsplit}{urlstring\optional{, @@ -79,12 +105,38 @@ separate the path segments and parameters. This function returns a 5-tuple: (addressing scheme, network location, path, query, fragment identifier). + +The return value is actually an instance of a subclass of +\pytype{tuple}. This class has the following additional read-only +convenience attributes: + +\begin{tableiv}{l|c|l|c}{member}{Attribute}{Index}{Value}{Value if not present} + \lineiv{scheme} {0} {URL scheme specifier} {empty string} + \lineiv{netloc} {1} {Network location part} {empty string} + \lineiv{path} {2} {Hierarchical path} {empty string} + \lineiv{query} {3} {Query component} {empty string} + \lineiv{fragment} {4} {Fragment identifier} {empty string} + \lineiv{username} { } {User name} {\constant{None}} + \lineiv{password} { } {Password} {\constant{None}} + \lineiv{hostname} { } {Host name (lower case)} {\constant{None}} + \lineiv{port} { } {Port number as integer, if present} {\constant{None}} +\end{tableiv} + +See section~\ref{urlparse-result-object}, ``Results of +\function{urlparse()} and \function{urlsplit()},'' for more +information on the result object. + \versionadded{2.2} +\versionchanged[Added attributes to return value]{2.5} \end{funcdesc} -\begin{funcdesc}{urlunsplit}{tuple} +\begin{funcdesc}{urlunsplit}{parts} Combine the elements of a tuple as returned by \function{urlsplit()} into a complete URL as a string. +The \var{parts} argument be any five-item iterable. +This may result in a slightly different, but equivalent URL, if the +URL that was parsed originally had unnecessary delimiters (for example, +a ? with an empty query; the RFC states that these are equivalent). \versionadded{2.2} \end{funcdesc} @@ -93,22 +145,16 @@ (\var{base}) with a ``relative URL'' (\var{url}). Informally, this uses components of the base URL, in particular the addressing scheme, the network location and (part of) the path, to provide missing -components in the relative URL. - -Example: - -\begin{verbatim} -urljoin('http://www.cwi.nl/%7Eguido/Python.html', 'FAQ.html') -\end{verbatim} - -yields the string +components in the relative URL. For example: \begin{verbatim} +>>> from urlparse import urljoin +>>> urljoin('http://www.cwi.nl/%7Eguido/Python.html', 'FAQ.html') 'http://www.cwi.nl/%7Eguido/FAQ.html' \end{verbatim} -The \var{allow_fragments} argument has the same meaning as for -\code{urlparse()}. +The \var{allow_fragments} argument has the same meaning and default as +for \function{urlparse()}. \end{funcdesc} \begin{funcdesc}{urldefrag}{url} @@ -133,3 +179,61 @@ both Uniform Resource Names (URNs) and Uniform Resource Locators (URLs).} \end{seealso} + + +\subsection{Results of \function{urlparse()} and \function{urlsplit()} + \label{urlparse-result-object}} + +The result objects from the \function{urlparse()} and +\function{urlsplit()} functions are subclasses of the \pytype{tuple} +type. These subclasses add the attributes described in those +functions, as well as provide an additional method: + +\begin{methoddesc}[ParseResult]{geturl}{} + Return the re-combined version of the original URL as a string. + This may differ from the original URL in that the scheme will always + be normalized to lower case and empty components may be dropped. + Specifically, empty parameters, queries, and fragment identifiers + will be removed. + + The result of this method is a fixpoint if passed back through the + original parsing function: + +\begin{verbatim} +>>> import urlparse +>>> url = 'HTTP://www.Python.org/doc/#' + +>>> r1 = urlparse.urlsplit(url) +>>> r1.geturl() +'http://www.Python.org/doc/' + +>>> r2 = urlparse.urlsplit(r1.geturl()) +>>> r2.geturl() +'http://www.Python.org/doc/' +\end{verbatim} + +\versionadded{2.5} +\end{methoddesc} + +The following classes provide the implementations of the parse results:: + +\begin{classdesc*}{BaseResult} + Base class for the concrete result classes. This provides most of + the attribute definitions. It does not provide a \method{geturl()} + method. It is derived from \class{tuple}, but does not override the + \method{__init__()} or \method{__new__()} methods. +\end{classdesc*} + + +\begin{classdesc}{ParseResult}{scheme, netloc, path, params, query, fragment} + Concrete class for \function{urlparse()} results. The + \method{__new__()} method is overridden to support checking that the + right number of arguments are passed. +\end{classdesc} + + +\begin{classdesc}{SplitResult}{scheme, netloc, path, query, fragment} + Concrete class for \function{urlsplit()} results. The + \method{__new__()} method is overridden to support checking that the + right number of arguments are passed. +\end{classdesc} Modified: python/trunk/Lib/test/test_urlparse.py ============================================================================== --- python/trunk/Lib/test/test_urlparse.py (original) +++ python/trunk/Lib/test/test_urlparse.py Sun Apr 2 00:14:43 2006 @@ -12,15 +12,53 @@ def checkRoundtrips(self, url, parsed, split): result = urlparse.urlparse(url) self.assertEqual(result, parsed) + t = (result.scheme, result.netloc, result.path, + result.params, result.query, result.fragment) + self.assertEqual(t, parsed) # put it back together and it should be the same result2 = urlparse.urlunparse(result) self.assertEqual(result2, url) + self.assertEqual(result2, result.geturl()) + + # the result of geturl() is a fixpoint; we can always parse it + # again to get the same result: + result3 = urlparse.urlparse(result.geturl()) + self.assertEqual(result3.geturl(), result.geturl()) + self.assertEqual(result3, result) + self.assertEqual(result3.scheme, result.scheme) + self.assertEqual(result3.netloc, result.netloc) + self.assertEqual(result3.path, result.path) + self.assertEqual(result3.params, result.params) + self.assertEqual(result3.query, result.query) + self.assertEqual(result3.fragment, result.fragment) + self.assertEqual(result3.username, result.username) + self.assertEqual(result3.password, result.password) + self.assertEqual(result3.hostname, result.hostname) + self.assertEqual(result3.port, result.port) # check the roundtrip using urlsplit() as well result = urlparse.urlsplit(url) self.assertEqual(result, split) + t = (result.scheme, result.netloc, result.path, + result.query, result.fragment) + self.assertEqual(t, split) result2 = urlparse.urlunsplit(result) self.assertEqual(result2, url) + self.assertEqual(result2, result.geturl()) + + # check the fixpoint property of re-parsing the result of geturl() + result3 = urlparse.urlsplit(result.geturl()) + self.assertEqual(result3.geturl(), result.geturl()) + self.assertEqual(result3, result) + self.assertEqual(result3.scheme, result.scheme) + self.assertEqual(result3.netloc, result.netloc) + self.assertEqual(result3.path, result.path) + self.assertEqual(result3.query, result.query) + self.assertEqual(result3.fragment, result.fragment) + self.assertEqual(result3.username, result.username) + self.assertEqual(result3.password, result.password) + self.assertEqual(result3.hostname, result.hostname) + self.assertEqual(result3.port, result.port) def test_roundtrips(self): testcases = [ @@ -187,6 +225,69 @@ ]: self.assertEqual(urlparse.urldefrag(url), (defrag, frag)) + def test_urlsplit_attributes(self): + url = "HTTP://WWW.PYTHON.ORG/doc/#frag" + p = urlparse.urlsplit(url) + self.assertEqual(p.scheme, "http") + self.assertEqual(p.netloc, "WWW.PYTHON.ORG") + self.assertEqual(p.path, "/doc/") + self.assertEqual(p.query, "") + self.assertEqual(p.fragment, "frag") + self.assertEqual(p.username, None) + self.assertEqual(p.password, None) + self.assertEqual(p.hostname, "www.python.org") + self.assertEqual(p.port, None) + # geturl() won't return exactly the original URL in this case + # since the scheme is always case-normalized + #self.assertEqual(p.geturl(), url) + + url = "http://User:Pass at www.python.org:080/doc/?query=yes#frag" + p = urlparse.urlsplit(url) + self.assertEqual(p.scheme, "http") + self.assertEqual(p.netloc, "User:Pass at www.python.org:080") + self.assertEqual(p.path, "/doc/") + self.assertEqual(p.query, "query=yes") + self.assertEqual(p.fragment, "frag") + self.assertEqual(p.username, "User") + self.assertEqual(p.password, "Pass") + self.assertEqual(p.hostname, "www.python.org") + self.assertEqual(p.port, 80) + self.assertEqual(p.geturl(), url) + + def test_attributes_bad_port(self): + """Check handling of non-integer ports.""" + p = urlparse.urlsplit("http://www.example.net:foo") + self.assertEqual(p.netloc, "www.example.net:foo") + self.assertRaises(ValueError, lambda: p.port) + + p = urlparse.urlparse("http://www.example.net:foo") + self.assertEqual(p.netloc, "www.example.net:foo") + self.assertRaises(ValueError, lambda: p.port) + + def test_attributes_without_netloc(self): + # This example is straight from RFC 3261. It looks like it + # should allow the username, hostname, and port to be filled + # in, but doesn't. Since it's a URI and doesn't use the + # scheme://netloc syntax, the netloc and related attributes + # should be left empty. + uri = "sip:alice at atlanta.com;maddr=239.255.255.1;ttl=15" + p = urlparse.urlsplit(uri) + self.assertEqual(p.netloc, "") + self.assertEqual(p.username, None) + self.assertEqual(p.password, None) + self.assertEqual(p.hostname, None) + self.assertEqual(p.port, None) + self.assertEqual(p.geturl(), uri) + + p = urlparse.urlparse(uri) + self.assertEqual(p.netloc, "") + self.assertEqual(p.username, None) + self.assertEqual(p.password, None) + self.assertEqual(p.hostname, None) + self.assertEqual(p.port, None) + self.assertEqual(p.geturl(), uri) + + def test_main(): test_support.run_unittest(UrlParseTestCase) Modified: python/trunk/Lib/urlparse.py ============================================================================== --- python/trunk/Lib/urlparse.py (original) +++ python/trunk/Lib/urlparse.py Sun Apr 2 00:14:43 2006 @@ -41,7 +41,111 @@ _parse_cache = {} -def urlparse(url, scheme='', allow_fragments=1): +class BaseResult(tuple): + """Base class for the parsed result objects. + + This provides the attributes shared by the two derived result + objects as read-only properties. The derived classes are + responsible for checking the right number of arguments were + supplied to the constructor. + + """ + + __slots__ = () + + # Attributes that access the basic components of the URL: + + @property + def scheme(self): + return self[0] + + @property + def netloc(self): + return self[1] + + @property + def path(self): + return self[2] + + @property + def query(self): + return self[-2] + + @property + def fragment(self): + return self[-1] + + # Additional attributes that provide access to parsed-out portions + # of the netloc: + + @property + def username(self): + netloc = self.netloc + if "@" in netloc: + userinfo = netloc.split("@", 1)[0] + if ":" in userinfo: + userinfo = userinfo.split(":", 1)[0] + return userinfo + return None + + @property + def password(self): + netloc = self.netloc + if "@" in netloc: + userinfo = netloc.split("@", 1)[0] + if ":" in userinfo: + return userinfo.split(":", 1)[1] + return None + + @property + def hostname(self): + netloc = self.netloc + if "@" in netloc: + netloc = netloc.split("@", 1)[1] + if ":" in netloc: + netloc = netloc.split(":", 1)[0] + return netloc.lower() or None + + @property + def port(self): + netloc = self.netloc + if "@" in netloc: + netloc = netloc.split("@", 1)[1] + if ":" in netloc: + port = netloc.split(":", 1)[1] + return int(port, 10) + return None + + +class SplitResult(BaseResult): + + __slots__ = () + + def __new__(cls, scheme, netloc, path, query, fragment): + return BaseResult.__new__( + cls, (scheme, netloc, path, query, fragment)) + + def geturl(self): + return urlunsplit(self) + + +class ParseResult(BaseResult): + + __slots__ = () + + def __new__(cls, scheme, netloc, path, params, query, fragment): + return BaseResult.__new__( + cls, (scheme, netloc, path, params, query, fragment)) + + @property + def params(self): + return self[3] + + def geturl(self): + return urlunparse(self) + + +def urlparse(url, scheme='', allow_fragments=True): """Parse a URL into 6 components: :///;?# Return a 6-tuple: (scheme, netloc, path, params, query, fragment). @@ -53,7 +157,7 @@ url, params = _splitparams(url) else: params = '' - return scheme, netloc, url, params, query, fragment + return ParseResult(scheme, netloc, url, params, query, fragment) def _splitparams(url): if '/' in url: @@ -73,12 +177,13 @@ delim = len(url) return url[start:delim], url[delim:] -def urlsplit(url, scheme='', allow_fragments=1): +def urlsplit(url, scheme='', allow_fragments=True): """Parse a URL into 5 components: :///?# Return a 5-tuple: (scheme, netloc, path, query, fragment). Note that we don't break the components up in smaller bits (e.g. netloc is a single string) and we don't expand % escapes.""" + allow_fragments = bool(allow_fragments) key = url, scheme, allow_fragments cached = _parse_cache.get(key, None) if cached: @@ -97,9 +202,9 @@ url, fragment = url.split('#', 1) if '?' in url: url, query = url.split('?', 1) - tuple = scheme, netloc, url, query, fragment - _parse_cache[key] = tuple - return tuple + v = SplitResult(scheme, netloc, url, query, fragment) + _parse_cache[key] = v + return v for c in url[:i]: if c not in scheme_chars: break @@ -111,9 +216,9 @@ url, fragment = url.split('#', 1) if scheme in uses_query and '?' in url: url, query = url.split('?', 1) - tuple = scheme, netloc, url, query, fragment - _parse_cache[key] = tuple - return tuple + v = SplitResult(scheme, netloc, url, query, fragment) + _parse_cache[key] = v + return v def urlunparse((scheme, netloc, url, params, query, fragment)): """Put a parsed URL back together again. This may result in a @@ -136,7 +241,7 @@ url = url + '#' + fragment return url -def urljoin(base, url, allow_fragments = 1): +def urljoin(base, url, allow_fragments=True): """Join a base URL and a possibly relative URL to form an absolute interpretation of the latter.""" if not base: Modified: python/trunk/Misc/NEWS ============================================================================== --- python/trunk/Misc/NEWS (original) +++ python/trunk/Misc/NEWS Sun Apr 2 00:14:43 2006 @@ -489,6 +489,9 @@ Library ------- +- Patch #624325: urlparse.urlparse() and urlparse.urlsplit() results + now sport attributes that provide access to the parts of the result. + - Patch #1462498: sgmllib now handles entity and character references in attribute values. From python-checkins at python.org Sun Apr 2 03:46:33 2006 From: python-checkins at python.org (andrew.kuchling) Date: Sun, 2 Apr 2006 03:46:33 +0200 (CEST) Subject: [Python-checkins] r43547 - python/trunk/Doc/whatsnew/whatsnew25.tex Message-ID: <20060402014633.98C301E4008@bag.python.org> Author: andrew.kuchling Date: Sun Apr 2 03:46:32 2006 New Revision: 43547 Modified: python/trunk/Doc/whatsnew/whatsnew25.tex Log: Write various sections; I haven't been able to check whether the TeX markup is correct Modified: python/trunk/Doc/whatsnew/whatsnew25.tex ============================================================================== --- python/trunk/Doc/whatsnew/whatsnew25.tex (original) +++ python/trunk/Doc/whatsnew/whatsnew25.tex Sun Apr 2 03:46:32 2006 @@ -2,6 +2,11 @@ \usepackage{distutils} % $Id$ +% Fix XXX comments +% Distutils upload +% The easy_install stuff +% xml.etree section +% added sqlite3 \title{What's New in Python 2.5} \release{0.0} @@ -55,11 +60,12 @@ x = true_value if condition else false_value \end{verbatim} -Evaluation is still lazy as in existing Boolean expression, so the -evaluation jumps around a bit. The \var{condition} expression is -evaluated first, and the \var{true_value} expression is evaluated only -if the condition was true. Similarly, the \var{false_value} -expression is only evaluated when the condition is false. +Evaluation is still lazy as in existing Boolean expressions, so the +order of evaluation jumps around a bit. The \var{condition} +expression in the middle is evaluated first, and the \var{true_value} +expression is evaluated only if the condition was true. Similarly, +the \var{false_value} expression is only evaluated when the condition +is false. This syntax may seem strange and backwards; why does the condition go in the \emph{middle} of the expression, and not in the front as in C's @@ -206,7 +212,96 @@ %====================================================================== \section{PEP 328: Absolute and Relative Imports} -% XXX write this +The simpler part of PEP 328 was implemented in Python 2.4: parentheses +could now be used to enclose the names imported from a module using +the \code{from ... import ...} statement, making it easier to import +many different names. + +The more complicated part has been implemented in Python 2.5: +importing a module can be specified to use absolute or +package-relative imports. The plan is to move toward making absolute +imports the default in future versions of Python. + +Let's say you have a package directory like this: +\begin{verbatim} +pkg/ +pkg/__init__.py +pkg/main.py +pkg/string.py +\end{verbatim} + +This defines a package named \module{pkg} containing the +\module{pkg.main} and \module{pkg.string} submodules. + +Consider the code in the \file{main.py} module. What happens if it +executes the statement \code{import string}? In Python 2.4 and +earlier, it will first look in the package's directory to perform a +relative import, finds \file{pkg/string.py}, imports the contents of +that file as the \module{pkg.string} module, and that module is bound +to the name \samp{string} in the \module{pkg.main} module's namespace. + +That's fine if \module{pkg.string} was what you wanted. But what if +you wanted Python's standard \module{string} module? There's no clean +way to ignore \module{pkg.string} and look for the standard module; +generally you had to look at the contents of \code{sys.modules}, which +is slightly unclean. +Holger Krekel's py.std package provides a tidier way to perform +imports from the standard library, \code{from py.std import string}, +% XXX correct attribution? +% XXX is that import correct? +but that package isn't available on all Python installations. + +Reading code which relies on relative imports is also less clear, +because a reader may be confused about which module, \module{string} +or \module{pkg.string}, is intended to be used. Python users soon +learned not to duplicate the names of standard library modules in the +names of their packages' submodules, but you can't protect against +having your submodule's name being used for a new module added in a +future version of Python. + +In Python 2.5, you can switch \keyword{import}'s behaviour to +absolute imports using a \code{from __future__ import absolute_import} +directive. This absolute-import behaviour will become the default in +a future version (probably Python 2.6). Once absolute-imports +are the default, \code{import string} will +always find the standard library's version. +It's suggested that users should begin using absolute imports as much +as possible, so it's preferable to begin writing \code{from pkg import +string} in your code. + +Relative imports are still possible by adding a leading period +to the module name when using the \code{from ... import} form: + +\begin{verbatim} +# Import names from pkg.string +from .string import name1, name2 +# Import pkg.string +from . import string +\end{verbatim} + +This imports the \module{string} module relative to the current +package, so in \module{pkg.main} this will import \var{name1} and +\var{name2} from \module{pkg.string}. Additional leading periods +perform the relative import starting from the parent of the current +package. For example, code in the \module{A.B.C} module can do: + +\begin{verbatim} +from . import D # Imports A.B.D +from .. import E # Imports A.E +from ..F import G # Imports A.F.G +\end{verbatim} + +Leading periods cannot be used with the \code{import \var{modname}} +form of the import statement, only the \code{from ... import} form. + +\begin{seealso} + +\seepep{328}{Imports: Multi-Line and Absolute/Relative}{PEP written +by Aahz; implemented by XXX.} + +\seeurl{XXX}{py.std} + +\end{seealso} %====================================================================== @@ -236,19 +331,62 @@ %====================================================================== \section{PEP 341: Unified try/except/finally} -% XXX write this +Until Python 2.5, the \keyword{try} statement came in two +flavours. You could use a \keyword{finally} block to ensure that code +is always executed, or a number of \keyword{except} blocks to catch an +exception. You couldn't combine both \keyword{except} blocks and a +\keyword{finally} block, because generating the right bytecode for the +combined version was complicated and it wasn't clear what the +semantics of the combined should be. + +GvR spent some time working with Java, which does support the +equivalent of combining \keyword{except} blocks and a +\keyword{finally} block, and this clarified what the statement should +mean. In Python 2.5, you can now write: + +\begin{verbatim} +try: + block-1 ... +except Exception1: + handler-1 ... +except Exception2: + handler-2 ... +else: + else-block +finally: + final-block +\end{verbatim} + +The code in \var{block-1} is executed. If the code raises an +exception, the handlers are tried in order: \var{handler-1}, +\var{handler-2}, ... If no exception is raised, the \var{else-block} +is executed. No matter what happened previously, the +\var{final-block} is executed once the code block is complete and any +raised exceptions handled. Even if there's an error in an exception +handler or the \var{else-block} and a new exception is raised, the +\var{final-block} is still executed. + +\begin{seealso} + +\seepep{341}{Unifying try-except and try-finally}{PEP written by Georg Brandl; +implementation by Thomas Lee. +XXX check implementor -- http://python.org/sf/1355913 +} + +\end{seealso} %====================================================================== \section{PEP 342: New Generator Features} +Python 2.5 adds a simple way to pass values \emph{into} a generator. As introduced in Python 2.3, generators only produce output; once a generator's code is invoked to create an iterator, there's no way to pass any new information into the function when its execution is -resumed. Hackish solutions to this include making the generator's -code look at a global variable and then changing the global variable's +resumed. Sometimes the ability to pass in some information would be +useful. Hackish solutions to this include making the generator's code +look at a global variable and then changing the global variable's value, or passing in some mutable object that callers then modify. -Python 2.5 adds the ability to pass values \emph{into} a generator. To refresh your memory of basic generators, here's a simple example: @@ -362,8 +500,22 @@ subroutines. Subroutines are entered at one point and exited at another point (the top of the function, and a \keyword{return statement}), but coroutines can be entered, exited, and resumed at -many different points (the \keyword{yield} statements). +many different points (the \keyword{yield} statements). We'll have to +figure out patterns for using coroutines effectively in Python. +The addition of the \method{close()} method has one side effect that +isn't obvious. \method{close()} is called when a generator is +garbage-collected, so this means the generator's code gets one last +chance to run before the generator is destroyed, and this last chance +means that \code{try...finally} statements in generators can now be +guaranteed to work; the \keyword{finally} clause will now always get a +chance to run. The syntactic restriction that you couldn't mix +\keyword{yield} statements with a \code{try...finally} suite has +therefore been removed. This seems like a minor bit of language +trivia, but using generators and \code{try...finally} is actually +necessary in order to implement the \keyword{with} statement +described by PEP 343. We'll look at this new statement in the following +section. \begin{seealso} @@ -385,14 +537,104 @@ %====================================================================== \section{PEP 343: The 'with' statement} +The \keyword{with} statement allows a clearer +version of code that uses \code{try...finally} blocks + +First, I'll discuss the statement as it will commonly be used, and +then I'll discuss the detailed implementation and how to write objects +(called ``context managers'') that can be used with this statement. +Most people, who will only use \keyword{with} in company with an +existing object, don't need to know these details, but can +Authors of new context managers will need to understand the + +The \keyword{with} statement is a new control-flow structure whose +basic structure is: + +\begin{verbatim} +with expression as variable: + with-block +\end{verbatim} + +The expression is evaluated, and it should result in a type of object +that's called a context manager. The context manager can return a +value that will be bound to the name \var{variable}. (Note carefully: +\var{variable} is \emph{not} assigned the result of \var{expression}. +One method of the context manager is run before \var{with-block} is +executed, and another method is run after the block is done, even if +the block raised an exception. + +To enable the statement in Python 2.5, you need +to add the following directive to your module: + +\begin{verbatim} +from __future__ import with_statement +\end{verbatim} + +Some standard Python objects can now behave as context managers. For +example, file objects: + +\begin{verbatim} +with open('/etc/passwd', 'r') as f: + for line in f: + print line + +# f has been automatically closed at this point. +\end{verbatim} + +The \module{threading} module's locks and condition variables +also support the new statement: + +\begin{verbatim} +lock = threading.Lock() +with lock: + # Critical section of code + ... +\end{verbatim} + +The lock is acquired before the block is executed, and released once +the block is complete. + +The \module{decimal} module's contexts, which encapsulate the desired +precision and rounding characteristics for computations, can also be +used as context managers. + +\begin{verbatim} +import decimal + +v1 = decimal.Decimal('578') + +# Displays with default precision of 28 digits +print v1.sqrt() + +with decimal.Context(prec=16): + # All code in this block uses a precision of 16 digits. + # The original context is restored on exiting the block. + print v1.sqrt() +\end{verbatim} + +\subsection{Writing Context Managers} + % XXX write this +The new \module{contextlib} module provides some functions and a +decorator that are useful for writing context managers. +% XXX describe further + +\begin{seealso} + +\seepep{343}{The ``with'' statement}{PEP written by +Guido van Rossum and Nick Coghlan. } + +\end{seealso} + %====================================================================== \section{PEP 352: Exceptions as New-Style Classes} -Exception classes can now be new-style classes, not just classic classes, -and the built-in \exception{Exception} class and all +Exception classes can now be new-style classes, not just classic +classes, and the built-in \exception{Exception} class and all the +standard built-in exceptions (\exception{NameError}, +\exception{ValueError}, etc.) are now new-style classes. The inheritance hierarchy for exceptions has been rearranged a bit. In 2.5, the inheritance relationships are: @@ -455,7 +697,47 @@ %====================================================================== \section{PEP 357: The '__index__' method} -% XXX write this +The NumPy developers had a problem that could only be solved by adding +a new special method, \method{__index__}. When using slice notation, +as in \code{[\var{start}:\var{stop}:\var{step}], the values of the +\var{start}, \var{stop}, and \var{step} indexes must all be either +integers or long integers. NumPy defines a variety of specialized +integer types corresponding to unsigned and signed integers of 8, 16, +32, and 64 bits, but there was no way to signal that these types could +be used as slice indexes. + +Slicing can't just use the existing \method{__int__} method because +that method is also used to implement coercion to integers. If +slicing used \method{__int__}, floating-point numbers would also +become legal slice indexes and that's clearly an undesirable +behaviour. + +Instead, a new special method called \method{__index__} was added. It +takes no arguments and returns an integer giving the slice index to +use. For example: + +\begin{verbatim} +class C: + def __index__ (self): + return self.value +\end{verbatim} + +The return value must be either a Python integer or long integer. +The interpreter will check that the type returned is correct, and +raises a \exception{TypeError} if this requirement isn't met. + +A corresponding \member{nb_index} slot was added to the C-level +\ctype{PyNumberMethods} structure to let C extensions implement this +protocol. \cfunction{PyNumber_Index(\var{obj})} can be used in +extension code to call the \method{__index__} function and retrieve +its result. + +\begin{seealso} + +\seepep{357}{Allowing Any Object to be Used for Slicing}{PEP written +(XXX and implemented?) by Travis Oliphant.} + +\end{seealso} %====================================================================== @@ -503,6 +785,8 @@ \end{verbatim} (Implemented by Brett Cannon.) +% XXX __missing__ hook in dictionaries + \end{itemize} @@ -536,6 +820,14 @@ and as a result sets will use a third less memory and are somewhat faster. (Implemented by Raymond Hettinger.) +\item The performance of some Unicode operations has been improved. +% XXX provide details? + +\item The code generator's peephole optimizer now performs +simple constant folding in expressions. If you write something like +\code{a = 2+3}, the code generator will do the arithmetic and produce +code corresponding to \code{a = 5}. + \end{itemize} The net result of the 2.5 optimizations is that Python 2.5 runs the @@ -557,6 +849,7 @@ % ctypes added % collections.deque now has .remove() +% collections.defaultdict % the cPickle module no longer accepts the deprecated None option in the % args tuple returned by __reduce__(). @@ -566,6 +859,11 @@ % datetime.datetime() now has a strptime class method which can be used to % create datetime object using a string and format. +% fileinput: opening hook used to control how files are opened. +% .input() now has a mode parameter +% now has a fileno() function +% accepts Unicode filenames + \item In the \module{gc} module, the new \function{get_count()} function returns a 3-tuple containing the current collection counts for the three GC generations. This is accounting information for the garbage @@ -574,14 +872,6 @@ function now takes an optional \var{generation} argument of 0, 1, or 2 to specify which generation to collect. -\item A new \module{hashlib} module has been added to replace the -\module{md5} and \module{sha} modules. \module{hashlib} adds support -for additional secure hashes (SHA-224, SHA-256, SHA-384, and SHA-512). -When available, the module uses OpenSSL for fast platform optimized -implementations of algorithms. The old \module{md5} and \module{sha} -modules still exist as wrappers around hashlib to preserve backwards -compatibility. (Contributed by Gregory P. Smith.) - \item The \function{nsmallest()} and \function{nlargest()} functions in the \module{heapq} module now support a \code{key} keyword argument similar to the one @@ -635,6 +925,17 @@ \function{os.lseek()} function. Two new constants for locking are \member{os.O_SHLOCK} and \member{os.O_EXLOCK}. +Two new functions, \function{wait3()} and \function{wait4()}, were +added. They're similar the \function{waitpid()} function which waits +for a child process to exit and returns a tuple of the process ID and +its exit status, but \function{wait3()} and \function{wait4()} return +additional information. \function{wait3()} doesn't take a process ID +as input, so it waits for any child process to exit and returns a +3-tuple of \var{process-id}, \var{exit-status}, \var{resource-usage} +as returned from the \function{resource.getrusage()} function. +\function{wait4(\var{pid})} does take a process ID. +(Contributed by XXX.) + On FreeBSD, the \function{os.stat()} function now returns times with nanosecond resolution, and the returned object now has \member{st_gen} and \member{st_birthtime}. @@ -660,10 +961,16 @@ In Python code, netlink addresses are represented as a tuple of 2 integers, \code{(\var{pid}, \var{group_mask})}. +Socket objects also gained accessor methods \method{getfamily()}, +\method{gettype()}, and \method{getproto()} methods to retrieve the +family, type, and protocol values for the socket. + \item New module: \module{spwd} provides functions for accessing the shadow password database on systems that support it. % XXX give example +% XXX patch #1382163: sys.subversion, Py_GetBuildNumber() + \item The \class{TarFile} class in the \module{tarfile} module now has an \method{extractall()} method that extracts all members from the archive into the current working directory. It's also possible to set @@ -680,6 +987,8 @@ by some specifications, so it's still available as \member{unicodedata.db_3_2_0}. +% patch #754022: Greatly enhanced webbrowser.py (by Oleg Broytmann). + \item A new package \module{xml.etree} has been added, which contains a subset of the ElementTree XML library. Available modules are \module{ElementTree}, \module{ElementPath}, and @@ -698,14 +1007,63 @@ %====================================================================== -% whole new modules get described in \subsections here +% whole new modules get described in subsections here % XXX new distutils features: upload -% XXX should hashlib perhaps be described here instead? -% XXX should xml.etree perhaps be described here instead? +%\subsection{The ElementTree package} + +\subsection{The hashlib package} +A new \module{hashlib} module has been added to replace the +\module{md5} and \module{sha} modules. \module{hashlib} adds support +for additional secure hashes (SHA-224, SHA-256, SHA-384, and SHA-512). +When available, the module uses OpenSSL for fast platform optimized +implementations of algorithms. + +The old \module{md5} and \module{sha} modules still exist as wrappers +around hashlib to preserve backwards compatibility. The new module's +interface is very close to that of the old modules, but not identical. +The most significant difference is that the constructor functions +for creating new hashing objects are named differently. +\begin{verbatim} +# Old versions +h = md5.md5() +h = md5.new() + +# New version +h = hashlib.md5() + +# Old versions +h = sha.sha() +h = sha.new() + +# New version +h = hashlib.sha1() + +# Hash that weren't previously available +h = hashlib.sha224() +h = hashlib.sha256() +h = hashlib.sha384() +h = hashlib.sha512() + +# Alternative form +h = hashlib.new('md5') # Provide algorithm as a string +\end{verbatim} + +Once a hash object has been created, its methods are the same as before: +\method{update(\var{string})} hashes the specified string into the +current digest state, \method{digest()} and \method{hexdigest()} +return the digest value as a binary string or a string of hex digits, +and \method{copy()} returns a new hashing object with the same digest state. + +This module was contributed by Gregory P. Smith. + + +%\subsection{The sqlite3 package} + +% XXX write these sections % ====================================================================== \section{Build and C API Changes} @@ -714,8 +1072,9 @@ \begin{itemize} -\item The design of the bytecode compiler has changed a great deal, no -longer generating bytecode by traversing the parse tree. Instead +% XXX PEP 353: ssize_t +\item The design of the bytecode compiler has changed a great deal, to +no longer generate bytecode by traversing the parse tree. Instead the parse tree is converted to an abstract syntax tree (or AST), and it is the abstract syntax tree that's traversed to produce the bytecode. @@ -753,9 +1112,9 @@ %====================================================================== -\subsection{Port-Specific Changes} +%\subsection{Port-Specific Changes} -Platform-specific changes go here. +%Platform-specific changes go here. %====================================================================== @@ -779,6 +1138,12 @@ the operating system. (Implemented by Evan Jones, and reworked by Tim Peters.) +\item Coverity, a company that markets a source code analysis tool + called Prevent, provided the results of their examination of the Python + source code. The analysis found a number of refcounting bugs, often + in error-handling code. These bugs have been fixed. + % XXX provide reference? + \end{itemize} From python-checkins at python.org Sun Apr 2 03:47:39 2006 From: python-checkins at python.org (andrew.kuchling) Date: Sun, 2 Apr 2006 03:47:39 +0200 (CEST) Subject: [Python-checkins] r43548 - python/trunk/Doc/lib/libhashlib.tex Message-ID: <20060402014739.1A61B1E400D@bag.python.org> Author: andrew.kuchling Date: Sun Apr 2 03:47:38 2006 New Revision: 43548 Modified: python/trunk/Doc/lib/libhashlib.tex Log: Grammar fix Modified: python/trunk/Doc/lib/libhashlib.tex ============================================================================== --- python/trunk/Doc/lib/libhashlib.tex (original) +++ python/trunk/Doc/lib/libhashlib.tex Sun Apr 2 03:47:38 2006 @@ -31,7 +31,7 @@ Constructors for hash algorithms that are always present in this module are \function{md5()}, \function{sha1()}, \function{sha224()}, \function{sha256()}, \function{sha384()}, and \function{sha512()}. Additional algorithms may also -be available depending upon the OpenSSL library python uses on your platform. +be available depending upon the OpenSSL library that Python uses on your platform. \index{OpenSSL} For example, to obtain the digest of the string \code{'Nobody inspects From dynkin at gmail.com Sun Apr 2 04:46:57 2006 From: dynkin at gmail.com (George Yoshida) Date: Sun, 2 Apr 2006 11:46:57 +0900 Subject: [Python-checkins] r43547 - python/trunk/Doc/whatsnew/whatsnew25.tex In-Reply-To: <20060402014633.98C301E4008@bag.python.org> References: <20060402014633.98C301E4008@bag.python.org> Message-ID: <2f188ee80604011846r559fa5e9ted0fb508d186d002@mail.gmail.com> On 4/2/06, andrew.kuchling wrote: > Author: andrew.kuchling > Date: Sun Apr 2 03:46:32 2006 > New Revision: 43547 > > Modified: > python/trunk/Doc/whatsnew/whatsnew25.tex > Log: > Write various sections; I haven't been able to check whether the TeX markup is correct > With this fix, at least the compilation should succeed, although I haven't checked every piece of the update in detail. Index: Doc/whatsnew/whatsnew25.tex =================================================================== --- Doc/whatsnew/whatsnew25.tex (revision 43548) +++ Doc/whatsnew/whatsnew25.tex (working copy) @@ -699,7 +699,7 @@ The NumPy developers had a problem that could only be solved by adding a new special method, \method{__index__}. When using slice notation, -as in \code{[\var{start}:\var{stop}:\var{step}], the values of the +as in \code{[\var{start}:\var{stop}:\var{step}]}, the values of the \var{start}, \var{stop}, and \var{step} indexes must all be either integers or long integers. NumPy defines a variety of specialized integer types corresponding to unsigned and signed integers of 8, 16, george at arp:/opt/python$ From python-checkins at python.org Sun Apr 2 05:30:07 2006 From: python-checkins at python.org (fred.drake) Date: Sun, 2 Apr 2006 05:30:07 +0200 (CEST) Subject: [Python-checkins] r43549 - python/trunk/Doc/whatsnew/whatsnew25.tex Message-ID: <20060402033007.780581E4008@bag.python.org> Author: fred.drake Date: Sun Apr 2 05:30:06 2006 New Revision: 43549 Modified: python/trunk/Doc/whatsnew/whatsnew25.tex Log: fix markup error Modified: python/trunk/Doc/whatsnew/whatsnew25.tex ============================================================================== --- python/trunk/Doc/whatsnew/whatsnew25.tex (original) +++ python/trunk/Doc/whatsnew/whatsnew25.tex Sun Apr 2 05:30:06 2006 @@ -699,7 +699,7 @@ The NumPy developers had a problem that could only be solved by adding a new special method, \method{__index__}. When using slice notation, -as in \code{[\var{start}:\var{stop}:\var{step}], the values of the +as in \code{[\var{start}:\var{stop}:\var{step}]}, the values of the \var{start}, \var{stop}, and \var{step} indexes must all be either integers or long integers. NumPy defines a variety of specialized integer types corresponding to unsigned and signed integers of 8, 16, From fdrake at acm.org Sun Apr 2 05:30:16 2006 From: fdrake at acm.org (Fred L. Drake, Jr.) Date: Sat, 1 Apr 2006 22:30:16 -0500 Subject: [Python-checkins] r43547 - python/trunk/Doc/whatsnew/whatsnew25.tex In-Reply-To: <2f188ee80604011846r559fa5e9ted0fb508d186d002@mail.gmail.com> References: <20060402014633.98C301E4008@bag.python.org> <2f188ee80604011846r559fa5e9ted0fb508d186d002@mail.gmail.com> Message-ID: <200604012230.17192.fdrake@acm.org> On Saturday 01 April 2006 21:46, George Yoshida wrote: > With this fix, at least the compilation should succeed, although I > haven't checked every piece of the update in detail. Committed, thanks! -Fred -- Fred L. Drake, Jr. From python-checkins at python.org Sun Apr 2 09:59:56 2006 From: python-checkins at python.org (fredrik.lundh) Date: Sun, 2 Apr 2006 09:59:56 +0200 (CEST) Subject: [Python-checkins] r43550 - python/trunk/Doc/tut/tut.tex Message-ID: <20060402075956.7D06A1E4005@bag.python.org> Author: fredrik.lundh Date: Sun Apr 2 09:59:55 2006 New Revision: 43550 Modified: python/trunk/Doc/tut/tut.tex Log: end the sentence... Modified: python/trunk/Doc/tut/tut.tex ============================================================================== --- python/trunk/Doc/tut/tut.tex (original) +++ python/trunk/Doc/tut/tut.tex Sun Apr 2 09:59:55 2006 @@ -5347,7 +5347,7 @@ \item \citetitle[../ref/ref.html]{Language Reference}: A detailed explanation of Python's syntax and semantics. It's heavy reading, -but is useful as a +but is useful as a complete guide to the language itself. \end{itemize} From python-checkins at python.org Sun Apr 2 12:48:08 2006 From: python-checkins at python.org (armin.rigo) Date: Sun, 2 Apr 2006 12:48:08 +0200 (CEST) Subject: [Python-checkins] r43551 - python/branches/release24-maint/Lib/inspect.py Message-ID: <20060402104808.643181E4005@bag.python.org> Author: armin.rigo Date: Sun Apr 2 12:48:07 2006 New Revision: 43551 Modified: python/branches/release24-maint/Lib/inspect.py Log: Backport of r39612: some more fixes for inspect.getsource(). Tests not included (the format changes and I can't be bothered to port them). Modified: python/branches/release24-maint/Lib/inspect.py ============================================================================== --- python/branches/release24-maint/Lib/inspect.py (original) +++ python/branches/release24-maint/Lib/inspect.py Sun Apr 2 12:48:07 2006 @@ -484,19 +484,6 @@ comments[-1:] = [] return string.join(comments, '') -class ListReader: - """Provide a readline() method to return lines from a list of strings.""" - def __init__(self, lines): - self.lines = lines - self.index = 0 - - def readline(self): - i = self.index - if i < len(self.lines): - self.index = i + 1 - return self.lines[i] - else: return '' - class EndOfBlock(Exception): pass class BlockFinder: @@ -506,40 +493,46 @@ self.islambda = False self.started = False self.passline = False - self.last = 0 + self.last = 1 def tokeneater(self, type, token, (srow, scol), (erow, ecol), line): if not self.started: + # look for the first "def", "class" or "lambda" if token in ("def", "class", "lambda"): if token == "lambda": self.islambda = True self.started = True - self.passline = True + self.passline = True # skip to the end of the line elif type == tokenize.NEWLINE: - self.passline = False + self.passline = False # stop skipping when a NEWLINE is seen self.last = srow + if self.islambda: # lambdas always end at the first NEWLINE + raise EndOfBlock elif self.passline: pass - elif self.islambda: - raise EndOfBlock, self.last elif type == tokenize.INDENT: self.indent = self.indent + 1 self.passline = True elif type == tokenize.DEDENT: self.indent = self.indent - 1 - if self.indent == 0: - raise EndOfBlock, self.last - elif type == tokenize.NAME and scol == 0: - raise EndOfBlock, self.last + # the end of matching indent/dedent pairs end a block + # (note that this only works for "def"/"class" blocks, + # not e.g. for "if: else:" or "try: finally:" blocks) + if self.indent <= 0: + raise EndOfBlock + elif self.indent == 0 and type not in (tokenize.COMMENT, tokenize.NL): + # any other token on the same indentation level end the previous + # block as well, except the pseudo-tokens COMMENT and NL. + raise EndOfBlock def getblock(lines): """Extract the block of code at the top of the given list of lines.""" + blockfinder = BlockFinder() try: - tokenize.tokenize(ListReader(lines).readline, BlockFinder().tokeneater) - except EndOfBlock, eob: - return lines[:eob.args[0]] - # Fooling the indent/dedent logic implies a one-line definition - return lines[:1] + tokenize.tokenize(iter(lines).next, blockfinder.tokeneater) + except (EndOfBlock, IndentationError): + pass + return lines[:blockfinder.last] def getsourcelines(object): """Return a list of source lines and starting line number for an object. From python-checkins at python.org Sun Apr 2 22:37:17 2006 From: python-checkins at python.org (georg.brandl) Date: Sun, 2 Apr 2006 22:37:17 +0200 (CEST) Subject: [Python-checkins] r43552 - python/trunk/Lib/urllib2.py Message-ID: <20060402203717.8CE381E4004@bag.python.org> Author: georg.brandl Date: Sun Apr 2 22:37:17 2006 New Revision: 43552 Modified: python/trunk/Lib/urllib2.py Log: bug #1462706: guard against host not having FQDN hostname Modified: python/trunk/Lib/urllib2.py ============================================================================== --- python/trunk/Lib/urllib2.py (original) +++ python/trunk/Lib/urllib2.py Sun Apr 2 22:37:17 2006 @@ -1130,8 +1130,11 @@ names = None def get_names(self): if FileHandler.names is None: - FileHandler.names = (socket.gethostbyname('localhost'), - socket.gethostbyname(socket.gethostname())) + try: + FileHandler.names = (socket.gethostbyname('localhost'), + socket.gethostbyname(socket.gethostname())) + except socket.gaierror: + FileHandler.names = (socket.gethostbyname('localhost'),) return FileHandler.names # not entirely sure what the rules are here From python-checkins at python.org Sun Apr 2 22:45:35 2006 From: python-checkins at python.org (georg.brandl) Date: Sun, 2 Apr 2006 22:45:35 +0200 (CEST) Subject: [Python-checkins] r43553 - in python/trunk/Lib: test/test_urllib2.py urllib2.py Message-ID: <20060402204535.840571E401A@bag.python.org> Author: georg.brandl Date: Sun Apr 2 22:45:34 2006 New Revision: 43553 Modified: python/trunk/Lib/test/test_urllib2.py python/trunk/Lib/urllib2.py Log: Patch #1462790: fix urllib2 ProxyHandler for host:port proxies Modified: python/trunk/Lib/test/test_urllib2.py ============================================================================== --- python/trunk/Lib/test/test_urllib2.py (original) +++ python/trunk/Lib/test/test_urllib2.py Sun Apr 2 22:45:34 2006 @@ -13,8 +13,7 @@ # parse_keqv_list, parse_http_list (I'm leaving this for Anthony Baxter # and Greg Stein, since they're doing Digest Authentication) # Authentication stuff (ditto) -# ProxyHandler, CustomProxy, CustomProxyHandler (I don't use a proxy) -# GopherHandler (haven't used gopher for a decade or so...) +# CustomProxy, CustomProxyHandler class TrivialTests(unittest.TestCase): def test_trivial(self): @@ -90,6 +89,7 @@ return self.handle(self.meth_name, self.action, *args) class MockHandler: + handler_order = 500 def __init__(self, methods): self._define_methods(methods) def _define_methods(self, methods): @@ -154,7 +154,7 @@ for meths in meth_spec: class MockHandlerSubclass(MockHandler): pass h = MockHandlerSubclass(meths) - h.handler_order = count + h.handler_order += count h.add_parent(opener) count = count + 1 handlers.append(h) @@ -642,6 +642,23 @@ o.open("http://www.example.com/") self.assert_(not hh.req.has_header("Cookie")) + def test_proxy(self): + o = OpenerDirector() + ph = urllib2.ProxyHandler(dict(http="proxy.example.com:3128")) + o.add_handler(ph) + meth_spec = [ + [("http_open", "return response")] + ] + handlers = add_ordered_mock_handlers(o, meth_spec) + + req = Request("http://acme.example.com/") + self.assertEqual(req.get_host(), "acme.example.com") + r = o.open(req) + self.assertEqual(req.get_host(), "proxy.example.com:3128") + + self.assertEqual([(handlers[0], "http_open")], + [tup[0:2] for tup in o.calls]) + class MiscTests(unittest.TestCase): @@ -827,6 +844,7 @@ def test_main(verbose=None): + test_support.run_doctest(urllib2, verbose) tests = (TrivialTests, OpenerDirectorTests, HandlerTests, Modified: python/trunk/Lib/urllib2.py ============================================================================== --- python/trunk/Lib/urllib2.py (original) +++ python/trunk/Lib/urllib2.py Sun Apr 2 22:45:34 2006 @@ -119,7 +119,8 @@ # support for FileHandler, proxies via environment variables from urllib import localhost, url2pathname, getproxies -__version__ = "2.5" +# used in User-Agent header sent +__version__ = sys.version[:3] _opener = None def urlopen(url, data=None): @@ -563,6 +564,80 @@ "lead to an infinite loop.\n" \ "The last 30x error message was:\n" + +def _parse_proxy(proxy): + """Return (scheme, user, password, host/port) given a URL or an authority. + + If a URL is supplied, it must have an authority (host:port) component. + According to RFC 3986, having an authority component means the URL must + have two slashes after the scheme: + + >>> _parse_proxy('file:/ftp.example.com/') + Traceback (most recent call last): + ValueError: proxy URL with no authority: 'file:/ftp.example.com/' + + The first three items of the returned tuple may be None. + + Examples of authority parsing: + + >>> _parse_proxy('proxy.example.com') + (None, None, None, 'proxy.example.com') + >>> _parse_proxy('proxy.example.com:3128') + (None, None, None, 'proxy.example.com:3128') + + The authority component may optionally include userinfo (assumed to be + username:password): + + >>> _parse_proxy('joe:password at proxy.example.com') + (None, 'joe', 'password', 'proxy.example.com') + >>> _parse_proxy('joe:password at proxy.example.com:3128') + (None, 'joe', 'password', 'proxy.example.com:3128') + + Same examples, but with URLs instead: + + >>> _parse_proxy('http://proxy.example.com/') + ('http', None, None, 'proxy.example.com') + >>> _parse_proxy('http://proxy.example.com:3128/') + ('http', None, None, 'proxy.example.com:3128') + >>> _parse_proxy('http://joe:password at proxy.example.com/') + ('http', 'joe', 'password', 'proxy.example.com') + >>> _parse_proxy('http://joe:password at proxy.example.com:3128') + ('http', 'joe', 'password', 'proxy.example.com:3128') + + Everything after the authority is ignored: + + >>> _parse_proxy('ftp://joe:password at proxy.example.com/rubbish:3128') + ('ftp', 'joe', 'password', 'proxy.example.com') + + Test for no trailing '/' case: + + >>> _parse_proxy('http://joe:password at proxy.example.com') + ('http', 'joe', 'password', 'proxy.example.com') + + """ + from urlparse import _splitnetloc + scheme, r_scheme = splittype(proxy) + if not r_scheme.startswith("/"): + # authority + scheme = None + authority = proxy + else: + # URL + if not r_scheme.startswith("//"): + raise ValueError("proxy URL with no authority: %r" % proxy) + # We have an authority, so for RFC 3986-compliant URLs (by ss 3. + # and 3.3.), path is empty or starts with '/' + end = r_scheme.find("/", 2) + if end == -1: + end = None + authority = r_scheme[2:end] + userinfo, hostport = splituser(authority) + if userinfo is not None: + user, password = splitpasswd(userinfo) + else: + user = password = None + return scheme, user, password, hostport + class ProxyHandler(BaseHandler): # Proxies must be in front handler_order = 100 @@ -579,30 +654,25 @@ def proxy_open(self, req, proxy, type): orig_type = req.get_type() - type, r_type = splittype(proxy) - if not type or r_type.isdigit(): - # proxy is specified without protocol - type = orig_type - host = proxy - else: - host, r_host = splithost(r_type) - user_pass, host = splituser(host) - user, password = splitpasswd(user_pass) + proxy_type, user, password, hostport = _parse_proxy(proxy) + if proxy_type is None: + proxy_type = orig_type if user and password: - user, password = user_pass.split(':', 1) - user_pass = base64.encodestring('%s:%s' % (unquote(user), - unquote(password))).strip() - req.add_header('Proxy-authorization', 'Basic ' + user_pass) - host = unquote(host) - req.set_proxy(host, type) - if orig_type == type: + user_pass = '%s:%s' % (unquote(user), unquote(password)) + creds = base64.encodestring(user_pass).strip() + req.add_header('Proxy-authorization', 'Basic ' + creds) + hostport = unquote(hostport) + req.set_proxy(hostport, proxy_type) + if orig_type == proxy_type: # let other handlers take care of it - # XXX this only makes sense if the proxy is before the - # other handlers return None else: # need to start over, because the other handlers don't # grok the proxy's URL type + # e.g. if we have a constructor arg proxies like so: + # {'http': 'ftp://proxy.example.com'}, we may end up turning + # a request for http://acme.example.com/a into one for + # ftp://proxy.example.com/a return self.parent.open(req) # feature suggested by Duncan Booth From python-checkins at python.org Sun Apr 2 22:48:12 2006 From: python-checkins at python.org (georg.brandl) Date: Sun, 2 Apr 2006 22:48:12 +0200 (CEST) Subject: [Python-checkins] r43554 - python/trunk/Lib/urllib2.py Message-ID: <20060402204812.2D1571E4004@bag.python.org> Author: georg.brandl Date: Sun Apr 2 22:48:11 2006 New Revision: 43554 Modified: python/trunk/Lib/urllib2.py Log: Patch #1463012: remove not working undocumented classes from urllib2 Modified: python/trunk/Lib/urllib2.py ============================================================================== --- python/trunk/Lib/urllib2.py (original) +++ python/trunk/Lib/urllib2.py Sun Apr 2 22:48:11 2006 @@ -14,7 +14,7 @@ HTTP 301, 302, 303 and 307 redirect errors, and the HTTPDigestAuthHandler deals with digest authentication. -urlopen(url, data=None) -- basic usage is that same as original +urlopen(url, data=None) -- basic usage is the same as original urllib. pass the url and optionally data to post to an HTTP URL, and get a file-like object back. One difference is that you can also pass a Request instance instead of URL. Raises a URLError (subclass of @@ -77,16 +77,13 @@ # the handler knows that the problem was, e.g., that it didn't know # that hash algo that requested in the challenge, it would be good to # pass that information along to the client, too. - -# XXX to do: -# name! -# documentation (getting there) -# complex proxies -# abstract factory for opener # ftp errors aren't handled cleanly -# gopher can return a socket.error # check digest against correct (i.e. non-apache) implementation +# Possible extensions: +# complex proxies XXX not sure what exactly was meant by this +# abstract factory for opener + import base64 import ftplib import httplib @@ -111,8 +108,7 @@ except ImportError: from StringIO import StringIO -# not sure how many of these need to be gotten rid of -from urllib import (unwrap, unquote, splittype, splithost, quote, +from urllib import (unwrap, unquote, splittype, splithost, addinfourl, splitport, splitgophertype, splitquery, splitattr, ftpwrapper, noheaders, splituser, splitpasswd, splitvalue) @@ -331,8 +327,9 @@ pass def _call_chain(self, chain, kind, meth_name, *args): - # XXX raise an exception if no one else should try to handle - # this url. return None if you can't but someone else could. + # Handlers raise an exception if no one else should try to handle + # the request, or return None if they can't but another handler + # could. Otherwise, they return the response. handlers = chain.get(kind, ()) for handler in handlers: func = getattr(handler, meth_name) @@ -675,50 +672,6 @@ # ftp://proxy.example.com/a return self.parent.open(req) -# feature suggested by Duncan Booth -# XXX custom is not a good name -class CustomProxy: - # either pass a function to the constructor or override handle - def __init__(self, proto, func=None, proxy_addr=None): - self.proto = proto - self.func = func - self.addr = proxy_addr - - def handle(self, req): - if self.func and self.func(req): - return 1 - - def get_proxy(self): - return self.addr - -class CustomProxyHandler(BaseHandler): - # Proxies must be in front - handler_order = 100 - - def __init__(self, *proxies): - self.proxies = {} - - def proxy_open(self, req): - proto = req.get_type() - try: - proxies = self.proxies[proto] - except KeyError: - return None - for p in proxies: - if p.handle(req): - req.set_proxy(p.get_proxy()) - return self.parent.open(req) - return None - - def do_proxy(self, p, req): - return self.parent.open(req) - - def add_proxy(self, cpo): - if cpo.proto in self.proxies: - self.proxies[cpo.proto].append(cpo) - else: - self.proxies[cpo.proto] = [cpo] - class HTTPPasswordMgr: def __init__(self): self.passwd = {} @@ -1333,6 +1286,7 @@ class GopherHandler(BaseHandler): def gopher_open(self, req): + # XXX can raise socket.error import gopherlib # this raises DeprecationWarning in 2.5 host = req.get_host() if not host: @@ -1348,25 +1302,3 @@ else: fp = gopherlib.send_selector(selector, host) return addinfourl(fp, noheaders(), req.get_full_url()) - -#bleck! don't use this yet -class OpenerFactory: - - default_handlers = [UnknownHandler, HTTPHandler, - HTTPDefaultErrorHandler, HTTPRedirectHandler, - FTPHandler, FileHandler] - handlers = [] - replacement_handlers = [] - - def add_handler(self, h): - self.handlers = self.handlers + [h] - - def replace_handler(self, h): - pass - - def build_opener(self): - opener = OpenerDirector() - for ph in self.default_handlers: - if inspect.isclass(ph): - ph = ph() - opener.add_handler(ph) From nnorwitz at gmail.com Sun Apr 2 22:54:41 2006 From: nnorwitz at gmail.com (Neal Norwitz) Date: Sun, 2 Apr 2006 12:54:41 -0800 Subject: [Python-checkins] r43554 - python/trunk/Lib/urllib2.py In-Reply-To: <20060402204812.2D1571E4004@bag.python.org> References: <20060402204812.2D1571E4004@bag.python.org> Message-ID: On 4/2/06, georg.brandl wrote: > Author: georg.brandl > Date: Sun Apr 2 22:48:11 2006 > New Revision: 43554 > > Modified: > python/trunk/Lib/urllib2.py > Log: > Patch #1463012: remove not working undocumented classes from urllib2 > > > > Modified: python/trunk/Lib/urllib2.py > ============================================================================== > --- python/trunk/Lib/urllib2.py (original) > +++ python/trunk/Lib/urllib2.py Sun Apr 2 22:48:11 2006 > @@ -111,8 +108,7 @@ > except ImportError: > from StringIO import StringIO > > -# not sure how many of these need to be gotten rid of > -from urllib import (unwrap, unquote, splittype, splithost, quote, > +from urllib import (unwrap, unquote, splittype, splithost, I'm curious why quote was removed? ISTM that people might be relying on urllib2.quote, even if by mistake (ie, they meant urllib.quote). Although, Google didn't seem to find anything when I searched for urllib2.quote. n From python-checkins at python.org Sun Apr 2 23:09:52 2006 From: python-checkins at python.org (georg.brandl) Date: Sun, 2 Apr 2006 23:09:52 +0200 (CEST) Subject: [Python-checkins] r43555 - python/trunk/Lib/lib-tk/Tkinter.py Message-ID: <20060402210952.39C461E4004@bag.python.org> Author: georg.brandl Date: Sun Apr 2 23:09:51 2006 New Revision: 43555 Modified: python/trunk/Lib/lib-tk/Tkinter.py Log: Remove "disgusting hack" in favour of closure (patch #1462235) Modified: python/trunk/Lib/lib-tk/Tkinter.py ============================================================================== --- python/trunk/Lib/lib-tk/Tkinter.py (original) +++ python/trunk/Lib/lib-tk/Tkinter.py Sun Apr 2 23:09:51 2006 @@ -449,18 +449,15 @@ # I'd rather use time.sleep(ms*0.001) self.tk.call('after', ms) else: - # XXX Disgusting hack to clean up after calling func - tmp = [] - def callit(func=func, args=args, self=self, tmp=tmp): + def callit(): try: func(*args) finally: try: - self.deletecommand(tmp[0]) + self.deletecommand(name) except TclError: pass name = self._register(callit) - tmp.append(name) return self.tk.call('after', ms, name) def after_idle(self, func, *args): """Call FUNC once if the Tcl main loop has no event to From python-checkins at python.org Sun Apr 2 23:13:13 2006 From: python-checkins at python.org (georg.brandl) Date: Sun, 2 Apr 2006 23:13:13 +0200 (CEST) Subject: [Python-checkins] r43556 - python/trunk/Lib/urllib2.py Message-ID: <20060402211313.7E94B1E4004@bag.python.org> Author: georg.brandl Date: Sun Apr 2 23:13:13 2006 New Revision: 43556 Modified: python/trunk/Lib/urllib2.py Log: Readd urllib.quote import as it doesn't cause any harm. Modified: python/trunk/Lib/urllib2.py ============================================================================== --- python/trunk/Lib/urllib2.py (original) +++ python/trunk/Lib/urllib2.py Sun Apr 2 23:13:13 2006 @@ -108,7 +108,7 @@ except ImportError: from StringIO import StringIO -from urllib import (unwrap, unquote, splittype, splithost, +from urllib import (unwrap, unquote, splittype, splithost, quote, addinfourl, splitport, splitgophertype, splitquery, splitattr, ftpwrapper, noheaders, splituser, splitpasswd, splitvalue) From g.brandl at gmx.net Sun Apr 2 23:13:23 2006 From: g.brandl at gmx.net (Georg Brandl) Date: Sun, 02 Apr 2006 23:13:23 +0200 Subject: [Python-checkins] r43554 - python/trunk/Lib/urllib2.py In-Reply-To: References: <20060402204812.2D1571E4004@bag.python.org> Message-ID: Neal Norwitz wrote: > On 4/2/06, georg.brandl wrote: >> Author: georg.brandl >> Date: Sun Apr 2 22:48:11 2006 >> New Revision: 43554 >> >> Modified: >> python/trunk/Lib/urllib2.py >> Log: >> Patch #1463012: remove not working undocumented classes from urllib2 >> >> >> >> Modified: python/trunk/Lib/urllib2.py >> ============================================================================== >> --- python/trunk/Lib/urllib2.py (original) >> +++ python/trunk/Lib/urllib2.py Sun Apr 2 22:48:11 2006 >> @@ -111,8 +108,7 @@ >> except ImportError: >> from StringIO import StringIO >> >> -# not sure how many of these need to be gotten rid of >> -from urllib import (unwrap, unquote, splittype, splithost, quote, >> +from urllib import (unwrap, unquote, splittype, splithost, > > I'm curious why quote was removed? ISTM that people might be relying > on urllib2.quote, even if by mistake (ie, they meant urllib.quote). > Although, Google didn't seem to find anything when I searched for > urllib2.quote. Granted, that would be gratuitous breakage. Readded in rev. 43556. Georg From python-checkins at python.org Sun Apr 2 23:18:27 2006 From: python-checkins at python.org (georg.brandl) Date: Sun, 2 Apr 2006 23:18:27 +0200 (CEST) Subject: [Python-checkins] r43557 - python/trunk/Doc/lib/libos.tex Message-ID: <20060402211827.7F8591E4004@bag.python.org> Author: georg.brandl Date: Sun Apr 2 23:18:27 2006 New Revision: 43557 Modified: python/trunk/Doc/lib/libos.tex Log: Document recent fdopen() change. Modified: python/trunk/Doc/lib/libos.tex ============================================================================== --- python/trunk/Doc/lib/libos.tex (original) +++ python/trunk/Doc/lib/libos.tex Sun Apr 2 23:18:27 2006 @@ -343,6 +343,10 @@ \versionchanged[When specified, the \var{mode} argument must now start with one of the letters \character{r}, \character{w}, or \character{a}, otherwise a \exception{ValueError} is raised]{2.3} +\versionchanged[On \UNIX, when the \var{mode} argument starts with + \character{a}, the \var{O_APPEND} flag is set on the file descriptor + (which the \cfunction{fdopen()} implementation already does on most + platforms)]{2.5} \end{funcdesc} \begin{funcdesc}{popen}{command\optional{, mode\optional{, bufsize}}} @@ -1732,7 +1736,7 @@ return suitable process handles. \end{funcdesc} -\begin{funcdesc}{wait3}{\{optional{options}} +\begin{funcdesc}{wait3}{\optional{options}} Similar to \function{waitpid()}, except no process id argument is given and a 3-element tuple containing the child's process id, exit status indication, and resource usage information is returned. Refer to From buildbot at python.org Sun Apr 2 23:35:36 2006 From: buildbot at python.org (buildbot at python.org) Date: Sun, 02 Apr 2006 21:35:36 +0000 Subject: [Python-checkins] buildbot failure in x86 XP-2 trunk Message-ID: <20060402213537.16C551E4004@bag.python.org> The Buildbot has detected a new failure of x86 XP-2 trunk. Full details are available at: http://www.python.org/dev/buildbot/all/x86%20XP-2%20trunk/builds/169 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: georg.brandl BUILD FAILED: failed failed slave lost sincerely, -The Buildbot From python-checkins at python.org Mon Apr 3 00:11:10 2006 From: python-checkins at python.org (walter.doerwald) Date: Mon, 3 Apr 2006 00:11:10 +0200 (CEST) Subject: [Python-checkins] r43558 - python/trunk/Lib/calendar.py Message-ID: <20060402221110.522991E4004@bag.python.org> Author: walter.doerwald Date: Mon Apr 3 00:11:10 2006 New Revision: 43558 Modified: python/trunk/Lib/calendar.py Log: Always use firstweekday module 7. Modified: python/trunk/Lib/calendar.py ============================================================================== --- python/trunk/Lib/calendar.py (original) +++ python/trunk/Lib/calendar.py Mon Apr 3 00:11:10 2006 @@ -152,7 +152,7 @@ while True: yield date date += oneday - if date.month != month and date.weekday() == self.firstweekday: + if date.month != month and date.weekday() == self.firstweekday%7: break def itermonthdays2(self, year, month): From neal at metaslash.com Mon Apr 3 03:09:47 2006 From: neal at metaslash.com (Neal Norwitz) Date: Sun, 2 Apr 2006 21:09:47 -0400 Subject: [Python-checkins] Python Regression Test Failures all (1) Message-ID: <20060403010947.GA9396@python.psfb.org> test_grammar test_opcodes test_operations test_builtin test_exceptions test_types test_MimeWriter test_StringIO test___all__ test___future__ test__locale test_aepack test_aepack skipped -- No module named aepack test_al test_al skipped -- No module named al test_anydbm test_applesingle test_applesingle skipped -- No module named macostools test_array test_ast test_asynchat test_atexit test_audioop test_augassign test_base64 test_bastion test_binascii test_binhex test_binop test_bisect test_bool test_bsddb test_bsddb185 test_bsddb185 skipped -- No module named bsddb185 test_bsddb3 test_unicode_file test_unicode_file skipped -- No Unicode filesystem semantics on this platform. test_unicodedata test_unittest test_univnewlines test_unpack test_urllib test_urllib2 test_urllib2net test_urllibnet test_urlparse test_userdict test_userlist test_userstring test_uu test_wait3 test_wait4 test_warnings test_wave test_weakref test_whichdb test_winreg test_winreg skipped -- No module named _winreg test_winsound test_winsound skipped -- No module named winsound test_with test_xdrlib test_xml_etree test_xml_etree_c test_xmllib test_xmlrpc test_xpickle test_xrange test_zipfile test_zipimport test_zlib 288 tests OK. 1 test failed: test_bsddb3 20 tests skipped: test_aepack test_al test_applesingle test_bsddb185 test_cd test_cl test_gl test_imgfile test_ioctl test_macfs test_macostools test_nis test_pep277 test_plistlib test_scriptpackages test_sunaudiodev test_tcl test_unicode_file test_winreg test_winsound 1 skip unexpected on linux2: test_ioctl [424302 refs] From neal at metaslash.com Mon Apr 3 03:21:49 2006 From: neal at metaslash.com (Neal Norwitz) Date: Sun, 2 Apr 2006 21:21:49 -0400 Subject: [Python-checkins] Python Regression Test Failures all (1) Message-ID: <20060403012149.GA11058@python.psfb.org> test_grammar test_opcodes test_operations test_builtin test_exceptions test_types test_MimeWriter test_StringIO test___all__ test___future__ test__locale test_aepack test_aepack skipped -- No module named aepack test_al test_al skipped -- No module named al test_anydbm test_applesingle test_applesingle skipped -- No module named macostools test_array test_ast test_asynchat test_atexit test_audioop test_augassign test_base64 test_bastion test_binascii test_binhex test_binop test_bisect test_bool test_bsddb test_bsddb185 test_bsddb185 skipped -- No module named bsddb185 test_bsddb3 test_bufio test_bz2 test_cProfile test_calendar test_call test_capi test_cd test_cd skipped -- No module named cd test_cfgparser test_cgi test_charmapcodec test_cl test_cl skipped -- No module named cl test_class test_cmath test_cmd_line test_code test_codeccallbacks test_codecencodings_cn test_codecencodings_hk test_codecencodings_jp test_codecencodings_kr test_codecencodings_tw test_codecmaps_cn test_codecmaps_hk test_codecmaps_jp test_codecmaps_kr test_codecmaps_tw test_codecs test_codeop test_coding test_coercion test_colorsys test_commands test_compare test_compile test_compiler test_complex test_contains test_contextlib test_cookie test_cookielib test_copy test_copy_reg test_cpickle test_crypt test_csv test_ctypes test_datetime test_dbm test_decimal test_decorators test_defaultdict test_deque test_descr test_descrtut test_dict test_difflib test_dircache test_dis test_distutils test_dl test_doctest test_doctest2 test_dumbdbm test_dummy_thread test_dummy_threading test_email test_email_codecs test_email_renamed test_enumerate test_eof test_errno test_exception_variations test_extcall test_fcntl test_file test_filecmp test_fileinput test_float test_fnmatch test_fork1 test_format test_fpformat test_frozen test_funcattrs test_functional test_future test_gc test_gdbm test_generators test_genexps test_getargs test_getargs2 test_getopt test_gettext test_gl test_gl skipped -- No module named gl test_glob test_global test_grp test_gzip test_hash test_hashlib test_heapq test_hexoct test_hmac test_hotshot test_htmllib test_htmlparser test_httplib test_imageop test_imaplib test_imgfile test_imgfile skipped -- No module named imgfile test_imp test_import test_importhooks test_index test_inspect test_ioctl test_ioctl skipped -- Unable to open /dev/tty test_isinstance test_iter test_iterlen test_itertools test_largefile test_list test_locale test_logging test_long test_long_future test_longexp test_macfs test_macfs skipped -- No module named macfs test_macostools test_macostools skipped -- No module named macostools test_macpath test_mailbox test_marshal test_math test_md5 test_mhlib test_mimetools test_mimetypes test_minidom test_mmap test_module test_multibytecodec test_multibytecodec_support test_multifile test_mutants test_netrc test_new test_nis test_nis skipped -- Local domain name not set test_normalization test_ntpath test_openpty test_operator test_optparse test_os test_parser test_peepholer test_pep247 test_pep263 test_pep277 test_pep277 skipped -- test works only on NT+ test_pep292 test_pep352 test_pickle test_pickletools test_pkg test_pkgimport test_platform test_plistlib test_plistlib skipped -- No module named plistlib test_poll test_popen [9898 refs] [9898 refs] [9898 refs] test_popen2 test_posix test_posixpath test_pow test_pprint test_profile test_profilehooks test_pty test_pwd test_pyclbr test_pyexpat test_queue test_quopri [10984 refs] [10984 refs] test_random test_re test_repr test_resource test_rfc822 test_rgbimg test_richcmp test_robotparser test_runpy test_sax test_scope test_scriptpackages test_scriptpackages skipped -- No module named aetools test_select test_set test_sets test_sgmllib test_sha test_shelve test_shlex test_shutil test_signal test_site test_slice test_socket test_socket_ssl test_socketserver test_softspace test_sort test_sqlite test_str test_strftime test_string test_stringprep test_strop test_strptime test_struct test_structseq test_subprocess [9893 refs] [9895 refs] [9893 refs] [9893 refs] [9893 refs] [9893 refs] [9893 refs] [9894 refs] [9894 refs] [9893 refs] [9894 refs] [9893 refs] [10110 refs] [9894 refs] [9894 refs] [9894 refs] [9894 refs] [9894 refs] [9894 refs] [9894 refs] this bit of output is from a test of stdout in a different process ... [9894 refs] [9893 refs] [10110 refs] test_sunaudiodev test_sunaudiodev skipped -- No module named sunaudiodev test_sundry test_symtable test_syntax test_sys [9893 refs] [9893 refs] test_tarfile test_tcl test_tcl skipped -- No module named _tkinter test_tempfile [9895 refs] test_textwrap test_thread test_threaded_import test_threadedtempfile test_threading test_threading_local test_threadsignals test_time test_timeout test_tokenize test_trace test_traceback test_transformer test_tuple test_ucn test_unary test_unicode test_unicode_file test_unicode_file skipped -- No Unicode filesystem semantics on this platform. test_unicodedata test_unittest test_univnewlines test_unpack test_urllib test_urllib2 test_urllib2net test_urllibnet test_urlparse test_userdict test_userlist test_userstring test_uu test_wait3 test_wait4 test_warnings test_wave test_weakref test_whichdb test_winreg test_winreg skipped -- No module named _winreg test_winsound test_winsound skipped -- No module named winsound test_with test_xdrlib test_xml_etree test_xml_etree_c test_xmllib test_xmlrpc test_xpickle test_xrange test_zipfile test_zipimport test_zlib 289 tests OK. 20 tests skipped: test_aepack test_al test_applesingle test_bsddb185 test_cd test_cl test_gl test_imgfile test_ioctl test_macfs test_macostools test_nis test_pep277 test_plistlib test_scriptpackages test_sunaudiodev test_tcl test_unicode_file test_winreg test_winsound 1 skip unexpected on linux2: test_ioctl [424300 refs] _uu test_wait3 test_wait4 test_warnings test_wave test_weakref test_whichdb test_winreg test_winreg skipped -- No module named _winreg test_winsound test_winsound skipped -- No module named winsound test_with test_xdrlib test_xml_etree test_xml_etree_c test_xmllib test_xmlrpc test_xpickle test_xrange test_zipfile test_zipimport test_zlib 288 tests OK. 1 test failed: test_bsddb3 20 tests skipped: test_aepack test_al test_applesingle test_bsddb185 test_cd test_cl test_gl test_imgfile test_ioctl test_macfs test_macostools test_nis test_pep277 test_plistlib test_scriptpackages test_sunaudiodev test_tcl test_unicode_file test_winreg test_winsound 1 skip unexpected on linux2: test_ioctl [424302 refs] From anthony at interlink.com.au Mon Apr 3 03:47:25 2006 From: anthony at interlink.com.au (Anthony Baxter) Date: Mon, 3 Apr 2006 12:47:25 +1100 Subject: [Python-checkins] TRUNK FREEZE. 2.5a1, 00:00 UTC, Wednesday 5th of April. Message-ID: <200604031147.27455.anthony@interlink.com.au> Now that the bug day has been and gone, it's time to cut 2.5a1. Please consider the trunk FROZEN from 00:00 UTC/GMT on Wednesday the 5th of April. I'll post again when it's unfrozen. Please help in not making the release manager cry because the trunk is broken. Thanks, Anthony From python-checkins at python.org Mon Apr 3 04:20:50 2006 From: python-checkins at python.org (anthony.baxter) Date: Mon, 3 Apr 2006 04:20:50 +0200 (CEST) Subject: [Python-checkins] r43559 - python/trunk/setup.py Message-ID: <20060403022050.06B0E1E4004@bag.python.org> Author: anthony.baxter Date: Mon Apr 3 04:20:49 2006 New Revision: 43559 Modified: python/trunk/setup.py Log: cleaned up setup.py code for sqlite3, based on patch from Gerhard Haering. Modified: python/trunk/setup.py ============================================================================== --- python/trunk/setup.py (original) +++ python/trunk/setup.py Mon Apr 3 04:20:49 2006 @@ -692,50 +692,42 @@ # The sqlite interface sqlite_setup_debug = True # verbose debug prints from this script? - # We hunt for "#define SQLITE_VERSION_NUMBER nnnnn" - # We need to find a version >= 3002002 (> sqlite version 3.2.2) + # We hunt for #define SQLITE_VERSION "n.n.n" + # We need to find >= sqlite version 3.0.8 sqlite_incdir = sqlite_libdir = None - sqlite_inc_paths = [ '/usr/include', + sqlite_inc_paths = [ '/usr/include', '/usr/include/sqlite', '/usr/include/sqlite3', '/usr/local/include', '/usr/local/include/sqlite', '/usr/local/include/sqlite3', ] - MIN_SQLITE_VERSION_NUMBER = 3000008 - MIN_SQLITE_VERSION = "3.0.8" + MIN_SQLITE_VERSION_NUMBER = (3, 0, 8) + MIN_SQLITE_VERSION = ".".join([str(x) + for x in MIN_SQLITE_VERSION_NUMBER]) for d in sqlite_inc_paths + inc_dirs: f = os.path.join(d, "sqlite3.h") if os.path.exists(f): if sqlite_setup_debug: print "sqlite: found %s"%f - f = open(f).read() - m = re.search(r"#define\WSQLITE_VERSION_NUMBER\W(\d+)", f) + incf = open(f).read() + m = re.search( + r'\s*.*#\s*.*define\s.*SQLITE_VERSION\W*"(.*)"', incf) if m: - sqlite_version = int(m.group(1)) - if sqlite_version >= MIN_SQLITE_VERSION_NUMBER: + sqlite_version = m.group(1) + sqlite_version_tuple = tuple([int(x) + for x in sqlite_version.split(".")]) + if sqlite_version_tuple >= MIN_SQLITE_VERSION_NUMBER: # we win! print "%s/sqlite3.h: version %s"%(d, sqlite_version) sqlite_incdir = d break - elif sqlite_version == 3000000: - # Bug in a common version out there where - # SQLITE_VERSION_NUMBER was set incorrectly - if sqlite_setup_debug: - print "found buggy SQLITE_VERSION_NUMBER, checking" - m = re.search(r'#define\WSQLITE_VERSION\W+"([\.\d]+)"', - f) - if m: - sqlite_version = m.group(1) - if sqlite_version >= MIN_SQLITE_VERSION: - print "%s/sqlite3.h: version %s"%(d, - sqlite_version) - sqlite_incdir = d - break else: - if sqlite_setup_debug: + if sqlite_setup_debug: print "%s: version %d is too old, need >= %s"%(d, sqlite_version, MIN_SQLITE_VERSION) - + elif sqlite_setup_debug: + print "sqlite: %s had no SQLITE_VERSION"%(f,) + if sqlite_incdir: sqlite_dirs_to_check = [ os.path.join(sqlite_incdir, '..', 'lib64'), @@ -763,19 +755,19 @@ PYSQLITE_VERSION = "2.1.3" sqlite_defines = [] if sys.platform != "win32": - sqlite_defines.append(('PYSQLITE_VERSION', + sqlite_defines.append(('PYSQLITE_VERSION', '"%s"' % PYSQLITE_VERSION)) else: - sqlite_defines.append(('PYSQLITE_VERSION', + sqlite_defines.append(('PYSQLITE_VERSION', '\\"'+PYSQLITE_VERSION+'\\"')) - sqlite_defines.append(('PY_MAJOR_VERSION', + sqlite_defines.append(('PY_MAJOR_VERSION', str(sys.version_info[0]))) - sqlite_defines.append(('PY_MINOR_VERSION', + sqlite_defines.append(('PY_MINOR_VERSION', str(sys.version_info[1]))) exts.append(Extension('_sqlite3', sqlite_srcs, define_macros=sqlite_defines, - include_dirs=["Modules/_sqlite", + include_dirs=["Modules/_sqlite", sqlite_incdir], library_dirs=sqlite_libdir, runtime_library_dirs=sqlite_libdir, From neal at metaslash.com Mon Apr 3 04:19:16 2006 From: neal at metaslash.com (Neal Norwitz) Date: Sun, 2 Apr 2006 22:19:16 -0400 Subject: [Python-checkins] Python Regression Test Failures all (1) Message-ID: <20060403021916.GA11331@python.psfb.org> test_grammar test_opcodes test_operations test_builtin test_exceptions test_types test_MimeWriter test_StringIO test___all__ test___future__ test__locale test_aepack test_aepack skipped -- No module named aepack test_al test_al skipped -- No module named al test_anydbm test_applesingle test_applesingle skipped -- No module named macostools test_array test_ast test_asynchat test_atexit test_audioop test_augassign test_base64 test_bastion test_binascii test_binhex test_binop test_bisect test_bool test_bsddb test_bsddb185 test_bsddb185 skipped -- No module named bsddb185 test_bsddb3 test_bufio test_bz2 test_cProfile test_calendar test_call test_capi test_cd test_cd skipped -- No module named cd test_cfgparser test_cgi test_charmapcodec test_cl test_cl skipped -- No module named cl test_class test_cmath test_cmd_line test_code test_codeccallbacks test_codecencodings_cn test_codecencodings_hk test_codecencodings_jp test_codecencodings_kr test_codecencodings_tw test_codecmaps_cn test_codecmaps_hk test_codecmaps_jp test_codecmaps_kr test_codecmaps_tw test_codecs test_codeop test_coding test_coercion test_colorsys test_commands test_compare test_compile test_compiler test_complex test_contains test_contextlib test_cookie test_cookielib test_copy test_copy_reg test_cpickle test_crypt test_csv test_ctypes test_datetime test_dbm test_decimal test_decorators test_defaultdict test_deque test_descr test_descrtut test_dict test_difflib test_dircache test_dis test_distutils test_dl test_doctest test_doctest2 test_dumbdbm test_dummy_thread test_dummy_threading test_email test_email_codecs test_email_renamed test_enumerate test_eof test_errno test_exception_variations test_extcall test_fcntl test_file test_filecmp test_fileinput test_float test_fnmatch test_fork1 test_format test_fpformat test_frozen test_funcattrs test_functional test_future test_gc test_gdbm test_generators test_genexps test_getargs test_getargs2 test_getopt test_gettext test_gl test_gl skipped -- No module named gl test_glob test_global test_grp test_gzip test_hash test_hashlib test_heapq test_hexoct test_hmac test_hotshot test_htmllib test_htmlparser test_httplib test_imageop test_imaplib test_imgfile test_imgfile skipped -- No module named imgfile test_imp test_import test_importhooks test_index test_inspect test_ioctl test_ioctl skipped -- Unable to open /dev/tty test_isinstance test_iter test_iterlen test_itertools test_largefile test_list test_locale test_logging test_long test_long_future test_longexp test_macfs test_macfs skipped -- No module named macfs test_macostools test_macostools skipped -- No module named macostools test_macpath test_mailbox test_marshal test_math test_md5 test_mhlib test_mimetools test_mimetypes test_minidom test_mmap test_module test_multibytecodec test_multibytecodec_support test_multifile test_mutants test_netrc test_new test_nis test_nis skipped -- Local domain name not set test_normalization test_ntpath test_openpty test_operator test_optparse test_os test_parser test_peepholer test_pep247 test_pep263 test_pep277 test_pep277 skipped -- test works only on NT+ test_pep292 test_pep352 test_pickle test_pickletools test_pkg test_pkgimport test_platform test_plistlib test_plistlib skipped -- No module named plistlib test_poll test_popen [9898 refs] [9898 refs] [9898 refs] test_popen2 test_posix test_posixpath test_pow test_pprint test_profile test_profilehooks test_pty test_pwd test_pyclbr test_pyexpat test_queue test_quopri [10984 refs] [10984 refs] test_random test_re test_repr test_resource test_rfc822 test_rgbimg test_richcmp test_robotparser test_runpy test_sax test_scope test_scriptpackages test_scriptpackages skipped -- No module named aetools test_select test_set test_sets test_sgmllib test_sha test_shelve test_shlex test_shutil test_signal test_site test_slice test_socket test_socket_ssl test_socketserver test_softspace test_sort test_sqlite test_str test_strftime test_string test_stringprep test_strop test_strptime test_struct test_structseq test_subprocess [9893 refs] [9895 refs] [9893 refs] [9893 refs] [9893 refs] [9893 refs] [9893 refs] [9894 refs] [9894 refs] [9893 refs] [9894 refs] [9893 refs] [10110 refs] [9894 refs] [9894 refs] [9894 refs] [9894 refs] [9894 refs] [9894 refs] [9894 refs] this bit of output is from a test of stdout in a different process ... [9894 refs] [9893 refs] [10110 refs] test_sunaudiodev test_sunaudiodev skipped -- No module named sunaudiodev test_sundry test_symtable test_syntax test_sys [9893 refs] [9893 refs] test_tarfile test_tcl test_tcl skipped -- No module named _tkinter test_tempfile [9895 refs] test_textwrap test_thread test_threaded_import test_threadedtempfile test_threading test_threading_local test_threadsignals test_time test_timeout test_tokenize test_trace test_traceback test_transformer test_tuple test_ucn test_unary test_unicode test_unicode_file test_unicode_file skipped -- No Unicode filesystem semantics on this platform. test_unicodedata test_unittest test_univnewlines test_unpack test_urllib test_urllib2 test_urllib2net test_urllibnet test_urlparse test_userdict test_userlist test_userstring test_uu test_wait3 test_wait4 test_warnings test_wave test_weakref test_whichdb test_winreg test_winreg skipped -- No module named _winreg test_winsound test_winsound skipped -- No module named winsound test_with test_xdrlib test_xml_etree test_xml_etree_c test_xmllib test_xmlrpc test_xpickle test_xrange test_zipfile test_zipimport test_zlib 289 tests OK. 20 tests skipped: test_aepack test_al test_applesingle test_bsddb185 test_cd test_cl test_gl test_imgfile test_ioctl test_macfs test_macostools test_nis test_pep277 test_plistlib test_scriptpackages test_sunaudiodev test_tcl test_unicode_file test_winreg test_winsound 1 skip unexpected on linux2: test_ioctl [424300 refs] _uu test_wait3 test_wait4 test_warnings test_wave test_weakref test_whichdb test_winreg test_winreg skipped -- No module named _winreg test_winsound test_winsound skipped -- No module named winsound test_with test_xdrlib test_xml_etree test_xml_etree_c test_xmllib test_xmlrpc test_xpickle test_xrange test_zipfile test_zipimport test_zlib 288 tests OK. 1 test failed: test_bsddb3 20 tests skipped: test_aepack test_al test_applesingle test_bsddb185 test_cd test_cl test_gl test_imgfile test_ioctl test_macfs test_macostools test_nis test_pep277 test_plistlib test_scriptpackages test_sunaudiodev test_tcl test_unicode_file test_winreg test_winsound 1 skip unexpected on linux2: test_ioctl [424302 refs] From python-checkins at python.org Mon Apr 3 04:46:45 2006 From: python-checkins at python.org (tim.peters) Date: Mon, 3 Apr 2006 04:46:45 +0200 (CEST) Subject: [Python-checkins] r43560 - in python/trunk/Lib: sgmllib.py test/test_traceback.py test/test_urllib2.py Message-ID: <20060403024645.036D01E4004@bag.python.org> Author: tim.peters Date: Mon Apr 3 04:46:44 2006 New Revision: 43560 Modified: python/trunk/Lib/sgmllib.py python/trunk/Lib/test/test_traceback.py python/trunk/Lib/test/test_urllib2.py Log: Whitespace normalization. Modified: python/trunk/Lib/sgmllib.py ============================================================================== --- python/trunk/Lib/sgmllib.py (original) +++ python/trunk/Lib/sgmllib.py Mon Apr 3 04:46:44 2006 @@ -270,7 +270,7 @@ if not rest: attrvalue = attrname else: - if (attrvalue[:1] == "'" == attrvalue[-1:] or + if (attrvalue[:1] == "'" == attrvalue[-1:] or attrvalue[:1] == '"' == attrvalue[-1:]): # strip quotes attrvalue = attrvalue[1:-1] Modified: python/trunk/Lib/test/test_traceback.py ============================================================================== --- python/trunk/Lib/test/test_traceback.py (original) +++ python/trunk/Lib/test/test_traceback.py Mon Apr 3 04:46:44 2006 @@ -23,7 +23,7 @@ def syntax_error_without_caret(self): # XXX why doesn't compile raise the same traceback? import test.badsyntax_nocaret - + def syntax_error_bad_indentation(self): compile("def spam():\n print 1\n print 2", "?", "exec") Modified: python/trunk/Lib/test/test_urllib2.py ============================================================================== --- python/trunk/Lib/test/test_urllib2.py (original) +++ python/trunk/Lib/test/test_urllib2.py Mon Apr 3 04:46:44 2006 @@ -355,12 +355,12 @@ "file://%s%s" % (socket.gethostbyname('localhost'), urlpath), ] try: - localaddr = socket.gethostbyname(socket.gethostname()) + localaddr = socket.gethostbyname(socket.gethostname()) except socket.gaierror: localaddr = '' if localaddr: urls.append("file://%s%s" % (localaddr, urlpath)) - + for url in urls: f = open(TESTFN, "wb") try: From buildbot at python.org Mon Apr 3 04:56:40 2006 From: buildbot at python.org (buildbot at python.org) Date: Mon, 03 Apr 2006 02:56:40 +0000 Subject: [Python-checkins] buildbot warnings in alpha Tru64 5.1 trunk Message-ID: <20060403025640.662681E4004@bag.python.org> The Buildbot has detected a new failure of alpha Tru64 5.1 trunk. Full details are available at: http://www.python.org/dev/buildbot/all/alpha%20Tru64%205.1%20trunk/builds/74 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: anthony.baxter Build Had Warnings: warnings test sincerely, -The Buildbot From python-checkins at python.org Mon Apr 3 06:45:36 2006 From: python-checkins at python.org (neal.norwitz) Date: Mon, 3 Apr 2006 06:45:36 +0200 (CEST) Subject: [Python-checkins] r43561 - in python/trunk/Lib: compiler/__init__.py compiler/ast.py compiler/transformer.py test/test_dict.py test/test_import.py test/test_sets.py test/test_urllib2.py Message-ID: <20060403044536.711181E4004@bag.python.org> Author: neal.norwitz Date: Mon Apr 3 06:45:34 2006 New Revision: 43561 Modified: python/trunk/Lib/compiler/__init__.py python/trunk/Lib/compiler/ast.py python/trunk/Lib/compiler/transformer.py python/trunk/Lib/test/test_dict.py python/trunk/Lib/test/test_import.py python/trunk/Lib/test/test_sets.py python/trunk/Lib/test/test_urllib2.py Log: Use absolute imports Modified: python/trunk/Lib/compiler/__init__.py ============================================================================== --- python/trunk/Lib/compiler/__init__.py (original) +++ python/trunk/Lib/compiler/__init__.py Mon Apr 3 06:45:34 2006 @@ -21,6 +21,6 @@ Generates a .pyc file by compiling filename. """ -from transformer import parse, parseFile -from visitor import walk -from pycodegen import compile, compileFile +from compiler.transformer import parse, parseFile +from compiler.visitor import walk +from compiler.pycodegen import compile, compileFile Modified: python/trunk/Lib/compiler/ast.py ============================================================================== --- python/trunk/Lib/compiler/ast.py (original) +++ python/trunk/Lib/compiler/ast.py Mon Apr 3 06:45:34 2006 @@ -2,7 +2,7 @@ This file is automatically generated by Tools/compiler/astgen.py """ -from consts import CO_VARARGS, CO_VARKEYWORDS +from compiler.consts import CO_VARARGS, CO_VARKEYWORDS def flatten(seq): l = [] Modified: python/trunk/Lib/compiler/transformer.py ============================================================================== --- python/trunk/Lib/compiler/transformer.py (original) +++ python/trunk/Lib/compiler/transformer.py Mon Apr 3 06:45:34 2006 @@ -34,8 +34,8 @@ class WalkerError(StandardError): pass -from consts import CO_VARARGS, CO_VARKEYWORDS -from consts import OP_ASSIGN, OP_DELETE, OP_APPLY +from compiler.consts import CO_VARARGS, CO_VARKEYWORDS +from compiler.consts import OP_ASSIGN, OP_DELETE, OP_APPLY def parseFile(path): f = open(path, "U") Modified: python/trunk/Lib/test/test_dict.py ============================================================================== --- python/trunk/Lib/test/test_dict.py (original) +++ python/trunk/Lib/test/test_dict.py Mon Apr 3 06:45:34 2006 @@ -445,7 +445,7 @@ self.fail_("g[42] didn't raise KeyError") -import mapping_tests +from test import mapping_tests class GeneralMappingTests(mapping_tests.BasicTestMappingProtocol): type2test = dict Modified: python/trunk/Lib/test/test_import.py ============================================================================== --- python/trunk/Lib/test/test_import.py (original) +++ python/trunk/Lib/test/test_import.py Mon Apr 3 06:45:34 2006 @@ -15,7 +15,7 @@ raise TestFailed("import of RAnDoM should have failed (case mismatch)") # Another brief digression to test the accuracy of manifest float constants. -import double_const # don't blink -- that *was* the test +from test import double_const # don't blink -- that *was* the test def remove_files(name): for f in (name + os.extsep + "py", Modified: python/trunk/Lib/test/test_sets.py ============================================================================== --- python/trunk/Lib/test/test_sets.py (original) +++ python/trunk/Lib/test/test_sets.py Mon Apr 3 06:45:34 2006 @@ -819,7 +819,8 @@ __test__ = {'libreftest' : libreftest} def test_main(verbose=None): - import test_sets, doctest + import doctest + from test import test_sets test_support.run_unittest( TestSetOfSets, TestExceptionPropagation, Modified: python/trunk/Lib/test/test_urllib2.py ============================================================================== --- python/trunk/Lib/test/test_urllib2.py (original) +++ python/trunk/Lib/test/test_urllib2.py Mon Apr 3 06:45:34 2006 @@ -632,7 +632,7 @@ from urllib2 import build_opener, HTTPHandler, HTTPError, \ HTTPCookieProcessor - from test_cookielib import interact_netscape + from test.test_cookielib import interact_netscape cj = CookieJar() interact_netscape(cj, "http://www.example.com/", "spam=eggs") From python-checkins at python.org Mon Apr 3 06:46:05 2006 From: python-checkins at python.org (neal.norwitz) Date: Mon, 3 Apr 2006 06:46:05 +0200 (CEST) Subject: [Python-checkins] r43562 - python/trunk/Objects/sliceobject.c Message-ID: <20060403044605.1CFCA1E4004@bag.python.org> Author: neal.norwitz Date: Mon Apr 3 06:46:04 2006 New Revision: 43562 Modified: python/trunk/Objects/sliceobject.c Log: Use Py_ssize_t in slices Modified: python/trunk/Objects/sliceobject.c ============================================================================== --- python/trunk/Objects/sliceobject.c (original) +++ python/trunk/Objects/sliceobject.c Mon Apr 3 06:46:04 2006 @@ -107,20 +107,20 @@ *step = 1; } else { if (!PyInt_Check(r->step)) return -1; - *step = PyInt_AsLong(r->step); + *step = PyInt_AsSsize_t(r->step); } if (r->start == Py_None) { *start = *step < 0 ? length-1 : 0; } else { if (!PyInt_Check(r->start)) return -1; - *start = PyInt_AsLong(r->start); + *start = PyInt_AsSsize_t(r->start); if (*start < 0) *start += length; } if (r->stop == Py_None) { *stop = *step < 0 ? -1 : length; } else { if (!PyInt_Check(r->stop)) return -1; - *stop = PyInt_AsLong(r->stop); + *stop = PyInt_AsSsize_t(r->stop); if (*stop < 0) *stop += length; } if (*stop > length) return -1; @@ -252,7 +252,7 @@ { Py_ssize_t ilen, start, stop, step, slicelength; - ilen = PyInt_AsLong(len); + ilen = PyInt_AsSsize_t(len); if (ilen == -1 && PyErr_Occurred()) { return NULL; @@ -263,7 +263,7 @@ return NULL; } - return Py_BuildValue("(iii)", start, stop, step); + return Py_BuildValue("(nnn)", start, stop, step); } PyDoc_STRVAR(slice_indices_doc, From python-checkins at python.org Mon Apr 3 06:46:29 2006 From: python-checkins at python.org (neal.norwitz) Date: Mon, 3 Apr 2006 06:46:29 +0200 (CEST) Subject: [Python-checkins] r43563 - python/trunk/Modules/arraymodule.c Message-ID: <20060403044629.18B831E4004@bag.python.org> Author: neal.norwitz Date: Mon Apr 3 06:46:28 2006 New Revision: 43563 Modified: python/trunk/Modules/arraymodule.c Log: Whitespace: break long line Modified: python/trunk/Modules/arraymodule.c ============================================================================== --- python/trunk/Modules/arraymodule.c (original) +++ python/trunk/Modules/arraymodule.c Mon Apr 3 06:46:28 2006 @@ -186,7 +186,8 @@ if (!PyArg_Parse(v, "u#;array item must be unicode character", &p, &len)) return -1; if (len != 1) { - PyErr_SetString(PyExc_TypeError, "array item must be unicode character"); + PyErr_SetString(PyExc_TypeError, + "array item must be unicode character"); return -1; } if (i >= 0) From python-checkins at python.org Mon Apr 3 06:48:38 2006 From: python-checkins at python.org (neal.norwitz) Date: Mon, 3 Apr 2006 06:48:38 +0200 (CEST) Subject: [Python-checkins] r43564 - in python/trunk: Doc/lib/libfuncs.tex Misc/NEWS Python/bltinmodule.c Message-ID: <20060403044838.514C71E4004@bag.python.org> Author: neal.norwitz Date: Mon Apr 3 06:48:37 2006 New Revision: 43564 Modified: python/trunk/Doc/lib/libfuncs.tex python/trunk/Misc/NEWS python/trunk/Python/bltinmodule.c Log: Accept keyword arguments for __import__ and doc the addition of the level param from PEP 328. Modified: python/trunk/Doc/lib/libfuncs.tex ============================================================================== --- python/trunk/Doc/lib/libfuncs.tex (original) +++ python/trunk/Doc/lib/libfuncs.tex Mon Apr 3 06:48:37 2006 @@ -6,7 +6,7 @@ \setindexsubitem{(built-in function)} -\begin{funcdesc}{__import__}{name\optional{, globals\optional{, locals\optional{, fromlist}}}} +\begin{funcdesc}{__import__}{name\optional{, globals\optional{, locals\optional{, fromlist\optional{, level}}}}} This function is invoked by the \keyword{import}\stindex{import} statement. It mainly exists so that you can replace it with another function that has a compatible interface, in order to change the @@ -20,9 +20,9 @@ For example, the statement \samp{import spam} results in the following call: \code{__import__('spam',} \code{globals(),} - \code{locals(), [])}; the statement \samp{from spam.ham import eggs} + \code{locals(), [], -1)}; the statement \samp{from spam.ham import eggs} results in \samp{__import__('spam.ham', globals(), locals(), - ['eggs'])}. Note that even though \code{locals()} and + ['eggs'], -1)}. Note that even though \code{locals()} and \code{['eggs']} are passed in as arguments, the \function{__import__()} function does not set the local variable named \code{eggs}; this is done by subsequent code that is generated @@ -52,6 +52,15 @@ mod = getattr(mod, comp) return mod \end{verbatim} + + \var{level} specifies whether to use absolute or relative imports. + The default is \code{-1} which indicates both absolute and relative + imports will be attempted. \code{0} means only perform absolute imports. + Positive values for \var{level} indicate the number of parent directories + to search relative to the directory of the module calling + \function{__import__}. +\versionchanged[The level parameter was added]{2.5} +\versionchanged[Keyword support for parameters was added]{2.5} \end{funcdesc} \begin{funcdesc}{abs}{x} Modified: python/trunk/Misc/NEWS ============================================================================== --- python/trunk/Misc/NEWS (original) +++ python/trunk/Misc/NEWS Mon Apr 3 06:48:37 2006 @@ -12,6 +12,8 @@ Core and builtins ----------------- +- __import__ accepts keyword arguments. + - Patch #1460496: round() now accepts keyword arguments. - Fixed bug #1459029 - unicode reprs were double-escaped. Modified: python/trunk/Python/bltinmodule.c ============================================================================== --- python/trunk/Python/bltinmodule.c (original) +++ python/trunk/Python/bltinmodule.c Mon Apr 3 06:48:37 2006 @@ -31,23 +31,25 @@ static PyObject *filtertuple (PyObject *, PyObject *); static PyObject * -builtin___import__(PyObject *self, PyObject *args) +builtin___import__(PyObject *self, PyObject *args, PyObject *kwds) { + static char *kwlist[] = {"name", "globals", "locals", "fromlist", + "level", 0}; char *name; PyObject *globals = NULL; PyObject *locals = NULL; PyObject *fromlist = NULL; int level = -1; - if (!PyArg_ParseTuple(args, "s|OOOi:__import__", - &name, &globals, &locals, &fromlist, &level)) + if (!PyArg_ParseTupleAndKeywords(args, kwds, "s|OOOi:__import__", + kwlist, &name, &globals, &locals, &fromlist, &level)) return NULL; return PyImport_ImportModuleLevel(name, globals, locals, fromlist, level); } PyDoc_STRVAR(import_doc, -"__import__(name, globals, locals, fromlist) -> module\n\ +"__import__(name, globals={}, locals={}, fromlist=[], level=-1) -> module\n\ \n\ Import a module. The globals are only used to determine the context;\n\ they are not modified. The locals are currently unused. The fromlist\n\ @@ -55,7 +57,10 @@ empty list to emulate ``import name''.\n\ When importing a module from a package, note that __import__('A.B', ...)\n\ returns package A when fromlist is empty, but its submodule B when\n\ -fromlist is not empty."); +fromlist is not empty. Level is used to determine whether to perform \n\ +absolute or relative imports. -1 is the original strategy of attempting\n\ +both absolute and relative imports, 0 is absolute, a positive number\n\ +is the number of parent directories to search relative to the current module."); static PyObject * @@ -2210,7 +2215,7 @@ static PyMethodDef builtin_methods[] = { - {"__import__", builtin___import__, METH_VARARGS, import_doc}, + {"__import__", (PyCFunction)builtin___import__, METH_VARARGS | METH_KEYWORDS, import_doc}, {"abs", builtin_abs, METH_O, abs_doc}, {"all", builtin_all, METH_O, all_doc}, {"any", builtin_any, METH_O, any_doc}, From python-checkins at python.org Mon Apr 3 06:50:59 2006 From: python-checkins at python.org (neal.norwitz) Date: Mon, 3 Apr 2006 06:50:59 +0200 (CEST) Subject: [Python-checkins] r43565 - in python/trunk: Parser/asdl_c.py Python/Python-ast.c Message-ID: <20060403045059.9A4811E4004@bag.python.org> Author: neal.norwitz Date: Mon Apr 3 06:50:58 2006 New Revision: 43565 Modified: python/trunk/Parser/asdl_c.py python/trunk/Python/Python-ast.c Log: * Fix a refleak of *_attributes. * Cleanup formatting a bit (add spaces). * Move static var initialized inside init_types() since that's the only place it's used. Modified: python/trunk/Parser/asdl_c.py ============================================================================== --- python/trunk/Parser/asdl_c.py (original) +++ python/trunk/Parser/asdl_c.py Mon Apr 3 06:50:58 2006 @@ -413,10 +413,10 @@ static int add_attributes(PyTypeObject* type, char**attrs, int num_fields) { - int i; + int i, result; PyObject *s, *l = PyList_New(num_fields); if (!l) return 0; - for(i=0; i < num_fields; i++) { + for(i = 0; i < num_fields; i++) { s = PyString_FromString(attrs[i]); if (!s) { Py_DECREF(l); @@ -424,7 +424,9 @@ } PyList_SET_ITEM(l, i, s); } - return PyObject_SetAttrString((PyObject*)type, "_attributes", l) >=0; + result = PyObject_SetAttrString((PyObject*)type, "_attributes", l) >= 0; + Py_DECREF(l); + return result; } static PyObject* ast2obj_list(asdl_seq *seq, PyObject* (*func)(void*)) @@ -465,9 +467,9 @@ } """, 0, reflow=False) - self.emit("static int initialized;", 0) self.emit("static int init_types(void)",0) self.emit("{", 0) + self.emit("static int initialized;", 1) self.emit("if (initialized) return 1;", 1) self.emit('AST_type = make_type("AST", &PyBaseObject_Type, NULL, 0);', 1) for dfn in mod.dfns: @@ -543,7 +545,7 @@ self.addObj(cons.name) def addObj(self, name): - self.emit('if(PyDict_SetItemString(d, "%s", (PyObject*)%s_type) < 0) return;' % (name, name), 1) + self.emit('if (PyDict_SetItemString(d, "%s", (PyObject*)%s_type) < 0) return;' % (name, name), 1) _SPECIALIZED_SEQUENCES = ('stmt', 'expr') Modified: python/trunk/Python/Python-ast.c ============================================================================== --- python/trunk/Python/Python-ast.c (original) +++ python/trunk/Python/Python-ast.c Mon Apr 3 06:50:58 2006 @@ -381,10 +381,10 @@ static int add_attributes(PyTypeObject* type, char**attrs, int num_fields) { - int i; + int i, result; PyObject *s, *l = PyList_New(num_fields); if (!l) return 0; - for(i=0; i < num_fields; i++) { + for(i = 0; i < num_fields; i++) { s = PyString_FromString(attrs[i]); if (!s) { Py_DECREF(l); @@ -392,7 +392,9 @@ } PyList_SET_ITEM(l, i, s); } - return PyObject_SetAttrString((PyObject*)type, "_attributes", l) >=0; + result = PyObject_SetAttrString((PyObject*)type, "_attributes", l) >= 0; + Py_DECREF(l); + return result; } static PyObject* ast2obj_list(asdl_seq *seq, PyObject* (*func)(void*)) @@ -432,9 +434,9 @@ return PyInt_FromLong(b); } -static int initialized; static int init_types(void) { + static int initialized; if (initialized) return 1; AST_type = make_type("AST", &PyBaseObject_Type, NULL, 0); mod_type = make_type("mod", AST_type, NULL, 0); @@ -3033,146 +3035,146 @@ return; if (PyModule_AddStringConstant(m, "__version__", "42753") < 0) return; - if(PyDict_SetItemString(d, "mod", (PyObject*)mod_type) < 0) return; - if(PyDict_SetItemString(d, "Module", (PyObject*)Module_type) < 0) - return; - if(PyDict_SetItemString(d, "Interactive", (PyObject*)Interactive_type) - < 0) return; - if(PyDict_SetItemString(d, "Expression", (PyObject*)Expression_type) < - 0) return; - if(PyDict_SetItemString(d, "Suite", (PyObject*)Suite_type) < 0) return; - if(PyDict_SetItemString(d, "stmt", (PyObject*)stmt_type) < 0) return; - if(PyDict_SetItemString(d, "FunctionDef", (PyObject*)FunctionDef_type) - < 0) return; - if(PyDict_SetItemString(d, "ClassDef", (PyObject*)ClassDef_type) < 0) - return; - if(PyDict_SetItemString(d, "Return", (PyObject*)Return_type) < 0) - return; - if(PyDict_SetItemString(d, "Delete", (PyObject*)Delete_type) < 0) - return; - if(PyDict_SetItemString(d, "Assign", (PyObject*)Assign_type) < 0) - return; - if(PyDict_SetItemString(d, "AugAssign", (PyObject*)AugAssign_type) < 0) - return; - if(PyDict_SetItemString(d, "Print", (PyObject*)Print_type) < 0) return; - if(PyDict_SetItemString(d, "For", (PyObject*)For_type) < 0) return; - if(PyDict_SetItemString(d, "While", (PyObject*)While_type) < 0) return; - if(PyDict_SetItemString(d, "If", (PyObject*)If_type) < 0) return; - if(PyDict_SetItemString(d, "With", (PyObject*)With_type) < 0) return; - if(PyDict_SetItemString(d, "Raise", (PyObject*)Raise_type) < 0) return; - if(PyDict_SetItemString(d, "TryExcept", (PyObject*)TryExcept_type) < 0) - return; - if(PyDict_SetItemString(d, "TryFinally", (PyObject*)TryFinally_type) < - 0) return; - if(PyDict_SetItemString(d, "Assert", (PyObject*)Assert_type) < 0) - return; - if(PyDict_SetItemString(d, "Import", (PyObject*)Import_type) < 0) - return; - if(PyDict_SetItemString(d, "ImportFrom", (PyObject*)ImportFrom_type) < - 0) return; - if(PyDict_SetItemString(d, "Exec", (PyObject*)Exec_type) < 0) return; - if(PyDict_SetItemString(d, "Global", (PyObject*)Global_type) < 0) - return; - if(PyDict_SetItemString(d, "Expr", (PyObject*)Expr_type) < 0) return; - if(PyDict_SetItemString(d, "Pass", (PyObject*)Pass_type) < 0) return; - if(PyDict_SetItemString(d, "Break", (PyObject*)Break_type) < 0) return; - if(PyDict_SetItemString(d, "Continue", (PyObject*)Continue_type) < 0) - return; - if(PyDict_SetItemString(d, "expr", (PyObject*)expr_type) < 0) return; - if(PyDict_SetItemString(d, "BoolOp", (PyObject*)BoolOp_type) < 0) - return; - if(PyDict_SetItemString(d, "BinOp", (PyObject*)BinOp_type) < 0) return; - if(PyDict_SetItemString(d, "UnaryOp", (PyObject*)UnaryOp_type) < 0) - return; - if(PyDict_SetItemString(d, "Lambda", (PyObject*)Lambda_type) < 0) - return; - if(PyDict_SetItemString(d, "IfExp", (PyObject*)IfExp_type) < 0) return; - if(PyDict_SetItemString(d, "Dict", (PyObject*)Dict_type) < 0) return; - if(PyDict_SetItemString(d, "ListComp", (PyObject*)ListComp_type) < 0) - return; - if(PyDict_SetItemString(d, "GeneratorExp", - (PyObject*)GeneratorExp_type) < 0) return; - if(PyDict_SetItemString(d, "Yield", (PyObject*)Yield_type) < 0) return; - if(PyDict_SetItemString(d, "Compare", (PyObject*)Compare_type) < 0) - return; - if(PyDict_SetItemString(d, "Call", (PyObject*)Call_type) < 0) return; - if(PyDict_SetItemString(d, "Repr", (PyObject*)Repr_type) < 0) return; - if(PyDict_SetItemString(d, "Num", (PyObject*)Num_type) < 0) return; - if(PyDict_SetItemString(d, "Str", (PyObject*)Str_type) < 0) return; - if(PyDict_SetItemString(d, "Attribute", (PyObject*)Attribute_type) < 0) - return; - if(PyDict_SetItemString(d, "Subscript", (PyObject*)Subscript_type) < 0) - return; - if(PyDict_SetItemString(d, "Name", (PyObject*)Name_type) < 0) return; - if(PyDict_SetItemString(d, "List", (PyObject*)List_type) < 0) return; - if(PyDict_SetItemString(d, "Tuple", (PyObject*)Tuple_type) < 0) return; - if(PyDict_SetItemString(d, "expr_context", - (PyObject*)expr_context_type) < 0) return; - if(PyDict_SetItemString(d, "Load", (PyObject*)Load_type) < 0) return; - if(PyDict_SetItemString(d, "Store", (PyObject*)Store_type) < 0) return; - if(PyDict_SetItemString(d, "Del", (PyObject*)Del_type) < 0) return; - if(PyDict_SetItemString(d, "AugLoad", (PyObject*)AugLoad_type) < 0) - return; - if(PyDict_SetItemString(d, "AugStore", (PyObject*)AugStore_type) < 0) - return; - if(PyDict_SetItemString(d, "Param", (PyObject*)Param_type) < 0) return; - if(PyDict_SetItemString(d, "slice", (PyObject*)slice_type) < 0) return; - if(PyDict_SetItemString(d, "Ellipsis", (PyObject*)Ellipsis_type) < 0) - return; - if(PyDict_SetItemString(d, "Slice", (PyObject*)Slice_type) < 0) return; - if(PyDict_SetItemString(d, "ExtSlice", (PyObject*)ExtSlice_type) < 0) - return; - if(PyDict_SetItemString(d, "Index", (PyObject*)Index_type) < 0) return; - if(PyDict_SetItemString(d, "boolop", (PyObject*)boolop_type) < 0) - return; - if(PyDict_SetItemString(d, "And", (PyObject*)And_type) < 0) return; - if(PyDict_SetItemString(d, "Or", (PyObject*)Or_type) < 0) return; - if(PyDict_SetItemString(d, "operator", (PyObject*)operator_type) < 0) - return; - if(PyDict_SetItemString(d, "Add", (PyObject*)Add_type) < 0) return; - if(PyDict_SetItemString(d, "Sub", (PyObject*)Sub_type) < 0) return; - if(PyDict_SetItemString(d, "Mult", (PyObject*)Mult_type) < 0) return; - if(PyDict_SetItemString(d, "Div", (PyObject*)Div_type) < 0) return; - if(PyDict_SetItemString(d, "Mod", (PyObject*)Mod_type) < 0) return; - if(PyDict_SetItemString(d, "Pow", (PyObject*)Pow_type) < 0) return; - if(PyDict_SetItemString(d, "LShift", (PyObject*)LShift_type) < 0) - return; - if(PyDict_SetItemString(d, "RShift", (PyObject*)RShift_type) < 0) - return; - if(PyDict_SetItemString(d, "BitOr", (PyObject*)BitOr_type) < 0) return; - if(PyDict_SetItemString(d, "BitXor", (PyObject*)BitXor_type) < 0) - return; - if(PyDict_SetItemString(d, "BitAnd", (PyObject*)BitAnd_type) < 0) - return; - if(PyDict_SetItemString(d, "FloorDiv", (PyObject*)FloorDiv_type) < 0) - return; - if(PyDict_SetItemString(d, "unaryop", (PyObject*)unaryop_type) < 0) - return; - if(PyDict_SetItemString(d, "Invert", (PyObject*)Invert_type) < 0) - return; - if(PyDict_SetItemString(d, "Not", (PyObject*)Not_type) < 0) return; - if(PyDict_SetItemString(d, "UAdd", (PyObject*)UAdd_type) < 0) return; - if(PyDict_SetItemString(d, "USub", (PyObject*)USub_type) < 0) return; - if(PyDict_SetItemString(d, "cmpop", (PyObject*)cmpop_type) < 0) return; - if(PyDict_SetItemString(d, "Eq", (PyObject*)Eq_type) < 0) return; - if(PyDict_SetItemString(d, "NotEq", (PyObject*)NotEq_type) < 0) return; - if(PyDict_SetItemString(d, "Lt", (PyObject*)Lt_type) < 0) return; - if(PyDict_SetItemString(d, "LtE", (PyObject*)LtE_type) < 0) return; - if(PyDict_SetItemString(d, "Gt", (PyObject*)Gt_type) < 0) return; - if(PyDict_SetItemString(d, "GtE", (PyObject*)GtE_type) < 0) return; - if(PyDict_SetItemString(d, "Is", (PyObject*)Is_type) < 0) return; - if(PyDict_SetItemString(d, "IsNot", (PyObject*)IsNot_type) < 0) return; - if(PyDict_SetItemString(d, "In", (PyObject*)In_type) < 0) return; - if(PyDict_SetItemString(d, "NotIn", (PyObject*)NotIn_type) < 0) return; - if(PyDict_SetItemString(d, "comprehension", - (PyObject*)comprehension_type) < 0) return; - if(PyDict_SetItemString(d, "excepthandler", - (PyObject*)excepthandler_type) < 0) return; - if(PyDict_SetItemString(d, "arguments", (PyObject*)arguments_type) < 0) - return; - if(PyDict_SetItemString(d, "keyword", (PyObject*)keyword_type) < 0) - return; - if(PyDict_SetItemString(d, "alias", (PyObject*)alias_type) < 0) return; + if (PyDict_SetItemString(d, "mod", (PyObject*)mod_type) < 0) return; + if (PyDict_SetItemString(d, "Module", (PyObject*)Module_type) < 0) + return; + if (PyDict_SetItemString(d, "Interactive", (PyObject*)Interactive_type) + < 0) return; + if (PyDict_SetItemString(d, "Expression", (PyObject*)Expression_type) < + 0) return; + if (PyDict_SetItemString(d, "Suite", (PyObject*)Suite_type) < 0) return; + if (PyDict_SetItemString(d, "stmt", (PyObject*)stmt_type) < 0) return; + if (PyDict_SetItemString(d, "FunctionDef", (PyObject*)FunctionDef_type) + < 0) return; + if (PyDict_SetItemString(d, "ClassDef", (PyObject*)ClassDef_type) < 0) + return; + if (PyDict_SetItemString(d, "Return", (PyObject*)Return_type) < 0) + return; + if (PyDict_SetItemString(d, "Delete", (PyObject*)Delete_type) < 0) + return; + if (PyDict_SetItemString(d, "Assign", (PyObject*)Assign_type) < 0) + return; + if (PyDict_SetItemString(d, "AugAssign", (PyObject*)AugAssign_type) < + 0) return; + if (PyDict_SetItemString(d, "Print", (PyObject*)Print_type) < 0) return; + if (PyDict_SetItemString(d, "For", (PyObject*)For_type) < 0) return; + if (PyDict_SetItemString(d, "While", (PyObject*)While_type) < 0) return; + if (PyDict_SetItemString(d, "If", (PyObject*)If_type) < 0) return; + if (PyDict_SetItemString(d, "With", (PyObject*)With_type) < 0) return; + if (PyDict_SetItemString(d, "Raise", (PyObject*)Raise_type) < 0) return; + if (PyDict_SetItemString(d, "TryExcept", (PyObject*)TryExcept_type) < + 0) return; + if (PyDict_SetItemString(d, "TryFinally", (PyObject*)TryFinally_type) < + 0) return; + if (PyDict_SetItemString(d, "Assert", (PyObject*)Assert_type) < 0) + return; + if (PyDict_SetItemString(d, "Import", (PyObject*)Import_type) < 0) + return; + if (PyDict_SetItemString(d, "ImportFrom", (PyObject*)ImportFrom_type) < + 0) return; + if (PyDict_SetItemString(d, "Exec", (PyObject*)Exec_type) < 0) return; + if (PyDict_SetItemString(d, "Global", (PyObject*)Global_type) < 0) + return; + if (PyDict_SetItemString(d, "Expr", (PyObject*)Expr_type) < 0) return; + if (PyDict_SetItemString(d, "Pass", (PyObject*)Pass_type) < 0) return; + if (PyDict_SetItemString(d, "Break", (PyObject*)Break_type) < 0) return; + if (PyDict_SetItemString(d, "Continue", (PyObject*)Continue_type) < 0) + return; + if (PyDict_SetItemString(d, "expr", (PyObject*)expr_type) < 0) return; + if (PyDict_SetItemString(d, "BoolOp", (PyObject*)BoolOp_type) < 0) + return; + if (PyDict_SetItemString(d, "BinOp", (PyObject*)BinOp_type) < 0) return; + if (PyDict_SetItemString(d, "UnaryOp", (PyObject*)UnaryOp_type) < 0) + return; + if (PyDict_SetItemString(d, "Lambda", (PyObject*)Lambda_type) < 0) + return; + if (PyDict_SetItemString(d, "IfExp", (PyObject*)IfExp_type) < 0) return; + if (PyDict_SetItemString(d, "Dict", (PyObject*)Dict_type) < 0) return; + if (PyDict_SetItemString(d, "ListComp", (PyObject*)ListComp_type) < 0) + return; + if (PyDict_SetItemString(d, "GeneratorExp", + (PyObject*)GeneratorExp_type) < 0) return; + if (PyDict_SetItemString(d, "Yield", (PyObject*)Yield_type) < 0) return; + if (PyDict_SetItemString(d, "Compare", (PyObject*)Compare_type) < 0) + return; + if (PyDict_SetItemString(d, "Call", (PyObject*)Call_type) < 0) return; + if (PyDict_SetItemString(d, "Repr", (PyObject*)Repr_type) < 0) return; + if (PyDict_SetItemString(d, "Num", (PyObject*)Num_type) < 0) return; + if (PyDict_SetItemString(d, "Str", (PyObject*)Str_type) < 0) return; + if (PyDict_SetItemString(d, "Attribute", (PyObject*)Attribute_type) < + 0) return; + if (PyDict_SetItemString(d, "Subscript", (PyObject*)Subscript_type) < + 0) return; + if (PyDict_SetItemString(d, "Name", (PyObject*)Name_type) < 0) return; + if (PyDict_SetItemString(d, "List", (PyObject*)List_type) < 0) return; + if (PyDict_SetItemString(d, "Tuple", (PyObject*)Tuple_type) < 0) return; + if (PyDict_SetItemString(d, "expr_context", + (PyObject*)expr_context_type) < 0) return; + if (PyDict_SetItemString(d, "Load", (PyObject*)Load_type) < 0) return; + if (PyDict_SetItemString(d, "Store", (PyObject*)Store_type) < 0) return; + if (PyDict_SetItemString(d, "Del", (PyObject*)Del_type) < 0) return; + if (PyDict_SetItemString(d, "AugLoad", (PyObject*)AugLoad_type) < 0) + return; + if (PyDict_SetItemString(d, "AugStore", (PyObject*)AugStore_type) < 0) + return; + if (PyDict_SetItemString(d, "Param", (PyObject*)Param_type) < 0) return; + if (PyDict_SetItemString(d, "slice", (PyObject*)slice_type) < 0) return; + if (PyDict_SetItemString(d, "Ellipsis", (PyObject*)Ellipsis_type) < 0) + return; + if (PyDict_SetItemString(d, "Slice", (PyObject*)Slice_type) < 0) return; + if (PyDict_SetItemString(d, "ExtSlice", (PyObject*)ExtSlice_type) < 0) + return; + if (PyDict_SetItemString(d, "Index", (PyObject*)Index_type) < 0) return; + if (PyDict_SetItemString(d, "boolop", (PyObject*)boolop_type) < 0) + return; + if (PyDict_SetItemString(d, "And", (PyObject*)And_type) < 0) return; + if (PyDict_SetItemString(d, "Or", (PyObject*)Or_type) < 0) return; + if (PyDict_SetItemString(d, "operator", (PyObject*)operator_type) < 0) + return; + if (PyDict_SetItemString(d, "Add", (PyObject*)Add_type) < 0) return; + if (PyDict_SetItemString(d, "Sub", (PyObject*)Sub_type) < 0) return; + if (PyDict_SetItemString(d, "Mult", (PyObject*)Mult_type) < 0) return; + if (PyDict_SetItemString(d, "Div", (PyObject*)Div_type) < 0) return; + if (PyDict_SetItemString(d, "Mod", (PyObject*)Mod_type) < 0) return; + if (PyDict_SetItemString(d, "Pow", (PyObject*)Pow_type) < 0) return; + if (PyDict_SetItemString(d, "LShift", (PyObject*)LShift_type) < 0) + return; + if (PyDict_SetItemString(d, "RShift", (PyObject*)RShift_type) < 0) + return; + if (PyDict_SetItemString(d, "BitOr", (PyObject*)BitOr_type) < 0) return; + if (PyDict_SetItemString(d, "BitXor", (PyObject*)BitXor_type) < 0) + return; + if (PyDict_SetItemString(d, "BitAnd", (PyObject*)BitAnd_type) < 0) + return; + if (PyDict_SetItemString(d, "FloorDiv", (PyObject*)FloorDiv_type) < 0) + return; + if (PyDict_SetItemString(d, "unaryop", (PyObject*)unaryop_type) < 0) + return; + if (PyDict_SetItemString(d, "Invert", (PyObject*)Invert_type) < 0) + return; + if (PyDict_SetItemString(d, "Not", (PyObject*)Not_type) < 0) return; + if (PyDict_SetItemString(d, "UAdd", (PyObject*)UAdd_type) < 0) return; + if (PyDict_SetItemString(d, "USub", (PyObject*)USub_type) < 0) return; + if (PyDict_SetItemString(d, "cmpop", (PyObject*)cmpop_type) < 0) return; + if (PyDict_SetItemString(d, "Eq", (PyObject*)Eq_type) < 0) return; + if (PyDict_SetItemString(d, "NotEq", (PyObject*)NotEq_type) < 0) return; + if (PyDict_SetItemString(d, "Lt", (PyObject*)Lt_type) < 0) return; + if (PyDict_SetItemString(d, "LtE", (PyObject*)LtE_type) < 0) return; + if (PyDict_SetItemString(d, "Gt", (PyObject*)Gt_type) < 0) return; + if (PyDict_SetItemString(d, "GtE", (PyObject*)GtE_type) < 0) return; + if (PyDict_SetItemString(d, "Is", (PyObject*)Is_type) < 0) return; + if (PyDict_SetItemString(d, "IsNot", (PyObject*)IsNot_type) < 0) return; + if (PyDict_SetItemString(d, "In", (PyObject*)In_type) < 0) return; + if (PyDict_SetItemString(d, "NotIn", (PyObject*)NotIn_type) < 0) return; + if (PyDict_SetItemString(d, "comprehension", + (PyObject*)comprehension_type) < 0) return; + if (PyDict_SetItemString(d, "excepthandler", + (PyObject*)excepthandler_type) < 0) return; + if (PyDict_SetItemString(d, "arguments", (PyObject*)arguments_type) < + 0) return; + if (PyDict_SetItemString(d, "keyword", (PyObject*)keyword_type) < 0) + return; + if (PyDict_SetItemString(d, "alias", (PyObject*)alias_type) < 0) return; } From python-checkins at python.org Mon Apr 3 06:52:08 2006 From: python-checkins at python.org (neal.norwitz) Date: Mon, 3 Apr 2006 06:52:08 +0200 (CEST) Subject: [Python-checkins] r43566 - python/trunk/setup.py Message-ID: <20060403045208.D89F81E4004@bag.python.org> Author: neal.norwitz Date: Mon Apr 3 06:52:05 2006 New Revision: 43566 Modified: python/trunk/setup.py Log: Remove some duplicated code for handling Mac modules. No functional change (intended). Also stoped setting srcdir twice. Modified: python/trunk/setup.py ============================================================================== --- python/trunk/setup.py (original) +++ python/trunk/setup.py Mon Apr 3 06:52:05 2006 @@ -999,115 +999,73 @@ carbon_extra_compile_args = [] # Mac OS X specific modules. - exts.append( Extension('_CF', ['cf/_CFmodule.c', 'cf/pycfbridge.c'], - extra_compile_args=carbon_extra_compile_args, - extra_link_args=['-framework', 'CoreFoundation']) ) - exts.append( Extension('ColorPicker', ['ColorPickermodule.c'], - extra_compile_args=carbon_extra_compile_args, - extra_link_args=['-framework', 'Carbon']) ) - exts.append( Extension('autoGIL', ['autoGIL.c'], - extra_compile_args=carbon_extra_compile_args, - extra_link_args=['-framework', 'CoreFoundation']) ) - exts.append( Extension('gestalt', ['gestaltmodule.c'], - extra_compile_args=carbon_extra_compile_args, - extra_link_args=['-framework', 'Carbon']) ) - exts.append( Extension('MacOS', ['macosmodule.c'], - extra_compile_args=carbon_extra_compile_args, - extra_link_args=['-framework', 'Carbon']) ) - exts.append( Extension('OSATerminology', ['OSATerminology.c'], - extra_compile_args=carbon_extra_compile_args, - extra_link_args=['-framework', 'Carbon']) ) - exts.append( Extension('icglue', ['icgluemodule.c'], - extra_compile_args=carbon_extra_compile_args, - extra_link_args=['-framework', 'Carbon']) ) - exts.append( Extension('_Res', ['res/_Resmodule.c'], - extra_compile_args=carbon_extra_compile_args, - extra_link_args=['-framework', 'Carbon']) ) - exts.append( Extension('_Snd', ['snd/_Sndmodule.c'], - extra_compile_args=carbon_extra_compile_args, - extra_link_args=['-framework', 'Carbon']) ) - exts.append( Extension('Nav', ['Nav.c'], - extra_compile_args=carbon_extra_compile_args, - extra_link_args=['-framework', 'Carbon']) ) - exts.append( Extension('_AE', ['ae/_AEmodule.c'], - extra_compile_args=carbon_extra_compile_args, - extra_link_args=['-framework', 'Carbon']) ) - exts.append( Extension('_AH', ['ah/_AHmodule.c'], - extra_compile_args=carbon_extra_compile_args, - extra_link_args=['-framework', 'Carbon']) ) - exts.append( Extension('_App', ['app/_Appmodule.c'], - extra_compile_args=carbon_extra_compile_args, - extra_link_args=['-framework', 'Carbon']) ) - exts.append( Extension('_CarbonEvt', ['carbonevt/_CarbonEvtmodule.c'], - extra_compile_args=carbon_extra_compile_args, - extra_link_args=['-framework', 'Carbon']) ) - exts.append( Extension('_CG', ['cg/_CGmodule.c'], - extra_compile_args=carbon_extra_compile_args, - extra_link_args=['-framework', 'ApplicationServices']) ) - exts.append( Extension('_Cm', ['cm/_Cmmodule.c'], - extra_compile_args=carbon_extra_compile_args, - extra_link_args=['-framework', 'Carbon']) ) - exts.append( Extension('_Ctl', ['ctl/_Ctlmodule.c'], - extra_compile_args=carbon_extra_compile_args, - extra_link_args=['-framework', 'Carbon']) ) - exts.append( Extension('_Dlg', ['dlg/_Dlgmodule.c'], - extra_compile_args=carbon_extra_compile_args, - extra_link_args=['-framework', 'Carbon']) ) - exts.append( Extension('_Drag', ['drag/_Dragmodule.c'], - extra_compile_args=carbon_extra_compile_args, - extra_link_args=['-framework', 'Carbon']) ) - exts.append( Extension('_Evt', ['evt/_Evtmodule.c'], - extra_compile_args=carbon_extra_compile_args, - extra_link_args=['-framework', 'Carbon']) ) - exts.append( Extension('_File', ['file/_Filemodule.c'], - extra_compile_args=carbon_extra_compile_args, - extra_link_args=['-framework', 'Carbon']) ) - exts.append( Extension('_Folder', ['folder/_Foldermodule.c'], - extra_compile_args=carbon_extra_compile_args, - extra_link_args=['-framework', 'Carbon']) ) - exts.append( Extension('_Fm', ['fm/_Fmmodule.c'], - extra_compile_args=carbon_extra_compile_args, - extra_link_args=['-framework', 'Carbon']) ) - exts.append( Extension('_Help', ['help/_Helpmodule.c'], - extra_compile_args=carbon_extra_compile_args, - extra_link_args=['-framework', 'Carbon']) ) - exts.append( Extension('_Icn', ['icn/_Icnmodule.c'], - extra_compile_args=carbon_extra_compile_args, - extra_link_args=['-framework', 'Carbon']) ) - exts.append( Extension('_IBCarbon', ['ibcarbon/_IBCarbon.c'], - extra_compile_args=carbon_extra_compile_args, - extra_link_args=['-framework', 'Carbon']) ) - exts.append( Extension('_Launch', ['launch/_Launchmodule.c'], - extra_compile_args=carbon_extra_compile_args, - extra_link_args=['-framework', 'ApplicationServices']) ) - exts.append( Extension('_List', ['list/_Listmodule.c'], - extra_compile_args=carbon_extra_compile_args, - extra_link_args=['-framework', 'Carbon']) ) - exts.append( Extension('_Menu', ['menu/_Menumodule.c'], - extra_compile_args=carbon_extra_compile_args, - extra_link_args=['-framework', 'Carbon']) ) - exts.append( Extension('_Mlte', ['mlte/_Mltemodule.c'], - extra_compile_args=carbon_extra_compile_args, - extra_link_args=['-framework', 'Carbon']) ) - exts.append( Extension('_OSA', ['osa/_OSAmodule.c'], - extra_compile_args=carbon_extra_compile_args, - extra_link_args=['-framework', 'Carbon']) ) - exts.append( Extension('_Qd', ['qd/_Qdmodule.c'], - extra_compile_args=carbon_extra_compile_args, - extra_link_args=['-framework', 'Carbon']) ) - exts.append( Extension('_Qdoffs', ['qdoffs/_Qdoffsmodule.c'], - extra_compile_args=carbon_extra_compile_args, - extra_link_args=['-framework', 'Carbon']) ) + def macSrcExists(name1, name2=''): + if not name1: + return None + names = (name1,) + if name2: + names = (name1, name2) + path = os.path.join(srcdir, 'Mac', 'Modules', *names) + return os.path.exists(path) + + def addMacExtension(name, kwds, extra_srcs=[]): + dirname = '' + if name[0] == '_': + dirname = name[1:].lower() + cname = name + '.c' + cmodulename = name + 'module.c' + # Check for NNN.c, NNNmodule.c, _nnn/NNN.c, _nnn/NNNmodule.c + if macSrcExists(cname): + srcs = [cname] + elif macSrcExists(cmodulename): + srcs = [cmodulename] + elif macSrcExists(dirname, cname): + # XXX(nnorwitz): If all the names ended with module, we + # wouldn't need this condition. ibcarbon is the only one. + srcs = [os.path.join(dirname, cname)] + elif macSrcExists(dirname, cmodulename): + srcs = [os.path.join(dirname, cmodulename)] + else: + raise RuntimeError("%s not found" % name) + + # Here's the whole point: add the extension with sources + exts.append(Extension(name, srcs + extra_srcs, **kwds)) + + # Core Foundation + core_kwds = {'extra_compile_args': carbon_extra_compile_args, + 'extra_link_args': ['-framework', 'CoreFoundation'], + } + addMacExtension('_CF', core_kwds, ['cf/pycfbridge.c']) + addMacExtension('autoGIL', core_kwds) + + # Carbon + carbon_kwds = {'extra_compile_args': carbon_extra_compile_args, + 'extra_link_args': ['-framework', 'Carbon'], + } + CARBON_EXTS = ['ColorPicker', 'gestalt', 'MacOS', 'Nav', + 'OSATerminology', 'icglue', + # All these are in subdirs + '_AE', '_AH', '_App', '_CarbonEvt', '_Cm', '_Ctl', + '_Dlg', '_Drag', '_Evt', '_File', '_Folder', '_Fm', + '_Help', '_Icn', '_IBCarbon', '_List', + '_Menu', '_Mlte', '_OSA', '_Res', '_Qd', '_Qdoffs', + '_Scrap', '_Snd', '_TE', '_Win', + ] + for name in CARBON_EXTS: + addMacExtension(name, carbon_kwds) + + # Application Services & QuickTime + app_kwds = {'extra_compile_args': carbon_extra_compile_args, + 'extra_link_args': ['-framework','ApplicationServices'], + } + addMacExtension('_Launch', app_kwds) + addMacExtension('_CG', app_kwds) + exts.append( Extension('_Qt', ['qt/_Qtmodule.c'], extra_compile_args=carbon_extra_compile_args, extra_link_args=['-framework', 'QuickTime', '-framework', 'Carbon']) ) - exts.append( Extension('_Scrap', ['scrap/_Scrapmodule.c'], - extra_compile_args=carbon_extra_compile_args, - extra_link_args=['-framework', 'Carbon']) ) - exts.append( Extension('_TE', ['te/_TEmodule.c'], - extra_compile_args=carbon_extra_compile_args, - extra_link_args=['-framework', 'Carbon']) ) + # As there is no standardized place (yet) to put # user-installed Mac libraries on OSX, we search for "waste" # in parent directories of the Python source tree. You @@ -1119,7 +1077,6 @@ waste_libs = find_library_file(self.compiler, "WASTE", [], ["../"*n + "waste/Static Libraries" for n in (0,1,2,3,4)]) if waste_incs != None and waste_libs != None: - (srcdir,) = sysconfig.get_config_vars('srcdir') exts.append( Extension('waste', ['waste/wastemodule.c'] + [ os.path.join(srcdir, d) for d in @@ -1132,9 +1089,6 @@ libraries = ['WASTE'], extra_link_args = ['-framework', 'Carbon'], ) ) - exts.append( Extension('_Win', ['win/_Winmodule.c'], - extra_compile_args=carbon_extra_compile_args, - extra_link_args=['-framework', 'Carbon']) ) self.extensions.extend(exts) From python-checkins at python.org Mon Apr 3 07:27:06 2006 From: python-checkins at python.org (neal.norwitz) Date: Mon, 3 Apr 2006 07:27:06 +0200 (CEST) Subject: [Python-checkins] r43567 - python/trunk/Lib/test/test_file.py Message-ID: <20060403052706.678B71E4004@bag.python.org> Author: neal.norwitz Date: Mon Apr 3 07:27:05 2006 New Revision: 43567 Modified: python/trunk/Lib/test/test_file.py Log: Skip the test for sys.stdin.seek(-1) on OSF/1 (Tru64) since it does Bad Things like cause the interpreter to exit abruptly. If there's a way to fix this, it would be good to really fix it. It could just be the operation of the std C library and we just aren't supposed to do that. When the test case is skipped, we print a message so the user can check for themselves. Modified: python/trunk/Lib/test/test_file.py ============================================================================== --- python/trunk/Lib/test/test_file.py (original) +++ python/trunk/Lib/test/test_file.py Mon Apr 3 07:27:05 2006 @@ -100,12 +100,18 @@ print "writelines accepted sequence of non-string objects" f.close() -try: - sys.stdin.seek(-1) -except IOError: - pass +# This causes the interpreter to exit on OSF1 v5.1. +if sys.platform != 'osf1V5': + try: + sys.stdin.seek(-1) + except IOError: + pass + else: + print "should not be able to seek on sys.stdin" else: - print "should not be able to seek on sys.stdin" + print >>sys.__stdout__, ( + ' Skipping sys.stdin.seek(-1), it may crash the interpreter.' + ' Test manually.') try: sys.stdin.truncate() From python-checkins at python.org Mon Apr 3 07:28:31 2006 From: python-checkins at python.org (neal.norwitz) Date: Mon, 3 Apr 2006 07:28:31 +0200 (CEST) Subject: [Python-checkins] r43568 - python/trunk/Lib/test/test_pty.py Message-ID: <20060403052831.EDD6D1E4004@bag.python.org> Author: neal.norwitz Date: Mon Apr 3 07:28:31 2006 New Revision: 43568 Modified: python/trunk/Lib/test/test_pty.py Log: Fix test_pty on OSF/1 (Tru64). The problem is that the newline gets converted to CR CR NL. There may be a way to fix this with tcsetattr, but I couldn't find it. There was a similar problem on IRIX. Just normalize the output and compare that. Will backport. Modified: python/trunk/Lib/test/test_pty.py ============================================================================== --- python/trunk/Lib/test/test_pty.py (original) +++ python/trunk/Lib/test/test_pty.py Mon Apr 3 07:28:31 2006 @@ -18,6 +18,27 @@ def debug(msg): pass +def normalize_output(data): + # Some operating systems do conversions on newline. We could possibly + # fix that by doing the appropriate termios.tcsetattr()s. I couldn't + # figure out the right combo on Tru64 and I don't have an IRIX box. + # So just normalize the output and doc the problem O/Ses by allowing + # certain combinations for some platforms, but avoid allowing other + # differences (like extra whitespace, trailing garbage, etc.) + + # This is about the best we can do without getting some feedback + # from someone more knowledgable. + + # OSF/1 (Tru64) apparently turns \n into \r\r\n. + if data.endswith('\r\r\n'): + return data[:-3] + '\n' + + # IRIX apparently turns \n into \r\n. + if data.endswith('\r\n'): + return data[:-2] + '\n' + + return data + # Marginal testing of pty suite. Cannot do extensive 'do or fail' testing # because pty code is not too portable. @@ -36,19 +57,16 @@ if not os.isatty(slave_fd) and sys.platform not in fickle_isatty: raise TestFailed, "slave_fd is not a tty" - # IRIX apparently turns \n into \r\n. Allow that, but avoid allowing other - # differences (like extra whitespace, trailing garbage, etc.) - debug("Writing to slave_fd") os.write(slave_fd, TEST_STRING_1) s1 = os.read(master_fd, 1024) - sys.stdout.write(s1.replace("\r\n", "\n")) + sys.stdout.write(normalize_output(s1)) debug("Writing chunked output") os.write(slave_fd, TEST_STRING_2[:5]) os.write(slave_fd, TEST_STRING_2[5:]) s2 = os.read(master_fd, 1024) - sys.stdout.write(s2.replace("\r\n", "\n")) + sys.stdout.write(normalize_output(s2)) os.close(slave_fd) os.close(master_fd) From python-checkins at python.org Mon Apr 3 07:29:03 2006 From: python-checkins at python.org (martin.v.loewis) Date: Mon, 3 Apr 2006 07:29:03 +0200 (CEST) Subject: [Python-checkins] r43569 - python/trunk/PCbuild/pcbuild.sln python/trunk/PCbuild/pythoncore.vcproj Message-ID: <20060403052903.BC7751E4004@bag.python.org> Author: martin.v.loewis Date: Mon Apr 3 07:29:03 2006 New Revision: 43569 Modified: python/trunk/PCbuild/pcbuild.sln python/trunk/PCbuild/pythoncore.vcproj Log: Patch #1462700: Make _ctypes_test depend on _ctypes. Use same GUID for pythoncore in project and solution. Modified: python/trunk/PCbuild/pcbuild.sln ============================================================================== --- python/trunk/PCbuild/pcbuild.sln (original) +++ python/trunk/PCbuild/pcbuild.sln Mon Apr 3 07:29:03 2006 @@ -96,6 +96,7 @@ EndProject Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "_ctypes_test", "_ctypes_test.vcproj", "{8CF334D9-4F82-42EB-97AF-83592C5AFD2F}" ProjectSection(ProjectDependencies) = postProject + {F22F40F4-D318-40DC-96B3-88DC81CE0894} = {F22F40F4-D318-40DC-96B3-88DC81CE0894} EndProjectSection EndProject Global Modified: python/trunk/PCbuild/pythoncore.vcproj ============================================================================== --- python/trunk/PCbuild/pythoncore.vcproj (original) +++ python/trunk/PCbuild/pythoncore.vcproj Mon Apr 3 07:29:03 2006 @@ -3,7 +3,7 @@ ProjectType="Visual C++" Version="7.10" Name="pythoncore" - ProjectGUID="{7AFA1F0B-A8A1-455A-A832-BF263404BBEF}" + ProjectGUID="{CF7AC3D1-E2DF-41D2-BEA6-1E2556CDEA26}" RootNamespace="pythoncore" SccProjectName="pythoncore" SccLocalPath=".."> From python-checkins at python.org Mon Apr 3 07:30:46 2006 From: python-checkins at python.org (neal.norwitz) Date: Mon, 3 Apr 2006 07:30:46 +0200 (CEST) Subject: [Python-checkins] r43570 - python/branches/release24-maint/Lib/test/test_pty.py Message-ID: <20060403053046.0A6681E4004@bag.python.org> Author: neal.norwitz Date: Mon Apr 3 07:30:45 2006 New Revision: 43570 Modified: python/branches/release24-maint/Lib/test/test_pty.py Log: Backport 43568: Fix test_pty on OSF/1 (Tru64). The problem is that the newline gets converted to CR CR NL. There may be a way to fix this with tcsetattr, but I couldn't find it. There was a similar problem on IRIX. Just normalize the output and compare that. Modified: python/branches/release24-maint/Lib/test/test_pty.py ============================================================================== --- python/branches/release24-maint/Lib/test/test_pty.py (original) +++ python/branches/release24-maint/Lib/test/test_pty.py Mon Apr 3 07:30:45 2006 @@ -18,6 +18,27 @@ def debug(msg): pass +def normalize_output(data): + # Some operating systems do conversions on newline. We could possibly + # fix that by doing the appropriate termios.tcsetattr()s. I couldn't + # figure out the right combo on Tru64 and I don't have an IRIX box. + # So just normalize the output and doc the problem O/Ses by allowing + # certain combinations for some platforms, but avoid allowing other + # differences (like extra whitespace, trailing garbage, etc.) + + # This is about the best we can do without getting some feedback + # from someone more knowledgable. + + # OSF/1 (Tru64) apparently turns \n into \r\r\n. + if data.endswith('\r\r\n'): + return data[:-3] + '\n' + + # IRIX apparently turns \n into \r\n. + if data.endswith('\r\n'): + return data[:-2] + '\n' + + return data + # Marginal testing of pty suite. Cannot do extensive 'do or fail' testing # because pty code is not too portable. @@ -36,19 +57,16 @@ if not os.isatty(slave_fd) and sys.platform not in fickle_isatty: raise TestFailed, "slave_fd is not a tty" - # IRIX apparently turns \n into \r\n. Allow that, but avoid allowing other - # differences (like extra whitespace, trailing garbage, etc.) - debug("Writing to slave_fd") os.write(slave_fd, TEST_STRING_1) s1 = os.read(master_fd, 1024) - sys.stdout.write(s1.replace("\r\n", "\n")) + sys.stdout.write(normalize_output(s1)) debug("Writing chunked output") os.write(slave_fd, TEST_STRING_2[:5]) os.write(slave_fd, TEST_STRING_2[5:]) s2 = os.read(master_fd, 1024) - sys.stdout.write(s2.replace("\r\n", "\n")) + sys.stdout.write(normalize_output(s2)) os.close(slave_fd) os.close(master_fd) From python-checkins at python.org Mon Apr 3 07:55:10 2006 From: python-checkins at python.org (martin.v.loewis) Date: Mon, 3 Apr 2006 07:55:10 +0200 (CEST) Subject: [Python-checkins] r43571 - external/sqlite-source-3.3.4 external/sqlite-source-3.3.4/alter.c external/sqlite-source-3.3.4/analyze.c external/sqlite-source-3.3.4/attach.c external/sqlite-source-3.3.4/auth.c external/sqlite-source-3.3.4/btree.c external/sqlite-source-3.3.4/btree.h external/sqlite-source-3.3.4/build.c external/sqlite-source-3.3.4/callback.c external/sqlite-source-3.3.4/complete.c external/sqlite-source-3.3.4/date.c external/sqlite-source-3.3.4/delete.c external/sqlite-source-3.3.4/expr.c external/sqlite-source-3.3.4/func.c external/sqlite-source-3.3.4/hash.c external/sqlite-source-3.3.4/hash.h external/sqlite-source-3.3.4/insert.c external/sqlite-source-3.3.4/keywordhash.h external/sqlite-source-3.3.4/legacy.c external/sqlite-source-3.3.4/main.c external/sqlite-source-3.3.4/opcodes.c external/sqlite-source-3.3.4/opcodes.h external/sqlite-source-3.3.4/os.c external/sqlite-source-3.3.4/os.h external/sqlite-source-3.3.4/os_common.h external/sqlite-source-3.3.4/os_unix.c external/sqlite-source-3.3.4/os_win.c external/sqlite-source-3.3.4/pager.c external/sqlite-source-3.3.4/pager.h external/sqlite-source-3.3.4/parse.c external/sqlite-source-3.3.4/parse.h external/sqlite-source-3.3.4/pragma.c external/sqlite-source-3.3.4/prepare.c external/sqlite-source-3.3.4/printf.c external/sqlite-source-3.3.4/random.c external/sqlite-source-3.3.4/select.c external/sqlite-source-3.3.4/shell.c external/sqlite-source-3.3.4/sqlite3.def external/sqlite-source-3.3.4/sqlite3.h external/sqlite-source-3.3.4/sqliteInt.h external/sqlite-source-3.3.4/table.c external/sqlite-source-3.3.4/tclsqlite.c external/sqlite-source-3.3.4/tokenize.c external/sqlite-source-3.3.4/trigger.c external/sqlite-source-3.3.4/update.c external/sqlite-source-3.3.4/utf.c external/sqlite-source-3.3.4/util.c external/sqlite-source-3.3.4/vacuum.c external/sqlite-source-3.3.4/vdbe.c external/sqlite-source-3.3.4/vdbe.h external/sqlite-source-3.3.4/vdbeInt.h external/sqlite-source-3.3.4/vdbeapi.c external/sqlite-source-3.3.4/vdbeaux.c external/sqlite-source-3.3.4/vdbefifo.c external/sqlite-source-3.3.4/vdbemem.c external/sqlite-source-3.3.4/where.c Message-ID: <20060403055510.CE0931E400D@bag.python.org> Author: martin.v.loewis Date: Mon Apr 3 07:54:59 2006 New Revision: 43571 Added: external/sqlite-source-3.3.4/ external/sqlite-source-3.3.4/alter.c external/sqlite-source-3.3.4/analyze.c external/sqlite-source-3.3.4/attach.c external/sqlite-source-3.3.4/auth.c external/sqlite-source-3.3.4/btree.c external/sqlite-source-3.3.4/btree.h external/sqlite-source-3.3.4/build.c external/sqlite-source-3.3.4/callback.c external/sqlite-source-3.3.4/complete.c external/sqlite-source-3.3.4/date.c external/sqlite-source-3.3.4/delete.c external/sqlite-source-3.3.4/expr.c external/sqlite-source-3.3.4/func.c external/sqlite-source-3.3.4/hash.c external/sqlite-source-3.3.4/hash.h external/sqlite-source-3.3.4/insert.c external/sqlite-source-3.3.4/keywordhash.h external/sqlite-source-3.3.4/legacy.c external/sqlite-source-3.3.4/main.c external/sqlite-source-3.3.4/opcodes.c external/sqlite-source-3.3.4/opcodes.h external/sqlite-source-3.3.4/os.c external/sqlite-source-3.3.4/os.h external/sqlite-source-3.3.4/os_common.h external/sqlite-source-3.3.4/os_unix.c external/sqlite-source-3.3.4/os_win.c external/sqlite-source-3.3.4/pager.c external/sqlite-source-3.3.4/pager.h external/sqlite-source-3.3.4/parse.c external/sqlite-source-3.3.4/parse.h external/sqlite-source-3.3.4/pragma.c external/sqlite-source-3.3.4/prepare.c external/sqlite-source-3.3.4/printf.c external/sqlite-source-3.3.4/random.c external/sqlite-source-3.3.4/select.c external/sqlite-source-3.3.4/shell.c external/sqlite-source-3.3.4/sqlite3.def external/sqlite-source-3.3.4/sqlite3.h external/sqlite-source-3.3.4/sqliteInt.h external/sqlite-source-3.3.4/table.c external/sqlite-source-3.3.4/tclsqlite.c external/sqlite-source-3.3.4/tokenize.c external/sqlite-source-3.3.4/trigger.c external/sqlite-source-3.3.4/update.c external/sqlite-source-3.3.4/utf.c external/sqlite-source-3.3.4/util.c external/sqlite-source-3.3.4/vacuum.c external/sqlite-source-3.3.4/vdbe.c external/sqlite-source-3.3.4/vdbe.h external/sqlite-source-3.3.4/vdbeInt.h external/sqlite-source-3.3.4/vdbeapi.c external/sqlite-source-3.3.4/vdbeaux.c external/sqlite-source-3.3.4/vdbefifo.c external/sqlite-source-3.3.4/vdbemem.c external/sqlite-source-3.3.4/where.c Log: Import http://www.sqlite.org/sqlite-source-3_3_4.zip Added: external/sqlite-source-3.3.4/alter.c ============================================================================== --- (empty file) +++ external/sqlite-source-3.3.4/alter.c Mon Apr 3 07:54:59 2006 @@ -0,0 +1,562 @@ +/* +** 2005 February 15 +** +** The author disclaims copyright to this source code. In place of +** a legal notice, here is a blessing: +** +** May you do good and not evil. +** May you find forgiveness for yourself and forgive others. +** May you share freely, never taking more than you give. +** +************************************************************************* +** This file contains C code routines that used to generate VDBE code +** that implements the ALTER TABLE command. +** +** $Id: alter.c,v 1.20 2006/02/09 02:56:03 drh Exp $ +*/ +#include "sqliteInt.h" +#include + +/* +** The code in this file only exists if we are not omitting the +** ALTER TABLE logic from the build. +*/ +#ifndef SQLITE_OMIT_ALTERTABLE + + +/* +** This function is used by SQL generated to implement the +** ALTER TABLE command. The first argument is the text of a CREATE TABLE or +** CREATE INDEX command. The second is a table name. The table name in +** the CREATE TABLE or CREATE INDEX statement is replaced with the second +** argument and the result returned. Examples: +** +** sqlite_rename_table('CREATE TABLE abc(a, b, c)', 'def') +** -> 'CREATE TABLE def(a, b, c)' +** +** sqlite_rename_table('CREATE INDEX i ON abc(a)', 'def') +** -> 'CREATE INDEX i ON def(a, b, c)' +*/ +static void renameTableFunc( + sqlite3_context *context, + int argc, + sqlite3_value **argv +){ + unsigned char const *zSql = sqlite3_value_text(argv[0]); + unsigned char const *zTableName = sqlite3_value_text(argv[1]); + + int token; + Token tname; + unsigned char const *zCsr = zSql; + int len = 0; + char *zRet; + + /* The principle used to locate the table name in the CREATE TABLE + ** statement is that the table name is the first token that is immediatedly + ** followed by a left parenthesis - TK_LP. + */ + if( zSql ){ + do { + /* Store the token that zCsr points to in tname. */ + tname.z = zCsr; + tname.n = len; + + /* Advance zCsr to the next token. Store that token type in 'token', + ** and it's length in 'len' (to be used next iteration of this loop). + */ + do { + zCsr += len; + len = sqlite3GetToken(zCsr, &token); + } while( token==TK_SPACE ); + assert( len>0 ); + } while( token!=TK_LP ); + + zRet = sqlite3MPrintf("%.*s%Q%s", tname.z - zSql, zSql, + zTableName, tname.z+tname.n); + sqlite3_result_text(context, zRet, -1, sqlite3FreeX); + } +} + +#ifndef SQLITE_OMIT_TRIGGER +/* This function is used by SQL generated to implement the ALTER TABLE +** ALTER TABLE command. The first argument is the text of a CREATE TRIGGER +** statement. The second is a table name. The table name in the CREATE +** TRIGGER statement is replaced with the second argument and the result +** returned. This is analagous to renameTableFunc() above, except for CREATE +** TRIGGER, not CREATE INDEX and CREATE TABLE. +*/ +static void renameTriggerFunc( + sqlite3_context *context, + int argc, + sqlite3_value **argv +){ + unsigned char const *zSql = sqlite3_value_text(argv[0]); + unsigned char const *zTableName = sqlite3_value_text(argv[1]); + + int token; + Token tname; + int dist = 3; + unsigned char const *zCsr = zSql; + int len = 0; + char *zRet; + + /* The principle used to locate the table name in the CREATE TRIGGER + ** statement is that the table name is the first token that is immediatedly + ** preceded by either TK_ON or TK_DOT and immediatedly followed by one + ** of TK_WHEN, TK_BEGIN or TK_FOR. + */ + if( zSql ){ + do { + /* Store the token that zCsr points to in tname. */ + tname.z = zCsr; + tname.n = len; + + /* Advance zCsr to the next token. Store that token type in 'token', + ** and it's length in 'len' (to be used next iteration of this loop). + */ + do { + zCsr += len; + len = sqlite3GetToken(zCsr, &token); + }while( token==TK_SPACE ); + assert( len>0 ); + + /* Variable 'dist' stores the number of tokens read since the most + ** recent TK_DOT or TK_ON. This means that when a WHEN, FOR or BEGIN + ** token is read and 'dist' equals 2, the condition stated above + ** to be met. + ** + ** Note that ON cannot be a database, table or column name, so + ** there is no need to worry about syntax like + ** "CREATE TRIGGER ... ON ON.ON BEGIN ..." etc. + */ + dist++; + if( token==TK_DOT || token==TK_ON ){ + dist = 0; + } + } while( dist!=2 || (token!=TK_WHEN && token!=TK_FOR && token!=TK_BEGIN) ); + + /* Variable tname now contains the token that is the old table-name + ** in the CREATE TRIGGER statement. + */ + zRet = sqlite3MPrintf("%.*s%Q%s", tname.z - zSql, zSql, + zTableName, tname.z+tname.n); + sqlite3_result_text(context, zRet, -1, sqlite3FreeX); + } +} +#endif /* !SQLITE_OMIT_TRIGGER */ + +/* +** Register built-in functions used to help implement ALTER TABLE +*/ +void sqlite3AlterFunctions(sqlite3 *db){ + static const struct { + char *zName; + signed char nArg; + void (*xFunc)(sqlite3_context*,int,sqlite3_value **); + } aFuncs[] = { + { "sqlite_rename_table", 2, renameTableFunc}, +#ifndef SQLITE_OMIT_TRIGGER + { "sqlite_rename_trigger", 2, renameTriggerFunc}, +#endif + }; + int i; + + for(i=0; idb->aDb[1].pSchema; /* Temp db schema */ + + /* If the table is not located in the temp-db (in which case NULL is + ** returned, loop through the tables list of triggers. For each trigger + ** that is not part of the temp-db schema, add a clause to the WHERE + ** expression being built up in zWhere. + */ + if( pTab->pSchema!=pTempSchema ){ + for( pTrig=pTab->pTrigger; pTrig; pTrig=pTrig->pNext ){ + if( pTrig->pSchema==pTempSchema ){ + if( !zWhere ){ + zWhere = sqlite3MPrintf("name=%Q", pTrig->name); + }else{ + tmp = zWhere; + zWhere = sqlite3MPrintf("%s OR name=%Q", zWhere, pTrig->name); + sqliteFree(tmp); + } + } + } + } + return zWhere; +} + +/* +** Generate code to drop and reload the internal representation of table +** pTab from the database, including triggers and temporary triggers. +** Argument zName is the name of the table in the database schema at +** the time the generated code is executed. This can be different from +** pTab->zName if this function is being called to code part of an +** "ALTER TABLE RENAME TO" statement. +*/ +static void reloadTableSchema(Parse *pParse, Table *pTab, const char *zName){ + Vdbe *v; + char *zWhere; + int iDb; /* Index of database containing pTab */ +#ifndef SQLITE_OMIT_TRIGGER + Trigger *pTrig; +#endif + + v = sqlite3GetVdbe(pParse); + if( !v ) return; + iDb = sqlite3SchemaToIndex(pParse->db, pTab->pSchema); + assert( iDb>=0 ); + +#ifndef SQLITE_OMIT_TRIGGER + /* Drop any table triggers from the internal schema. */ + for(pTrig=pTab->pTrigger; pTrig; pTrig=pTrig->pNext){ + int iTrigDb = sqlite3SchemaToIndex(pParse->db, pTrig->pSchema); + assert( iTrigDb==iDb || iTrigDb==1 ); + sqlite3VdbeOp3(v, OP_DropTrigger, iTrigDb, 0, pTrig->name, 0); + } +#endif + + /* Drop the table and index from the internal schema */ + sqlite3VdbeOp3(v, OP_DropTable, iDb, 0, pTab->zName, 0); + + /* Reload the table, index and permanent trigger schemas. */ + zWhere = sqlite3MPrintf("tbl_name=%Q", zName); + if( !zWhere ) return; + sqlite3VdbeOp3(v, OP_ParseSchema, iDb, 0, zWhere, P3_DYNAMIC); + +#ifndef SQLITE_OMIT_TRIGGER + /* Now, if the table is not stored in the temp database, reload any temp + ** triggers. Don't use IN(...) in case SQLITE_OMIT_SUBQUERY is defined. + */ + if( (zWhere=whereTempTriggers(pParse, pTab))!=0 ){ + sqlite3VdbeOp3(v, OP_ParseSchema, 1, 0, zWhere, P3_DYNAMIC); + } +#endif +} + +/* +** Generate code to implement the "ALTER TABLE xxx RENAME TO yyy" +** command. +*/ +void sqlite3AlterRenameTable( + Parse *pParse, /* Parser context. */ + SrcList *pSrc, /* The table to rename. */ + Token *pName /* The new table name. */ +){ + int iDb; /* Database that contains the table */ + char *zDb; /* Name of database iDb */ + Table *pTab; /* Table being renamed */ + char *zName = 0; /* NULL-terminated version of pName */ + sqlite3 *db = pParse->db; /* Database connection */ + Vdbe *v; +#ifndef SQLITE_OMIT_TRIGGER + char *zWhere = 0; /* Where clause to locate temp triggers */ +#endif + + if( sqlite3MallocFailed() ) goto exit_rename_table; + assert( pSrc->nSrc==1 ); + + pTab = sqlite3LocateTable(pParse, pSrc->a[0].zName, pSrc->a[0].zDatabase); + if( !pTab ) goto exit_rename_table; + iDb = sqlite3SchemaToIndex(pParse->db, pTab->pSchema); + zDb = db->aDb[iDb].zName; + + /* Get a NULL terminated version of the new table name. */ + zName = sqlite3NameFromToken(pName); + if( !zName ) goto exit_rename_table; + + /* Check that a table or index named 'zName' does not already exist + ** in database iDb. If so, this is an error. + */ + if( sqlite3FindTable(db, zName, zDb) || sqlite3FindIndex(db, zName, zDb) ){ + sqlite3ErrorMsg(pParse, + "there is already another table or index with this name: %s", zName); + goto exit_rename_table; + } + + /* Make sure it is not a system table being altered, or a reserved name + ** that the table is being renamed to. + */ + if( strlen(pTab->zName)>6 && 0==sqlite3StrNICmp(pTab->zName, "sqlite_", 7) ){ + sqlite3ErrorMsg(pParse, "table %s may not be altered", pTab->zName); + goto exit_rename_table; + } + if( SQLITE_OK!=sqlite3CheckObjectName(pParse, zName) ){ + goto exit_rename_table; + } + +#ifndef SQLITE_OMIT_AUTHORIZATION + /* Invoke the authorization callback. */ + if( sqlite3AuthCheck(pParse, SQLITE_ALTER_TABLE, zDb, pTab->zName, 0) ){ + goto exit_rename_table; + } +#endif + + /* Begin a transaction and code the VerifyCookie for database iDb. + ** Then modify the schema cookie (since the ALTER TABLE modifies the + ** schema). + */ + v = sqlite3GetVdbe(pParse); + if( v==0 ){ + goto exit_rename_table; + } + sqlite3BeginWriteOperation(pParse, 0, iDb); + sqlite3ChangeCookie(db, v, iDb); + + /* Modify the sqlite_master table to use the new table name. */ + sqlite3NestedParse(pParse, + "UPDATE %Q.%s SET " +#ifdef SQLITE_OMIT_TRIGGER + "sql = sqlite_rename_table(sql, %Q), " +#else + "sql = CASE " + "WHEN type = 'trigger' THEN sqlite_rename_trigger(sql, %Q)" + "ELSE sqlite_rename_table(sql, %Q) END, " +#endif + "tbl_name = %Q, " + "name = CASE " + "WHEN type='table' THEN %Q " + "WHEN name LIKE 'sqlite_autoindex%%' AND type='index' THEN " + "'sqlite_autoindex_' || %Q || substr(name, %d+18,10) " + "ELSE name END " + "WHERE tbl_name=%Q AND " + "(type='table' OR type='index' OR type='trigger');", + zDb, SCHEMA_TABLE(iDb), zName, zName, zName, +#ifndef SQLITE_OMIT_TRIGGER + zName, +#endif + zName, strlen(pTab->zName), pTab->zName + ); + +#ifndef SQLITE_OMIT_AUTOINCREMENT + /* If the sqlite_sequence table exists in this database, then update + ** it with the new table name. + */ + if( sqlite3FindTable(db, "sqlite_sequence", zDb) ){ + sqlite3NestedParse(pParse, + "UPDATE %Q.sqlite_sequence set name = %Q WHERE name = %Q", + zDb, zName, pTab->zName); + } +#endif + +#ifndef SQLITE_OMIT_TRIGGER + /* If there are TEMP triggers on this table, modify the sqlite_temp_master + ** table. Don't do this if the table being ALTERed is itself located in + ** the temp database. + */ + if( (zWhere=whereTempTriggers(pParse, pTab))!=0 ){ + sqlite3NestedParse(pParse, + "UPDATE sqlite_temp_master SET " + "sql = sqlite_rename_trigger(sql, %Q), " + "tbl_name = %Q " + "WHERE %s;", zName, zName, zWhere); + sqliteFree(zWhere); + } +#endif + + /* Drop and reload the internal table schema. */ + reloadTableSchema(pParse, pTab, zName); + +exit_rename_table: + sqlite3SrcListDelete(pSrc); + sqliteFree(zName); +} + + +/* +** This function is called after an "ALTER TABLE ... ADD" statement +** has been parsed. Argument pColDef contains the text of the new +** column definition. +** +** The Table structure pParse->pNewTable was extended to include +** the new column during parsing. +*/ +void sqlite3AlterFinishAddColumn(Parse *pParse, Token *pColDef){ + Table *pNew; /* Copy of pParse->pNewTable */ + Table *pTab; /* Table being altered */ + int iDb; /* Database number */ + const char *zDb; /* Database name */ + const char *zTab; /* Table name */ + char *zCol; /* Null-terminated column definition */ + Column *pCol; /* The new column */ + Expr *pDflt; /* Default value for the new column */ + + if( pParse->nErr ) return; + pNew = pParse->pNewTable; + assert( pNew ); + + iDb = sqlite3SchemaToIndex(pParse->db, pNew->pSchema); + zDb = pParse->db->aDb[iDb].zName; + zTab = pNew->zName; + pCol = &pNew->aCol[pNew->nCol-1]; + pDflt = pCol->pDflt; + pTab = sqlite3FindTable(pParse->db, zTab, zDb); + assert( pTab ); + +#ifndef SQLITE_OMIT_AUTHORIZATION + /* Invoke the authorization callback. */ + if( sqlite3AuthCheck(pParse, SQLITE_ALTER_TABLE, zDb, pTab->zName, 0) ){ + return; + } +#endif + + /* If the default value for the new column was specified with a + ** literal NULL, then set pDflt to 0. This simplifies checking + ** for an SQL NULL default below. + */ + if( pDflt && pDflt->op==TK_NULL ){ + pDflt = 0; + } + + /* Check that the new column is not specified as PRIMARY KEY or UNIQUE. + ** If there is a NOT NULL constraint, then the default value for the + ** column must not be NULL. + */ + if( pCol->isPrimKey ){ + sqlite3ErrorMsg(pParse, "Cannot add a PRIMARY KEY column"); + return; + } + if( pNew->pIndex ){ + sqlite3ErrorMsg(pParse, "Cannot add a UNIQUE column"); + return; + } + if( pCol->notNull && !pDflt ){ + sqlite3ErrorMsg(pParse, + "Cannot add a NOT NULL column with default value NULL"); + return; + } + + /* Ensure the default expression is something that sqlite3ValueFromExpr() + ** can handle (i.e. not CURRENT_TIME etc.) + */ + if( pDflt ){ + sqlite3_value *pVal; + if( sqlite3ValueFromExpr(pDflt, SQLITE_UTF8, SQLITE_AFF_NONE, &pVal) ){ + /* malloc() has failed */ + return; + } + if( !pVal ){ + sqlite3ErrorMsg(pParse, "Cannot add a column with non-constant default"); + return; + } + sqlite3ValueFree(pVal); + } + + /* Modify the CREATE TABLE statement. */ + zCol = sqliteStrNDup((char*)pColDef->z, pColDef->n); + if( zCol ){ + char *zEnd = &zCol[pColDef->n-1]; + while( (zEnd>zCol && *zEnd==';') || isspace(*(unsigned char *)zEnd) ){ + *zEnd-- = '\0'; + } + sqlite3NestedParse(pParse, + "UPDATE %Q.%s SET " + "sql = substr(sql,1,%d) || ', ' || %Q || substr(sql,%d,length(sql)) " + "WHERE type = 'table' AND name = %Q", + zDb, SCHEMA_TABLE(iDb), pNew->addColOffset, zCol, pNew->addColOffset+1, + zTab + ); + sqliteFree(zCol); + } + + /* If the default value of the new column is NULL, then set the file + ** format to 2. If the default value of the new column is not NULL, + ** the file format becomes 3. + */ + sqlite3MinimumFileFormat(pParse, iDb, pDflt ? 3 : 2); + + /* Reload the schema of the modified table. */ + reloadTableSchema(pParse, pTab, pTab->zName); +} + +/* +** This function is called by the parser after the table-name in +** an "ALTER TABLE ADD" statement is parsed. Argument +** pSrc is the full-name of the table being altered. +** +** This routine makes a (partial) copy of the Table structure +** for the table being altered and sets Parse.pNewTable to point +** to it. Routines called by the parser as the column definition +** is parsed (i.e. sqlite3AddColumn()) add the new Column data to +** the copy. The copy of the Table structure is deleted by tokenize.c +** after parsing is finished. +** +** Routine sqlite3AlterFinishAddColumn() will be called to complete +** coding the "ALTER TABLE ... ADD" statement. +*/ +void sqlite3AlterBeginAddColumn(Parse *pParse, SrcList *pSrc){ + Table *pNew; + Table *pTab; + Vdbe *v; + int iDb; + int i; + int nAlloc; + + /* Look up the table being altered. */ + assert( pParse->pNewTable==0 ); + if( sqlite3MallocFailed() ) goto exit_begin_add_column; + pTab = sqlite3LocateTable(pParse, pSrc->a[0].zName, pSrc->a[0].zDatabase); + if( !pTab ) goto exit_begin_add_column; + + /* Make sure this is not an attempt to ALTER a view. */ + if( pTab->pSelect ){ + sqlite3ErrorMsg(pParse, "Cannot add a column to a view"); + goto exit_begin_add_column; + } + + assert( pTab->addColOffset>0 ); + iDb = sqlite3SchemaToIndex(pParse->db, pTab->pSchema); + + /* Put a copy of the Table struct in Parse.pNewTable for the + ** sqlite3AddColumn() function and friends to modify. + */ + pNew = (Table *)sqliteMalloc(sizeof(Table)); + if( !pNew ) goto exit_begin_add_column; + pParse->pNewTable = pNew; + pNew->nRef = 1; + pNew->nCol = pTab->nCol; + assert( pNew->nCol>0 ); + nAlloc = (((pNew->nCol-1)/8)*8)+8; + assert( nAlloc>=pNew->nCol && nAlloc%8==0 && nAlloc-pNew->nCol<8 ); + pNew->aCol = (Column *)sqliteMalloc(sizeof(Column)*nAlloc); + pNew->zName = sqliteStrDup(pTab->zName); + if( !pNew->aCol || !pNew->zName ){ + goto exit_begin_add_column; + } + memcpy(pNew->aCol, pTab->aCol, sizeof(Column)*pNew->nCol); + for(i=0; inCol; i++){ + Column *pCol = &pNew->aCol[i]; + pCol->zName = sqliteStrDup(pCol->zName); + pCol->zColl = 0; + pCol->zType = 0; + pCol->pDflt = 0; + } + pNew->pSchema = pParse->db->aDb[iDb].pSchema; + pNew->addColOffset = pTab->addColOffset; + pNew->nRef = 1; + + /* Begin a transaction and increment the schema cookie. */ + sqlite3BeginWriteOperation(pParse, 0, iDb); + v = sqlite3GetVdbe(pParse); + if( !v ) goto exit_begin_add_column; + sqlite3ChangeCookie(pParse->db, v, iDb); + +exit_begin_add_column: + sqlite3SrcListDelete(pSrc); + return; +} +#endif /* SQLITE_ALTER_TABLE */ Added: external/sqlite-source-3.3.4/analyze.c ============================================================================== --- (empty file) +++ external/sqlite-source-3.3.4/analyze.c Mon Apr 3 07:54:59 2006 @@ -0,0 +1,403 @@ +/* +** 2005 July 8 +** +** The author disclaims copyright to this source code. In place of +** a legal notice, here is a blessing: +** +** May you do good and not evil. +** May you find forgiveness for yourself and forgive others. +** May you share freely, never taking more than you give. +** +************************************************************************* +** This file contains code associated with the ANALYZE command. +** +** @(#) $Id: analyze.c,v 1.16 2006/01/10 17:58:23 danielk1977 Exp $ +*/ +#ifndef SQLITE_OMIT_ANALYZE +#include "sqliteInt.h" + +/* +** This routine generates code that opens the sqlite_stat1 table on cursor +** iStatCur. +** +** If the sqlite_stat1 tables does not previously exist, it is created. +** If it does previously exist, all entires associated with table zWhere +** are removed. If zWhere==0 then all entries are removed. +*/ +static void openStatTable( + Parse *pParse, /* Parsing context */ + int iDb, /* The database we are looking in */ + int iStatCur, /* Open the sqlite_stat1 table on this cursor */ + const char *zWhere /* Delete entries associated with this table */ +){ + sqlite3 *db = pParse->db; + Db *pDb; + int iRootPage; + Table *pStat; + Vdbe *v = sqlite3GetVdbe(pParse); + + pDb = &db->aDb[iDb]; + if( (pStat = sqlite3FindTable(db, "sqlite_stat1", pDb->zName))==0 ){ + /* The sqlite_stat1 tables does not exist. Create it. + ** Note that a side-effect of the CREATE TABLE statement is to leave + ** the rootpage of the new table on the top of the stack. This is + ** important because the OpenWrite opcode below will be needing it. */ + sqlite3NestedParse(pParse, + "CREATE TABLE %Q.sqlite_stat1(tbl,idx,stat)", + pDb->zName + ); + iRootPage = 0; /* Cause rootpage to be taken from top of stack */ + }else if( zWhere ){ + /* The sqlite_stat1 table exists. Delete all entries associated with + ** the table zWhere. */ + sqlite3NestedParse(pParse, + "DELETE FROM %Q.sqlite_stat1 WHERE tbl=%Q", + pDb->zName, zWhere + ); + iRootPage = pStat->tnum; + }else{ + /* The sqlite_stat1 table already exists. Delete all rows. */ + iRootPage = pStat->tnum; + sqlite3VdbeAddOp(v, OP_Clear, pStat->tnum, iDb); + } + + /* Open the sqlite_stat1 table for writing. Unless it was created + ** by this vdbe program, lock it for writing at the shared-cache level. + ** If this vdbe did create the sqlite_stat1 table, then it must have + ** already obtained a schema-lock, making the write-lock redundant. + */ + if( iRootPage>0 ){ + sqlite3TableLock(pParse, iDb, iRootPage, 1, "sqlite_stat1"); + } + sqlite3VdbeAddOp(v, OP_Integer, iDb, 0); + sqlite3VdbeAddOp(v, OP_OpenWrite, iStatCur, iRootPage); + sqlite3VdbeAddOp(v, OP_SetNumColumns, iStatCur, 3); +} + +/* +** Generate code to do an analysis of all indices associated with +** a single table. +*/ +static void analyzeOneTable( + Parse *pParse, /* Parser context */ + Table *pTab, /* Table whose indices are to be analyzed */ + int iStatCur, /* Cursor that writes to the sqlite_stat1 table */ + int iMem /* Available memory locations begin here */ +){ + Index *pIdx; /* An index to being analyzed */ + int iIdxCur; /* Cursor number for index being analyzed */ + int nCol; /* Number of columns in the index */ + Vdbe *v; /* The virtual machine being built up */ + int i; /* Loop counter */ + int topOfLoop; /* The top of the loop */ + int endOfLoop; /* The end of the loop */ + int addr; /* The address of an instruction */ + int iDb; /* Index of database containing pTab */ + + v = sqlite3GetVdbe(pParse); + if( pTab==0 || pTab->pIndex==0 ){ + /* Do no analysis for tables that have no indices */ + return; + } + + iDb = sqlite3SchemaToIndex(pParse->db, pTab->pSchema); + assert( iDb>=0 ); +#ifndef SQLITE_OMIT_AUTHORIZATION + if( sqlite3AuthCheck(pParse, SQLITE_ANALYZE, pTab->zName, 0, + pParse->db->aDb[iDb].zName ) ){ + return; + } +#endif + + /* Establish a read-lock on the table at the shared-cache level. */ + sqlite3TableLock(pParse, iDb, pTab->tnum, 0, pTab->zName); + + iIdxCur = pParse->nTab; + for(pIdx=pTab->pIndex; pIdx; pIdx=pIdx->pNext){ + KeyInfo *pKey = sqlite3IndexKeyinfo(pParse, pIdx); + + /* Open a cursor to the index to be analyzed + */ + assert( iDb==sqlite3SchemaToIndex(pParse->db, pIdx->pSchema) ); + sqlite3VdbeAddOp(v, OP_Integer, iDb, 0); + VdbeComment((v, "# %s", pIdx->zName)); + sqlite3VdbeOp3(v, OP_OpenRead, iIdxCur, pIdx->tnum, + (char *)pKey, P3_KEYINFO_HANDOFF); + nCol = pIdx->nColumn; + if( iMem+nCol*2>=pParse->nMem ){ + pParse->nMem = iMem+nCol*2+1; + } + sqlite3VdbeAddOp(v, OP_SetNumColumns, iIdxCur, nCol+1); + + /* Memory cells are used as follows: + ** + ** mem[iMem]: The total number of rows in the table. + ** mem[iMem+1]: Number of distinct values in column 1 + ** ... + ** mem[iMem+nCol]: Number of distinct values in column N + ** mem[iMem+nCol+1] Last observed value of column 1 + ** ... + ** mem[iMem+nCol+nCol]: Last observed value of column N + ** + ** Cells iMem through iMem+nCol are initialized to 0. The others + ** are initialized to NULL. + */ + for(i=0; i<=nCol; i++){ + sqlite3VdbeAddOp(v, OP_MemInt, 0, iMem+i); + } + for(i=0; i0 then it is always the case the D>0 so division by zero + ** is never possible. + */ + sqlite3VdbeAddOp(v, OP_MemLoad, iMem, 0); + addr = sqlite3VdbeAddOp(v, OP_IfNot, 0, 0); + sqlite3VdbeAddOp(v, OP_NewRowid, iStatCur, 0); + sqlite3VdbeOp3(v, OP_String8, 0, 0, pTab->zName, 0); + sqlite3VdbeOp3(v, OP_String8, 0, 0, pIdx->zName, 0); + sqlite3VdbeAddOp(v, OP_MemLoad, iMem, 0); + sqlite3VdbeOp3(v, OP_String8, 0, 0, " ", 0); + for(i=0; idb; + Schema *pSchema = db->aDb[iDb].pSchema; /* Schema of database iDb */ + HashElem *k; + int iStatCur; + int iMem; + + sqlite3BeginWriteOperation(pParse, 0, iDb); + iStatCur = pParse->nTab++; + openStatTable(pParse, iDb, iStatCur, 0); + iMem = pParse->nMem; + for(k=sqliteHashFirst(&pSchema->tblHash); k; k=sqliteHashNext(k)){ + Table *pTab = (Table*)sqliteHashData(k); + analyzeOneTable(pParse, pTab, iStatCur, iMem); + } + loadAnalysis(pParse, iDb); +} + +/* +** Generate code that will do an analysis of a single table in +** a database. +*/ +static void analyzeTable(Parse *pParse, Table *pTab){ + int iDb; + int iStatCur; + + assert( pTab!=0 ); + iDb = sqlite3SchemaToIndex(pParse->db, pTab->pSchema); + sqlite3BeginWriteOperation(pParse, 0, iDb); + iStatCur = pParse->nTab++; + openStatTable(pParse, iDb, iStatCur, pTab->zName); + analyzeOneTable(pParse, pTab, iStatCur, pParse->nMem); + loadAnalysis(pParse, iDb); +} + +/* +** Generate code for the ANALYZE command. The parser calls this routine +** when it recognizes an ANALYZE command. +** +** ANALYZE -- 1 +** ANALYZE -- 2 +** ANALYZE ?.? -- 3 +** +** Form 1 causes all indices in all attached databases to be analyzed. +** Form 2 analyzes all indices the single database named. +** Form 3 analyzes all indices associated with the named table. +*/ +void sqlite3Analyze(Parse *pParse, Token *pName1, Token *pName2){ + sqlite3 *db = pParse->db; + int iDb; + int i; + char *z, *zDb; + Table *pTab; + Token *pTableName; + + /* Read the database schema. If an error occurs, leave an error message + ** and code in pParse and return NULL. */ + if( SQLITE_OK!=sqlite3ReadSchema(pParse) ){ + return; + } + + if( pName1==0 ){ + /* Form 1: Analyze everything */ + for(i=0; inDb; i++){ + if( i==1 ) continue; /* Do not analyze the TEMP database */ + analyzeDatabase(pParse, i); + } + }else if( pName2==0 || pName2->n==0 ){ + /* Form 2: Analyze the database or table named */ + iDb = sqlite3FindDb(db, pName1); + if( iDb>=0 ){ + analyzeDatabase(pParse, iDb); + }else{ + z = sqlite3NameFromToken(pName1); + pTab = sqlite3LocateTable(pParse, z, 0); + sqliteFree(z); + if( pTab ){ + analyzeTable(pParse, pTab); + } + } + }else{ + /* Form 3: Analyze the fully qualified table name */ + iDb = sqlite3TwoPartName(pParse, pName1, pName2, &pTableName); + if( iDb>=0 ){ + zDb = db->aDb[iDb].zName; + z = sqlite3NameFromToken(pTableName); + pTab = sqlite3LocateTable(pParse, z, zDb); + sqliteFree(z); + if( pTab ){ + analyzeTable(pParse, pTab); + } + } + } +} + +/* +** Used to pass information from the analyzer reader through to the +** callback routine. +*/ +typedef struct analysisInfo analysisInfo; +struct analysisInfo { + sqlite3 *db; + const char *zDatabase; +}; + +/* +** This callback is invoked once for each index when reading the +** sqlite_stat1 table. +** +** argv[0] = name of the index +** argv[1] = results of analysis - on integer for each column +*/ +static int analysisLoader(void *pData, int argc, char **argv, char **azNotUsed){ + analysisInfo *pInfo = (analysisInfo*)pData; + Index *pIndex; + int i, c; + unsigned int v; + const char *z; + + assert( argc==2 ); + if( argv==0 || argv[0]==0 || argv[1]==0 ){ + return 0; + } + pIndex = sqlite3FindIndex(pInfo->db, argv[0], pInfo->zDatabase); + if( pIndex==0 ){ + return 0; + } + z = argv[1]; + for(i=0; *z && i<=pIndex->nColumn; i++){ + v = 0; + while( (c=z[0])>='0' && c<='9' ){ + v = v*10 + c - '0'; + z++; + } + pIndex->aiRowEst[i] = v; + if( *z==' ' ) z++; + } + return 0; +} + +/* +** Load the content of the sqlite_stat1 table into the index hash tables. +*/ +void sqlite3AnalysisLoad(sqlite3 *db, int iDb){ + analysisInfo sInfo; + HashElem *i; + char *zSql; + + /* Clear any prior statistics */ + for(i=sqliteHashFirst(&db->aDb[iDb].pSchema->idxHash);i;i=sqliteHashNext(i)){ + Index *pIdx = sqliteHashData(i); + sqlite3DefaultRowEst(pIdx); + } + + /* Check to make sure the sqlite_stat1 table existss */ + sInfo.db = db; + sInfo.zDatabase = db->aDb[iDb].zName; + if( sqlite3FindTable(db, "sqlite_stat1", sInfo.zDatabase)==0 ){ + return; + } + + + /* Load new statistics out of the sqlite_stat1 table */ + zSql = sqlite3MPrintf("SELECT idx, stat FROM %Q.sqlite_stat1", + sInfo.zDatabase); + sqlite3SafetyOff(db); + sqlite3_exec(db, zSql, analysisLoader, &sInfo, 0); + sqlite3SafetyOn(db); + sqliteFree(zSql); +} + + +#endif /* SQLITE_OMIT_ANALYZE */ Added: external/sqlite-source-3.3.4/attach.c ============================================================================== --- (empty file) +++ external/sqlite-source-3.3.4/attach.c Mon Apr 3 07:54:59 2006 @@ -0,0 +1,493 @@ +/* +** 2003 April 6 +** +** The author disclaims copyright to this source code. In place of +** a legal notice, here is a blessing: +** +** May you do good and not evil. +** May you find forgiveness for yourself and forgive others. +** May you share freely, never taking more than you give. +** +************************************************************************* +** This file contains code used to implement the ATTACH and DETACH commands. +** +** $Id: attach.c,v 1.49 2006/01/24 12:09:18 danielk1977 Exp $ +*/ +#include "sqliteInt.h" + +/* +** Resolve an expression that was part of an ATTACH or DETACH statement. This +** is slightly different from resolving a normal SQL expression, because simple +** identifiers are treated as strings, not possible column names or aliases. +** +** i.e. if the parser sees: +** +** ATTACH DATABASE abc AS def +** +** it treats the two expressions as literal strings 'abc' and 'def' instead of +** looking for columns of the same name. +** +** This only applies to the root node of pExpr, so the statement: +** +** ATTACH DATABASE abc||def AS 'db2' +** +** will fail because neither abc or def can be resolved. +*/ +static int resolveAttachExpr(NameContext *pName, Expr *pExpr) +{ + int rc = SQLITE_OK; + if( pExpr ){ + if( pExpr->op!=TK_ID ){ + rc = sqlite3ExprResolveNames(pName, pExpr); + }else{ + pExpr->op = TK_STRING; + } + } + return rc; +} + +/* +** An SQL user-function registered to do the work of an ATTACH statement. The +** three arguments to the function come directly from an attach statement: +** +** ATTACH DATABASE x AS y KEY z +** +** SELECT sqlite_attach(x, y, z) +** +** If the optional "KEY z" syntax is omitted, an SQL NULL is passed as the +** third argument. +*/ +static void attachFunc( + sqlite3_context *context, + int argc, + sqlite3_value **argv +){ + int i; + int rc = 0; + sqlite3 *db = sqlite3_user_data(context); + const char *zName; + const char *zFile; + Db *aNew; + char zErr[128]; + char *zErrDyn = 0; + + zFile = (const char *)sqlite3_value_text(argv[0]); + zName = (const char *)sqlite3_value_text(argv[1]); + + /* Check for the following errors: + ** + ** * Too many attached databases, + ** * Transaction currently open + ** * Specified database name already being used. + */ + if( db->nDb>=MAX_ATTACHED+2 ){ + sqlite3_snprintf( + 127, zErr, "too many attached databases - max %d", MAX_ATTACHED + ); + goto attach_error; + } + if( !db->autoCommit ){ + strcpy(zErr, "cannot ATTACH database within transaction"); + goto attach_error; + } + for(i=0; inDb; i++){ + char *z = db->aDb[i].zName; + if( z && sqlite3StrICmp(z, zName)==0 ){ + sqlite3_snprintf(127, zErr, "database %s is already in use", zName); + goto attach_error; + } + } + + /* Allocate the new entry in the db->aDb[] array and initialise the schema + ** hash tables. + */ + if( db->aDb==db->aDbStatic ){ + aNew = sqliteMalloc( sizeof(db->aDb[0])*3 ); + if( aNew==0 ){ + return; + } + memcpy(aNew, db->aDb, sizeof(db->aDb[0])*2); + }else{ + aNew = sqliteRealloc(db->aDb, sizeof(db->aDb[0])*(db->nDb+1) ); + if( aNew==0 ){ + return; + } + } + db->aDb = aNew; + aNew = &db->aDb[db->nDb++]; + memset(aNew, 0, sizeof(*aNew)); + + /* Open the database file. If the btree is successfully opened, use + ** it to obtain the database schema. At this point the schema may + ** or may not be initialised. + */ + rc = sqlite3BtreeFactory(db, zFile, 0, MAX_PAGES, &aNew->pBt); + if( rc==SQLITE_OK ){ + aNew->pSchema = sqlite3SchemaGet(aNew->pBt); + if( !aNew->pSchema ){ + rc = SQLITE_NOMEM; + }else if( aNew->pSchema->file_format && aNew->pSchema->enc!=ENC(db) ){ + strcpy(zErr, + "attached databases must use the same text encoding as main database"); + goto attach_error; + } + } + aNew->zName = sqliteStrDup(zName); + aNew->safety_level = 3; + +#if SQLITE_HAS_CODEC + { + extern int sqlite3CodecAttach(sqlite3*, int, void*, int); + extern void sqlite3CodecGetKey(sqlite3*, int, void**, int*); + int nKey; + char *zKey; + int t = sqlite3_value_type(argv[2]); + switch( t ){ + case SQLITE_INTEGER: + case SQLITE_FLOAT: + zErrDyn = sqliteStrDup("Invalid key value"); + rc = SQLITE_ERROR; + break; + + case SQLITE_TEXT: + case SQLITE_BLOB: + nKey = sqlite3_value_bytes(argv[2]); + zKey = (char *)sqlite3_value_blob(argv[2]); + sqlite3CodecAttach(db, db->nDb-1, zKey, nKey); + break; + + case SQLITE_NULL: + /* No key specified. Use the key from the main database */ + sqlite3CodecGetKey(db, 0, (void**)&zKey, &nKey); + sqlite3CodecAttach(db, db->nDb-1, zKey, nKey); + break; + } + } +#endif + + /* If the file was opened successfully, read the schema for the new database. + ** If this fails, or if opening the file failed, then close the file and + ** remove the entry from the db->aDb[] array. i.e. put everything back the way + ** we found it. + */ + if( rc==SQLITE_OK ){ + sqlite3SafetyOn(db); + rc = sqlite3Init(db, &zErrDyn); + sqlite3SafetyOff(db); + } + if( rc ){ + int iDb = db->nDb - 1; + assert( iDb>=2 ); + if( db->aDb[iDb].pBt ){ + sqlite3BtreeClose(db->aDb[iDb].pBt); + db->aDb[iDb].pBt = 0; + db->aDb[iDb].pSchema = 0; + } + sqlite3ResetInternalSchema(db, 0); + db->nDb = iDb; + sqlite3_snprintf(127, zErr, "unable to open database: %s", zFile); + goto attach_error; + } + + return; + +attach_error: + /* Return an error if we get here */ + if( zErrDyn ){ + sqlite3_result_error(context, zErrDyn, -1); + sqliteFree(zErrDyn); + }else{ + zErr[sizeof(zErr)-1] = 0; + sqlite3_result_error(context, zErr, -1); + } +} + +/* +** An SQL user-function registered to do the work of an DETACH statement. The +** three arguments to the function come directly from a detach statement: +** +** DETACH DATABASE x +** +** SELECT sqlite_detach(x) +*/ +static void detachFunc( + sqlite3_context *context, + int argc, + sqlite3_value **argv +){ + const char *zName = (const char *)sqlite3_value_text(argv[0]); + sqlite3 *db = sqlite3_user_data(context); + int i; + Db *pDb = 0; + char zErr[128]; + + assert(zName); + for(i=0; inDb; i++){ + pDb = &db->aDb[i]; + if( pDb->pBt==0 ) continue; + if( sqlite3StrICmp(pDb->zName, zName)==0 ) break; + } + + if( i>=db->nDb ){ + sqlite3_snprintf(sizeof(zErr), zErr, "no such database: %s", zName); + goto detach_error; + } + if( i<2 ){ + sqlite3_snprintf(sizeof(zErr), zErr, "cannot detach database %s", zName); + goto detach_error; + } + if( !db->autoCommit ){ + strcpy(zErr, "cannot DETACH database within transaction"); + goto detach_error; + } + + sqlite3BtreeClose(pDb->pBt); + pDb->pBt = 0; + pDb->pSchema = 0; + sqlite3ResetInternalSchema(db, 0); + return; + +detach_error: + sqlite3_result_error(context, zErr, -1); +} + +/* +** This procedure generates VDBE code for a single invocation of either the +** sqlite_detach() or sqlite_attach() SQL user functions. +*/ +static void codeAttach( + Parse *pParse, /* The parser context */ + int type, /* Either SQLITE_ATTACH or SQLITE_DETACH */ + const char *zFunc, /* Either "sqlite_attach" or "sqlite_detach */ + int nFunc, /* Number of args to pass to zFunc */ + Expr *pAuthArg, /* Expression to pass to authorization callback */ + Expr *pFilename, /* Name of database file */ + Expr *pDbname, /* Name of the database to use internally */ + Expr *pKey /* Database key for encryption extension */ +){ + int rc; + NameContext sName; + Vdbe *v; + FuncDef *pFunc; + sqlite3* db = pParse->db; + +#ifndef SQLITE_OMIT_AUTHORIZATION + assert( sqlite3MallocFailed() || pAuthArg ); + if( pAuthArg ){ + char *zAuthArg = sqlite3NameFromToken(&pAuthArg->span); + if( !zAuthArg ){ + goto attach_end; + } + rc = sqlite3AuthCheck(pParse, type, zAuthArg, 0, 0); + sqliteFree(zAuthArg); + if(rc!=SQLITE_OK ){ + goto attach_end; + } + } +#endif /* SQLITE_OMIT_AUTHORIZATION */ + + memset(&sName, 0, sizeof(NameContext)); + sName.pParse = pParse; + + if( + SQLITE_OK!=(rc = resolveAttachExpr(&sName, pFilename)) || + SQLITE_OK!=(rc = resolveAttachExpr(&sName, pDbname)) || + SQLITE_OK!=(rc = resolveAttachExpr(&sName, pKey)) + ){ + pParse->nErr++; + goto attach_end; + } + + v = sqlite3GetVdbe(pParse); + sqlite3ExprCode(pParse, pFilename); + sqlite3ExprCode(pParse, pDbname); + sqlite3ExprCode(pParse, pKey); + + assert( v || sqlite3MallocFailed() ); + if( v ){ + sqlite3VdbeAddOp(v, OP_Function, 0, nFunc); + pFunc = sqlite3FindFunction(db, zFunc, strlen(zFunc), nFunc, SQLITE_UTF8,0); + sqlite3VdbeChangeP3(v, -1, (char *)pFunc, P3_FUNCDEF); + + /* Code an OP_Expire. For an ATTACH statement, set P1 to true (expire this + ** statement only). For DETACH, set it to false (expire all existing + ** statements). + */ + sqlite3VdbeAddOp(v, OP_Expire, (type==SQLITE_ATTACH), 0); + } + +attach_end: + sqlite3ExprDelete(pFilename); + sqlite3ExprDelete(pDbname); + sqlite3ExprDelete(pKey); +} + +/* +** Called by the parser to compile a DETACH statement. +** +** DETACH pDbname +*/ +void sqlite3Detach(Parse *pParse, Expr *pDbname){ + codeAttach(pParse, SQLITE_DETACH, "sqlite_detach", 1, pDbname, 0, 0, pDbname); +} + +/* +** Called by the parser to compile an ATTACH statement. +** +** ATTACH p AS pDbname KEY pKey +*/ +void sqlite3Attach(Parse *pParse, Expr *p, Expr *pDbname, Expr *pKey){ + codeAttach(pParse, SQLITE_ATTACH, "sqlite_attach", 3, p, p, pDbname, pKey); +} + +/* +** Register the functions sqlite_attach and sqlite_detach. +*/ +void sqlite3AttachFunctions(sqlite3 *db){ + static const int enc = SQLITE_UTF8; + sqlite3CreateFunc(db, "sqlite_attach", 3, enc, db, attachFunc, 0, 0); + sqlite3CreateFunc(db, "sqlite_detach", 1, enc, db, detachFunc, 0, 0); +} + +/* +** Initialize a DbFixer structure. This routine must be called prior +** to passing the structure to one of the sqliteFixAAAA() routines below. +** +** The return value indicates whether or not fixation is required. TRUE +** means we do need to fix the database references, FALSE means we do not. +*/ +int sqlite3FixInit( + DbFixer *pFix, /* The fixer to be initialized */ + Parse *pParse, /* Error messages will be written here */ + int iDb, /* This is the database that must be used */ + const char *zType, /* "view", "trigger", or "index" */ + const Token *pName /* Name of the view, trigger, or index */ +){ + sqlite3 *db; + + if( iDb<0 || iDb==1 ) return 0; + db = pParse->db; + assert( db->nDb>iDb ); + pFix->pParse = pParse; + pFix->zDb = db->aDb[iDb].zName; + pFix->zType = zType; + pFix->pName = pName; + return 1; +} + +/* +** The following set of routines walk through the parse tree and assign +** a specific database to all table references where the database name +** was left unspecified in the original SQL statement. The pFix structure +** must have been initialized by a prior call to sqlite3FixInit(). +** +** These routines are used to make sure that an index, trigger, or +** view in one database does not refer to objects in a different database. +** (Exception: indices, triggers, and views in the TEMP database are +** allowed to refer to anything.) If a reference is explicitly made +** to an object in a different database, an error message is added to +** pParse->zErrMsg and these routines return non-zero. If everything +** checks out, these routines return 0. +*/ +int sqlite3FixSrcList( + DbFixer *pFix, /* Context of the fixation */ + SrcList *pList /* The Source list to check and modify */ +){ + int i; + const char *zDb; + struct SrcList_item *pItem; + + if( pList==0 ) return 0; + zDb = pFix->zDb; + for(i=0, pItem=pList->a; inSrc; i++, pItem++){ + if( pItem->zDatabase==0 ){ + pItem->zDatabase = sqliteStrDup(zDb); + }else if( sqlite3StrICmp(pItem->zDatabase,zDb)!=0 ){ + sqlite3ErrorMsg(pFix->pParse, + "%s %T cannot reference objects in database %s", + pFix->zType, pFix->pName, pItem->zDatabase); + return 1; + } +#if !defined(SQLITE_OMIT_VIEW) || !defined(SQLITE_OMIT_TRIGGER) + if( sqlite3FixSelect(pFix, pItem->pSelect) ) return 1; + if( sqlite3FixExpr(pFix, pItem->pOn) ) return 1; +#endif + } + return 0; +} +#if !defined(SQLITE_OMIT_VIEW) || !defined(SQLITE_OMIT_TRIGGER) +int sqlite3FixSelect( + DbFixer *pFix, /* Context of the fixation */ + Select *pSelect /* The SELECT statement to be fixed to one database */ +){ + while( pSelect ){ + if( sqlite3FixExprList(pFix, pSelect->pEList) ){ + return 1; + } + if( sqlite3FixSrcList(pFix, pSelect->pSrc) ){ + return 1; + } + if( sqlite3FixExpr(pFix, pSelect->pWhere) ){ + return 1; + } + if( sqlite3FixExpr(pFix, pSelect->pHaving) ){ + return 1; + } + pSelect = pSelect->pPrior; + } + return 0; +} +int sqlite3FixExpr( + DbFixer *pFix, /* Context of the fixation */ + Expr *pExpr /* The expression to be fixed to one database */ +){ + while( pExpr ){ + if( sqlite3FixSelect(pFix, pExpr->pSelect) ){ + return 1; + } + if( sqlite3FixExprList(pFix, pExpr->pList) ){ + return 1; + } + if( sqlite3FixExpr(pFix, pExpr->pRight) ){ + return 1; + } + pExpr = pExpr->pLeft; + } + return 0; +} +int sqlite3FixExprList( + DbFixer *pFix, /* Context of the fixation */ + ExprList *pList /* The expression to be fixed to one database */ +){ + int i; + struct ExprList_item *pItem; + if( pList==0 ) return 0; + for(i=0, pItem=pList->a; inExpr; i++, pItem++){ + if( sqlite3FixExpr(pFix, pItem->pExpr) ){ + return 1; + } + } + return 0; +} +#endif + +#ifndef SQLITE_OMIT_TRIGGER +int sqlite3FixTriggerStep( + DbFixer *pFix, /* Context of the fixation */ + TriggerStep *pStep /* The trigger step be fixed to one database */ +){ + while( pStep ){ + if( sqlite3FixSelect(pFix, pStep->pSelect) ){ + return 1; + } + if( sqlite3FixExpr(pFix, pStep->pWhere) ){ + return 1; + } + if( sqlite3FixExprList(pFix, pStep->pExprList) ){ + return 1; + } + pStep = pStep->pNext; + } + return 0; +} +#endif Added: external/sqlite-source-3.3.4/auth.c ============================================================================== --- (empty file) +++ external/sqlite-source-3.3.4/auth.c Mon Apr 3 07:54:59 2006 @@ -0,0 +1,232 @@ +/* +** 2003 January 11 +** +** The author disclaims copyright to this source code. In place of +** a legal notice, here is a blessing: +** +** May you do good and not evil. +** May you find forgiveness for yourself and forgive others. +** May you share freely, never taking more than you give. +** +************************************************************************* +** This file contains code used to implement the sqlite3_set_authorizer() +** API. This facility is an optional feature of the library. Embedded +** systems that do not need this facility may omit it by recompiling +** the library with -DSQLITE_OMIT_AUTHORIZATION=1 +** +** $Id: auth.c,v 1.24 2006/01/13 13:55:45 drh Exp $ +*/ +#include "sqliteInt.h" + +/* +** All of the code in this file may be omitted by defining a single +** macro. +*/ +#ifndef SQLITE_OMIT_AUTHORIZATION + +/* +** Set or clear the access authorization function. +** +** The access authorization function is be called during the compilation +** phase to verify that the user has read and/or write access permission on +** various fields of the database. The first argument to the auth function +** is a copy of the 3rd argument to this routine. The second argument +** to the auth function is one of these constants: +** +** SQLITE_CREATE_INDEX +** SQLITE_CREATE_TABLE +** SQLITE_CREATE_TEMP_INDEX +** SQLITE_CREATE_TEMP_TABLE +** SQLITE_CREATE_TEMP_TRIGGER +** SQLITE_CREATE_TEMP_VIEW +** SQLITE_CREATE_TRIGGER +** SQLITE_CREATE_VIEW +** SQLITE_DELETE +** SQLITE_DROP_INDEX +** SQLITE_DROP_TABLE +** SQLITE_DROP_TEMP_INDEX +** SQLITE_DROP_TEMP_TABLE +** SQLITE_DROP_TEMP_TRIGGER +** SQLITE_DROP_TEMP_VIEW +** SQLITE_DROP_TRIGGER +** SQLITE_DROP_VIEW +** SQLITE_INSERT +** SQLITE_PRAGMA +** SQLITE_READ +** SQLITE_SELECT +** SQLITE_TRANSACTION +** SQLITE_UPDATE +** +** The third and fourth arguments to the auth function are the name of +** the table and the column that are being accessed. The auth function +** should return either SQLITE_OK, SQLITE_DENY, or SQLITE_IGNORE. If +** SQLITE_OK is returned, it means that access is allowed. SQLITE_DENY +** means that the SQL statement will never-run - the sqlite3_exec() call +** will return with an error. SQLITE_IGNORE means that the SQL statement +** should run but attempts to read the specified column will return NULL +** and attempts to write the column will be ignored. +** +** Setting the auth function to NULL disables this hook. The default +** setting of the auth function is NULL. +*/ +int sqlite3_set_authorizer( + sqlite3 *db, + int (*xAuth)(void*,int,const char*,const char*,const char*,const char*), + void *pArg +){ + db->xAuth = xAuth; + db->pAuthArg = pArg; + sqlite3ExpirePreparedStatements(db); + return SQLITE_OK; +} + +/* +** Write an error message into pParse->zErrMsg that explains that the +** user-supplied authorization function returned an illegal value. +*/ +static void sqliteAuthBadReturnCode(Parse *pParse, int rc){ + sqlite3ErrorMsg(pParse, "illegal return value (%d) from the " + "authorization function - should be SQLITE_OK, SQLITE_IGNORE, " + "or SQLITE_DENY", rc); + pParse->rc = SQLITE_ERROR; +} + +/* +** The pExpr should be a TK_COLUMN expression. The table referred to +** is in pTabList or else it is the NEW or OLD table of a trigger. +** Check to see if it is OK to read this particular column. +** +** If the auth function returns SQLITE_IGNORE, change the TK_COLUMN +** instruction into a TK_NULL. If the auth function returns SQLITE_DENY, +** then generate an error. +*/ +void sqlite3AuthRead( + Parse *pParse, /* The parser context */ + Expr *pExpr, /* The expression to check authorization on */ + SrcList *pTabList /* All table that pExpr might refer to */ +){ + sqlite3 *db = pParse->db; + int rc; + Table *pTab; /* The table being read */ + const char *zCol; /* Name of the column of the table */ + int iSrc; /* Index in pTabList->a[] of table being read */ + const char *zDBase; /* Name of database being accessed */ + TriggerStack *pStack; /* The stack of current triggers */ + int iDb; /* The index of the database the expression refers to */ + + if( db->xAuth==0 ) return; + if( pExpr->op==TK_AS ) return; + assert( pExpr->op==TK_COLUMN ); + iDb = sqlite3SchemaToIndex(pParse->db, pExpr->pSchema); + if( iDb<0 ){ + /* An attempt to read a column out of a subquery or other + ** temporary table. */ + return; + } + for(iSrc=0; pTabList && iSrcnSrc; iSrc++){ + if( pExpr->iTable==pTabList->a[iSrc].iCursor ) break; + } + if( iSrc>=0 && pTabList && iSrcnSrc ){ + pTab = pTabList->a[iSrc].pTab; + }else if( (pStack = pParse->trigStack)!=0 ){ + /* This must be an attempt to read the NEW or OLD pseudo-tables + ** of a trigger. + */ + assert( pExpr->iTable==pStack->newIdx || pExpr->iTable==pStack->oldIdx ); + pTab = pStack->pTab; + }else{ + return; + } + if( pTab==0 ) return; + if( pExpr->iColumn>=0 ){ + assert( pExpr->iColumnnCol ); + zCol = pTab->aCol[pExpr->iColumn].zName; + }else if( pTab->iPKey>=0 ){ + assert( pTab->iPKeynCol ); + zCol = pTab->aCol[pTab->iPKey].zName; + }else{ + zCol = "ROWID"; + } + assert( iDb>=0 && iDbnDb ); + zDBase = db->aDb[iDb].zName; + rc = db->xAuth(db->pAuthArg, SQLITE_READ, pTab->zName, zCol, zDBase, + pParse->zAuthContext); + if( rc==SQLITE_IGNORE ){ + pExpr->op = TK_NULL; + }else if( rc==SQLITE_DENY ){ + if( db->nDb>2 || iDb!=0 ){ + sqlite3ErrorMsg(pParse, "access to %s.%s.%s is prohibited", + zDBase, pTab->zName, zCol); + }else{ + sqlite3ErrorMsg(pParse, "access to %s.%s is prohibited",pTab->zName,zCol); + } + pParse->rc = SQLITE_AUTH; + }else if( rc!=SQLITE_OK ){ + sqliteAuthBadReturnCode(pParse, rc); + } +} + +/* +** Do an authorization check using the code and arguments given. Return +** either SQLITE_OK (zero) or SQLITE_IGNORE or SQLITE_DENY. If SQLITE_DENY +** is returned, then the error count and error message in pParse are +** modified appropriately. +*/ +int sqlite3AuthCheck( + Parse *pParse, + int code, + const char *zArg1, + const char *zArg2, + const char *zArg3 +){ + sqlite3 *db = pParse->db; + int rc; + + /* Don't do any authorization checks if the database is initialising. */ + if( db->init.busy ){ + return SQLITE_OK; + } + + if( db->xAuth==0 ){ + return SQLITE_OK; + } + rc = db->xAuth(db->pAuthArg, code, zArg1, zArg2, zArg3, pParse->zAuthContext); + if( rc==SQLITE_DENY ){ + sqlite3ErrorMsg(pParse, "not authorized"); + pParse->rc = SQLITE_AUTH; + }else if( rc!=SQLITE_OK && rc!=SQLITE_IGNORE ){ + rc = SQLITE_DENY; + sqliteAuthBadReturnCode(pParse, rc); + } + return rc; +} + +/* +** Push an authorization context. After this routine is called, the +** zArg3 argument to authorization callbacks will be zContext until +** popped. Or if pParse==0, this routine is a no-op. +*/ +void sqlite3AuthContextPush( + Parse *pParse, + AuthContext *pContext, + const char *zContext +){ + pContext->pParse = pParse; + if( pParse ){ + pContext->zAuthContext = pParse->zAuthContext; + pParse->zAuthContext = zContext; + } +} + +/* +** Pop an authorization context that was previously pushed +** by sqlite3AuthContextPush +*/ +void sqlite3AuthContextPop(AuthContext *pContext){ + if( pContext->pParse ){ + pContext->pParse->zAuthContext = pContext->zAuthContext; + pContext->pParse = 0; + } +} + +#endif /* SQLITE_OMIT_AUTHORIZATION */ Added: external/sqlite-source-3.3.4/btree.c ============================================================================== --- (empty file) +++ external/sqlite-source-3.3.4/btree.c Mon Apr 3 07:54:59 2006 @@ -0,0 +1,6639 @@ +/* +** 2004 April 6 +** +** The author disclaims copyright to this source code. In place of +** a legal notice, here is a blessing: +** +** May you do good and not evil. +** May you find forgiveness for yourself and forgive others. +** May you share freely, never taking more than you give. +** +************************************************************************* +** $Id: btree.c,v 1.314 2006/02/11 01:25:51 drh Exp $ +** +** This file implements a external (disk-based) database using BTrees. +** For a detailed discussion of BTrees, refer to +** +** Donald E. Knuth, THE ART OF COMPUTER PROGRAMMING, Volume 3: +** "Sorting And Searching", pages 473-480. Addison-Wesley +** Publishing Company, Reading, Massachusetts. +** +** The basic idea is that each page of the file contains N database +** entries and N+1 pointers to subpages. +** +** ---------------------------------------------------------------- +** | Ptr(0) | Key(0) | Ptr(1) | Key(1) | ... | Key(N) | Ptr(N+1) | +** ---------------------------------------------------------------- +** +** All of the keys on the page that Ptr(0) points to have values less +** than Key(0). All of the keys on page Ptr(1) and its subpages have +** values greater than Key(0) and less than Key(1). All of the keys +** on Ptr(N+1) and its subpages have values greater than Key(N). And +** so forth. +** +** Finding a particular key requires reading O(log(M)) pages from the +** disk where M is the number of entries in the tree. +** +** In this implementation, a single file can hold one or more separate +** BTrees. Each BTree is identified by the index of its root page. The +** key and data for any entry are combined to form the "payload". A +** fixed amount of payload can be carried directly on the database +** page. If the payload is larger than the preset amount then surplus +** bytes are stored on overflow pages. The payload for an entry +** and the preceding pointer are combined to form a "Cell". Each +** page has a small header which contains the Ptr(N+1) pointer and other +** information such as the size of key and data. +** +** FORMAT DETAILS +** +** The file is divided into pages. The first page is called page 1, +** the second is page 2, and so forth. A page number of zero indicates +** "no such page". The page size can be anything between 512 and 65536. +** Each page can be either a btree page, a freelist page or an overflow +** page. +** +** The first page is always a btree page. The first 100 bytes of the first +** page contain a special header (the "file header") that describes the file. +** The format of the file header is as follows: +** +** OFFSET SIZE DESCRIPTION +** 0 16 Header string: "SQLite format 3\000" +** 16 2 Page size in bytes. +** 18 1 File format write version +** 19 1 File format read version +** 20 1 Bytes of unused space at the end of each page +** 21 1 Max embedded payload fraction +** 22 1 Min embedded payload fraction +** 23 1 Min leaf payload fraction +** 24 4 File change counter +** 28 4 Reserved for future use +** 32 4 First freelist page +** 36 4 Number of freelist pages in the file +** 40 60 15 4-byte meta values passed to higher layers +** +** All of the integer values are big-endian (most significant byte first). +** +** The file change counter is incremented when the database is changed more +** than once within the same second. This counter, together with the +** modification time of the file, allows other processes to know +** when the file has changed and thus when they need to flush their +** cache. +** +** The max embedded payload fraction is the amount of the total usable +** space in a page that can be consumed by a single cell for standard +** B-tree (non-LEAFDATA) tables. A value of 255 means 100%. The default +** is to limit the maximum cell size so that at least 4 cells will fit +** on one page. Thus the default max embedded payload fraction is 64. +** +** If the payload for a cell is larger than the max payload, then extra +** payload is spilled to overflow pages. Once an overflow page is allocated, +** as many bytes as possible are moved into the overflow pages without letting +** the cell size drop below the min embedded payload fraction. +** +** The min leaf payload fraction is like the min embedded payload fraction +** except that it applies to leaf nodes in a LEAFDATA tree. The maximum +** payload fraction for a LEAFDATA tree is always 100% (or 255) and it +** not specified in the header. +** +** Each btree pages is divided into three sections: The header, the +** cell pointer array, and the cell area area. Page 1 also has a 100-byte +** file header that occurs before the page header. +** +** |----------------| +** | file header | 100 bytes. Page 1 only. +** |----------------| +** | page header | 8 bytes for leaves. 12 bytes for interior nodes +** |----------------| +** | cell pointer | | 2 bytes per cell. Sorted order. +** | array | | Grows downward +** | | v +** |----------------| +** | unallocated | +** | space | +** |----------------| ^ Grows upwards +** | cell content | | Arbitrary order interspersed with freeblocks. +** | area | | and free space fragments. +** |----------------| +** +** The page headers looks like this: +** +** OFFSET SIZE DESCRIPTION +** 0 1 Flags. 1: intkey, 2: zerodata, 4: leafdata, 8: leaf +** 1 2 byte offset to the first freeblock +** 3 2 number of cells on this page +** 5 2 first byte of the cell content area +** 7 1 number of fragmented free bytes +** 8 4 Right child (the Ptr(N+1) value). Omitted on leaves. +** +** The flags define the format of this btree page. The leaf flag means that +** this page has no children. The zerodata flag means that this page carries +** only keys and no data. The intkey flag means that the key is a integer +** which is stored in the key size entry of the cell header rather than in +** the payload area. +** +** The cell pointer array begins on the first byte after the page header. +** The cell pointer array contains zero or more 2-byte numbers which are +** offsets from the beginning of the page to the cell content in the cell +** content area. The cell pointers occur in sorted order. The system strives +** to keep free space after the last cell pointer so that new cells can +** be easily added without having to defragment the page. +** +** Cell content is stored at the very end of the page and grows toward the +** beginning of the page. +** +** Unused space within the cell content area is collected into a linked list of +** freeblocks. Each freeblock is at least 4 bytes in size. The byte offset +** to the first freeblock is given in the header. Freeblocks occur in +** increasing order. Because a freeblock must be at least 4 bytes in size, +** any group of 3 or fewer unused bytes in the cell content area cannot +** exist on the freeblock chain. A group of 3 or fewer free bytes is called +** a fragment. The total number of bytes in all fragments is recorded. +** in the page header at offset 7. +** +** SIZE DESCRIPTION +** 2 Byte offset of the next freeblock +** 2 Bytes in this freeblock +** +** Cells are of variable length. Cells are stored in the cell content area at +** the end of the page. Pointers to the cells are in the cell pointer array +** that immediately follows the page header. Cells is not necessarily +** contiguous or in order, but cell pointers are contiguous and in order. +** +** Cell content makes use of variable length integers. A variable +** length integer is 1 to 9 bytes where the lower 7 bits of each +** byte are used. The integer consists of all bytes that have bit 8 set and +** the first byte with bit 8 clear. The most significant byte of the integer +** appears first. A variable-length integer may not be more than 9 bytes long. +** As a special case, all 8 bytes of the 9th byte are used as data. This +** allows a 64-bit integer to be encoded in 9 bytes. +** +** 0x00 becomes 0x00000000 +** 0x7f becomes 0x0000007f +** 0x81 0x00 becomes 0x00000080 +** 0x82 0x00 becomes 0x00000100 +** 0x80 0x7f becomes 0x0000007f +** 0x8a 0x91 0xd1 0xac 0x78 becomes 0x12345678 +** 0x81 0x81 0x81 0x81 0x01 becomes 0x10204081 +** +** Variable length integers are used for rowids and to hold the number of +** bytes of key and data in a btree cell. +** +** The content of a cell looks like this: +** +** SIZE DESCRIPTION +** 4 Page number of the left child. Omitted if leaf flag is set. +** var Number of bytes of data. Omitted if the zerodata flag is set. +** var Number of bytes of key. Or the key itself if intkey flag is set. +** * Payload +** 4 First page of the overflow chain. Omitted if no overflow +** +** Overflow pages form a linked list. Each page except the last is completely +** filled with data (pagesize - 4 bytes). The last page can have as little +** as 1 byte of data. +** +** SIZE DESCRIPTION +** 4 Page number of next overflow page +** * Data +** +** Freelist pages come in two subtypes: trunk pages and leaf pages. The +** file header points to first in a linked list of trunk page. Each trunk +** page points to multiple leaf pages. The content of a leaf page is +** unspecified. A trunk page looks like this: +** +** SIZE DESCRIPTION +** 4 Page number of next trunk page +** 4 Number of leaf pointers on this page +** * zero or more pages numbers of leaves +*/ +#include "sqliteInt.h" +#include "pager.h" +#include "btree.h" +#include "os.h" +#include + +/* Round up a number to the next larger multiple of 8. This is used +** to force 8-byte alignment on 64-bit architectures. +*/ +#define ROUND8(x) ((x+7)&~7) + + +/* The following value is the maximum cell size assuming a maximum page +** size give above. +*/ +#define MX_CELL_SIZE(pBt) (pBt->pageSize-8) + +/* The maximum number of cells on a single page of the database. This +** assumes a minimum cell size of 3 bytes. Such small cells will be +** exceedingly rare, but they are possible. +*/ +#define MX_CELL(pBt) ((pBt->pageSize-8)/3) + +/* Forward declarations */ +typedef struct MemPage MemPage; +typedef struct BtLock BtLock; + +/* +** This is a magic string that appears at the beginning of every +** SQLite database in order to identify the file as a real database. +** +** You can change this value at compile-time by specifying a +** -DSQLITE_FILE_HEADER="..." on the compiler command-line. The +** header must be exactly 16 bytes including the zero-terminator so +** the string itself should be 15 characters long. If you change +** the header, then your custom library will not be able to read +** databases generated by the standard tools and the standard tools +** will not be able to read databases created by your custom library. +*/ +#ifndef SQLITE_FILE_HEADER /* 123456789 123456 */ +# define SQLITE_FILE_HEADER "SQLite format 3" +#endif +static const char zMagicHeader[] = SQLITE_FILE_HEADER; + +/* +** Page type flags. An ORed combination of these flags appear as the +** first byte of every BTree page. +*/ +#define PTF_INTKEY 0x01 +#define PTF_ZERODATA 0x02 +#define PTF_LEAFDATA 0x04 +#define PTF_LEAF 0x08 + +/* +** As each page of the file is loaded into memory, an instance of the following +** structure is appended and initialized to zero. This structure stores +** information about the page that is decoded from the raw file page. +** +** The pParent field points back to the parent page. This allows us to +** walk up the BTree from any leaf to the root. Care must be taken to +** unref() the parent page pointer when this page is no longer referenced. +** The pageDestructor() routine handles that chore. +*/ +struct MemPage { + u8 isInit; /* True if previously initialized. MUST BE FIRST! */ + u8 idxShift; /* True if Cell indices have changed */ + u8 nOverflow; /* Number of overflow cell bodies in aCell[] */ + u8 intKey; /* True if intkey flag is set */ + u8 leaf; /* True if leaf flag is set */ + u8 zeroData; /* True if table stores keys only */ + u8 leafData; /* True if tables stores data on leaves only */ + u8 hasData; /* True if this page stores data */ + u8 hdrOffset; /* 100 for page 1. 0 otherwise */ + u8 childPtrSize; /* 0 if leaf==1. 4 if leaf==0 */ + u16 maxLocal; /* Copy of Btree.maxLocal or Btree.maxLeaf */ + u16 minLocal; /* Copy of Btree.minLocal or Btree.minLeaf */ + u16 cellOffset; /* Index in aData of first cell pointer */ + u16 idxParent; /* Index in parent of this node */ + u16 nFree; /* Number of free bytes on the page */ + u16 nCell; /* Number of cells on this page, local and ovfl */ + struct _OvflCell { /* Cells that will not fit on aData[] */ + u8 *pCell; /* Pointers to the body of the overflow cell */ + u16 idx; /* Insert this cell before idx-th non-overflow cell */ + } aOvfl[5]; + BtShared *pBt; /* Pointer back to BTree structure */ + u8 *aData; /* Pointer back to the start of the page */ + Pgno pgno; /* Page number for this page */ + MemPage *pParent; /* The parent of this page. NULL for root */ +}; + +/* +** The in-memory image of a disk page has the auxiliary information appended +** to the end. EXTRA_SIZE is the number of bytes of space needed to hold +** that extra information. +*/ +#define EXTRA_SIZE sizeof(MemPage) + +/* Btree handle */ +struct Btree { + sqlite3 *pSqlite; + BtShared *pBt; + u8 inTrans; /* TRANS_NONE, TRANS_READ or TRANS_WRITE */ +}; + +/* +** Btree.inTrans may take one of the following values. +** +** If the shared-data extension is enabled, there may be multiple users +** of the Btree structure. At most one of these may open a write transaction, +** but any number may have active read transactions. Variable Btree.pDb +** points to the handle that owns any current write-transaction. +*/ +#define TRANS_NONE 0 +#define TRANS_READ 1 +#define TRANS_WRITE 2 + +/* +** Everything we need to know about an open database +*/ +struct BtShared { + Pager *pPager; /* The page cache */ + BtCursor *pCursor; /* A list of all open cursors */ + MemPage *pPage1; /* First page of the database */ + u8 inStmt; /* True if we are in a statement subtransaction */ + u8 readOnly; /* True if the underlying file is readonly */ + u8 maxEmbedFrac; /* Maximum payload as % of total page size */ + u8 minEmbedFrac; /* Minimum payload as % of total page size */ + u8 minLeafFrac; /* Minimum leaf payload as % of total page size */ + u8 pageSizeFixed; /* True if the page size can no longer be changed */ +#ifndef SQLITE_OMIT_AUTOVACUUM + u8 autoVacuum; /* True if database supports auto-vacuum */ +#endif + u16 pageSize; /* Total number of bytes on a page */ + u16 usableSize; /* Number of usable bytes on each page */ + int maxLocal; /* Maximum local payload in non-LEAFDATA tables */ + int minLocal; /* Minimum local payload in non-LEAFDATA tables */ + int maxLeaf; /* Maximum local payload in a LEAFDATA table */ + int minLeaf; /* Minimum local payload in a LEAFDATA table */ + BusyHandler *pBusyHandler; /* Callback for when there is lock contention */ + u8 inTransaction; /* Transaction state */ + int nRef; /* Number of references to this structure */ + int nTransaction; /* Number of open transactions (read + write) */ + void *pSchema; /* Pointer to space allocated by sqlite3BtreeSchema() */ + void (*xFreeSchema)(void*); /* Destructor for BtShared.pSchema */ +#ifndef SQLITE_OMIT_SHARED_CACHE + BtLock *pLock; /* List of locks held on this shared-btree struct */ + BtShared *pNext; /* Next in ThreadData.pBtree linked list */ +#endif +}; + +/* +** An instance of the following structure is used to hold information +** about a cell. The parseCellPtr() function fills in this structure +** based on information extract from the raw disk page. +*/ +typedef struct CellInfo CellInfo; +struct CellInfo { + u8 *pCell; /* Pointer to the start of cell content */ + i64 nKey; /* The key for INTKEY tables, or number of bytes in key */ + u32 nData; /* Number of bytes of data */ + u16 nHeader; /* Size of the cell content header in bytes */ + u16 nLocal; /* Amount of payload held locally */ + u16 iOverflow; /* Offset to overflow page number. Zero if no overflow */ + u16 nSize; /* Size of the cell content on the main b-tree page */ +}; + +/* +** A cursor is a pointer to a particular entry in the BTree. +** The entry is identified by its MemPage and the index in +** MemPage.aCell[] of the entry. +*/ +struct BtCursor { + Btree *pBtree; /* The Btree to which this cursor belongs */ + BtCursor *pNext, *pPrev; /* Forms a linked list of all cursors */ + int (*xCompare)(void*,int,const void*,int,const void*); /* Key comp func */ + void *pArg; /* First arg to xCompare() */ + Pgno pgnoRoot; /* The root page of this tree */ + MemPage *pPage; /* Page that contains the entry */ + int idx; /* Index of the entry in pPage->aCell[] */ + CellInfo info; /* A parse of the cell we are pointing at */ + u8 wrFlag; /* True if writable */ + u8 eState; /* One of the CURSOR_XXX constants (see below) */ +#ifndef SQLITE_OMIT_SHARED_CACHE + void *pKey; + i64 nKey; + int skip; /* (skip<0) -> Prev() is a no-op. (skip>0) -> Next() is */ +#endif +}; + +/* +** Potential values for BtCursor.eState. The first two values (VALID and +** INVALID) may occur in any build. The third (REQUIRESEEK) may only occur +** if sqlite was compiled without the OMIT_SHARED_CACHE symbol defined. +** +** CURSOR_VALID: +** Cursor points to a valid entry. getPayload() etc. may be called. +** +** CURSOR_INVALID: +** Cursor does not point to a valid entry. This can happen (for example) +** because the table is empty or because BtreeCursorFirst() has not been +** called. +** +** CURSOR_REQUIRESEEK: +** The table that this cursor was opened on still exists, but has been +** modified since the cursor was last used. The cursor position is saved +** in variables BtCursor.pKey and BtCursor.nKey. When a cursor is in +** this state, restoreOrClearCursorPosition() can be called to attempt to +** seek the cursor to the saved position. +*/ +#define CURSOR_INVALID 0 +#define CURSOR_VALID 1 +#define CURSOR_REQUIRESEEK 2 + +/* +** The TRACE macro will print high-level status information about the +** btree operation when the global variable sqlite3_btree_trace is +** enabled. +*/ +#if SQLITE_TEST +# define TRACE(X) if( sqlite3_btree_trace )\ + { sqlite3DebugPrintf X; fflush(stdout); } +#else +# define TRACE(X) +#endif +int sqlite3_btree_trace=0; /* True to enable tracing */ + +/* +** Forward declaration +*/ +static int checkReadLocks(BtShared*,Pgno,BtCursor*); + +/* +** Read or write a two- and four-byte big-endian integer values. +*/ +static u32 get2byte(unsigned char *p){ + return (p[0]<<8) | p[1]; +} +static u32 get4byte(unsigned char *p){ + return (p[0]<<24) | (p[1]<<16) | (p[2]<<8) | p[3]; +} +static void put2byte(unsigned char *p, u32 v){ + p[0] = v>>8; + p[1] = v; +} +static void put4byte(unsigned char *p, u32 v){ + p[0] = v>>24; + p[1] = v>>16; + p[2] = v>>8; + p[3] = v; +} + +/* +** Routines to read and write variable-length integers. These used to +** be defined locally, but now we use the varint routines in the util.c +** file. +*/ +#define getVarint sqlite3GetVarint +/* #define getVarint32 sqlite3GetVarint32 */ +#define getVarint32(A,B) ((*B=*(A))<=0x7f?1:sqlite3GetVarint32(A,B)) +#define putVarint sqlite3PutVarint + +/* The database page the PENDING_BYTE occupies. This page is never used. +** TODO: This macro is very similary to PAGER_MJ_PGNO() in pager.c. They +** should possibly be consolidated (presumably in pager.h). +*/ +#define PENDING_BYTE_PAGE(pBt) ((PENDING_BYTE/(pBt)->pageSize)+1) + +/* +** A linked list of the following structures is stored at BtShared.pLock. +** Locks are added (or upgraded from READ_LOCK to WRITE_LOCK) when a cursor +** is opened on the table with root page BtShared.iTable. Locks are removed +** from this list when a transaction is committed or rolled back, or when +** a btree handle is closed. +*/ +struct BtLock { + Btree *pBtree; /* Btree handle holding this lock */ + Pgno iTable; /* Root page of table */ + u8 eLock; /* READ_LOCK or WRITE_LOCK */ + BtLock *pNext; /* Next in BtShared.pLock list */ +}; + +/* Candidate values for BtLock.eLock */ +#define READ_LOCK 1 +#define WRITE_LOCK 2 + +#ifdef SQLITE_OMIT_SHARED_CACHE + /* + ** The functions queryTableLock(), lockTable() and unlockAllTables() + ** manipulate entries in the BtShared.pLock linked list used to store + ** shared-cache table level locks. If the library is compiled with the + ** shared-cache feature disabled, then there is only ever one user + ** of each BtShared structure and so this locking is not necessary. + ** So define the lock related functions as no-ops. + */ + #define queryTableLock(a,b,c) SQLITE_OK + #define lockTable(a,b,c) SQLITE_OK + #define unlockAllTables(a) + #define restoreOrClearCursorPosition(a,b) SQLITE_OK + #define saveAllCursors(a,b,c) SQLITE_OK + +#else + +static void releasePage(MemPage *pPage); + +/* +** Save the current cursor position in the variables BtCursor.nKey +** and BtCursor.pKey. The cursor's state is set to CURSOR_REQUIRESEEK. +*/ +static int saveCursorPosition(BtCursor *pCur){ + int rc; + + assert( CURSOR_VALID==pCur->eState ); + assert( 0==pCur->pKey ); + + rc = sqlite3BtreeKeySize(pCur, &pCur->nKey); + + /* If this is an intKey table, then the above call to BtreeKeySize() + ** stores the integer key in pCur->nKey. In this case this value is + ** all that is required. Otherwise, if pCur is not open on an intKey + ** table, then malloc space for and store the pCur->nKey bytes of key + ** data. + */ + if( rc==SQLITE_OK && 0==pCur->pPage->intKey){ + void *pKey = sqliteMalloc(pCur->nKey); + if( pKey ){ + rc = sqlite3BtreeKey(pCur, 0, pCur->nKey, pKey); + if( rc==SQLITE_OK ){ + pCur->pKey = pKey; + }else{ + sqliteFree(pKey); + } + }else{ + rc = SQLITE_NOMEM; + } + } + assert( !pCur->pPage->intKey || !pCur->pKey ); + + if( rc==SQLITE_OK ){ + releasePage(pCur->pPage); + pCur->pPage = 0; + pCur->eState = CURSOR_REQUIRESEEK; + } + + return rc; +} + +/* +** Save the positions of all cursors except pExcept open on the table +** with root-page iRoot. Usually, this is called just before cursor +** pExcept is used to modify the table (BtreeDelete() or BtreeInsert()). +*/ +static int saveAllCursors(BtShared *pBt, Pgno iRoot, BtCursor *pExcept){ + BtCursor *p; + if( sqlite3ThreadDataReadOnly()->useSharedData ){ + for(p=pBt->pCursor; p; p=p->pNext){ + if( p!=pExcept && (0==iRoot || p->pgnoRoot==iRoot) && + p->eState==CURSOR_VALID ){ + int rc = saveCursorPosition(p); + if( SQLITE_OK!=rc ){ + return rc; + } + } + } + } + return SQLITE_OK; +} + +/* +** Restore the cursor to the position it was in (or as close to as possible) +** when saveCursorPosition() was called. Note that this call deletes the +** saved position info stored by saveCursorPosition(), so there can be +** at most one effective restoreOrClearCursorPosition() call after each +** saveCursorPosition(). +** +** If the second argument argument - doSeek - is false, then instead of +** returning the cursor to it's saved position, any saved position is deleted +** and the cursor state set to CURSOR_INVALID. +*/ +static int restoreOrClearCursorPositionX(BtCursor *pCur, int doSeek){ + int rc = SQLITE_OK; + assert( sqlite3ThreadDataReadOnly()->useSharedData ); + assert( pCur->eState==CURSOR_REQUIRESEEK ); + pCur->eState = CURSOR_INVALID; + if( doSeek ){ + rc = sqlite3BtreeMoveto(pCur, pCur->pKey, pCur->nKey, &pCur->skip); + } + if( rc==SQLITE_OK ){ + sqliteFree(pCur->pKey); + pCur->pKey = 0; + assert( CURSOR_VALID==pCur->eState || CURSOR_INVALID==pCur->eState ); + } + return rc; +} + +#define restoreOrClearCursorPosition(p,x) \ + (p->eState==CURSOR_REQUIRESEEK?restoreOrClearCursorPositionX(p,x):SQLITE_OK) + +/* +** Query to see if btree handle p may obtain a lock of type eLock +** (READ_LOCK or WRITE_LOCK) on the table with root-page iTab. Return +** SQLITE_OK if the lock may be obtained (by calling lockTable()), or +** SQLITE_LOCKED if not. +*/ +static int queryTableLock(Btree *p, Pgno iTab, u8 eLock){ + BtShared *pBt = p->pBt; + BtLock *pIter; + + /* This is a no-op if the shared-cache is not enabled */ + if( 0==sqlite3ThreadDataReadOnly()->useSharedData ){ + return SQLITE_OK; + } + + /* This (along with lockTable()) is where the ReadUncommitted flag is + ** dealt with. If the caller is querying for a read-lock and the flag is + ** set, it is unconditionally granted - even if there are write-locks + ** on the table. If a write-lock is requested, the ReadUncommitted flag + ** is not considered. + ** + ** In function lockTable(), if a read-lock is demanded and the + ** ReadUncommitted flag is set, no entry is added to the locks list + ** (BtShared.pLock). + ** + ** To summarize: If the ReadUncommitted flag is set, then read cursors do + ** not create or respect table locks. The locking procedure for a + ** write-cursor does not change. + */ + if( + !p->pSqlite || + 0==(p->pSqlite->flags&SQLITE_ReadUncommitted) || + eLock==WRITE_LOCK || + iTab==MASTER_ROOT + ){ + for(pIter=pBt->pLock; pIter; pIter=pIter->pNext){ + if( pIter->pBtree!=p && pIter->iTable==iTab && + (pIter->eLock!=eLock || eLock!=READ_LOCK) ){ + return SQLITE_LOCKED; + } + } + } + return SQLITE_OK; +} + +/* +** Add a lock on the table with root-page iTable to the shared-btree used +** by Btree handle p. Parameter eLock must be either READ_LOCK or +** WRITE_LOCK. +** +** SQLITE_OK is returned if the lock is added successfully. SQLITE_BUSY and +** SQLITE_NOMEM may also be returned. +*/ +static int lockTable(Btree *p, Pgno iTable, u8 eLock){ + BtShared *pBt = p->pBt; + BtLock *pLock = 0; + BtLock *pIter; + + /* This is a no-op if the shared-cache is not enabled */ + if( 0==sqlite3ThreadDataReadOnly()->useSharedData ){ + return SQLITE_OK; + } + + assert( SQLITE_OK==queryTableLock(p, iTable, eLock) ); + + /* If the read-uncommitted flag is set and a read-lock is requested, + ** return early without adding an entry to the BtShared.pLock list. See + ** comment in function queryTableLock() for more info on handling + ** the ReadUncommitted flag. + */ + if( + (p->pSqlite) && + (p->pSqlite->flags&SQLITE_ReadUncommitted) && + (eLock==READ_LOCK) && + iTable!=MASTER_ROOT + ){ + return SQLITE_OK; + } + + /* First search the list for an existing lock on this table. */ + for(pIter=pBt->pLock; pIter; pIter=pIter->pNext){ + if( pIter->iTable==iTable && pIter->pBtree==p ){ + pLock = pIter; + break; + } + } + + /* If the above search did not find a BtLock struct associating Btree p + ** with table iTable, allocate one and link it into the list. + */ + if( !pLock ){ + pLock = (BtLock *)sqliteMalloc(sizeof(BtLock)); + if( !pLock ){ + return SQLITE_NOMEM; + } + pLock->iTable = iTable; + pLock->pBtree = p; + pLock->pNext = pBt->pLock; + pBt->pLock = pLock; + } + + /* Set the BtLock.eLock variable to the maximum of the current lock + ** and the requested lock. This means if a write-lock was already held + ** and a read-lock requested, we don't incorrectly downgrade the lock. + */ + assert( WRITE_LOCK>READ_LOCK ); + if( eLock>pLock->eLock ){ + pLock->eLock = eLock; + } + + return SQLITE_OK; +} + +/* +** Release all the table locks (locks obtained via calls to the lockTable() +** procedure) held by Btree handle p. +*/ +static void unlockAllTables(Btree *p){ + BtLock **ppIter = &p->pBt->pLock; + + /* If the shared-cache extension is not enabled, there should be no + ** locks in the BtShared.pLock list, making this procedure a no-op. Assert + ** that this is the case. + */ + assert( sqlite3ThreadDataReadOnly()->useSharedData || 0==*ppIter ); + + while( *ppIter ){ + BtLock *pLock = *ppIter; + if( pLock->pBtree==p ){ + *ppIter = pLock->pNext; + sqliteFree(pLock); + }else{ + ppIter = &pLock->pNext; + } + } +} +#endif /* SQLITE_OMIT_SHARED_CACHE */ + +#ifndef SQLITE_OMIT_AUTOVACUUM +/* +** These macros define the location of the pointer-map entry for a +** database page. The first argument to each is the number of usable +** bytes on each page of the database (often 1024). The second is the +** page number to look up in the pointer map. +** +** PTRMAP_PAGENO returns the database page number of the pointer-map +** page that stores the required pointer. PTRMAP_PTROFFSET returns +** the offset of the requested map entry. +** +** If the pgno argument passed to PTRMAP_PAGENO is a pointer-map page, +** then pgno is returned. So (pgno==PTRMAP_PAGENO(pgsz, pgno)) can be +** used to test if pgno is a pointer-map page. PTRMAP_ISPAGE implements +** this test. +*/ +#define PTRMAP_PAGENO(pBt, pgno) ptrmapPageno(pBt, pgno) +#define PTRMAP_PTROFFSET(pBt, pgno) (5*(pgno-ptrmapPageno(pBt, pgno)-1)) +#define PTRMAP_ISPAGE(pBt, pgno) (PTRMAP_PAGENO((pBt),(pgno))==(pgno)) + +static Pgno ptrmapPageno(BtShared *pBt, Pgno pgno){ + int nPagesPerMapPage = (pBt->usableSize/5)+1; + int iPtrMap = (pgno-2)/nPagesPerMapPage; + int ret = (iPtrMap*nPagesPerMapPage) + 2; + if( ret==PENDING_BYTE_PAGE(pBt) ){ + ret++; + } + return ret; +} + +/* +** The pointer map is a lookup table that identifies the parent page for +** each child page in the database file. The parent page is the page that +** contains a pointer to the child. Every page in the database contains +** 0 or 1 parent pages. (In this context 'database page' refers +** to any page that is not part of the pointer map itself.) Each pointer map +** entry consists of a single byte 'type' and a 4 byte parent page number. +** The PTRMAP_XXX identifiers below are the valid types. +** +** The purpose of the pointer map is to facility moving pages from one +** position in the file to another as part of autovacuum. When a page +** is moved, the pointer in its parent must be updated to point to the +** new location. The pointer map is used to locate the parent page quickly. +** +** PTRMAP_ROOTPAGE: The database page is a root-page. The page-number is not +** used in this case. +** +** PTRMAP_FREEPAGE: The database page is an unused (free) page. The page-number +** is not used in this case. +** +** PTRMAP_OVERFLOW1: The database page is the first page in a list of +** overflow pages. The page number identifies the page that +** contains the cell with a pointer to this overflow page. +** +** PTRMAP_OVERFLOW2: The database page is the second or later page in a list of +** overflow pages. The page-number identifies the previous +** page in the overflow page list. +** +** PTRMAP_BTREE: The database page is a non-root btree page. The page number +** identifies the parent page in the btree. +*/ +#define PTRMAP_ROOTPAGE 1 +#define PTRMAP_FREEPAGE 2 +#define PTRMAP_OVERFLOW1 3 +#define PTRMAP_OVERFLOW2 4 +#define PTRMAP_BTREE 5 + +/* +** Write an entry into the pointer map. +** +** This routine updates the pointer map entry for page number 'key' +** so that it maps to type 'eType' and parent page number 'pgno'. +** An error code is returned if something goes wrong, otherwise SQLITE_OK. +*/ +static int ptrmapPut(BtShared *pBt, Pgno key, u8 eType, Pgno parent){ + u8 *pPtrmap; /* The pointer map page */ + Pgno iPtrmap; /* The pointer map page number */ + int offset; /* Offset in pointer map page */ + int rc; + + /* The master-journal page number must never be used as a pointer map page */ + assert( 0==PTRMAP_ISPAGE(pBt, PENDING_BYTE_PAGE(pBt)) ); + + assert( pBt->autoVacuum ); + if( key==0 ){ + return SQLITE_CORRUPT_BKPT; + } + iPtrmap = PTRMAP_PAGENO(pBt, key); + rc = sqlite3pager_get(pBt->pPager, iPtrmap, (void **)&pPtrmap); + if( rc!=SQLITE_OK ){ + return rc; + } + offset = PTRMAP_PTROFFSET(pBt, key); + + if( eType!=pPtrmap[offset] || get4byte(&pPtrmap[offset+1])!=parent ){ + TRACE(("PTRMAP_UPDATE: %d->(%d,%d)\n", key, eType, parent)); + rc = sqlite3pager_write(pPtrmap); + if( rc==SQLITE_OK ){ + pPtrmap[offset] = eType; + put4byte(&pPtrmap[offset+1], parent); + } + } + + sqlite3pager_unref(pPtrmap); + return rc; +} + +/* +** Read an entry from the pointer map. +** +** This routine retrieves the pointer map entry for page 'key', writing +** the type and parent page number to *pEType and *pPgno respectively. +** An error code is returned if something goes wrong, otherwise SQLITE_OK. +*/ +static int ptrmapGet(BtShared *pBt, Pgno key, u8 *pEType, Pgno *pPgno){ + int iPtrmap; /* Pointer map page index */ + u8 *pPtrmap; /* Pointer map page data */ + int offset; /* Offset of entry in pointer map */ + int rc; + + iPtrmap = PTRMAP_PAGENO(pBt, key); + rc = sqlite3pager_get(pBt->pPager, iPtrmap, (void **)&pPtrmap); + if( rc!=0 ){ + return rc; + } + + offset = PTRMAP_PTROFFSET(pBt, key); + if( pEType ) *pEType = pPtrmap[offset]; + if( pPgno ) *pPgno = get4byte(&pPtrmap[offset+1]); + + sqlite3pager_unref(pPtrmap); + if( *pEType<1 || *pEType>5 ) return SQLITE_CORRUPT_BKPT; + return SQLITE_OK; +} + +#endif /* SQLITE_OMIT_AUTOVACUUM */ + +/* +** Given a btree page and a cell index (0 means the first cell on +** the page, 1 means the second cell, and so forth) return a pointer +** to the cell content. +** +** This routine works only for pages that do not contain overflow cells. +*/ +static u8 *findCell(MemPage *pPage, int iCell){ + u8 *data = pPage->aData; + assert( iCell>=0 ); + assert( iCellhdrOffset+3]) ); + return data + get2byte(&data[pPage->cellOffset+2*iCell]); +} + +/* +** This a more complex version of findCell() that works for +** pages that do contain overflow cells. See insert +*/ +static u8 *findOverflowCell(MemPage *pPage, int iCell){ + int i; + for(i=pPage->nOverflow-1; i>=0; i--){ + int k; + struct _OvflCell *pOvfl; + pOvfl = &pPage->aOvfl[i]; + k = pOvfl->idx; + if( k<=iCell ){ + if( k==iCell ){ + return pOvfl->pCell; + } + iCell--; + } + } + return findCell(pPage, iCell); +} + +/* +** Parse a cell content block and fill in the CellInfo structure. There +** are two versions of this function. parseCell() takes a cell index +** as the second argument and parseCellPtr() takes a pointer to the +** body of the cell as its second argument. +*/ +static void parseCellPtr( + MemPage *pPage, /* Page containing the cell */ + u8 *pCell, /* Pointer to the cell text. */ + CellInfo *pInfo /* Fill in this structure */ +){ + int n; /* Number bytes in cell content header */ + u32 nPayload; /* Number of bytes of cell payload */ + + pInfo->pCell = pCell; + assert( pPage->leaf==0 || pPage->leaf==1 ); + n = pPage->childPtrSize; + assert( n==4-4*pPage->leaf ); + if( pPage->hasData ){ + n += getVarint32(&pCell[n], &nPayload); + }else{ + nPayload = 0; + } + pInfo->nData = nPayload; + if( pPage->intKey ){ + n += getVarint(&pCell[n], (u64 *)&pInfo->nKey); + }else{ + u32 x; + n += getVarint32(&pCell[n], &x); + pInfo->nKey = x; + nPayload += x; + } + pInfo->nHeader = n; + if( nPayload<=pPage->maxLocal ){ + /* This is the (easy) common case where the entire payload fits + ** on the local page. No overflow is required. + */ + int nSize; /* Total size of cell content in bytes */ + pInfo->nLocal = nPayload; + pInfo->iOverflow = 0; + nSize = nPayload + n; + if( nSize<4 ){ + nSize = 4; /* Minimum cell size is 4 */ + } + pInfo->nSize = nSize; + }else{ + /* If the payload will not fit completely on the local page, we have + ** to decide how much to store locally and how much to spill onto + ** overflow pages. The strategy is to minimize the amount of unused + ** space on overflow pages while keeping the amount of local storage + ** in between minLocal and maxLocal. + ** + ** Warning: changing the way overflow payload is distributed in any + ** way will result in an incompatible file format. + */ + int minLocal; /* Minimum amount of payload held locally */ + int maxLocal; /* Maximum amount of payload held locally */ + int surplus; /* Overflow payload available for local storage */ + + minLocal = pPage->minLocal; + maxLocal = pPage->maxLocal; + surplus = minLocal + (nPayload - minLocal)%(pPage->pBt->usableSize - 4); + if( surplus <= maxLocal ){ + pInfo->nLocal = surplus; + }else{ + pInfo->nLocal = minLocal; + } + pInfo->iOverflow = pInfo->nLocal + n; + pInfo->nSize = pInfo->iOverflow + 4; + } +} +static void parseCell( + MemPage *pPage, /* Page containing the cell */ + int iCell, /* The cell index. First cell is 0 */ + CellInfo *pInfo /* Fill in this structure */ +){ + parseCellPtr(pPage, findCell(pPage, iCell), pInfo); +} + +/* +** Compute the total number of bytes that a Cell needs in the cell +** data area of the btree-page. The return number includes the cell +** data header and the local payload, but not any overflow page or +** the space used by the cell pointer. +*/ +#ifndef NDEBUG +static int cellSize(MemPage *pPage, int iCell){ + CellInfo info; + parseCell(pPage, iCell, &info); + return info.nSize; +} +#endif +static int cellSizePtr(MemPage *pPage, u8 *pCell){ + CellInfo info; + parseCellPtr(pPage, pCell, &info); + return info.nSize; +} + +#ifndef SQLITE_OMIT_AUTOVACUUM +/* +** If the cell pCell, part of page pPage contains a pointer +** to an overflow page, insert an entry into the pointer-map +** for the overflow page. +*/ +static int ptrmapPutOvflPtr(MemPage *pPage, u8 *pCell){ + if( pCell ){ + CellInfo info; + parseCellPtr(pPage, pCell, &info); + if( (info.nData+(pPage->intKey?0:info.nKey))>info.nLocal ){ + Pgno ovfl = get4byte(&pCell[info.iOverflow]); + return ptrmapPut(pPage->pBt, ovfl, PTRMAP_OVERFLOW1, pPage->pgno); + } + } + return SQLITE_OK; +} +/* +** If the cell with index iCell on page pPage contains a pointer +** to an overflow page, insert an entry into the pointer-map +** for the overflow page. +*/ +static int ptrmapPutOvfl(MemPage *pPage, int iCell){ + u8 *pCell; + pCell = findOverflowCell(pPage, iCell); + return ptrmapPutOvflPtr(pPage, pCell); +} +#endif + + +/* +** Do sanity checking on a page. Throw an exception if anything is +** not right. +** +** This routine is used for internal error checking only. It is omitted +** from most builds. +*/ +#if defined(BTREE_DEBUG) && !defined(NDEBUG) && 0 +static void _pageIntegrity(MemPage *pPage){ + int usableSize; + u8 *data; + int i, j, idx, c, pc, hdr, nFree; + int cellOffset; + int nCell, cellLimit; + u8 *used; + + used = sqliteMallocRaw( pPage->pBt->pageSize ); + if( used==0 ) return; + usableSize = pPage->pBt->usableSize; + assert( pPage->aData==&((unsigned char*)pPage)[-pPage->pBt->pageSize] ); + hdr = pPage->hdrOffset; + assert( hdr==(pPage->pgno==1 ? 100 : 0) ); + assert( pPage->pgno==sqlite3pager_pagenumber(pPage->aData) ); + c = pPage->aData[hdr]; + if( pPage->isInit ){ + assert( pPage->leaf == ((c & PTF_LEAF)!=0) ); + assert( pPage->zeroData == ((c & PTF_ZERODATA)!=0) ); + assert( pPage->leafData == ((c & PTF_LEAFDATA)!=0) ); + assert( pPage->intKey == ((c & (PTF_INTKEY|PTF_LEAFDATA))!=0) ); + assert( pPage->hasData == + !(pPage->zeroData || (!pPage->leaf && pPage->leafData)) ); + assert( pPage->cellOffset==pPage->hdrOffset+12-4*pPage->leaf ); + assert( pPage->nCell = get2byte(&pPage->aData[hdr+3]) ); + } + data = pPage->aData; + memset(used, 0, usableSize); + for(i=0; ileaf*4; i++) used[i] = 1; + nFree = 0; + pc = get2byte(&data[hdr+1]); + while( pc ){ + int size; + assert( pc>0 && pcisInit==0 + || pPage->nFree==nFree+data[hdr+7]+cellLimit-(cellOffset+2*nCell) ); + cellOffset = pPage->cellOffset; + for(i=0; i0 && pcinTrans!=TRANS_NONE || p->pBt->nTransactionpBt->nRef ); \ + assert( p->pBt->nTransaction<=p->pBt->nRef ); \ + assert( p->pBt->inTransaction!=TRANS_NONE || p->pBt->nTransaction==0 ); \ + assert( p->pBt->inTransaction>=p->inTrans ); + +/* +** Defragment the page given. All Cells are moved to the +** end of the page and all free space is collected into one +** big FreeBlk that occurs in between the header and cell +** pointer array and the cell content area. +*/ +static int defragmentPage(MemPage *pPage){ + int i; /* Loop counter */ + int pc; /* Address of a i-th cell */ + int addr; /* Offset of first byte after cell pointer array */ + int hdr; /* Offset to the page header */ + int size; /* Size of a cell */ + int usableSize; /* Number of usable bytes on a page */ + int cellOffset; /* Offset to the cell pointer array */ + int brk; /* Offset to the cell content area */ + int nCell; /* Number of cells on the page */ + unsigned char *data; /* The page data */ + unsigned char *temp; /* Temp area for cell content */ + + assert( sqlite3pager_iswriteable(pPage->aData) ); + assert( pPage->pBt!=0 ); + assert( pPage->pBt->usableSize <= SQLITE_MAX_PAGE_SIZE ); + assert( pPage->nOverflow==0 ); + temp = sqliteMalloc( pPage->pBt->pageSize ); + if( temp==0 ) return SQLITE_NOMEM; + data = pPage->aData; + hdr = pPage->hdrOffset; + cellOffset = pPage->cellOffset; + nCell = pPage->nCell; + assert( nCell==get2byte(&data[hdr+3]) ); + usableSize = pPage->pBt->usableSize; + brk = get2byte(&data[hdr+5]); + memcpy(&temp[brk], &data[brk], usableSize - brk); + brk = usableSize; + for(i=0; ipBt->usableSize ); + size = cellSizePtr(pPage, &temp[pc]); + brk -= size; + memcpy(&data[brk], &temp[pc], size); + put2byte(pAddr, brk); + } + assert( brk>=cellOffset+2*nCell ); + put2byte(&data[hdr+5], brk); + data[hdr+1] = 0; + data[hdr+2] = 0; + data[hdr+7] = 0; + addr = cellOffset+2*nCell; + memset(&data[addr], 0, brk-addr); + sqliteFree(temp); + return SQLITE_OK; +} + +/* +** Allocate nByte bytes of space on a page. +** +** Return the index into pPage->aData[] of the first byte of +** the new allocation. Or return 0 if there is not enough free +** space on the page to satisfy the allocation request. +** +** If the page contains nBytes of free space but does not contain +** nBytes of contiguous free space, then this routine automatically +** calls defragementPage() to consolidate all free space before +** allocating the new chunk. +*/ +static int allocateSpace(MemPage *pPage, int nByte){ + int addr, pc, hdr; + int size; + int nFrag; + int top; + int nCell; + int cellOffset; + unsigned char *data; + + data = pPage->aData; + assert( sqlite3pager_iswriteable(data) ); + assert( pPage->pBt ); + if( nByte<4 ) nByte = 4; + if( pPage->nFreenOverflow>0 ) return 0; + pPage->nFree -= nByte; + hdr = pPage->hdrOffset; + + nFrag = data[hdr+7]; + if( nFrag<60 ){ + /* Search the freelist looking for a slot big enough to satisfy the + ** space request. */ + addr = hdr+1; + while( (pc = get2byte(&data[addr]))>0 ){ + size = get2byte(&data[pc+2]); + if( size>=nByte ){ + if( sizecellOffset; + if( nFrag>=60 || cellOffset + 2*nCell > top - nByte ){ + if( defragmentPage(pPage) ) return 0; + top = get2byte(&data[hdr+5]); + } + top -= nByte; + assert( cellOffset + 2*nCell <= top ); + put2byte(&data[hdr+5], top); + return top; +} + +/* +** Return a section of the pPage->aData to the freelist. +** The first byte of the new free block is pPage->aDisk[start] +** and the size of the block is "size" bytes. +** +** Most of the effort here is involved in coalesing adjacent +** free blocks into a single big free block. +*/ +static void freeSpace(MemPage *pPage, int start, int size){ + int addr, pbegin, hdr; + unsigned char *data = pPage->aData; + + assert( pPage->pBt!=0 ); + assert( sqlite3pager_iswriteable(data) ); + assert( start>=pPage->hdrOffset+6+(pPage->leaf?0:4) ); + assert( (start + size)<=pPage->pBt->usableSize ); + if( size<4 ) size = 4; + + /* Add the space back into the linked list of freeblocks */ + hdr = pPage->hdrOffset; + addr = hdr + 1; + while( (pbegin = get2byte(&data[addr]))0 ){ + assert( pbegin<=pPage->pBt->usableSize-4 ); + assert( pbegin>addr ); + addr = pbegin; + } + assert( pbegin<=pPage->pBt->usableSize-4 ); + assert( pbegin>addr || pbegin==0 ); + put2byte(&data[addr], start); + put2byte(&data[start], pbegin); + put2byte(&data[start+2], size); + pPage->nFree += size; + + /* Coalesce adjacent free blocks */ + addr = pPage->hdrOffset + 1; + while( (pbegin = get2byte(&data[addr]))>0 ){ + int pnext, psize; + assert( pbegin>addr ); + assert( pbegin<=pPage->pBt->usableSize-4 ); + pnext = get2byte(&data[pbegin]); + psize = get2byte(&data[pbegin+2]); + if( pbegin + psize + 3 >= pnext && pnext>0 ){ + int frag = pnext - (pbegin+psize); + assert( frag<=data[pPage->hdrOffset+7] ); + data[pPage->hdrOffset+7] -= frag; + put2byte(&data[pbegin], get2byte(&data[pnext])); + put2byte(&data[pbegin+2], pnext+get2byte(&data[pnext+2])-pbegin); + }else{ + addr = pbegin; + } + } + + /* If the cell content area begins with a freeblock, remove it. */ + if( data[hdr+1]==data[hdr+5] && data[hdr+2]==data[hdr+6] ){ + int top; + pbegin = get2byte(&data[hdr+1]); + memcpy(&data[hdr+1], &data[pbegin], 2); + top = get2byte(&data[hdr+5]); + put2byte(&data[hdr+5], top + get2byte(&data[pbegin+2])); + } +} + +/* +** Decode the flags byte (the first byte of the header) for a page +** and initialize fields of the MemPage structure accordingly. +*/ +static void decodeFlags(MemPage *pPage, int flagByte){ + BtShared *pBt; /* A copy of pPage->pBt */ + + assert( pPage->hdrOffset==(pPage->pgno==1 ? 100 : 0) ); + pPage->intKey = (flagByte & (PTF_INTKEY|PTF_LEAFDATA))!=0; + pPage->zeroData = (flagByte & PTF_ZERODATA)!=0; + pPage->leaf = (flagByte & PTF_LEAF)!=0; + pPage->childPtrSize = 4*(pPage->leaf==0); + pBt = pPage->pBt; + if( flagByte & PTF_LEAFDATA ){ + pPage->leafData = 1; + pPage->maxLocal = pBt->maxLeaf; + pPage->minLocal = pBt->minLeaf; + }else{ + pPage->leafData = 0; + pPage->maxLocal = pBt->maxLocal; + pPage->minLocal = pBt->minLocal; + } + pPage->hasData = !(pPage->zeroData || (!pPage->leaf && pPage->leafData)); +} + +/* +** Initialize the auxiliary information for a disk block. +** +** The pParent parameter must be a pointer to the MemPage which +** is the parent of the page being initialized. The root of a +** BTree has no parent and so for that page, pParent==NULL. +** +** Return SQLITE_OK on success. If we see that the page does +** not contain a well-formed database page, then return +** SQLITE_CORRUPT. Note that a return of SQLITE_OK does not +** guarantee that the page is well-formed. It only shows that +** we failed to detect any corruption. +*/ +static int initPage( + MemPage *pPage, /* The page to be initialized */ + MemPage *pParent /* The parent. Might be NULL */ +){ + int pc; /* Address of a freeblock within pPage->aData[] */ + int hdr; /* Offset to beginning of page header */ + u8 *data; /* Equal to pPage->aData */ + BtShared *pBt; /* The main btree structure */ + int usableSize; /* Amount of usable space on each page */ + int cellOffset; /* Offset from start of page to first cell pointer */ + int nFree; /* Number of unused bytes on the page */ + int top; /* First byte of the cell content area */ + + pBt = pPage->pBt; + assert( pBt!=0 ); + assert( pParent==0 || pParent->pBt==pBt ); + assert( pPage->pgno==sqlite3pager_pagenumber(pPage->aData) ); + assert( pPage->aData == &((unsigned char*)pPage)[-pBt->pageSize] ); + if( pPage->pParent!=pParent && (pPage->pParent!=0 || pPage->isInit) ){ + /* The parent page should never change unless the file is corrupt */ + return SQLITE_CORRUPT_BKPT; + } + if( pPage->isInit ) return SQLITE_OK; + if( pPage->pParent==0 && pParent!=0 ){ + pPage->pParent = pParent; + sqlite3pager_ref(pParent->aData); + } + hdr = pPage->hdrOffset; + data = pPage->aData; + decodeFlags(pPage, data[hdr]); + pPage->nOverflow = 0; + pPage->idxShift = 0; + usableSize = pBt->usableSize; + pPage->cellOffset = cellOffset = hdr + 12 - 4*pPage->leaf; + top = get2byte(&data[hdr+5]); + pPage->nCell = get2byte(&data[hdr+3]); + if( pPage->nCell>MX_CELL(pBt) ){ + /* To many cells for a single page. The page must be corrupt */ + return SQLITE_CORRUPT_BKPT; + } + if( pPage->nCell==0 && pParent!=0 && pParent->pgno!=1 ){ + /* All pages must have at least one cell, except for root pages */ + return SQLITE_CORRUPT_BKPT; + } + + /* Compute the total free space on the page */ + pc = get2byte(&data[hdr+1]); + nFree = data[hdr+7] + top - (cellOffset + 2*pPage->nCell); + while( pc>0 ){ + int next, size; + if( pc>usableSize-4 ){ + /* Free block is off the page */ + return SQLITE_CORRUPT_BKPT; + } + next = get2byte(&data[pc]); + size = get2byte(&data[pc+2]); + if( next>0 && next<=pc+size+3 ){ + /* Free blocks must be in accending order */ + return SQLITE_CORRUPT_BKPT; + } + nFree += size; + pc = next; + } + pPage->nFree = nFree; + if( nFree>=usableSize ){ + /* Free space cannot exceed total page size */ + return SQLITE_CORRUPT_BKPT; + } + + pPage->isInit = 1; + pageIntegrity(pPage); + return SQLITE_OK; +} + +/* +** Set up a raw page so that it looks like a database page holding +** no entries. +*/ +static void zeroPage(MemPage *pPage, int flags){ + unsigned char *data = pPage->aData; + BtShared *pBt = pPage->pBt; + int hdr = pPage->hdrOffset; + int first; + + assert( sqlite3pager_pagenumber(data)==pPage->pgno ); + assert( &data[pBt->pageSize] == (unsigned char*)pPage ); + assert( sqlite3pager_iswriteable(data) ); + memset(&data[hdr], 0, pBt->usableSize - hdr); + data[hdr] = flags; + first = hdr + 8 + 4*((flags&PTF_LEAF)==0); + memset(&data[hdr+1], 0, 4); + data[hdr+7] = 0; + put2byte(&data[hdr+5], pBt->usableSize); + pPage->nFree = pBt->usableSize - first; + decodeFlags(pPage, flags); + pPage->hdrOffset = hdr; + pPage->cellOffset = first; + pPage->nOverflow = 0; + pPage->idxShift = 0; + pPage->nCell = 0; + pPage->isInit = 1; + pageIntegrity(pPage); +} + +/* +** Get a page from the pager. Initialize the MemPage.pBt and +** MemPage.aData elements if needed. +*/ +static int getPage(BtShared *pBt, Pgno pgno, MemPage **ppPage){ + int rc; + unsigned char *aData; + MemPage *pPage; + rc = sqlite3pager_get(pBt->pPager, pgno, (void**)&aData); + if( rc ) return rc; + pPage = (MemPage*)&aData[pBt->pageSize]; + pPage->aData = aData; + pPage->pBt = pBt; + pPage->pgno = pgno; + pPage->hdrOffset = pPage->pgno==1 ? 100 : 0; + *ppPage = pPage; + return SQLITE_OK; +} + +/* +** Get a page from the pager and initialize it. This routine +** is just a convenience wrapper around separate calls to +** getPage() and initPage(). +*/ +static int getAndInitPage( + BtShared *pBt, /* The database file */ + Pgno pgno, /* Number of the page to get */ + MemPage **ppPage, /* Write the page pointer here */ + MemPage *pParent /* Parent of the page */ +){ + int rc; + if( pgno==0 ){ + return SQLITE_CORRUPT_BKPT; + } + rc = getPage(pBt, pgno, ppPage); + if( rc==SQLITE_OK && (*ppPage)->isInit==0 ){ + rc = initPage(*ppPage, pParent); + } + return rc; +} + +/* +** Release a MemPage. This should be called once for each prior +** call to getPage. +*/ +static void releasePage(MemPage *pPage){ + if( pPage ){ + assert( pPage->aData ); + assert( pPage->pBt ); + assert( &pPage->aData[pPage->pBt->pageSize]==(unsigned char*)pPage ); + sqlite3pager_unref(pPage->aData); + } +} + +/* +** This routine is called when the reference count for a page +** reaches zero. We need to unref the pParent pointer when that +** happens. +*/ +static void pageDestructor(void *pData, int pageSize){ + MemPage *pPage; + assert( (pageSize & 7)==0 ); + pPage = (MemPage*)&((char*)pData)[pageSize]; + if( pPage->pParent ){ + MemPage *pParent = pPage->pParent; + pPage->pParent = 0; + releasePage(pParent); + } + pPage->isInit = 0; +} + +/* +** During a rollback, when the pager reloads information into the cache +** so that the cache is restored to its original state at the start of +** the transaction, for each page restored this routine is called. +** +** This routine needs to reset the extra data section at the end of the +** page to agree with the restored data. +*/ +static void pageReinit(void *pData, int pageSize){ + MemPage *pPage; + assert( (pageSize & 7)==0 ); + pPage = (MemPage*)&((char*)pData)[pageSize]; + if( pPage->isInit ){ + pPage->isInit = 0; + initPage(pPage, pPage->pParent); + } +} + +/* +** Open a database file. +** +** zFilename is the name of the database file. If zFilename is NULL +** a new database with a random name is created. This randomly named +** database file will be deleted when sqlite3BtreeClose() is called. +*/ +int sqlite3BtreeOpen( + const char *zFilename, /* Name of the file containing the BTree database */ + sqlite3 *pSqlite, /* Associated database handle */ + Btree **ppBtree, /* Pointer to new Btree object written here */ + int flags /* Options */ +){ + BtShared *pBt; /* Shared part of btree structure */ + Btree *p; /* Handle to return */ + int rc; + int nReserve; + unsigned char zDbHeader[100]; +#if !defined(SQLITE_OMIT_SHARED_CACHE) && !defined(SQLITE_OMIT_DISKIO) + const ThreadData *pTsdro; +#endif + + /* Set the variable isMemdb to true for an in-memory database, or + ** false for a file-based database. This symbol is only required if + ** either of the shared-data or autovacuum features are compiled + ** into the library. + */ +#if !defined(SQLITE_OMIT_SHARED_CACHE) || !defined(SQLITE_OMIT_AUTOVACUUM) + #ifdef SQLITE_OMIT_MEMORYDB + const int isMemdb = !zFilename; + #else + const int isMemdb = !zFilename || (strcmp(zFilename, ":memory:")?0:1); + #endif +#endif + + p = sqliteMalloc(sizeof(Btree)); + if( !p ){ + return SQLITE_NOMEM; + } + p->inTrans = TRANS_NONE; + p->pSqlite = pSqlite; + + /* Try to find an existing Btree structure opened on zFilename. */ +#if !defined(SQLITE_OMIT_SHARED_CACHE) && !defined(SQLITE_OMIT_DISKIO) + pTsdro = sqlite3ThreadDataReadOnly(); + if( pTsdro->useSharedData && zFilename && !isMemdb ){ + char *zFullPathname = sqlite3OsFullPathname(zFilename); + if( !zFullPathname ){ + sqliteFree(p); + return SQLITE_NOMEM; + } + for(pBt=pTsdro->pBtree; pBt; pBt=pBt->pNext){ + assert( pBt->nRef>0 ); + if( 0==strcmp(zFullPathname, sqlite3pager_filename(pBt->pPager)) ){ + p->pBt = pBt; + *ppBtree = p; + pBt->nRef++; + sqliteFree(zFullPathname); + return SQLITE_OK; + } + } + sqliteFree(zFullPathname); + } +#endif + + /* + ** The following asserts make sure that structures used by the btree are + ** the right size. This is to guard against size changes that result + ** when compiling on a different architecture. + */ + assert( sizeof(i64)==8 ); + assert( sizeof(u64)==8 ); + assert( sizeof(u32)==4 ); + assert( sizeof(u16)==2 ); + assert( sizeof(Pgno)==4 ); + + pBt = sqliteMalloc( sizeof(*pBt) ); + if( pBt==0 ){ + *ppBtree = 0; + sqliteFree(p); + return SQLITE_NOMEM; + } + rc = sqlite3pager_open(&pBt->pPager, zFilename, EXTRA_SIZE, flags); + if( rc!=SQLITE_OK ){ + if( pBt->pPager ) sqlite3pager_close(pBt->pPager); + sqliteFree(pBt); + sqliteFree(p); + *ppBtree = 0; + return rc; + } + p->pBt = pBt; + + sqlite3pager_set_destructor(pBt->pPager, pageDestructor); + sqlite3pager_set_reiniter(pBt->pPager, pageReinit); + pBt->pCursor = 0; + pBt->pPage1 = 0; + pBt->readOnly = sqlite3pager_isreadonly(pBt->pPager); + sqlite3pager_read_fileheader(pBt->pPager, sizeof(zDbHeader), zDbHeader); + pBt->pageSize = get2byte(&zDbHeader[16]); + if( pBt->pageSize<512 || pBt->pageSize>SQLITE_MAX_PAGE_SIZE + || ((pBt->pageSize-1)&pBt->pageSize)!=0 ){ + pBt->pageSize = SQLITE_DEFAULT_PAGE_SIZE; + pBt->maxEmbedFrac = 64; /* 25% */ + pBt->minEmbedFrac = 32; /* 12.5% */ + pBt->minLeafFrac = 32; /* 12.5% */ +#ifndef SQLITE_OMIT_AUTOVACUUM + /* If the magic name ":memory:" will create an in-memory database, then + ** do not set the auto-vacuum flag, even if SQLITE_DEFAULT_AUTOVACUUM + ** is true. On the other hand, if SQLITE_OMIT_MEMORYDB has been defined, + ** then ":memory:" is just a regular file-name. Respect the auto-vacuum + ** default in this case. + */ + if( zFilename && !isMemdb ){ + pBt->autoVacuum = SQLITE_DEFAULT_AUTOVACUUM; + } +#endif + nReserve = 0; + }else{ + nReserve = zDbHeader[20]; + pBt->maxEmbedFrac = zDbHeader[21]; + pBt->minEmbedFrac = zDbHeader[22]; + pBt->minLeafFrac = zDbHeader[23]; + pBt->pageSizeFixed = 1; +#ifndef SQLITE_OMIT_AUTOVACUUM + pBt->autoVacuum = (get4byte(&zDbHeader[36 + 4*4])?1:0); +#endif + } + pBt->usableSize = pBt->pageSize - nReserve; + assert( (pBt->pageSize & 7)==0 ); /* 8-byte alignment of pageSize */ + sqlite3pager_set_pagesize(pBt->pPager, pBt->pageSize); + +#ifndef SQLITE_OMIT_SHARED_CACHE + /* Add the new btree to the linked list starting at ThreadData.pBtree. + ** There is no chance that a malloc() may fail inside of the + ** sqlite3ThreadData() call, as the ThreadData structure must have already + ** been allocated for pTsdro->useSharedData to be non-zero. + */ + if( pTsdro->useSharedData && zFilename && !isMemdb ){ + pBt->pNext = pTsdro->pBtree; + sqlite3ThreadData()->pBtree = pBt; + } +#endif + pBt->nRef = 1; + *ppBtree = p; + return SQLITE_OK; +} + +/* +** Close an open database and invalidate all cursors. +*/ +int sqlite3BtreeClose(Btree *p){ + BtShared *pBt = p->pBt; + BtCursor *pCur; + +#ifndef SQLITE_OMIT_SHARED_CACHE + ThreadData *pTsd; +#endif + + /* Close all cursors opened via this handle. */ + pCur = pBt->pCursor; + while( pCur ){ + BtCursor *pTmp = pCur; + pCur = pCur->pNext; + if( pTmp->pBtree==p ){ + sqlite3BtreeCloseCursor(pTmp); + } + } + + /* Rollback any active transaction and free the handle structure. + ** The call to sqlite3BtreeRollback() drops any table-locks held by + ** this handle. + */ + sqlite3BtreeRollback(p); + sqliteFree(p); + +#ifndef SQLITE_OMIT_SHARED_CACHE + /* If there are still other outstanding references to the shared-btree + ** structure, return now. The remainder of this procedure cleans + ** up the shared-btree. + */ + assert( pBt->nRef>0 ); + pBt->nRef--; + if( pBt->nRef ){ + return SQLITE_OK; + } + + /* Remove the shared-btree from the thread wide list. Call + ** ThreadDataReadOnly() and then cast away the const property of the + ** pointer to avoid allocating thread data if it is not really required. + */ + pTsd = (ThreadData *)sqlite3ThreadDataReadOnly(); + if( pTsd->pBtree==pBt ){ + assert( pTsd==sqlite3ThreadData() ); + pTsd->pBtree = pBt->pNext; + }else{ + BtShared *pPrev; + for(pPrev=pTsd->pBtree; pPrev && pPrev->pNext!=pBt; pPrev=pPrev->pNext); + if( pPrev ){ + assert( pTsd==sqlite3ThreadData() ); + pPrev->pNext = pBt->pNext; + } + } +#endif + + /* Close the pager and free the shared-btree structure */ + assert( !pBt->pCursor ); + sqlite3pager_close(pBt->pPager); + if( pBt->xFreeSchema && pBt->pSchema ){ + pBt->xFreeSchema(pBt->pSchema); + } + sqliteFree(pBt->pSchema); + sqliteFree(pBt); + return SQLITE_OK; +} + +/* +** Change the busy handler callback function. +*/ +int sqlite3BtreeSetBusyHandler(Btree *p, BusyHandler *pHandler){ + BtShared *pBt = p->pBt; + pBt->pBusyHandler = pHandler; + sqlite3pager_set_busyhandler(pBt->pPager, pHandler); + return SQLITE_OK; +} + +/* +** Change the limit on the number of pages allowed in the cache. +** +** The maximum number of cache pages is set to the absolute +** value of mxPage. If mxPage is negative, the pager will +** operate asynchronously - it will not stop to do fsync()s +** to insure data is written to the disk surface before +** continuing. Transactions still work if synchronous is off, +** and the database cannot be corrupted if this program +** crashes. But if the operating system crashes or there is +** an abrupt power failure when synchronous is off, the database +** could be left in an inconsistent and unrecoverable state. +** Synchronous is on by default so database corruption is not +** normally a worry. +*/ +int sqlite3BtreeSetCacheSize(Btree *p, int mxPage){ + BtShared *pBt = p->pBt; + sqlite3pager_set_cachesize(pBt->pPager, mxPage); + return SQLITE_OK; +} + +/* +** Change the way data is synced to disk in order to increase or decrease +** how well the database resists damage due to OS crashes and power +** failures. Level 1 is the same as asynchronous (no syncs() occur and +** there is a high probability of damage) Level 2 is the default. There +** is a very low but non-zero probability of damage. Level 3 reduces the +** probability of damage to near zero but with a write performance reduction. +*/ +#ifndef SQLITE_OMIT_PAGER_PRAGMAS +int sqlite3BtreeSetSafetyLevel(Btree *p, int level, int fullSync){ + BtShared *pBt = p->pBt; + sqlite3pager_set_safety_level(pBt->pPager, level, fullSync); + return SQLITE_OK; +} +#endif + +/* +** Return TRUE if the given btree is set to safety level 1. In other +** words, return TRUE if no sync() occurs on the disk files. +*/ +int sqlite3BtreeSyncDisabled(Btree *p){ + BtShared *pBt = p->pBt; + assert( pBt && pBt->pPager ); + return sqlite3pager_nosync(pBt->pPager); +} + +#if !defined(SQLITE_OMIT_PAGER_PRAGMAS) || !defined(SQLITE_OMIT_VACUUM) +/* +** Change the default pages size and the number of reserved bytes per page. +** +** The page size must be a power of 2 between 512 and 65536. If the page +** size supplied does not meet this constraint then the page size is not +** changed. +** +** Page sizes are constrained to be a power of two so that the region +** of the database file used for locking (beginning at PENDING_BYTE, +** the first byte past the 1GB boundary, 0x40000000) needs to occur +** at the beginning of a page. +** +** If parameter nReserve is less than zero, then the number of reserved +** bytes per page is left unchanged. +*/ +int sqlite3BtreeSetPageSize(Btree *p, int pageSize, int nReserve){ + BtShared *pBt = p->pBt; + if( pBt->pageSizeFixed ){ + return SQLITE_READONLY; + } + if( nReserve<0 ){ + nReserve = pBt->pageSize - pBt->usableSize; + } + if( pageSize>=512 && pageSize<=SQLITE_MAX_PAGE_SIZE && + ((pageSize-1)&pageSize)==0 ){ + assert( (pageSize & 7)==0 ); + assert( !pBt->pPage1 && !pBt->pCursor ); + pBt->pageSize = sqlite3pager_set_pagesize(pBt->pPager, pageSize); + } + pBt->usableSize = pBt->pageSize - nReserve; + return SQLITE_OK; +} + +/* +** Return the currently defined page size +*/ +int sqlite3BtreeGetPageSize(Btree *p){ + return p->pBt->pageSize; +} +int sqlite3BtreeGetReserve(Btree *p){ + return p->pBt->pageSize - p->pBt->usableSize; +} +#endif /* !defined(SQLITE_OMIT_PAGER_PRAGMAS) || !defined(SQLITE_OMIT_VACUUM) */ + +/* +** Change the 'auto-vacuum' property of the database. If the 'autoVacuum' +** parameter is non-zero, then auto-vacuum mode is enabled. If zero, it +** is disabled. The default value for the auto-vacuum property is +** determined by the SQLITE_DEFAULT_AUTOVACUUM macro. +*/ +int sqlite3BtreeSetAutoVacuum(Btree *p, int autoVacuum){ + BtShared *pBt = p->pBt;; +#ifdef SQLITE_OMIT_AUTOVACUUM + return SQLITE_READONLY; +#else + if( pBt->pageSizeFixed ){ + return SQLITE_READONLY; + } + pBt->autoVacuum = (autoVacuum?1:0); + return SQLITE_OK; +#endif +} + +/* +** Return the value of the 'auto-vacuum' property. If auto-vacuum is +** enabled 1 is returned. Otherwise 0. +*/ +int sqlite3BtreeGetAutoVacuum(Btree *p){ +#ifdef SQLITE_OMIT_AUTOVACUUM + return 0; +#else + return p->pBt->autoVacuum; +#endif +} + + +/* +** Get a reference to pPage1 of the database file. This will +** also acquire a readlock on that file. +** +** SQLITE_OK is returned on success. If the file is not a +** well-formed database file, then SQLITE_CORRUPT is returned. +** SQLITE_BUSY is returned if the database is locked. SQLITE_NOMEM +** is returned if we run out of memory. SQLITE_PROTOCOL is returned +** if there is a locking protocol violation. +*/ +static int lockBtree(BtShared *pBt){ + int rc, pageSize; + MemPage *pPage1; + if( pBt->pPage1 ) return SQLITE_OK; + rc = getPage(pBt, 1, &pPage1); + if( rc!=SQLITE_OK ) return rc; + + + /* Do some checking to help insure the file we opened really is + ** a valid database file. + */ + rc = SQLITE_NOTADB; + if( sqlite3pager_pagecount(pBt->pPager)>0 ){ + u8 *page1 = pPage1->aData; + if( memcmp(page1, zMagicHeader, 16)!=0 ){ + goto page1_init_failed; + } + if( page1[18]>1 || page1[19]>1 ){ + goto page1_init_failed; + } + pageSize = get2byte(&page1[16]); + if( ((pageSize-1)&pageSize)!=0 ){ + goto page1_init_failed; + } + assert( (pageSize & 7)==0 ); + pBt->pageSize = pageSize; + pBt->usableSize = pageSize - page1[20]; + if( pBt->usableSize<500 ){ + goto page1_init_failed; + } + pBt->maxEmbedFrac = page1[21]; + pBt->minEmbedFrac = page1[22]; + pBt->minLeafFrac = page1[23]; +#ifndef SQLITE_OMIT_AUTOVACUUM + pBt->autoVacuum = (get4byte(&page1[36 + 4*4])?1:0); +#endif + } + + /* maxLocal is the maximum amount of payload to store locally for + ** a cell. Make sure it is small enough so that at least minFanout + ** cells can will fit on one page. We assume a 10-byte page header. + ** Besides the payload, the cell must store: + ** 2-byte pointer to the cell + ** 4-byte child pointer + ** 9-byte nKey value + ** 4-byte nData value + ** 4-byte overflow page pointer + ** So a cell consists of a 2-byte poiner, a header which is as much as + ** 17 bytes long, 0 to N bytes of payload, and an optional 4 byte overflow + ** page pointer. + */ + pBt->maxLocal = (pBt->usableSize-12)*pBt->maxEmbedFrac/255 - 23; + pBt->minLocal = (pBt->usableSize-12)*pBt->minEmbedFrac/255 - 23; + pBt->maxLeaf = pBt->usableSize - 35; + pBt->minLeaf = (pBt->usableSize-12)*pBt->minLeafFrac/255 - 23; + if( pBt->minLocal>pBt->maxLocal || pBt->maxLocal<0 ){ + goto page1_init_failed; + } + assert( pBt->maxLeaf + 23 <= MX_CELL_SIZE(pBt) ); + pBt->pPage1 = pPage1; + return SQLITE_OK; + +page1_init_failed: + releasePage(pPage1); + pBt->pPage1 = 0; + return rc; +} + +/* +** This routine works like lockBtree() except that it also invokes the +** busy callback if there is lock contention. +*/ +static int lockBtreeWithRetry(Btree *pRef){ + int rc = SQLITE_OK; + if( pRef->inTrans==TRANS_NONE ){ + u8 inTransaction = pRef->pBt->inTransaction; + btreeIntegrity(pRef); + rc = sqlite3BtreeBeginTrans(pRef, 0); + pRef->pBt->inTransaction = inTransaction; + pRef->inTrans = TRANS_NONE; + if( rc==SQLITE_OK ){ + pRef->pBt->nTransaction--; + } + btreeIntegrity(pRef); + } + return rc; +} + + +/* +** If there are no outstanding cursors and we are not in the middle +** of a transaction but there is a read lock on the database, then +** this routine unrefs the first page of the database file which +** has the effect of releasing the read lock. +** +** If there are any outstanding cursors, this routine is a no-op. +** +** If there is a transaction in progress, this routine is a no-op. +*/ +static void unlockBtreeIfUnused(BtShared *pBt){ + if( pBt->inTransaction==TRANS_NONE && pBt->pCursor==0 && pBt->pPage1!=0 ){ + if( pBt->pPage1->aData==0 ){ + MemPage *pPage = pBt->pPage1; + pPage->aData = &((u8*)pPage)[-pBt->pageSize]; + pPage->pBt = pBt; + pPage->pgno = 1; + } + releasePage(pBt->pPage1); + pBt->pPage1 = 0; + pBt->inStmt = 0; + } +} + +/* +** Create a new database by initializing the first page of the +** file. +*/ +static int newDatabase(BtShared *pBt){ + MemPage *pP1; + unsigned char *data; + int rc; + if( sqlite3pager_pagecount(pBt->pPager)>0 ) return SQLITE_OK; + pP1 = pBt->pPage1; + assert( pP1!=0 ); + data = pP1->aData; + rc = sqlite3pager_write(data); + if( rc ) return rc; + memcpy(data, zMagicHeader, sizeof(zMagicHeader)); + assert( sizeof(zMagicHeader)==16 ); + put2byte(&data[16], pBt->pageSize); + data[18] = 1; + data[19] = 1; + data[20] = pBt->pageSize - pBt->usableSize; + data[21] = pBt->maxEmbedFrac; + data[22] = pBt->minEmbedFrac; + data[23] = pBt->minLeafFrac; + memset(&data[24], 0, 100-24); + zeroPage(pP1, PTF_INTKEY|PTF_LEAF|PTF_LEAFDATA ); + pBt->pageSizeFixed = 1; +#ifndef SQLITE_OMIT_AUTOVACUUM + if( pBt->autoVacuum ){ + put4byte(&data[36 + 4*4], 1); + } +#endif + return SQLITE_OK; +} + +/* +** Attempt to start a new transaction. A write-transaction +** is started if the second argument is nonzero, otherwise a read- +** transaction. If the second argument is 2 or more and exclusive +** transaction is started, meaning that no other process is allowed +** to access the database. A preexisting transaction may not be +** upgraded to exclusive by calling this routine a second time - the +** exclusivity flag only works for a new transaction. +** +** A write-transaction must be started before attempting any +** changes to the database. None of the following routines +** will work unless a transaction is started first: +** +** sqlite3BtreeCreateTable() +** sqlite3BtreeCreateIndex() +** sqlite3BtreeClearTable() +** sqlite3BtreeDropTable() +** sqlite3BtreeInsert() +** sqlite3BtreeDelete() +** sqlite3BtreeUpdateMeta() +** +** If an initial attempt to acquire the lock fails because of lock contention +** and the database was previously unlocked, then invoke the busy handler +** if there is one. But if there was previously a read-lock, do not +** invoke the busy handler - just return SQLITE_BUSY. SQLITE_BUSY is +** returned when there is already a read-lock in order to avoid a deadlock. +** +** Suppose there are two processes A and B. A has a read lock and B has +** a reserved lock. B tries to promote to exclusive but is blocked because +** of A's read lock. A tries to promote to reserved but is blocked by B. +** One or the other of the two processes must give way or there can be +** no progress. By returning SQLITE_BUSY and not invoking the busy callback +** when A already has a read lock, we encourage A to give up and let B +** proceed. +*/ +int sqlite3BtreeBeginTrans(Btree *p, int wrflag){ + BtShared *pBt = p->pBt; + int rc = SQLITE_OK; + + btreeIntegrity(p); + + /* If the btree is already in a write-transaction, or it + ** is already in a read-transaction and a read-transaction + ** is requested, this is a no-op. + */ + if( p->inTrans==TRANS_WRITE || (p->inTrans==TRANS_READ && !wrflag) ){ + return SQLITE_OK; + } + + /* Write transactions are not possible on a read-only database */ + if( pBt->readOnly && wrflag ){ + return SQLITE_READONLY; + } + + /* If another database handle has already opened a write transaction + ** on this shared-btree structure and a second write transaction is + ** requested, return SQLITE_BUSY. + */ + if( pBt->inTransaction==TRANS_WRITE && wrflag ){ + return SQLITE_BUSY; + } + + do { + if( pBt->pPage1==0 ){ + rc = lockBtree(pBt); + } + + if( rc==SQLITE_OK && wrflag ){ + rc = sqlite3pager_begin(pBt->pPage1->aData, wrflag>1); + if( rc==SQLITE_OK ){ + rc = newDatabase(pBt); + } + } + + if( rc==SQLITE_OK ){ + if( wrflag ) pBt->inStmt = 0; + }else{ + unlockBtreeIfUnused(pBt); + } + }while( rc==SQLITE_BUSY && pBt->inTransaction==TRANS_NONE && + sqlite3InvokeBusyHandler(pBt->pBusyHandler) ); + + if( rc==SQLITE_OK ){ + if( p->inTrans==TRANS_NONE ){ + pBt->nTransaction++; + } + p->inTrans = (wrflag?TRANS_WRITE:TRANS_READ); + if( p->inTrans>pBt->inTransaction ){ + pBt->inTransaction = p->inTrans; + } + } + + btreeIntegrity(p); + return rc; +} + +#ifndef SQLITE_OMIT_AUTOVACUUM + +/* +** Set the pointer-map entries for all children of page pPage. Also, if +** pPage contains cells that point to overflow pages, set the pointer +** map entries for the overflow pages as well. +*/ +static int setChildPtrmaps(MemPage *pPage){ + int i; /* Counter variable */ + int nCell; /* Number of cells in page pPage */ + int rc = SQLITE_OK; /* Return code */ + BtShared *pBt = pPage->pBt; + int isInitOrig = pPage->isInit; + Pgno pgno = pPage->pgno; + + initPage(pPage, 0); + nCell = pPage->nCell; + + for(i=0; ileaf ){ + Pgno childPgno = get4byte(pCell); + rc = ptrmapPut(pBt, childPgno, PTRMAP_BTREE, pgno); + if( rc!=SQLITE_OK ) goto set_child_ptrmaps_out; + } + } + + if( !pPage->leaf ){ + Pgno childPgno = get4byte(&pPage->aData[pPage->hdrOffset+8]); + rc = ptrmapPut(pBt, childPgno, PTRMAP_BTREE, pgno); + } + +set_child_ptrmaps_out: + pPage->isInit = isInitOrig; + return rc; +} + +/* +** Somewhere on pPage, which is guarenteed to be a btree page, not an overflow +** page, is a pointer to page iFrom. Modify this pointer so that it points to +** iTo. Parameter eType describes the type of pointer to be modified, as +** follows: +** +** PTRMAP_BTREE: pPage is a btree-page. The pointer points at a child +** page of pPage. +** +** PTRMAP_OVERFLOW1: pPage is a btree-page. The pointer points at an overflow +** page pointed to by one of the cells on pPage. +** +** PTRMAP_OVERFLOW2: pPage is an overflow-page. The pointer points at the next +** overflow page in the list. +*/ +static int modifyPagePointer(MemPage *pPage, Pgno iFrom, Pgno iTo, u8 eType){ + if( eType==PTRMAP_OVERFLOW2 ){ + /* The pointer is always the first 4 bytes of the page in this case. */ + if( get4byte(pPage->aData)!=iFrom ){ + return SQLITE_CORRUPT_BKPT; + } + put4byte(pPage->aData, iTo); + }else{ + int isInitOrig = pPage->isInit; + int i; + int nCell; + + initPage(pPage, 0); + nCell = pPage->nCell; + + for(i=0; iaData[pPage->hdrOffset+8])!=iFrom ){ + return SQLITE_CORRUPT_BKPT; + } + put4byte(&pPage->aData[pPage->hdrOffset+8], iTo); + } + + pPage->isInit = isInitOrig; + } + return SQLITE_OK; +} + + +/* +** Move the open database page pDbPage to location iFreePage in the +** database. The pDbPage reference remains valid. +*/ +static int relocatePage( + BtShared *pBt, /* Btree */ + MemPage *pDbPage, /* Open page to move */ + u8 eType, /* Pointer map 'type' entry for pDbPage */ + Pgno iPtrPage, /* Pointer map 'page-no' entry for pDbPage */ + Pgno iFreePage /* The location to move pDbPage to */ +){ + MemPage *pPtrPage; /* The page that contains a pointer to pDbPage */ + Pgno iDbPage = pDbPage->pgno; + Pager *pPager = pBt->pPager; + int rc; + + assert( eType==PTRMAP_OVERFLOW2 || eType==PTRMAP_OVERFLOW1 || + eType==PTRMAP_BTREE || eType==PTRMAP_ROOTPAGE ); + + /* Move page iDbPage from it's current location to page number iFreePage */ + TRACE(("AUTOVACUUM: Moving %d to free page %d (ptr page %d type %d)\n", + iDbPage, iFreePage, iPtrPage, eType)); + rc = sqlite3pager_movepage(pPager, pDbPage->aData, iFreePage); + if( rc!=SQLITE_OK ){ + return rc; + } + pDbPage->pgno = iFreePage; + + /* If pDbPage was a btree-page, then it may have child pages and/or cells + ** that point to overflow pages. The pointer map entries for all these + ** pages need to be changed. + ** + ** If pDbPage is an overflow page, then the first 4 bytes may store a + ** pointer to a subsequent overflow page. If this is the case, then + ** the pointer map needs to be updated for the subsequent overflow page. + */ + if( eType==PTRMAP_BTREE || eType==PTRMAP_ROOTPAGE ){ + rc = setChildPtrmaps(pDbPage); + if( rc!=SQLITE_OK ){ + return rc; + } + }else{ + Pgno nextOvfl = get4byte(pDbPage->aData); + if( nextOvfl!=0 ){ + rc = ptrmapPut(pBt, nextOvfl, PTRMAP_OVERFLOW2, iFreePage); + if( rc!=SQLITE_OK ){ + return rc; + } + } + } + + /* Fix the database pointer on page iPtrPage that pointed at iDbPage so + ** that it points at iFreePage. Also fix the pointer map entry for + ** iPtrPage. + */ + if( eType!=PTRMAP_ROOTPAGE ){ + rc = getPage(pBt, iPtrPage, &pPtrPage); + if( rc!=SQLITE_OK ){ + return rc; + } + rc = sqlite3pager_write(pPtrPage->aData); + if( rc!=SQLITE_OK ){ + releasePage(pPtrPage); + return rc; + } + rc = modifyPagePointer(pPtrPage, iDbPage, iFreePage, eType); + releasePage(pPtrPage); + if( rc==SQLITE_OK ){ + rc = ptrmapPut(pBt, iFreePage, eType, iPtrPage); + } + } + return rc; +} + +/* Forward declaration required by autoVacuumCommit(). */ +static int allocatePage(BtShared *, MemPage **, Pgno *, Pgno, u8); + +/* +** This routine is called prior to sqlite3pager_commit when a transaction +** is commited for an auto-vacuum database. +*/ +static int autoVacuumCommit(BtShared *pBt, Pgno *nTrunc){ + Pager *pPager = pBt->pPager; + Pgno nFreeList; /* Number of pages remaining on the free-list. */ + int nPtrMap; /* Number of pointer-map pages deallocated */ + Pgno origSize; /* Pages in the database file */ + Pgno finSize; /* Pages in the database file after truncation */ + int rc; /* Return code */ + u8 eType; + int pgsz = pBt->pageSize; /* Page size for this database */ + Pgno iDbPage; /* The database page to move */ + MemPage *pDbMemPage = 0; /* "" */ + Pgno iPtrPage; /* The page that contains a pointer to iDbPage */ + Pgno iFreePage; /* The free-list page to move iDbPage to */ + MemPage *pFreeMemPage = 0; /* "" */ + +#ifndef NDEBUG + int nRef = *sqlite3pager_stats(pPager); +#endif + + assert( pBt->autoVacuum ); + if( PTRMAP_ISPAGE(pBt, sqlite3pager_pagecount(pPager)) ){ + return SQLITE_CORRUPT_BKPT; + } + + /* Figure out how many free-pages are in the database. If there are no + ** free pages, then auto-vacuum is a no-op. + */ + nFreeList = get4byte(&pBt->pPage1->aData[36]); + if( nFreeList==0 ){ + *nTrunc = 0; + return SQLITE_OK; + } + + /* This block figures out how many pages there are in the database + ** now (variable origSize), and how many there will be after the + ** truncation (variable finSize). + ** + ** The final size is the original size, less the number of free pages + ** in the database, less any pointer-map pages that will no longer + ** be required, less 1 if the pending-byte page was part of the database + ** but is not after the truncation. + **/ + origSize = sqlite3pager_pagecount(pPager); + if( origSize==PENDING_BYTE_PAGE(pBt) ){ + origSize--; + } + nPtrMap = (nFreeList-origSize+PTRMAP_PAGENO(pBt, origSize)+pgsz/5)/(pgsz/5); + finSize = origSize - nFreeList - nPtrMap; + if( origSize>PENDING_BYTE_PAGE(pBt) && finSize<=PENDING_BYTE_PAGE(pBt) ){ + finSize--; + } + while( PTRMAP_ISPAGE(pBt, finSize) || finSize==PENDING_BYTE_PAGE(pBt) ){ + finSize--; + } + TRACE(("AUTOVACUUM: Begin (db size %d->%d)\n", origSize, finSize)); + + /* Variable 'finSize' will be the size of the file in pages after + ** the auto-vacuum has completed (the current file size minus the number + ** of pages on the free list). Loop through the pages that lie beyond + ** this mark, and if they are not already on the free list, move them + ** to a free page earlier in the file (somewhere before finSize). + */ + for( iDbPage=finSize+1; iDbPage<=origSize; iDbPage++ ){ + /* If iDbPage is a pointer map page, or the pending-byte page, skip it. */ + if( PTRMAP_ISPAGE(pBt, iDbPage) || iDbPage==PENDING_BYTE_PAGE(pBt) ){ + continue; + } + + rc = ptrmapGet(pBt, iDbPage, &eType, &iPtrPage); + if( rc!=SQLITE_OK ) goto autovacuum_out; + if( eType==PTRMAP_ROOTPAGE ){ + rc = SQLITE_CORRUPT_BKPT; + goto autovacuum_out; + } + + /* If iDbPage is free, do not swap it. */ + if( eType==PTRMAP_FREEPAGE ){ + continue; + } + rc = getPage(pBt, iDbPage, &pDbMemPage); + if( rc!=SQLITE_OK ) goto autovacuum_out; + + /* Find the next page in the free-list that is not already at the end + ** of the file. A page can be pulled off the free list using the + ** allocatePage() routine. + */ + do{ + if( pFreeMemPage ){ + releasePage(pFreeMemPage); + pFreeMemPage = 0; + } + rc = allocatePage(pBt, &pFreeMemPage, &iFreePage, 0, 0); + if( rc!=SQLITE_OK ){ + releasePage(pDbMemPage); + goto autovacuum_out; + } + assert( iFreePage<=origSize ); + }while( iFreePage>finSize ); + releasePage(pFreeMemPage); + pFreeMemPage = 0; + + /* Relocate the page into the body of the file. Note that although the + ** page has moved within the database file, the pDbMemPage pointer + ** remains valid. This means that this function can run without + ** invalidating cursors open on the btree. This is important in + ** shared-cache mode. + */ + rc = relocatePage(pBt, pDbMemPage, eType, iPtrPage, iFreePage); + releasePage(pDbMemPage); + if( rc!=SQLITE_OK ) goto autovacuum_out; + } + + /* The entire free-list has been swapped to the end of the file. So + ** truncate the database file to finSize pages and consider the + ** free-list empty. + */ + rc = sqlite3pager_write(pBt->pPage1->aData); + if( rc!=SQLITE_OK ) goto autovacuum_out; + put4byte(&pBt->pPage1->aData[32], 0); + put4byte(&pBt->pPage1->aData[36], 0); + if( rc!=SQLITE_OK ) goto autovacuum_out; + *nTrunc = finSize; + assert( finSize!=PENDING_BYTE_PAGE(pBt) ); + +autovacuum_out: + assert( nRef==*sqlite3pager_stats(pPager) ); + if( rc!=SQLITE_OK ){ + sqlite3pager_rollback(pPager); + } + return rc; +} +#endif + +/* +** Commit the transaction currently in progress. +** +** This will release the write lock on the database file. If there +** are no active cursors, it also releases the read lock. +*/ +int sqlite3BtreeCommit(Btree *p){ + BtShared *pBt = p->pBt; + + btreeIntegrity(p); + + /* If the handle has a write-transaction open, commit the shared-btrees + ** transaction and set the shared state to TRANS_READ. + */ + if( p->inTrans==TRANS_WRITE ){ + int rc; + assert( pBt->inTransaction==TRANS_WRITE ); + assert( pBt->nTransaction>0 ); + rc = sqlite3pager_commit(pBt->pPager); + if( rc!=SQLITE_OK ){ + return rc; + } + pBt->inTransaction = TRANS_READ; + pBt->inStmt = 0; + } + unlockAllTables(p); + + /* If the handle has any kind of transaction open, decrement the transaction + ** count of the shared btree. If the transaction count reaches 0, set + ** the shared state to TRANS_NONE. The unlockBtreeIfUnused() call below + ** will unlock the pager. + */ + if( p->inTrans!=TRANS_NONE ){ + pBt->nTransaction--; + if( 0==pBt->nTransaction ){ + pBt->inTransaction = TRANS_NONE; + } + } + + /* Set the handles current transaction state to TRANS_NONE and unlock + ** the pager if this call closed the only read or write transaction. + */ + p->inTrans = TRANS_NONE; + unlockBtreeIfUnused(pBt); + + btreeIntegrity(p); + return SQLITE_OK; +} + +#ifndef NDEBUG +/* +** Return the number of write-cursors open on this handle. This is for use +** in assert() expressions, so it is only compiled if NDEBUG is not +** defined. +*/ +static int countWriteCursors(BtShared *pBt){ + BtCursor *pCur; + int r = 0; + for(pCur=pBt->pCursor; pCur; pCur=pCur->pNext){ + if( pCur->wrFlag ) r++; + } + return r; +} +#endif + +#if defined(SQLITE_TEST) && defined(SQLITE_DEBUG) +/* +** Print debugging information about all cursors to standard output. +*/ +void sqlite3BtreeCursorList(Btree *p){ + BtCursor *pCur; + BtShared *pBt = p->pBt; + for(pCur=pBt->pCursor; pCur; pCur=pCur->pNext){ + MemPage *pPage = pCur->pPage; + char *zMode = pCur->wrFlag ? "rw" : "ro"; + sqlite3DebugPrintf("CURSOR %p rooted at %4d(%s) currently at %d.%d%s\n", + pCur, pCur->pgnoRoot, zMode, + pPage ? pPage->pgno : 0, pCur->idx, + (pCur->eState==CURSOR_VALID) ? "" : " eof" + ); + } +} +#endif + +/* +** Rollback the transaction in progress. All cursors will be +** invalided by this operation. Any attempt to use a cursor +** that was open at the beginning of this operation will result +** in an error. +** +** This will release the write lock on the database file. If there +** are no active cursors, it also releases the read lock. +*/ +int sqlite3BtreeRollback(Btree *p){ + int rc; + BtShared *pBt = p->pBt; + MemPage *pPage1; + + rc = saveAllCursors(pBt, 0, 0); +#ifndef SQLITE_OMIT_SHARED_CACHE + if( rc!=SQLITE_OK ){ + /* This is a horrible situation. An IO or malloc() error occured whilst + ** trying to save cursor positions. If this is an automatic rollback (as + ** the result of a constraint, malloc() failure or IO error) then + ** the cache may be internally inconsistent (not contain valid trees) so + ** we cannot simply return the error to the caller. Instead, abort + ** all queries that may be using any of the cursors that failed to save. + */ + while( pBt->pCursor ){ + sqlite3 *db = pBt->pCursor->pBtree->pSqlite; + if( db ){ + sqlite3AbortOtherActiveVdbes(db, 0); + } + } + } +#endif + btreeIntegrity(p); + unlockAllTables(p); + + if( p->inTrans==TRANS_WRITE ){ + int rc2; + + assert( TRANS_WRITE==pBt->inTransaction ); + rc2 = sqlite3pager_rollback(pBt->pPager); + if( rc2!=SQLITE_OK ){ + rc = rc2; + } + + /* The rollback may have destroyed the pPage1->aData value. So + ** call getPage() on page 1 again to make sure pPage1->aData is + ** set correctly. */ + if( getPage(pBt, 1, &pPage1)==SQLITE_OK ){ + releasePage(pPage1); + } + assert( countWriteCursors(pBt)==0 ); + pBt->inTransaction = TRANS_READ; + } + + if( p->inTrans!=TRANS_NONE ){ + assert( pBt->nTransaction>0 ); + pBt->nTransaction--; + if( 0==pBt->nTransaction ){ + pBt->inTransaction = TRANS_NONE; + } + } + + p->inTrans = TRANS_NONE; + pBt->inStmt = 0; + unlockBtreeIfUnused(pBt); + + btreeIntegrity(p); + return rc; +} + +/* +** Start a statement subtransaction. The subtransaction can +** can be rolled back independently of the main transaction. +** You must start a transaction before starting a subtransaction. +** The subtransaction is ended automatically if the main transaction +** commits or rolls back. +** +** Only one subtransaction may be active at a time. It is an error to try +** to start a new subtransaction if another subtransaction is already active. +** +** Statement subtransactions are used around individual SQL statements +** that are contained within a BEGIN...COMMIT block. If a constraint +** error occurs within the statement, the effect of that one statement +** can be rolled back without having to rollback the entire transaction. +*/ +int sqlite3BtreeBeginStmt(Btree *p){ + int rc; + BtShared *pBt = p->pBt; + if( (p->inTrans!=TRANS_WRITE) || pBt->inStmt ){ + return pBt->readOnly ? SQLITE_READONLY : SQLITE_ERROR; + } + assert( pBt->inTransaction==TRANS_WRITE ); + rc = pBt->readOnly ? SQLITE_OK : sqlite3pager_stmt_begin(pBt->pPager); + pBt->inStmt = 1; + return rc; +} + + +/* +** Commit the statment subtransaction currently in progress. If no +** subtransaction is active, this is a no-op. +*/ +int sqlite3BtreeCommitStmt(Btree *p){ + int rc; + BtShared *pBt = p->pBt; + if( pBt->inStmt && !pBt->readOnly ){ + rc = sqlite3pager_stmt_commit(pBt->pPager); + }else{ + rc = SQLITE_OK; + } + pBt->inStmt = 0; + return rc; +} + +/* +** Rollback the active statement subtransaction. If no subtransaction +** is active this routine is a no-op. +** +** All cursors will be invalidated by this operation. Any attempt +** to use a cursor that was open at the beginning of this operation +** will result in an error. +*/ +int sqlite3BtreeRollbackStmt(Btree *p){ + int rc = SQLITE_OK; + BtShared *pBt = p->pBt; + sqlite3MallocDisallow(); + if( pBt->inStmt && !pBt->readOnly ){ + rc = sqlite3pager_stmt_rollback(pBt->pPager); + assert( countWriteCursors(pBt)==0 ); + pBt->inStmt = 0; + } + sqlite3MallocAllow(); + return rc; +} + +/* +** Default key comparison function to be used if no comparison function +** is specified on the sqlite3BtreeCursor() call. +*/ +static int dfltCompare( + void *NotUsed, /* User data is not used */ + int n1, const void *p1, /* First key to compare */ + int n2, const void *p2 /* Second key to compare */ +){ + int c; + c = memcmp(p1, p2, n1pBt; + + *ppCur = 0; + if( wrFlag ){ + if( pBt->readOnly ){ + return SQLITE_READONLY; + } + if( checkReadLocks(pBt, iTable, 0) ){ + return SQLITE_LOCKED; + } + } + + if( pBt->pPage1==0 ){ + rc = lockBtreeWithRetry(p); + if( rc!=SQLITE_OK ){ + return rc; + } + } + pCur = sqliteMalloc( sizeof(*pCur) ); + if( pCur==0 ){ + rc = SQLITE_NOMEM; + goto create_cursor_exception; + } + pCur->pgnoRoot = (Pgno)iTable; + if( iTable==1 && sqlite3pager_pagecount(pBt->pPager)==0 ){ + rc = SQLITE_EMPTY; + goto create_cursor_exception; + } + rc = getAndInitPage(pBt, pCur->pgnoRoot, &pCur->pPage, 0); + if( rc!=SQLITE_OK ){ + goto create_cursor_exception; + } + + /* Now that no other errors can occur, finish filling in the BtCursor + ** variables, link the cursor into the BtShared list and set *ppCur (the + ** output argument to this function). + */ + pCur->xCompare = xCmp ? xCmp : dfltCompare; + pCur->pArg = pArg; + pCur->pBtree = p; + pCur->wrFlag = wrFlag; + pCur->pNext = pBt->pCursor; + if( pCur->pNext ){ + pCur->pNext->pPrev = pCur; + } + pBt->pCursor = pCur; + pCur->eState = CURSOR_INVALID; + *ppCur = pCur; + + return SQLITE_OK; +create_cursor_exception: + if( pCur ){ + releasePage(pCur->pPage); + sqliteFree(pCur); + } + unlockBtreeIfUnused(pBt); + return rc; +} + +#if 0 /* Not Used */ +/* +** Change the value of the comparison function used by a cursor. +*/ +void sqlite3BtreeSetCompare( + BtCursor *pCur, /* The cursor to whose comparison function is changed */ + int(*xCmp)(void*,int,const void*,int,const void*), /* New comparison func */ + void *pArg /* First argument to xCmp() */ +){ + pCur->xCompare = xCmp ? xCmp : dfltCompare; + pCur->pArg = pArg; +} +#endif + +/* +** Close a cursor. The read lock on the database file is released +** when the last cursor is closed. +*/ +int sqlite3BtreeCloseCursor(BtCursor *pCur){ + BtShared *pBt = pCur->pBtree->pBt; + restoreOrClearCursorPosition(pCur, 0); + if( pCur->pPrev ){ + pCur->pPrev->pNext = pCur->pNext; + }else{ + pBt->pCursor = pCur->pNext; + } + if( pCur->pNext ){ + pCur->pNext->pPrev = pCur->pPrev; + } + releasePage(pCur->pPage); + unlockBtreeIfUnused(pBt); + sqliteFree(pCur); + return SQLITE_OK; +} + +/* +** Make a temporary cursor by filling in the fields of pTempCur. +** The temporary cursor is not on the cursor list for the Btree. +*/ +static void getTempCursor(BtCursor *pCur, BtCursor *pTempCur){ + memcpy(pTempCur, pCur, sizeof(*pCur)); + pTempCur->pNext = 0; + pTempCur->pPrev = 0; + if( pTempCur->pPage ){ + sqlite3pager_ref(pTempCur->pPage->aData); + } +} + +/* +** Delete a temporary cursor such as was made by the CreateTemporaryCursor() +** function above. +*/ +static void releaseTempCursor(BtCursor *pCur){ + if( pCur->pPage ){ + sqlite3pager_unref(pCur->pPage->aData); + } +} + +/* +** Make sure the BtCursor.info field of the given cursor is valid. +** If it is not already valid, call parseCell() to fill it in. +** +** BtCursor.info is a cache of the information in the current cell. +** Using this cache reduces the number of calls to parseCell(). +*/ +static void getCellInfo(BtCursor *pCur){ + if( pCur->info.nSize==0 ){ + parseCell(pCur->pPage, pCur->idx, &pCur->info); + }else{ +#ifndef NDEBUG + CellInfo info; + memset(&info, 0, sizeof(info)); + parseCell(pCur->pPage, pCur->idx, &info); + assert( memcmp(&info, &pCur->info, sizeof(info))==0 ); +#endif + } +} + +/* +** Set *pSize to the size of the buffer needed to hold the value of +** the key for the current entry. If the cursor is not pointing +** to a valid entry, *pSize is set to 0. +** +** For a table with the INTKEY flag set, this routine returns the key +** itself, not the number of bytes in the key. +*/ +int sqlite3BtreeKeySize(BtCursor *pCur, i64 *pSize){ + int rc = restoreOrClearCursorPosition(pCur, 1); + if( rc==SQLITE_OK ){ + assert( pCur->eState==CURSOR_INVALID || pCur->eState==CURSOR_VALID ); + if( pCur->eState==CURSOR_INVALID ){ + *pSize = 0; + }else{ + getCellInfo(pCur); + *pSize = pCur->info.nKey; + } + } + return rc; +} + +/* +** Set *pSize to the number of bytes of data in the entry the +** cursor currently points to. Always return SQLITE_OK. +** Failure is not possible. If the cursor is not currently +** pointing to an entry (which can happen, for example, if +** the database is empty) then *pSize is set to 0. +*/ +int sqlite3BtreeDataSize(BtCursor *pCur, u32 *pSize){ + int rc = restoreOrClearCursorPosition(pCur, 1); + if( rc==SQLITE_OK ){ + assert( pCur->eState==CURSOR_INVALID || pCur->eState==CURSOR_VALID ); + if( pCur->eState==CURSOR_INVALID ){ + /* Not pointing at a valid entry - set *pSize to 0. */ + *pSize = 0; + }else{ + getCellInfo(pCur); + *pSize = pCur->info.nData; + } + } + return rc; +} + +/* +** Read payload information from the entry that the pCur cursor is +** pointing to. Begin reading the payload at "offset" and read +** a total of "amt" bytes. Put the result in zBuf. +** +** This routine does not make a distinction between key and data. +** It just reads bytes from the payload area. Data might appear +** on the main page or be scattered out on multiple overflow pages. +*/ +static int getPayload( + BtCursor *pCur, /* Cursor pointing to entry to read from */ + int offset, /* Begin reading this far into payload */ + int amt, /* Read this many bytes */ + unsigned char *pBuf, /* Write the bytes into this buffer */ + int skipKey /* offset begins at data if this is true */ +){ + unsigned char *aPayload; + Pgno nextPage; + int rc; + MemPage *pPage; + BtShared *pBt; + int ovflSize; + u32 nKey; + + assert( pCur!=0 && pCur->pPage!=0 ); + assert( pCur->eState==CURSOR_VALID ); + pBt = pCur->pBtree->pBt; + pPage = pCur->pPage; + pageIntegrity(pPage); + assert( pCur->idx>=0 && pCur->idxnCell ); + getCellInfo(pCur); + aPayload = pCur->info.pCell + pCur->info.nHeader; + if( pPage->intKey ){ + nKey = 0; + }else{ + nKey = pCur->info.nKey; + } + assert( offset>=0 ); + if( skipKey ){ + offset += nKey; + } + if( offset+amt > nKey+pCur->info.nData ){ + return SQLITE_ERROR; + } + if( offsetinfo.nLocal ){ + int a = amt; + if( a+offset>pCur->info.nLocal ){ + a = pCur->info.nLocal - offset; + } + memcpy(pBuf, &aPayload[offset], a); + if( a==amt ){ + return SQLITE_OK; + } + offset = 0; + pBuf += a; + amt -= a; + }else{ + offset -= pCur->info.nLocal; + } + ovflSize = pBt->usableSize - 4; + if( amt>0 ){ + nextPage = get4byte(&aPayload[pCur->info.nLocal]); + while( amt>0 && nextPage ){ + rc = sqlite3pager_get(pBt->pPager, nextPage, (void**)&aPayload); + if( rc!=0 ){ + return rc; + } + nextPage = get4byte(aPayload); + if( offset ovflSize ){ + a = ovflSize - offset; + } + memcpy(pBuf, &aPayload[offset+4], a); + offset = 0; + amt -= a; + pBuf += a; + }else{ + offset -= ovflSize; + } + sqlite3pager_unref(aPayload); + } + } + + if( amt>0 ){ + return SQLITE_CORRUPT_BKPT; + } + return SQLITE_OK; +} + +/* +** Read part of the key associated with cursor pCur. Exactly +** "amt" bytes will be transfered into pBuf[]. The transfer +** begins at "offset". +** +** Return SQLITE_OK on success or an error code if anything goes +** wrong. An error is returned if "offset+amt" is larger than +** the available payload. +*/ +int sqlite3BtreeKey(BtCursor *pCur, u32 offset, u32 amt, void *pBuf){ + int rc = restoreOrClearCursorPosition(pCur, 1); + if( rc==SQLITE_OK ){ + assert( pCur->eState==CURSOR_VALID ); + assert( pCur->pPage!=0 ); + if( pCur->pPage->intKey ){ + return SQLITE_CORRUPT_BKPT; + } + assert( pCur->pPage->intKey==0 ); + assert( pCur->idx>=0 && pCur->idxpPage->nCell ); + rc = getPayload(pCur, offset, amt, (unsigned char*)pBuf, 0); + } + return rc; +} + +/* +** Read part of the data associated with cursor pCur. Exactly +** "amt" bytes will be transfered into pBuf[]. The transfer +** begins at "offset". +** +** Return SQLITE_OK on success or an error code if anything goes +** wrong. An error is returned if "offset+amt" is larger than +** the available payload. +*/ +int sqlite3BtreeData(BtCursor *pCur, u32 offset, u32 amt, void *pBuf){ + int rc = restoreOrClearCursorPosition(pCur, 1); + if( rc==SQLITE_OK ){ + assert( pCur->eState==CURSOR_VALID ); + assert( pCur->pPage!=0 ); + assert( pCur->idx>=0 && pCur->idxpPage->nCell ); + rc = getPayload(pCur, offset, amt, pBuf, 1); + } + return rc; +} + +/* +** Return a pointer to payload information from the entry that the +** pCur cursor is pointing to. The pointer is to the beginning of +** the key if skipKey==0 and it points to the beginning of data if +** skipKey==1. The number of bytes of available key/data is written +** into *pAmt. If *pAmt==0, then the value returned will not be +** a valid pointer. +** +** This routine is an optimization. It is common for the entire key +** and data to fit on the local page and for there to be no overflow +** pages. When that is so, this routine can be used to access the +** key and data without making a copy. If the key and/or data spills +** onto overflow pages, then getPayload() must be used to reassembly +** the key/data and copy it into a preallocated buffer. +** +** The pointer returned by this routine looks directly into the cached +** page of the database. The data might change or move the next time +** any btree routine is called. +*/ +static const unsigned char *fetchPayload( + BtCursor *pCur, /* Cursor pointing to entry to read from */ + int *pAmt, /* Write the number of available bytes here */ + int skipKey /* read beginning at data if this is true */ +){ + unsigned char *aPayload; + MemPage *pPage; + u32 nKey; + int nLocal; + + assert( pCur!=0 && pCur->pPage!=0 ); + assert( pCur->eState==CURSOR_VALID ); + pPage = pCur->pPage; + pageIntegrity(pPage); + assert( pCur->idx>=0 && pCur->idxnCell ); + getCellInfo(pCur); + aPayload = pCur->info.pCell; + aPayload += pCur->info.nHeader; + if( pPage->intKey ){ + nKey = 0; + }else{ + nKey = pCur->info.nKey; + } + if( skipKey ){ + aPayload += nKey; + nLocal = pCur->info.nLocal - nKey; + }else{ + nLocal = pCur->info.nLocal; + if( nLocal>nKey ){ + nLocal = nKey; + } + } + *pAmt = nLocal; + return aPayload; +} + + +/* +** For the entry that cursor pCur is point to, return as +** many bytes of the key or data as are available on the local +** b-tree page. Write the number of available bytes into *pAmt. +** +** The pointer returned is ephemeral. The key/data may move +** or be destroyed on the next call to any Btree routine. +** +** These routines is used to get quick access to key and data +** in the common case where no overflow pages are used. +*/ +const void *sqlite3BtreeKeyFetch(BtCursor *pCur, int *pAmt){ + if( pCur->eState==CURSOR_VALID ){ + return (const void*)fetchPayload(pCur, pAmt, 0); + } + return 0; +} +const void *sqlite3BtreeDataFetch(BtCursor *pCur, int *pAmt){ + if( pCur->eState==CURSOR_VALID ){ + return (const void*)fetchPayload(pCur, pAmt, 1); + } + return 0; +} + + +/* +** Move the cursor down to a new child page. The newPgno argument is the +** page number of the child page to move to. +*/ +static int moveToChild(BtCursor *pCur, u32 newPgno){ + int rc; + MemPage *pNewPage; + MemPage *pOldPage; + BtShared *pBt = pCur->pBtree->pBt; + + assert( pCur->eState==CURSOR_VALID ); + rc = getAndInitPage(pBt, newPgno, &pNewPage, pCur->pPage); + if( rc ) return rc; + pageIntegrity(pNewPage); + pNewPage->idxParent = pCur->idx; + pOldPage = pCur->pPage; + pOldPage->idxShift = 0; + releasePage(pOldPage); + pCur->pPage = pNewPage; + pCur->idx = 0; + pCur->info.nSize = 0; + if( pNewPage->nCell<1 ){ + return SQLITE_CORRUPT_BKPT; + } + return SQLITE_OK; +} + +/* +** Return true if the page is the virtual root of its table. +** +** The virtual root page is the root page for most tables. But +** for the table rooted on page 1, sometime the real root page +** is empty except for the right-pointer. In such cases the +** virtual root page is the page that the right-pointer of page +** 1 is pointing to. +*/ +static int isRootPage(MemPage *pPage){ + MemPage *pParent = pPage->pParent; + if( pParent==0 ) return 1; + if( pParent->pgno>1 ) return 0; + if( get2byte(&pParent->aData[pParent->hdrOffset+3])==0 ) return 1; + return 0; +} + +/* +** Move the cursor up to the parent page. +** +** pCur->idx is set to the cell index that contains the pointer +** to the page we are coming from. If we are coming from the +** right-most child page then pCur->idx is set to one more than +** the largest cell index. +*/ +static void moveToParent(BtCursor *pCur){ + MemPage *pParent; + MemPage *pPage; + int idxParent; + + assert( pCur->eState==CURSOR_VALID ); + pPage = pCur->pPage; + assert( pPage!=0 ); + assert( !isRootPage(pPage) ); + pageIntegrity(pPage); + pParent = pPage->pParent; + assert( pParent!=0 ); + pageIntegrity(pParent); + idxParent = pPage->idxParent; + sqlite3pager_ref(pParent->aData); + releasePage(pPage); + pCur->pPage = pParent; + pCur->info.nSize = 0; + assert( pParent->idxShift==0 ); + pCur->idx = idxParent; +} + +/* +** Move the cursor to the root page +*/ +static int moveToRoot(BtCursor *pCur){ + MemPage *pRoot; + int rc = SQLITE_OK; + BtShared *pBt = pCur->pBtree->pBt; + + restoreOrClearCursorPosition(pCur, 0); + pRoot = pCur->pPage; + if( pRoot && pRoot->pgno==pCur->pgnoRoot ){ + assert( pRoot->isInit ); + }else{ + if( + SQLITE_OK!=(rc = getAndInitPage(pBt, pCur->pgnoRoot, &pRoot, 0)) + ){ + pCur->eState = CURSOR_INVALID; + return rc; + } + releasePage(pCur->pPage); + pageIntegrity(pRoot); + pCur->pPage = pRoot; + } + pCur->idx = 0; + pCur->info.nSize = 0; + if( pRoot->nCell==0 && !pRoot->leaf ){ + Pgno subpage; + assert( pRoot->pgno==1 ); + subpage = get4byte(&pRoot->aData[pRoot->hdrOffset+8]); + assert( subpage>0 ); + pCur->eState = CURSOR_VALID; + rc = moveToChild(pCur, subpage); + } + pCur->eState = ((pCur->pPage->nCell>0)?CURSOR_VALID:CURSOR_INVALID); + return rc; +} + +/* +** Move the cursor down to the left-most leaf entry beneath the +** entry to which it is currently pointing. +** +** The left-most leaf is the one with the smallest key - the first +** in ascending order. +*/ +static int moveToLeftmost(BtCursor *pCur){ + Pgno pgno; + int rc; + MemPage *pPage; + + assert( pCur->eState==CURSOR_VALID ); + while( !(pPage = pCur->pPage)->leaf ){ + assert( pCur->idx>=0 && pCur->idxnCell ); + pgno = get4byte(findCell(pPage, pCur->idx)); + rc = moveToChild(pCur, pgno); + if( rc ) return rc; + } + return SQLITE_OK; +} + +/* +** Move the cursor down to the right-most leaf entry beneath the +** page to which it is currently pointing. Notice the difference +** between moveToLeftmost() and moveToRightmost(). moveToLeftmost() +** finds the left-most entry beneath the *entry* whereas moveToRightmost() +** finds the right-most entry beneath the *page*. +** +** The right-most entry is the one with the largest key - the last +** key in ascending order. +*/ +static int moveToRightmost(BtCursor *pCur){ + Pgno pgno; + int rc; + MemPage *pPage; + + assert( pCur->eState==CURSOR_VALID ); + while( !(pPage = pCur->pPage)->leaf ){ + pgno = get4byte(&pPage->aData[pPage->hdrOffset+8]); + pCur->idx = pPage->nCell; + rc = moveToChild(pCur, pgno); + if( rc ) return rc; + } + pCur->idx = pPage->nCell - 1; + pCur->info.nSize = 0; + return SQLITE_OK; +} + +/* Move the cursor to the first entry in the table. Return SQLITE_OK +** on success. Set *pRes to 0 if the cursor actually points to something +** or set *pRes to 1 if the table is empty. +*/ +int sqlite3BtreeFirst(BtCursor *pCur, int *pRes){ + int rc; + rc = moveToRoot(pCur); + if( rc ) return rc; + if( pCur->eState==CURSOR_INVALID ){ + assert( pCur->pPage->nCell==0 ); + *pRes = 1; + return SQLITE_OK; + } + assert( pCur->pPage->nCell>0 ); + *pRes = 0; + rc = moveToLeftmost(pCur); + return rc; +} + +/* Move the cursor to the last entry in the table. Return SQLITE_OK +** on success. Set *pRes to 0 if the cursor actually points to something +** or set *pRes to 1 if the table is empty. +*/ +int sqlite3BtreeLast(BtCursor *pCur, int *pRes){ + int rc; + rc = moveToRoot(pCur); + if( rc ) return rc; + if( CURSOR_INVALID==pCur->eState ){ + assert( pCur->pPage->nCell==0 ); + *pRes = 1; + return SQLITE_OK; + } + assert( pCur->eState==CURSOR_VALID ); + *pRes = 0; + rc = moveToRightmost(pCur); + return rc; +} + +/* Move the cursor so that it points to an entry near pKey/nKey. +** Return a success code. +** +** For INTKEY tables, only the nKey parameter is used. pKey is +** ignored. For other tables, nKey is the number of bytes of data +** in pKey. The comparison function specified when the cursor was +** created is used to compare keys. +** +** If an exact match is not found, then the cursor is always +** left pointing at a leaf page which would hold the entry if it +** were present. The cursor might point to an entry that comes +** before or after the key. +** +** The result of comparing the key with the entry to which the +** cursor is written to *pRes if pRes!=NULL. The meaning of +** this value is as follows: +** +** *pRes<0 The cursor is left pointing at an entry that +** is smaller than pKey or if the table is empty +** and the cursor is therefore left point to nothing. +** +** *pRes==0 The cursor is left pointing at an entry that +** exactly matches pKey. +** +** *pRes>0 The cursor is left pointing at an entry that +** is larger than pKey. +*/ +int sqlite3BtreeMoveto(BtCursor *pCur, const void *pKey, i64 nKey, int *pRes){ + int rc; + int tryRightmost; + rc = moveToRoot(pCur); + if( rc ) return rc; + assert( pCur->pPage ); + assert( pCur->pPage->isInit ); + tryRightmost = pCur->pPage->intKey; + if( pCur->eState==CURSOR_INVALID ){ + *pRes = -1; + assert( pCur->pPage->nCell==0 ); + return SQLITE_OK; + } + for(;;){ + int lwr, upr; + Pgno chldPg; + MemPage *pPage = pCur->pPage; + int c = -1; /* pRes return if table is empty must be -1 */ + lwr = 0; + upr = pPage->nCell-1; + if( !pPage->intKey && pKey==0 ){ + return SQLITE_CORRUPT_BKPT; + } + pageIntegrity(pPage); + while( lwr<=upr ){ + void *pCellKey; + i64 nCellKey; + pCur->idx = (lwr+upr)/2; + pCur->info.nSize = 0; + if( pPage->intKey ){ + u8 *pCell; + if( tryRightmost ){ + pCur->idx = upr; + } + pCell = findCell(pPage, pCur->idx) + pPage->childPtrSize; + if( pPage->hasData ){ + u32 dummy; + pCell += getVarint32(pCell, &dummy); + } + getVarint(pCell, (u64 *)&nCellKey); + if( nCellKeynKey ){ + c = +1; + tryRightmost = 0; + }else{ + c = 0; + } + }else{ + int available; + pCellKey = (void *)fetchPayload(pCur, &available, 0); + nCellKey = pCur->info.nKey; + if( available>=nCellKey ){ + c = pCur->xCompare(pCur->pArg, nCellKey, pCellKey, nKey, pKey); + }else{ + pCellKey = sqliteMallocRaw( nCellKey ); + if( pCellKey==0 ) return SQLITE_NOMEM; + rc = sqlite3BtreeKey(pCur, 0, nCellKey, (void *)pCellKey); + c = pCur->xCompare(pCur->pArg, nCellKey, pCellKey, nKey, pKey); + sqliteFree(pCellKey); + if( rc ) return rc; + } + } + if( c==0 ){ + if( pPage->leafData && !pPage->leaf ){ + lwr = pCur->idx; + upr = lwr - 1; + break; + }else{ + if( pRes ) *pRes = 0; + return SQLITE_OK; + } + } + if( c<0 ){ + lwr = pCur->idx+1; + }else{ + upr = pCur->idx-1; + } + } + assert( lwr==upr+1 ); + assert( pPage->isInit ); + if( pPage->leaf ){ + chldPg = 0; + }else if( lwr>=pPage->nCell ){ + chldPg = get4byte(&pPage->aData[pPage->hdrOffset+8]); + }else{ + chldPg = get4byte(findCell(pPage, lwr)); + } + if( chldPg==0 ){ + assert( pCur->idx>=0 && pCur->idxpPage->nCell ); + if( pRes ) *pRes = c; + return SQLITE_OK; + } + pCur->idx = lwr; + pCur->info.nSize = 0; + rc = moveToChild(pCur, chldPg); + if( rc ){ + return rc; + } + } + /* NOT REACHED */ +} + +/* +** Return TRUE if the cursor is not pointing at an entry of the table. +** +** TRUE will be returned after a call to sqlite3BtreeNext() moves +** past the last entry in the table or sqlite3BtreePrev() moves past +** the first entry. TRUE is also returned if the table is empty. +*/ +int sqlite3BtreeEof(BtCursor *pCur){ + /* TODO: What if the cursor is in CURSOR_REQUIRESEEK but all table entries + ** have been deleted? This API will need to change to return an error code + ** as well as the boolean result value. + */ + return (CURSOR_VALID!=pCur->eState); +} + +/* +** Advance the cursor to the next entry in the database. If +** successful then set *pRes=0. If the cursor +** was already pointing to the last entry in the database before +** this routine was called, then set *pRes=1. +*/ +int sqlite3BtreeNext(BtCursor *pCur, int *pRes){ + int rc; + MemPage *pPage; + +#ifndef SQLITE_OMIT_SHARED_CACHE + rc = restoreOrClearCursorPosition(pCur, 1); + if( rc!=SQLITE_OK ){ + return rc; + } + if( pCur->skip>0 ){ + pCur->skip = 0; + *pRes = 0; + return SQLITE_OK; + } + pCur->skip = 0; +#endif + + assert( pRes!=0 ); + pPage = pCur->pPage; + if( CURSOR_INVALID==pCur->eState ){ + *pRes = 1; + return SQLITE_OK; + } + assert( pPage->isInit ); + assert( pCur->idxnCell ); + + pCur->idx++; + pCur->info.nSize = 0; + if( pCur->idx>=pPage->nCell ){ + if( !pPage->leaf ){ + rc = moveToChild(pCur, get4byte(&pPage->aData[pPage->hdrOffset+8])); + if( rc ) return rc; + rc = moveToLeftmost(pCur); + *pRes = 0; + return rc; + } + do{ + if( isRootPage(pPage) ){ + *pRes = 1; + pCur->eState = CURSOR_INVALID; + return SQLITE_OK; + } + moveToParent(pCur); + pPage = pCur->pPage; + }while( pCur->idx>=pPage->nCell ); + *pRes = 0; + if( pPage->leafData ){ + rc = sqlite3BtreeNext(pCur, pRes); + }else{ + rc = SQLITE_OK; + } + return rc; + } + *pRes = 0; + if( pPage->leaf ){ + return SQLITE_OK; + } + rc = moveToLeftmost(pCur); + return rc; +} + +/* +** Step the cursor to the back to the previous entry in the database. If +** successful then set *pRes=0. If the cursor +** was already pointing to the first entry in the database before +** this routine was called, then set *pRes=1. +*/ +int sqlite3BtreePrevious(BtCursor *pCur, int *pRes){ + int rc; + Pgno pgno; + MemPage *pPage; + +#ifndef SQLITE_OMIT_SHARED_CACHE + rc = restoreOrClearCursorPosition(pCur, 1); + if( rc!=SQLITE_OK ){ + return rc; + } + if( pCur->skip<0 ){ + pCur->skip = 0; + *pRes = 0; + return SQLITE_OK; + } + pCur->skip = 0; +#endif + + if( CURSOR_INVALID==pCur->eState ){ + *pRes = 1; + return SQLITE_OK; + } + + pPage = pCur->pPage; + assert( pPage->isInit ); + assert( pCur->idx>=0 ); + if( !pPage->leaf ){ + pgno = get4byte( findCell(pPage, pCur->idx) ); + rc = moveToChild(pCur, pgno); + if( rc ) return rc; + rc = moveToRightmost(pCur); + }else{ + while( pCur->idx==0 ){ + if( isRootPage(pPage) ){ + pCur->eState = CURSOR_INVALID; + *pRes = 1; + return SQLITE_OK; + } + moveToParent(pCur); + pPage = pCur->pPage; + } + pCur->idx--; + pCur->info.nSize = 0; + if( pPage->leafData && !pPage->leaf ){ + rc = sqlite3BtreePrevious(pCur, pRes); + }else{ + rc = SQLITE_OK; + } + } + *pRes = 0; + return rc; +} + +/* +** Allocate a new page from the database file. +** +** The new page is marked as dirty. (In other words, sqlite3pager_write() +** has already been called on the new page.) The new page has also +** been referenced and the calling routine is responsible for calling +** sqlite3pager_unref() on the new page when it is done. +** +** SQLITE_OK is returned on success. Any other return value indicates +** an error. *ppPage and *pPgno are undefined in the event of an error. +** Do not invoke sqlite3pager_unref() on *ppPage if an error is returned. +** +** If the "nearby" parameter is not 0, then a (feeble) effort is made to +** locate a page close to the page number "nearby". This can be used in an +** attempt to keep related pages close to each other in the database file, +** which in turn can make database access faster. +** +** If the "exact" parameter is not 0, and the page-number nearby exists +** anywhere on the free-list, then it is guarenteed to be returned. This +** is only used by auto-vacuum databases when allocating a new table. +*/ +static int allocatePage( + BtShared *pBt, + MemPage **ppPage, + Pgno *pPgno, + Pgno nearby, + u8 exact +){ + MemPage *pPage1; + int rc; + int n; /* Number of pages on the freelist */ + int k; /* Number of leaves on the trunk of the freelist */ + + pPage1 = pBt->pPage1; + n = get4byte(&pPage1->aData[36]); + if( n>0 ){ + /* There are pages on the freelist. Reuse one of those pages. */ + MemPage *pTrunk = 0; + Pgno iTrunk; + MemPage *pPrevTrunk = 0; + u8 searchList = 0; /* If the free-list must be searched for 'nearby' */ + + /* If the 'exact' parameter was true and a query of the pointer-map + ** shows that the page 'nearby' is somewhere on the free-list, then + ** the entire-list will be searched for that page. + */ +#ifndef SQLITE_OMIT_AUTOVACUUM + if( exact ){ + u8 eType; + assert( nearby>0 ); + assert( pBt->autoVacuum ); + rc = ptrmapGet(pBt, nearby, &eType, 0); + if( rc ) return rc; + if( eType==PTRMAP_FREEPAGE ){ + searchList = 1; + } + *pPgno = nearby; + } +#endif + + /* Decrement the free-list count by 1. Set iTrunk to the index of the + ** first free-list trunk page. iPrevTrunk is initially 1. + */ + rc = sqlite3pager_write(pPage1->aData); + if( rc ) return rc; + put4byte(&pPage1->aData[36], n-1); + + /* The code within this loop is run only once if the 'searchList' variable + ** is not true. Otherwise, it runs once for each trunk-page on the + ** free-list until the page 'nearby' is located. + */ + do { + pPrevTrunk = pTrunk; + if( pPrevTrunk ){ + iTrunk = get4byte(&pPrevTrunk->aData[0]); + }else{ + iTrunk = get4byte(&pPage1->aData[32]); + } + rc = getPage(pBt, iTrunk, &pTrunk); + if( rc ){ + releasePage(pPrevTrunk); + return rc; + } + + /* TODO: This should move to after the loop? */ + rc = sqlite3pager_write(pTrunk->aData); + if( rc ){ + releasePage(pTrunk); + releasePage(pPrevTrunk); + return rc; + } + + k = get4byte(&pTrunk->aData[4]); + if( k==0 && !searchList ){ + /* The trunk has no leaves and the list is not being searched. + ** So extract the trunk page itself and use it as the newly + ** allocated page */ + assert( pPrevTrunk==0 ); + *pPgno = iTrunk; + memcpy(&pPage1->aData[32], &pTrunk->aData[0], 4); + *ppPage = pTrunk; + pTrunk = 0; + TRACE(("ALLOCATE: %d trunk - %d free pages left\n", *pPgno, n-1)); + }else if( k>pBt->usableSize/4 - 8 ){ + /* Value of k is out of range. Database corruption */ + return SQLITE_CORRUPT_BKPT; +#ifndef SQLITE_OMIT_AUTOVACUUM + }else if( searchList && nearby==iTrunk ){ + /* The list is being searched and this trunk page is the page + ** to allocate, regardless of whether it has leaves. + */ + assert( *pPgno==iTrunk ); + *ppPage = pTrunk; + searchList = 0; + if( k==0 ){ + if( !pPrevTrunk ){ + memcpy(&pPage1->aData[32], &pTrunk->aData[0], 4); + }else{ + memcpy(&pPrevTrunk->aData[0], &pTrunk->aData[0], 4); + } + }else{ + /* The trunk page is required by the caller but it contains + ** pointers to free-list leaves. The first leaf becomes a trunk + ** page in this case. + */ + MemPage *pNewTrunk; + Pgno iNewTrunk = get4byte(&pTrunk->aData[8]); + rc = getPage(pBt, iNewTrunk, &pNewTrunk); + if( rc!=SQLITE_OK ){ + releasePage(pTrunk); + releasePage(pPrevTrunk); + return rc; + } + rc = sqlite3pager_write(pNewTrunk->aData); + if( rc!=SQLITE_OK ){ + releasePage(pNewTrunk); + releasePage(pTrunk); + releasePage(pPrevTrunk); + return rc; + } + memcpy(&pNewTrunk->aData[0], &pTrunk->aData[0], 4); + put4byte(&pNewTrunk->aData[4], k-1); + memcpy(&pNewTrunk->aData[8], &pTrunk->aData[12], (k-1)*4); + if( !pPrevTrunk ){ + put4byte(&pPage1->aData[32], iNewTrunk); + }else{ + put4byte(&pPrevTrunk->aData[0], iNewTrunk); + } + releasePage(pNewTrunk); + } + pTrunk = 0; + TRACE(("ALLOCATE: %d trunk - %d free pages left\n", *pPgno, n-1)); +#endif + }else{ + /* Extract a leaf from the trunk */ + int closest; + Pgno iPage; + unsigned char *aData = pTrunk->aData; + if( nearby>0 ){ + int i, dist; + closest = 0; + dist = get4byte(&aData[8]) - nearby; + if( dist<0 ) dist = -dist; + for(i=1; isqlite3pager_pagecount(pBt->pPager) ){ + /* Free page off the end of the file */ + return SQLITE_CORRUPT_BKPT; + } + TRACE(("ALLOCATE: %d was leaf %d of %d on trunk %d" + ": %d more free pages\n", + *pPgno, closest+1, k, pTrunk->pgno, n-1)); + if( closestaData); + rc = sqlite3pager_write((*ppPage)->aData); + if( rc!=SQLITE_OK ){ + releasePage(*ppPage); + } + } + searchList = 0; + } + } + releasePage(pPrevTrunk); + }while( searchList ); + releasePage(pTrunk); + }else{ + /* There are no pages on the freelist, so create a new page at the + ** end of the file */ + *pPgno = sqlite3pager_pagecount(pBt->pPager) + 1; + +#ifndef SQLITE_OMIT_AUTOVACUUM + if( pBt->autoVacuum && PTRMAP_ISPAGE(pBt, *pPgno) ){ + /* If *pPgno refers to a pointer-map page, allocate two new pages + ** at the end of the file instead of one. The first allocated page + ** becomes a new pointer-map page, the second is used by the caller. + */ + TRACE(("ALLOCATE: %d from end of file (pointer-map page)\n", *pPgno)); + assert( *pPgno!=PENDING_BYTE_PAGE(pBt) ); + (*pPgno)++; + } +#endif + + assert( *pPgno!=PENDING_BYTE_PAGE(pBt) ); + rc = getPage(pBt, *pPgno, ppPage); + if( rc ) return rc; + rc = sqlite3pager_write((*ppPage)->aData); + if( rc!=SQLITE_OK ){ + releasePage(*ppPage); + } + TRACE(("ALLOCATE: %d from end of file\n", *pPgno)); + } + + assert( *pPgno!=PENDING_BYTE_PAGE(pBt) ); + return rc; +} + +/* +** Add a page of the database file to the freelist. +** +** sqlite3pager_unref() is NOT called for pPage. +*/ +static int freePage(MemPage *pPage){ + BtShared *pBt = pPage->pBt; + MemPage *pPage1 = pBt->pPage1; + int rc, n, k; + + /* Prepare the page for freeing */ + assert( pPage->pgno>1 ); + pPage->isInit = 0; + releasePage(pPage->pParent); + pPage->pParent = 0; + + /* Increment the free page count on pPage1 */ + rc = sqlite3pager_write(pPage1->aData); + if( rc ) return rc; + n = get4byte(&pPage1->aData[36]); + put4byte(&pPage1->aData[36], n+1); + +#ifndef SQLITE_OMIT_AUTOVACUUM + /* If the database supports auto-vacuum, write an entry in the pointer-map + ** to indicate that the page is free. + */ + if( pBt->autoVacuum ){ + rc = ptrmapPut(pBt, pPage->pgno, PTRMAP_FREEPAGE, 0); + if( rc ) return rc; + } +#endif + + if( n==0 ){ + /* This is the first free page */ + rc = sqlite3pager_write(pPage->aData); + if( rc ) return rc; + memset(pPage->aData, 0, 8); + put4byte(&pPage1->aData[32], pPage->pgno); + TRACE(("FREE-PAGE: %d first\n", pPage->pgno)); + }else{ + /* Other free pages already exist. Retrive the first trunk page + ** of the freelist and find out how many leaves it has. */ + MemPage *pTrunk; + rc = getPage(pBt, get4byte(&pPage1->aData[32]), &pTrunk); + if( rc ) return rc; + k = get4byte(&pTrunk->aData[4]); + if( k>=pBt->usableSize/4 - 8 ){ + /* The trunk is full. Turn the page being freed into a new + ** trunk page with no leaves. */ + rc = sqlite3pager_write(pPage->aData); + if( rc ) return rc; + put4byte(pPage->aData, pTrunk->pgno); + put4byte(&pPage->aData[4], 0); + put4byte(&pPage1->aData[32], pPage->pgno); + TRACE(("FREE-PAGE: %d new trunk page replacing %d\n", + pPage->pgno, pTrunk->pgno)); + }else{ + /* Add the newly freed page as a leaf on the current trunk */ + rc = sqlite3pager_write(pTrunk->aData); + if( rc ) return rc; + put4byte(&pTrunk->aData[4], k+1); + put4byte(&pTrunk->aData[8+k*4], pPage->pgno); + sqlite3pager_dont_write(pBt->pPager, pPage->pgno); + TRACE(("FREE-PAGE: %d leaf on trunk page %d\n",pPage->pgno,pTrunk->pgno)); + } + releasePage(pTrunk); + } + return rc; +} + +/* +** Free any overflow pages associated with the given Cell. +*/ +static int clearCell(MemPage *pPage, unsigned char *pCell){ + BtShared *pBt = pPage->pBt; + CellInfo info; + Pgno ovflPgno; + int rc; + + parseCellPtr(pPage, pCell, &info); + if( info.iOverflow==0 ){ + return SQLITE_OK; /* No overflow pages. Return without doing anything */ + } + ovflPgno = get4byte(&pCell[info.iOverflow]); + while( ovflPgno!=0 ){ + MemPage *pOvfl; + if( ovflPgno>sqlite3pager_pagecount(pBt->pPager) ){ + return SQLITE_CORRUPT_BKPT; + } + rc = getPage(pBt, ovflPgno, &pOvfl); + if( rc ) return rc; + ovflPgno = get4byte(pOvfl->aData); + rc = freePage(pOvfl); + sqlite3pager_unref(pOvfl->aData); + if( rc ) return rc; + } + return SQLITE_OK; +} + +/* +** Create the byte sequence used to represent a cell on page pPage +** and write that byte sequence into pCell[]. Overflow pages are +** allocated and filled in as necessary. The calling procedure +** is responsible for making sure sufficient space has been allocated +** for pCell[]. +** +** Note that pCell does not necessary need to point to the pPage->aData +** area. pCell might point to some temporary storage. The cell will +** be constructed in this temporary area then copied into pPage->aData +** later. +*/ +static int fillInCell( + MemPage *pPage, /* The page that contains the cell */ + unsigned char *pCell, /* Complete text of the cell */ + const void *pKey, i64 nKey, /* The key */ + const void *pData,int nData, /* The data */ + int *pnSize /* Write cell size here */ +){ + int nPayload; + const u8 *pSrc; + int nSrc, n, rc; + int spaceLeft; + MemPage *pOvfl = 0; + MemPage *pToRelease = 0; + unsigned char *pPrior; + unsigned char *pPayload; + BtShared *pBt = pPage->pBt; + Pgno pgnoOvfl = 0; + int nHeader; + CellInfo info; + + /* Fill in the header. */ + nHeader = 0; + if( !pPage->leaf ){ + nHeader += 4; + } + if( pPage->hasData ){ + nHeader += putVarint(&pCell[nHeader], nData); + }else{ + nData = 0; + } + nHeader += putVarint(&pCell[nHeader], *(u64*)&nKey); + parseCellPtr(pPage, pCell, &info); + assert( info.nHeader==nHeader ); + assert( info.nKey==nKey ); + assert( info.nData==nData ); + + /* Fill in the payload */ + nPayload = nData; + if( pPage->intKey ){ + pSrc = pData; + nSrc = nData; + nData = 0; + }else{ + nPayload += nKey; + pSrc = pKey; + nSrc = nKey; + } + *pnSize = info.nSize; + spaceLeft = info.nLocal; + pPayload = &pCell[nHeader]; + pPrior = &pCell[info.iOverflow]; + + while( nPayload>0 ){ + if( spaceLeft==0 ){ +#ifndef SQLITE_OMIT_AUTOVACUUM + Pgno pgnoPtrmap = pgnoOvfl; /* Overflow page pointer-map entry page */ +#endif + rc = allocatePage(pBt, &pOvfl, &pgnoOvfl, pgnoOvfl, 0); +#ifndef SQLITE_OMIT_AUTOVACUUM + /* If the database supports auto-vacuum, and the second or subsequent + ** overflow page is being allocated, add an entry to the pointer-map + ** for that page now. The entry for the first overflow page will be + ** added later, by the insertCell() routine. + */ + if( pBt->autoVacuum && pgnoPtrmap!=0 && rc==SQLITE_OK ){ + rc = ptrmapPut(pBt, pgnoOvfl, PTRMAP_OVERFLOW2, pgnoPtrmap); + } +#endif + if( rc ){ + releasePage(pToRelease); + /* clearCell(pPage, pCell); */ + return rc; + } + put4byte(pPrior, pgnoOvfl); + releasePage(pToRelease); + pToRelease = pOvfl; + pPrior = pOvfl->aData; + put4byte(pPrior, 0); + pPayload = &pOvfl->aData[4]; + spaceLeft = pBt->usableSize - 4; + } + n = nPayload; + if( n>spaceLeft ) n = spaceLeft; + if( n>nSrc ) n = nSrc; + memcpy(pPayload, pSrc, n); + nPayload -= n; + pPayload += n; + pSrc += n; + nSrc -= n; + spaceLeft -= n; + if( nSrc==0 ){ + nSrc = nData; + pSrc = pData; + } + } + releasePage(pToRelease); + return SQLITE_OK; +} + +/* +** Change the MemPage.pParent pointer on the page whose number is +** given in the second argument so that MemPage.pParent holds the +** pointer in the third argument. +*/ +static int reparentPage(BtShared *pBt, Pgno pgno, MemPage *pNewParent, int idx){ + MemPage *pThis; + unsigned char *aData; + + if( pgno==0 ) return SQLITE_OK; + assert( pBt->pPager!=0 ); + aData = sqlite3pager_lookup(pBt->pPager, pgno); + if( aData ){ + pThis = (MemPage*)&aData[pBt->pageSize]; + assert( pThis->aData==aData ); + if( pThis->isInit ){ + if( pThis->pParent!=pNewParent ){ + if( pThis->pParent ) sqlite3pager_unref(pThis->pParent->aData); + pThis->pParent = pNewParent; + if( pNewParent ) sqlite3pager_ref(pNewParent->aData); + } + pThis->idxParent = idx; + } + sqlite3pager_unref(aData); + } + +#ifndef SQLITE_OMIT_AUTOVACUUM + if( pBt->autoVacuum ){ + return ptrmapPut(pBt, pgno, PTRMAP_BTREE, pNewParent->pgno); + } +#endif + return SQLITE_OK; +} + + + +/* +** Change the pParent pointer of all children of pPage to point back +** to pPage. +** +** In other words, for every child of pPage, invoke reparentPage() +** to make sure that each child knows that pPage is its parent. +** +** This routine gets called after you memcpy() one page into +** another. +*/ +static int reparentChildPages(MemPage *pPage){ + int i; + BtShared *pBt = pPage->pBt; + int rc = SQLITE_OK; + + if( pPage->leaf ) return SQLITE_OK; + + for(i=0; inCell; i++){ + u8 *pCell = findCell(pPage, i); + if( !pPage->leaf ){ + rc = reparentPage(pBt, get4byte(pCell), pPage, i); + if( rc!=SQLITE_OK ) return rc; + } + } + if( !pPage->leaf ){ + rc = reparentPage(pBt, get4byte(&pPage->aData[pPage->hdrOffset+8]), + pPage, i); + pPage->idxShift = 0; + } + return rc; +} + +/* +** Remove the i-th cell from pPage. This routine effects pPage only. +** The cell content is not freed or deallocated. It is assumed that +** the cell content has been copied someplace else. This routine just +** removes the reference to the cell from pPage. +** +** "sz" must be the number of bytes in the cell. +*/ +static void dropCell(MemPage *pPage, int idx, int sz){ + int i; /* Loop counter */ + int pc; /* Offset to cell content of cell being deleted */ + u8 *data; /* pPage->aData */ + u8 *ptr; /* Used to move bytes around within data[] */ + + assert( idx>=0 && idxnCell ); + assert( sz==cellSize(pPage, idx) ); + assert( sqlite3pager_iswriteable(pPage->aData) ); + data = pPage->aData; + ptr = &data[pPage->cellOffset + 2*idx]; + pc = get2byte(ptr); + assert( pc>10 && pc+sz<=pPage->pBt->usableSize ); + freeSpace(pPage, pc, sz); + for(i=idx+1; inCell; i++, ptr+=2){ + ptr[0] = ptr[2]; + ptr[1] = ptr[3]; + } + pPage->nCell--; + put2byte(&data[pPage->hdrOffset+3], pPage->nCell); + pPage->nFree += 2; + pPage->idxShift = 1; +} + +/* +** Insert a new cell on pPage at cell index "i". pCell points to the +** content of the cell. +** +** If the cell content will fit on the page, then put it there. If it +** will not fit, then make a copy of the cell content into pTemp if +** pTemp is not null. Regardless of pTemp, allocate a new entry +** in pPage->aOvfl[] and make it point to the cell content (either +** in pTemp or the original pCell) and also record its index. +** Allocating a new entry in pPage->aCell[] implies that +** pPage->nOverflow is incremented. +** +** If nSkip is non-zero, then do not copy the first nSkip bytes of the +** cell. The caller will overwrite them after this function returns. If +** nSkip is non-zero, then pCell may not point to an invalid memory location +** (but pCell+nSkip is always valid). +*/ +static int insertCell( + MemPage *pPage, /* Page into which we are copying */ + int i, /* New cell becomes the i-th cell of the page */ + u8 *pCell, /* Content of the new cell */ + int sz, /* Bytes of content in pCell */ + u8 *pTemp, /* Temp storage space for pCell, if needed */ + u8 nSkip /* Do not write the first nSkip bytes of the cell */ +){ + int idx; /* Where to write new cell content in data[] */ + int j; /* Loop counter */ + int top; /* First byte of content for any cell in data[] */ + int end; /* First byte past the last cell pointer in data[] */ + int ins; /* Index in data[] where new cell pointer is inserted */ + int hdr; /* Offset into data[] of the page header */ + int cellOffset; /* Address of first cell pointer in data[] */ + u8 *data; /* The content of the whole page */ + u8 *ptr; /* Used for moving information around in data[] */ + + assert( i>=0 && i<=pPage->nCell+pPage->nOverflow ); + assert( sz==cellSizePtr(pPage, pCell) ); + assert( sqlite3pager_iswriteable(pPage->aData) ); + if( pPage->nOverflow || sz+2>pPage->nFree ){ + if( pTemp ){ + memcpy(pTemp+nSkip, pCell+nSkip, sz-nSkip); + pCell = pTemp; + } + j = pPage->nOverflow++; + assert( jaOvfl)/sizeof(pPage->aOvfl[0]) ); + pPage->aOvfl[j].pCell = pCell; + pPage->aOvfl[j].idx = i; + pPage->nFree = 0; + }else{ + data = pPage->aData; + hdr = pPage->hdrOffset; + top = get2byte(&data[hdr+5]); + cellOffset = pPage->cellOffset; + end = cellOffset + 2*pPage->nCell + 2; + ins = cellOffset + 2*i; + if( end > top - sz ){ + int rc = defragmentPage(pPage); + if( rc!=SQLITE_OK ) return rc; + top = get2byte(&data[hdr+5]); + assert( end + sz <= top ); + } + idx = allocateSpace(pPage, sz); + assert( idx>0 ); + assert( end <= get2byte(&data[hdr+5]) ); + pPage->nCell++; + pPage->nFree -= 2; + memcpy(&data[idx+nSkip], pCell+nSkip, sz-nSkip); + for(j=end-2, ptr=&data[j]; j>ins; j-=2, ptr-=2){ + ptr[0] = ptr[-2]; + ptr[1] = ptr[-1]; + } + put2byte(&data[ins], idx); + put2byte(&data[hdr+3], pPage->nCell); + pPage->idxShift = 1; + pageIntegrity(pPage); +#ifndef SQLITE_OMIT_AUTOVACUUM + if( pPage->pBt->autoVacuum ){ + /* The cell may contain a pointer to an overflow page. If so, write + ** the entry for the overflow page into the pointer map. + */ + CellInfo info; + parseCellPtr(pPage, pCell, &info); + if( (info.nData+(pPage->intKey?0:info.nKey))>info.nLocal ){ + Pgno pgnoOvfl = get4byte(&pCell[info.iOverflow]); + int rc = ptrmapPut(pPage->pBt, pgnoOvfl, PTRMAP_OVERFLOW1, pPage->pgno); + if( rc!=SQLITE_OK ) return rc; + } + } +#endif + } + + return SQLITE_OK; +} + +/* +** Add a list of cells to a page. The page should be initially empty. +** The cells are guaranteed to fit on the page. +*/ +static void assemblePage( + MemPage *pPage, /* The page to be assemblied */ + int nCell, /* The number of cells to add to this page */ + u8 **apCell, /* Pointers to cell bodies */ + int *aSize /* Sizes of the cells */ +){ + int i; /* Loop counter */ + int totalSize; /* Total size of all cells */ + int hdr; /* Index of page header */ + int cellptr; /* Address of next cell pointer */ + int cellbody; /* Address of next cell body */ + u8 *data; /* Data for the page */ + + assert( pPage->nOverflow==0 ); + totalSize = 0; + for(i=0; inFree ); + assert( pPage->nCell==0 ); + cellptr = pPage->cellOffset; + data = pPage->aData; + hdr = pPage->hdrOffset; + put2byte(&data[hdr+3], nCell); + if( nCell ){ + cellbody = allocateSpace(pPage, totalSize); + assert( cellbody>0 ); + assert( pPage->nFree >= 2*nCell ); + pPage->nFree -= 2*nCell; + for(i=0; ipBt->usableSize ); + } + pPage->nCell = nCell; +} + +/* +** The following parameters determine how many adjacent pages get involved +** in a balancing operation. NN is the number of neighbors on either side +** of the page that participate in the balancing operation. NB is the +** total number of pages that participate, including the target page and +** NN neighbors on either side. +** +** The minimum value of NN is 1 (of course). Increasing NN above 1 +** (to 2 or 3) gives a modest improvement in SELECT and DELETE performance +** in exchange for a larger degradation in INSERT and UPDATE performance. +** The value of NN appears to give the best results overall. +*/ +#define NN 1 /* Number of neighbors on either side of pPage */ +#define NB (NN*2+1) /* Total pages involved in the balance */ + +/* Forward reference */ +static int balance(MemPage*, int); + +#ifndef SQLITE_OMIT_QUICKBALANCE +/* +** This version of balance() handles the common special case where +** a new entry is being inserted on the extreme right-end of the +** tree, in other words, when the new entry will become the largest +** entry in the tree. +** +** Instead of trying balance the 3 right-most leaf pages, just add +** a new page to the right-hand side and put the one new entry in +** that page. This leaves the right side of the tree somewhat +** unbalanced. But odds are that we will be inserting new entries +** at the end soon afterwards so the nearly empty page will quickly +** fill up. On average. +** +** pPage is the leaf page which is the right-most page in the tree. +** pParent is its parent. pPage must have a single overflow entry +** which is also the right-most entry on the page. +*/ +static int balance_quick(MemPage *pPage, MemPage *pParent){ + int rc; + MemPage *pNew; + Pgno pgnoNew; + u8 *pCell; + int szCell; + CellInfo info; + BtShared *pBt = pPage->pBt; + int parentIdx = pParent->nCell; /* pParent new divider cell index */ + int parentSize; /* Size of new divider cell */ + u8 parentCell[64]; /* Space for the new divider cell */ + + /* Allocate a new page. Insert the overflow cell from pPage + ** into it. Then remove the overflow cell from pPage. + */ + rc = allocatePage(pBt, &pNew, &pgnoNew, 0, 0); + if( rc!=SQLITE_OK ){ + return rc; + } + pCell = pPage->aOvfl[0].pCell; + szCell = cellSizePtr(pPage, pCell); + zeroPage(pNew, pPage->aData[0]); + assemblePage(pNew, 1, &pCell, &szCell); + pPage->nOverflow = 0; + + /* Set the parent of the newly allocated page to pParent. */ + pNew->pParent = pParent; + sqlite3pager_ref(pParent->aData); + + /* pPage is currently the right-child of pParent. Change this + ** so that the right-child is the new page allocated above and + ** pPage is the next-to-right child. + */ + assert( pPage->nCell>0 ); + parseCellPtr(pPage, findCell(pPage, pPage->nCell-1), &info); + rc = fillInCell(pParent, parentCell, 0, info.nKey, 0, 0, &parentSize); + if( rc!=SQLITE_OK ){ + return rc; + } + assert( parentSize<64 ); + rc = insertCell(pParent, parentIdx, parentCell, parentSize, 0, 4); + if( rc!=SQLITE_OK ){ + return rc; + } + put4byte(findOverflowCell(pParent,parentIdx), pPage->pgno); + put4byte(&pParent->aData[pParent->hdrOffset+8], pgnoNew); + +#ifndef SQLITE_OMIT_AUTOVACUUM + /* If this is an auto-vacuum database, update the pointer map + ** with entries for the new page, and any pointer from the + ** cell on the page to an overflow page. + */ + if( pBt->autoVacuum ){ + rc = ptrmapPut(pBt, pgnoNew, PTRMAP_BTREE, pParent->pgno); + if( rc!=SQLITE_OK ){ + return rc; + } + rc = ptrmapPutOvfl(pNew, 0); + if( rc!=SQLITE_OK ){ + return rc; + } + } +#endif + + /* Release the reference to the new page and balance the parent page, + ** in case the divider cell inserted caused it to become overfull. + */ + releasePage(pNew); + return balance(pParent, 0); +} +#endif /* SQLITE_OMIT_QUICKBALANCE */ + +/* +** The ISAUTOVACUUM macro is used within balance_nonroot() to determine +** if the database supports auto-vacuum or not. Because it is used +** within an expression that is an argument to another macro +** (sqliteMallocRaw), it is not possible to use conditional compilation. +** So, this macro is defined instead. +*/ +#ifndef SQLITE_OMIT_AUTOVACUUM +#define ISAUTOVACUUM (pBt->autoVacuum) +#else +#define ISAUTOVACUUM 0 +#endif + +/* +** This routine redistributes Cells on pPage and up to NN*2 siblings +** of pPage so that all pages have about the same amount of free space. +** Usually NN siblings on either side of pPage is used in the balancing, +** though more siblings might come from one side if pPage is the first +** or last child of its parent. If pPage has fewer than 2*NN siblings +** (something which can only happen if pPage is the root page or a +** child of root) then all available siblings participate in the balancing. +** +** The number of siblings of pPage might be increased or decreased by one or +** two in an effort to keep pages nearly full but not over full. The root page +** is special and is allowed to be nearly empty. If pPage is +** the root page, then the depth of the tree might be increased +** or decreased by one, as necessary, to keep the root page from being +** overfull or completely empty. +** +** Note that when this routine is called, some of the Cells on pPage +** might not actually be stored in pPage->aData[]. This can happen +** if the page is overfull. Part of the job of this routine is to +** make sure all Cells for pPage once again fit in pPage->aData[]. +** +** In the course of balancing the siblings of pPage, the parent of pPage +** might become overfull or underfull. If that happens, then this routine +** is called recursively on the parent. +** +** If this routine fails for any reason, it might leave the database +** in a corrupted state. So if this routine fails, the database should +** be rolled back. +*/ +static int balance_nonroot(MemPage *pPage){ + MemPage *pParent; /* The parent of pPage */ + BtShared *pBt; /* The whole database */ + int nCell = 0; /* Number of cells in apCell[] */ + int nMaxCells = 0; /* Allocated size of apCell, szCell, aFrom. */ + int nOld; /* Number of pages in apOld[] */ + int nNew; /* Number of pages in apNew[] */ + int nDiv; /* Number of cells in apDiv[] */ + int i, j, k; /* Loop counters */ + int idx; /* Index of pPage in pParent->aCell[] */ + int nxDiv; /* Next divider slot in pParent->aCell[] */ + int rc; /* The return code */ + int leafCorrection; /* 4 if pPage is a leaf. 0 if not */ + int leafData; /* True if pPage is a leaf of a LEAFDATA tree */ + int usableSpace; /* Bytes in pPage beyond the header */ + int pageFlags; /* Value of pPage->aData[0] */ + int subtotal; /* Subtotal of bytes in cells on one page */ + int iSpace = 0; /* First unused byte of aSpace[] */ + MemPage *apOld[NB]; /* pPage and up to two siblings */ + Pgno pgnoOld[NB]; /* Page numbers for each page in apOld[] */ + MemPage *apCopy[NB]; /* Private copies of apOld[] pages */ + MemPage *apNew[NB+2]; /* pPage and up to NB siblings after balancing */ + Pgno pgnoNew[NB+2]; /* Page numbers for each page in apNew[] */ + u8 *apDiv[NB]; /* Divider cells in pParent */ + int cntNew[NB+2]; /* Index in aCell[] of cell after i-th page */ + int szNew[NB+2]; /* Combined size of cells place on i-th page */ + u8 **apCell = 0; /* All cells begin balanced */ + int *szCell; /* Local size of all cells in apCell[] */ + u8 *aCopy[NB]; /* Space for holding data of apCopy[] */ + u8 *aSpace; /* Space to hold copies of dividers cells */ +#ifndef SQLITE_OMIT_AUTOVACUUM + u8 *aFrom = 0; +#endif + + /* + ** Find the parent page. + */ + assert( pPage->isInit ); + assert( sqlite3pager_iswriteable(pPage->aData) ); + pBt = pPage->pBt; + pParent = pPage->pParent; + assert( pParent ); + if( SQLITE_OK!=(rc = sqlite3pager_write(pParent->aData)) ){ + return rc; + } + TRACE(("BALANCE: begin page %d child of %d\n", pPage->pgno, pParent->pgno)); + +#ifndef SQLITE_OMIT_QUICKBALANCE + /* + ** A special case: If a new entry has just been inserted into a + ** table (that is, a btree with integer keys and all data at the leaves) + ** and the new entry is the right-most entry in the tree (it has the + ** largest key) then use the special balance_quick() routine for + ** balancing. balance_quick() is much faster and results in a tighter + ** packing of data in the common case. + */ + if( pPage->leaf && + pPage->intKey && + pPage->leafData && + pPage->nOverflow==1 && + pPage->aOvfl[0].idx==pPage->nCell && + pPage->pParent->pgno!=1 && + get4byte(&pParent->aData[pParent->hdrOffset+8])==pPage->pgno + ){ + /* + ** TODO: Check the siblings to the left of pPage. It may be that + ** they are not full and no new page is required. + */ + return balance_quick(pPage, pParent); + } +#endif + + /* + ** Find the cell in the parent page whose left child points back + ** to pPage. The "idx" variable is the index of that cell. If pPage + ** is the rightmost child of pParent then set idx to pParent->nCell + */ + if( pParent->idxShift ){ + Pgno pgno; + pgno = pPage->pgno; + assert( pgno==sqlite3pager_pagenumber(pPage->aData) ); + for(idx=0; idxnCell; idx++){ + if( get4byte(findCell(pParent, idx))==pgno ){ + break; + } + } + assert( idxnCell + || get4byte(&pParent->aData[pParent->hdrOffset+8])==pgno ); + }else{ + idx = pPage->idxParent; + } + + /* + ** Initialize variables so that it will be safe to jump + ** directly to balance_cleanup at any moment. + */ + nOld = nNew = 0; + sqlite3pager_ref(pParent->aData); + + /* + ** Find sibling pages to pPage and the cells in pParent that divide + ** the siblings. An attempt is made to find NN siblings on either + ** side of pPage. More siblings are taken from one side, however, if + ** pPage there are fewer than NN siblings on the other side. If pParent + ** has NB or fewer children then all children of pParent are taken. + */ + nxDiv = idx - NN; + if( nxDiv + NB > pParent->nCell ){ + nxDiv = pParent->nCell - NB + 1; + } + if( nxDiv<0 ){ + nxDiv = 0; + } + nDiv = 0; + for(i=0, k=nxDiv; inCell ){ + apDiv[i] = findCell(pParent, k); + nDiv++; + assert( !pParent->leaf ); + pgnoOld[i] = get4byte(apDiv[i]); + }else if( k==pParent->nCell ){ + pgnoOld[i] = get4byte(&pParent->aData[pParent->hdrOffset+8]); + }else{ + break; + } + rc = getAndInitPage(pBt, pgnoOld[i], &apOld[i], pParent); + if( rc ) goto balance_cleanup; + apOld[i]->idxParent = k; + apCopy[i] = 0; + assert( i==nOld ); + nOld++; + nMaxCells += 1+apOld[i]->nCell+apOld[i]->nOverflow; + } + + /* Make nMaxCells a multiple of 2 in order to preserve 8-byte + ** alignment */ + nMaxCells = (nMaxCells + 1)&~1; + + /* + ** Allocate space for memory structures + */ + apCell = sqliteMallocRaw( + nMaxCells*sizeof(u8*) /* apCell */ + + nMaxCells*sizeof(int) /* szCell */ + + ROUND8(sizeof(MemPage))*NB /* aCopy */ + + pBt->pageSize*(5+NB) /* aSpace */ + + (ISAUTOVACUUM ? nMaxCells : 0) /* aFrom */ + ); + if( apCell==0 ){ + rc = SQLITE_NOMEM; + goto balance_cleanup; + } + szCell = (int*)&apCell[nMaxCells]; + aCopy[0] = (u8*)&szCell[nMaxCells]; + assert( ((aCopy[0] - (u8*)apCell) & 7)==0 ); /* 8-byte alignment required */ + for(i=1; ipageSize+ROUND8(sizeof(MemPage))]; + assert( ((aCopy[i] - (u8*)apCell) & 7)==0 ); /* 8-byte alignment required */ + } + aSpace = &aCopy[NB-1][pBt->pageSize+ROUND8(sizeof(MemPage))]; + assert( ((aSpace - (u8*)apCell) & 7)==0 ); /* 8-byte alignment required */ +#ifndef SQLITE_OMIT_AUTOVACUUM + if( pBt->autoVacuum ){ + aFrom = &aSpace[5*pBt->pageSize]; + } +#endif + + /* + ** Make copies of the content of pPage and its siblings into aOld[]. + ** The rest of this function will use data from the copies rather + ** that the original pages since the original pages will be in the + ** process of being overwritten. + */ + for(i=0; ipageSize]; + p->aData = &((u8*)p)[-pBt->pageSize]; + memcpy(p->aData, apOld[i]->aData, pBt->pageSize + sizeof(MemPage)); + /* The memcpy() above changes the value of p->aData so we have to + ** set it again. */ + p->aData = &((u8*)p)[-pBt->pageSize]; + } + + /* + ** Load pointers to all cells on sibling pages and the divider cells + ** into the local apCell[] array. Make copies of the divider cells + ** into space obtained form aSpace[] and remove the the divider Cells + ** from pParent. + ** + ** If the siblings are on leaf pages, then the child pointers of the + ** divider cells are stripped from the cells before they are copied + ** into aSpace[]. In this way, all cells in apCell[] are without + ** child pointers. If siblings are not leaves, then all cell in + ** apCell[] include child pointers. Either way, all cells in apCell[] + ** are alike. + ** + ** leafCorrection: 4 if pPage is a leaf. 0 if pPage is not a leaf. + ** leafData: 1 if pPage holds key+data and pParent holds only keys. + */ + nCell = 0; + leafCorrection = pPage->leaf*4; + leafData = pPage->leafData && pPage->leaf; + for(i=0; inCell+pOld->nOverflow; + for(j=0; jautoVacuum ){ + int a; + aFrom[nCell] = i; + for(a=0; anOverflow; a++){ + if( pOld->aOvfl[a].pCell==apCell[nCell] ){ + aFrom[nCell] = 0xFF; + break; + } + } + } +#endif + nCell++; + } + if( ipageSize*5 ); + memcpy(pTemp, apDiv[i], sz); + apCell[nCell] = pTemp+leafCorrection; +#ifndef SQLITE_OMIT_AUTOVACUUM + if( pBt->autoVacuum ){ + aFrom[nCell] = 0xFF; + } +#endif + dropCell(pParent, nxDiv, sz); + szCell[nCell] -= leafCorrection; + assert( get4byte(pTemp)==pgnoOld[i] ); + if( !pOld->leaf ){ + assert( leafCorrection==0 ); + /* The right pointer of the child page pOld becomes the left + ** pointer of the divider cell */ + memcpy(apCell[nCell], &pOld->aData[pOld->hdrOffset+8], 4); + }else{ + assert( leafCorrection==4 ); + } + nCell++; + } + } + } + + /* + ** Figure out the number of pages needed to hold all nCell cells. + ** Store this number in "k". Also compute szNew[] which is the total + ** size of all cells on the i-th page and cntNew[] which is the index + ** in apCell[] of the cell that divides page i from page i+1. + ** cntNew[k] should equal nCell. + ** + ** Values computed by this block: + ** + ** k: The total number of sibling pages + ** szNew[i]: Spaced used on the i-th sibling page. + ** cntNew[i]: Index in apCell[] and szCell[] for the first cell to + ** the right of the i-th sibling page. + ** usableSpace: Number of bytes of space available on each sibling. + ** + */ + usableSpace = pBt->usableSize - 12 + leafCorrection; + for(subtotal=k=i=0; i usableSpace ){ + szNew[k] = subtotal - szCell[i]; + cntNew[k] = i; + if( leafData ){ i--; } + subtotal = 0; + k++; + } + } + szNew[k] = subtotal; + cntNew[k] = nCell; + k++; + + /* + ** The packing computed by the previous block is biased toward the siblings + ** on the left side. The left siblings are always nearly full, while the + ** right-most sibling might be nearly empty. This block of code attempts + ** to adjust the packing of siblings to get a better balance. + ** + ** This adjustment is more than an optimization. The packing above might + ** be so out of balance as to be illegal. For example, the right-most + ** sibling might be completely empty. This adjustment is not optional. + */ + for(i=k-1; i>0; i--){ + int szRight = szNew[i]; /* Size of sibling on the right */ + int szLeft = szNew[i-1]; /* Size of sibling on the left */ + int r; /* Index of right-most cell in left sibling */ + int d; /* Index of first cell to the left of right sibling */ + + r = cntNew[i-1] - 1; + d = r + 1 - leafData; + assert( d0) or we are the + ** a virtual root page. A virtual root page is when the real root + ** page is page 1 and we are the only child of that page. + */ + assert( cntNew[0]>0 || (pParent->pgno==1 && pParent->nCell==0) ); + + /* + ** Allocate k new pages. Reuse old pages where possible. + */ + assert( pPage->pgno>1 ); + pageFlags = pPage->aData[0]; + for(i=0; iaData); + if( rc ) goto balance_cleanup; + }else{ + rc = allocatePage(pBt, &pNew, &pgnoNew[i], pgnoNew[i-1], 0); + if( rc ) goto balance_cleanup; + apNew[i] = pNew; + } + nNew++; + zeroPage(pNew, pageFlags); + } + + /* Free any old pages that were not reused as new pages. + */ + while( ii ){ + int t; + MemPage *pT; + t = pgnoNew[i]; + pT = apNew[i]; + pgnoNew[i] = pgnoNew[minI]; + apNew[i] = apNew[minI]; + pgnoNew[minI] = t; + apNew[minI] = pT; + } + } + TRACE(("BALANCE: old: %d %d %d new: %d(%d) %d(%d) %d(%d) %d(%d) %d(%d)\n", + pgnoOld[0], + nOld>=2 ? pgnoOld[1] : 0, + nOld>=3 ? pgnoOld[2] : 0, + pgnoNew[0], szNew[0], + nNew>=2 ? pgnoNew[1] : 0, nNew>=2 ? szNew[1] : 0, + nNew>=3 ? pgnoNew[2] : 0, nNew>=3 ? szNew[2] : 0, + nNew>=4 ? pgnoNew[3] : 0, nNew>=4 ? szNew[3] : 0, + nNew>=5 ? pgnoNew[4] : 0, nNew>=5 ? szNew[4] : 0)); + + /* + ** Evenly distribute the data in apCell[] across the new pages. + ** Insert divider cells into pParent as necessary. + */ + j = 0; + for(i=0; ipgno==pgnoNew[i] ); + assemblePage(pNew, cntNew[i]-j, &apCell[j], &szCell[j]); + assert( pNew->nCell>0 || (nNew==1 && cntNew[0]==0) ); + assert( pNew->nOverflow==0 ); + +#ifndef SQLITE_OMIT_AUTOVACUUM + /* If this is an auto-vacuum database, update the pointer map entries + ** that point to the siblings that were rearranged. These can be: left + ** children of cells, the right-child of the page, or overflow pages + ** pointed to by cells. + */ + if( pBt->autoVacuum ){ + for(k=j; kpgno!=pNew->pgno ){ + rc = ptrmapPutOvfl(pNew, k-j); + if( rc!=SQLITE_OK ){ + goto balance_cleanup; + } + } + } + } +#endif + + j = cntNew[i]; + + /* If the sibling page assembled above was not the right-most sibling, + ** insert a divider cell into the parent page. + */ + if( ileaf ){ + memcpy(&pNew->aData[8], pCell, 4); + pTemp = 0; + }else if( leafData ){ + /* If the tree is a leaf-data tree, and the siblings are leaves, + ** then there is no divider cell in apCell[]. Instead, the divider + ** cell consists of the integer key for the right-most cell of + ** the sibling-page assembled above only. + */ + CellInfo info; + j--; + parseCellPtr(pNew, apCell[j], &info); + pCell = &aSpace[iSpace]; + fillInCell(pParent, pCell, 0, info.nKey, 0, 0, &sz); + iSpace += sz; + assert( iSpace<=pBt->pageSize*5 ); + pTemp = 0; + }else{ + pCell -= 4; + pTemp = &aSpace[iSpace]; + iSpace += sz; + assert( iSpace<=pBt->pageSize*5 ); + } + rc = insertCell(pParent, nxDiv, pCell, sz, pTemp, 4); + if( rc!=SQLITE_OK ) goto balance_cleanup; + put4byte(findOverflowCell(pParent,nxDiv), pNew->pgno); +#ifndef SQLITE_OMIT_AUTOVACUUM + /* If this is an auto-vacuum database, and not a leaf-data tree, + ** then update the pointer map with an entry for the overflow page + ** that the cell just inserted points to (if any). + */ + if( pBt->autoVacuum && !leafData ){ + rc = ptrmapPutOvfl(pParent, nxDiv); + if( rc!=SQLITE_OK ){ + goto balance_cleanup; + } + } +#endif + j++; + nxDiv++; + } + } + assert( j==nCell ); + if( (pageFlags & PTF_LEAF)==0 ){ + memcpy(&apNew[nNew-1]->aData[8], &apCopy[nOld-1]->aData[8], 4); + } + if( nxDiv==pParent->nCell+pParent->nOverflow ){ + /* Right-most sibling is the right-most child of pParent */ + put4byte(&pParent->aData[pParent->hdrOffset+8], pgnoNew[nNew-1]); + }else{ + /* Right-most sibling is the left child of the first entry in pParent + ** past the right-most divider entry */ + put4byte(findOverflowCell(pParent, nxDiv), pgnoNew[nNew-1]); + } + + /* + ** Reparent children of all cells. + */ + for(i=0; iisInit ); + /* assert( pPage->isInit ); // No! pPage might have been added to freelist */ + /* pageIntegrity(pPage); // No! pPage might have been added to freelist */ + rc = balance(pParent, 0); + + /* + ** Cleanup before returning. + */ +balance_cleanup: + sqliteFree(apCell); + for(i=0; ipgno, nOld, nNew, nCell)); + return rc; +} + +/* +** This routine is called for the root page of a btree when the root +** page contains no cells. This is an opportunity to make the tree +** shallower by one level. +*/ +static int balance_shallower(MemPage *pPage){ + MemPage *pChild; /* The only child page of pPage */ + Pgno pgnoChild; /* Page number for pChild */ + int rc = SQLITE_OK; /* Return code from subprocedures */ + BtShared *pBt; /* The main BTree structure */ + int mxCellPerPage; /* Maximum number of cells per page */ + u8 **apCell; /* All cells from pages being balanced */ + int *szCell; /* Local size of all cells */ + + assert( pPage->pParent==0 ); + assert( pPage->nCell==0 ); + pBt = pPage->pBt; + mxCellPerPage = MX_CELL(pBt); + apCell = sqliteMallocRaw( mxCellPerPage*(sizeof(u8*)+sizeof(int)) ); + if( apCell==0 ) return SQLITE_NOMEM; + szCell = (int*)&apCell[mxCellPerPage]; + if( pPage->leaf ){ + /* The table is completely empty */ + TRACE(("BALANCE: empty table %d\n", pPage->pgno)); + }else{ + /* The root page is empty but has one child. Transfer the + ** information from that one child into the root page if it + ** will fit. This reduces the depth of the tree by one. + ** + ** If the root page is page 1, it has less space available than + ** its child (due to the 100 byte header that occurs at the beginning + ** of the database fle), so it might not be able to hold all of the + ** information currently contained in the child. If this is the + ** case, then do not do the transfer. Leave page 1 empty except + ** for the right-pointer to the child page. The child page becomes + ** the virtual root of the tree. + */ + pgnoChild = get4byte(&pPage->aData[pPage->hdrOffset+8]); + assert( pgnoChild>0 ); + assert( pgnoChild<=sqlite3pager_pagecount(pPage->pBt->pPager) ); + rc = getPage(pPage->pBt, pgnoChild, &pChild); + if( rc ) goto end_shallow_balance; + if( pPage->pgno==1 ){ + rc = initPage(pChild, pPage); + if( rc ) goto end_shallow_balance; + assert( pChild->nOverflow==0 ); + if( pChild->nFree>=100 ){ + /* The child information will fit on the root page, so do the + ** copy */ + int i; + zeroPage(pPage, pChild->aData[0]); + for(i=0; inCell; i++){ + apCell[i] = findCell(pChild,i); + szCell[i] = cellSizePtr(pChild, apCell[i]); + } + assemblePage(pPage, pChild->nCell, apCell, szCell); + /* Copy the right-pointer of the child to the parent. */ + put4byte(&pPage->aData[pPage->hdrOffset+8], + get4byte(&pChild->aData[pChild->hdrOffset+8])); + freePage(pChild); + TRACE(("BALANCE: child %d transfer to page 1\n", pChild->pgno)); + }else{ + /* The child has more information that will fit on the root. + ** The tree is already balanced. Do nothing. */ + TRACE(("BALANCE: child %d will not fit on page 1\n", pChild->pgno)); + } + }else{ + memcpy(pPage->aData, pChild->aData, pPage->pBt->usableSize); + pPage->isInit = 0; + pPage->pParent = 0; + rc = initPage(pPage, 0); + assert( rc==SQLITE_OK ); + freePage(pChild); + TRACE(("BALANCE: transfer child %d into root %d\n", + pChild->pgno, pPage->pgno)); + } + rc = reparentChildPages(pPage); + assert( pPage->nOverflow==0 ); +#ifndef SQLITE_OMIT_AUTOVACUUM + if( pBt->autoVacuum ){ + int i; + for(i=0; inCell; i++){ + rc = ptrmapPutOvfl(pPage, i); + if( rc!=SQLITE_OK ){ + goto end_shallow_balance; + } + } + } +#endif + if( rc!=SQLITE_OK ) goto end_shallow_balance; + releasePage(pChild); + } +end_shallow_balance: + sqliteFree(apCell); + return rc; +} + + +/* +** The root page is overfull +** +** When this happens, Create a new child page and copy the +** contents of the root into the child. Then make the root +** page an empty page with rightChild pointing to the new +** child. Finally, call balance_internal() on the new child +** to cause it to split. +*/ +static int balance_deeper(MemPage *pPage){ + int rc; /* Return value from subprocedures */ + MemPage *pChild; /* Pointer to a new child page */ + Pgno pgnoChild; /* Page number of the new child page */ + BtShared *pBt; /* The BTree */ + int usableSize; /* Total usable size of a page */ + u8 *data; /* Content of the parent page */ + u8 *cdata; /* Content of the child page */ + int hdr; /* Offset to page header in parent */ + int brk; /* Offset to content of first cell in parent */ + + assert( pPage->pParent==0 ); + assert( pPage->nOverflow>0 ); + pBt = pPage->pBt; + rc = allocatePage(pBt, &pChild, &pgnoChild, pPage->pgno, 0); + if( rc ) return rc; + assert( sqlite3pager_iswriteable(pChild->aData) ); + usableSize = pBt->usableSize; + data = pPage->aData; + hdr = pPage->hdrOffset; + brk = get2byte(&data[hdr+5]); + cdata = pChild->aData; + memcpy(cdata, &data[hdr], pPage->cellOffset+2*pPage->nCell-hdr); + memcpy(&cdata[brk], &data[brk], usableSize-brk); + assert( pChild->isInit==0 ); + rc = initPage(pChild, pPage); + if( rc ) goto balancedeeper_out; + memcpy(pChild->aOvfl, pPage->aOvfl, pPage->nOverflow*sizeof(pPage->aOvfl[0])); + pChild->nOverflow = pPage->nOverflow; + if( pChild->nOverflow ){ + pChild->nFree = 0; + } + assert( pChild->nCell==pPage->nCell ); + zeroPage(pPage, pChild->aData[0] & ~PTF_LEAF); + put4byte(&pPage->aData[pPage->hdrOffset+8], pgnoChild); + TRACE(("BALANCE: copy root %d into %d\n", pPage->pgno, pChild->pgno)); +#ifndef SQLITE_OMIT_AUTOVACUUM + if( pBt->autoVacuum ){ + int i; + rc = ptrmapPut(pBt, pChild->pgno, PTRMAP_BTREE, pPage->pgno); + if( rc ) goto balancedeeper_out; + for(i=0; inCell; i++){ + rc = ptrmapPutOvfl(pChild, i); + if( rc!=SQLITE_OK ){ + return rc; + } + } + } +#endif + rc = balance_nonroot(pChild); + +balancedeeper_out: + releasePage(pChild); + return rc; +} + +/* +** Decide if the page pPage needs to be balanced. If balancing is +** required, call the appropriate balancing routine. +*/ +static int balance(MemPage *pPage, int insert){ + int rc = SQLITE_OK; + if( pPage->pParent==0 ){ + if( pPage->nOverflow>0 ){ + rc = balance_deeper(pPage); + } + if( rc==SQLITE_OK && pPage->nCell==0 ){ + rc = balance_shallower(pPage); + } + }else{ + if( pPage->nOverflow>0 || + (!insert && pPage->nFree>pPage->pBt->usableSize*2/3) ){ + rc = balance_nonroot(pPage); + } + } + return rc; +} + +/* +** This routine checks all cursors that point to table pgnoRoot. +** If any of those cursors other than pExclude were opened with +** wrFlag==0 then this routine returns SQLITE_LOCKED. If all +** cursors that point to pgnoRoot were opened with wrFlag==1 +** then this routine returns SQLITE_OK. +** +** In addition to checking for read-locks (where a read-lock +** means a cursor opened with wrFlag==0) this routine also moves +** all cursors other than pExclude so that they are pointing to the +** first Cell on root page. This is necessary because an insert +** or delete might change the number of cells on a page or delete +** a page entirely and we do not want to leave any cursors +** pointing to non-existant pages or cells. +*/ +static int checkReadLocks(BtShared *pBt, Pgno pgnoRoot, BtCursor *pExclude){ + BtCursor *p; + for(p=pBt->pCursor; p; p=p->pNext){ + u32 flags = (p->pBtree->pSqlite ? p->pBtree->pSqlite->flags : 0); + if( p->pgnoRoot!=pgnoRoot || p==pExclude ) continue; + if( p->wrFlag==0 && flags&SQLITE_ReadUncommitted ) continue; + if( p->wrFlag==0 ) return SQLITE_LOCKED; + if( p->pPage->pgno!=p->pgnoRoot ){ + moveToRoot(p); + } + } + return SQLITE_OK; +} + +/* +** Insert a new record into the BTree. The key is given by (pKey,nKey) +** and the data is given by (pData,nData). The cursor is used only to +** define what table the record should be inserted into. The cursor +** is left pointing at a random location. +** +** For an INTKEY table, only the nKey value of the key is used. pKey is +** ignored. For a ZERODATA table, the pData and nData are both ignored. +*/ +int sqlite3BtreeInsert( + BtCursor *pCur, /* Insert data into the table of this cursor */ + const void *pKey, i64 nKey, /* The key of the new record */ + const void *pData, int nData /* The data of the new record */ +){ + int rc; + int loc; + int szNew; + MemPage *pPage; + BtShared *pBt = pCur->pBtree->pBt; + unsigned char *oldCell; + unsigned char *newCell = 0; + + if( pBt->inTransaction!=TRANS_WRITE ){ + /* Must start a transaction before doing an insert */ + return pBt->readOnly ? SQLITE_READONLY : SQLITE_ERROR; + } + assert( !pBt->readOnly ); + if( !pCur->wrFlag ){ + return SQLITE_PERM; /* Cursor not open for writing */ + } + if( checkReadLocks(pBt, pCur->pgnoRoot, pCur) ){ + return SQLITE_LOCKED; /* The table pCur points to has a read lock */ + } + + /* Save the positions of any other cursors open on this table */ + restoreOrClearCursorPosition(pCur, 0); + if( + SQLITE_OK!=(rc = saveAllCursors(pBt, pCur->pgnoRoot, pCur)) || + SQLITE_OK!=(rc = sqlite3BtreeMoveto(pCur, pKey, nKey, &loc)) + ){ + return rc; + } + + pPage = pCur->pPage; + assert( pPage->intKey || nKey>=0 ); + assert( pPage->leaf || !pPage->leafData ); + TRACE(("INSERT: table=%d nkey=%lld ndata=%d page=%d %s\n", + pCur->pgnoRoot, nKey, nData, pPage->pgno, + loc==0 ? "overwrite" : "new entry")); + assert( pPage->isInit ); + rc = sqlite3pager_write(pPage->aData); + if( rc ) return rc; + newCell = sqliteMallocRaw( MX_CELL_SIZE(pBt) ); + if( newCell==0 ) return SQLITE_NOMEM; + rc = fillInCell(pPage, newCell, pKey, nKey, pData, nData, &szNew); + if( rc ) goto end_insert; + assert( szNew==cellSizePtr(pPage, newCell) ); + assert( szNew<=MX_CELL_SIZE(pBt) ); + if( loc==0 && CURSOR_VALID==pCur->eState ){ + int szOld; + assert( pCur->idx>=0 && pCur->idxnCell ); + oldCell = findCell(pPage, pCur->idx); + if( !pPage->leaf ){ + memcpy(newCell, oldCell, 4); + } + szOld = cellSizePtr(pPage, oldCell); + rc = clearCell(pPage, oldCell); + if( rc ) goto end_insert; + dropCell(pPage, pCur->idx, szOld); + }else if( loc<0 && pPage->nCell>0 ){ + assert( pPage->leaf ); + pCur->idx++; + pCur->info.nSize = 0; + }else{ + assert( pPage->leaf ); + } + rc = insertCell(pPage, pCur->idx, newCell, szNew, 0, 0); + if( rc!=SQLITE_OK ) goto end_insert; + rc = balance(pPage, 1); + /* sqlite3BtreePageDump(pCur->pBt, pCur->pgnoRoot, 1); */ + /* fflush(stdout); */ + if( rc==SQLITE_OK ){ + moveToRoot(pCur); + } +end_insert: + sqliteFree(newCell); + return rc; +} + +/* +** Delete the entry that the cursor is pointing to. The cursor +** is left pointing at a random location. +*/ +int sqlite3BtreeDelete(BtCursor *pCur){ + MemPage *pPage = pCur->pPage; + unsigned char *pCell; + int rc; + Pgno pgnoChild = 0; + BtShared *pBt = pCur->pBtree->pBt; + + assert( pPage->isInit ); + if( pBt->inTransaction!=TRANS_WRITE ){ + /* Must start a transaction before doing a delete */ + return pBt->readOnly ? SQLITE_READONLY : SQLITE_ERROR; + } + assert( !pBt->readOnly ); + if( pCur->idx >= pPage->nCell ){ + return SQLITE_ERROR; /* The cursor is not pointing to anything */ + } + if( !pCur->wrFlag ){ + return SQLITE_PERM; /* Did not open this cursor for writing */ + } + if( checkReadLocks(pBt, pCur->pgnoRoot, pCur) ){ + return SQLITE_LOCKED; /* The table pCur points to has a read lock */ + } + + /* Restore the current cursor position (a no-op if the cursor is not in + ** CURSOR_REQUIRESEEK state) and save the positions of any other cursors + ** open on the same table. Then call sqlite3pager_write() on the page + ** that the entry will be deleted from. + */ + if( + (rc = restoreOrClearCursorPosition(pCur, 1))!=0 || + (rc = saveAllCursors(pBt, pCur->pgnoRoot, pCur))!=0 || + (rc = sqlite3pager_write(pPage->aData))!=0 + ){ + return rc; + } + + /* Locate the cell within it's page and leave pCell pointing to the + ** data. The clearCell() call frees any overflow pages associated with the + ** cell. The cell itself is still intact. + */ + pCell = findCell(pPage, pCur->idx); + if( !pPage->leaf ){ + pgnoChild = get4byte(pCell); + } + rc = clearCell(pPage, pCell); + if( rc ) return rc; + + if( !pPage->leaf ){ + /* + ** The entry we are about to delete is not a leaf so if we do not + ** do something we will leave a hole on an internal page. + ** We have to fill the hole by moving in a cell from a leaf. The + ** next Cell after the one to be deleted is guaranteed to exist and + ** to be a leaf so we can use it. + */ + BtCursor leafCur; + unsigned char *pNext; + int szNext; /* The compiler warning is wrong: szNext is always + ** initialized before use. Adding an extra initialization + ** to silence the compiler slows down the code. */ + int notUsed; + unsigned char *tempCell = 0; + assert( !pPage->leafData ); + getTempCursor(pCur, &leafCur); + rc = sqlite3BtreeNext(&leafCur, ¬Used); + if( rc!=SQLITE_OK ){ + if( rc!=SQLITE_NOMEM ){ + rc = SQLITE_CORRUPT_BKPT; + } + } + if( rc==SQLITE_OK ){ + rc = sqlite3pager_write(leafCur.pPage->aData); + } + if( rc==SQLITE_OK ){ + TRACE(("DELETE: table=%d delete internal from %d replace from leaf %d\n", + pCur->pgnoRoot, pPage->pgno, leafCur.pPage->pgno)); + dropCell(pPage, pCur->idx, cellSizePtr(pPage, pCell)); + pNext = findCell(leafCur.pPage, leafCur.idx); + szNext = cellSizePtr(leafCur.pPage, pNext); + assert( MX_CELL_SIZE(pBt)>=szNext+4 ); + tempCell = sqliteMallocRaw( MX_CELL_SIZE(pBt) ); + if( tempCell==0 ){ + rc = SQLITE_NOMEM; + } + } + if( rc==SQLITE_OK ){ + rc = insertCell(pPage, pCur->idx, pNext-4, szNext+4, tempCell, 0); + } + if( rc==SQLITE_OK ){ + put4byte(findOverflowCell(pPage, pCur->idx), pgnoChild); + rc = balance(pPage, 0); + } + if( rc==SQLITE_OK ){ + dropCell(leafCur.pPage, leafCur.idx, szNext); + rc = balance(leafCur.pPage, 0); + } + sqliteFree(tempCell); + releaseTempCursor(&leafCur); + }else{ + TRACE(("DELETE: table=%d delete from leaf %d\n", + pCur->pgnoRoot, pPage->pgno)); + dropCell(pPage, pCur->idx, cellSizePtr(pPage, pCell)); + rc = balance(pPage, 0); + } + if( rc==SQLITE_OK ){ + moveToRoot(pCur); + } + return rc; +} + +/* +** Create a new BTree table. Write into *piTable the page +** number for the root page of the new table. +** +** The type of type is determined by the flags parameter. Only the +** following values of flags are currently in use. Other values for +** flags might not work: +** +** BTREE_INTKEY|BTREE_LEAFDATA Used for SQL tables with rowid keys +** BTREE_ZERODATA Used for SQL indices +*/ +int sqlite3BtreeCreateTable(Btree *p, int *piTable, int flags){ + BtShared *pBt = p->pBt; + MemPage *pRoot; + Pgno pgnoRoot; + int rc; + if( pBt->inTransaction!=TRANS_WRITE ){ + /* Must start a transaction first */ + return pBt->readOnly ? SQLITE_READONLY : SQLITE_ERROR; + } + assert( !pBt->readOnly ); + + /* It is illegal to create a table if any cursors are open on the + ** database. This is because in auto-vacuum mode the backend may + ** need to move a database page to make room for the new root-page. + ** If an open cursor was using the page a problem would occur. + */ + if( pBt->pCursor ){ + return SQLITE_LOCKED; + } + +#ifdef SQLITE_OMIT_AUTOVACUUM + rc = allocatePage(pBt, &pRoot, &pgnoRoot, 1, 0); + if( rc ) return rc; +#else + if( pBt->autoVacuum ){ + Pgno pgnoMove; /* Move a page here to make room for the root-page */ + MemPage *pPageMove; /* The page to move to. */ + + /* Read the value of meta[3] from the database to determine where the + ** root page of the new table should go. meta[3] is the largest root-page + ** created so far, so the new root-page is (meta[3]+1). + */ + rc = sqlite3BtreeGetMeta(p, 4, &pgnoRoot); + if( rc!=SQLITE_OK ) return rc; + pgnoRoot++; + + /* The new root-page may not be allocated on a pointer-map page, or the + ** PENDING_BYTE page. + */ + if( pgnoRoot==PTRMAP_PAGENO(pBt, pgnoRoot) || + pgnoRoot==PENDING_BYTE_PAGE(pBt) ){ + pgnoRoot++; + } + assert( pgnoRoot>=3 ); + + /* Allocate a page. The page that currently resides at pgnoRoot will + ** be moved to the allocated page (unless the allocated page happens + ** to reside at pgnoRoot). + */ + rc = allocatePage(pBt, &pPageMove, &pgnoMove, pgnoRoot, 1); + if( rc!=SQLITE_OK ){ + return rc; + } + + if( pgnoMove!=pgnoRoot ){ + u8 eType; + Pgno iPtrPage; + + releasePage(pPageMove); + rc = getPage(pBt, pgnoRoot, &pRoot); + if( rc!=SQLITE_OK ){ + return rc; + } + rc = ptrmapGet(pBt, pgnoRoot, &eType, &iPtrPage); + if( rc!=SQLITE_OK || eType==PTRMAP_ROOTPAGE || eType==PTRMAP_FREEPAGE ){ + releasePage(pRoot); + return rc; + } + assert( eType!=PTRMAP_ROOTPAGE ); + assert( eType!=PTRMAP_FREEPAGE ); + rc = sqlite3pager_write(pRoot->aData); + if( rc!=SQLITE_OK ){ + releasePage(pRoot); + return rc; + } + rc = relocatePage(pBt, pRoot, eType, iPtrPage, pgnoMove); + releasePage(pRoot); + if( rc!=SQLITE_OK ){ + return rc; + } + rc = getPage(pBt, pgnoRoot, &pRoot); + if( rc!=SQLITE_OK ){ + return rc; + } + rc = sqlite3pager_write(pRoot->aData); + if( rc!=SQLITE_OK ){ + releasePage(pRoot); + return rc; + } + }else{ + pRoot = pPageMove; + } + + /* Update the pointer-map and meta-data with the new root-page number. */ + rc = ptrmapPut(pBt, pgnoRoot, PTRMAP_ROOTPAGE, 0); + if( rc ){ + releasePage(pRoot); + return rc; + } + rc = sqlite3BtreeUpdateMeta(p, 4, pgnoRoot); + if( rc ){ + releasePage(pRoot); + return rc; + } + + }else{ + rc = allocatePage(pBt, &pRoot, &pgnoRoot, 1, 0); + if( rc ) return rc; + } +#endif + assert( sqlite3pager_iswriteable(pRoot->aData) ); + zeroPage(pRoot, flags | PTF_LEAF); + sqlite3pager_unref(pRoot->aData); + *piTable = (int)pgnoRoot; + return SQLITE_OK; +} + +/* +** Erase the given database page and all its children. Return +** the page to the freelist. +*/ +static int clearDatabasePage( + BtShared *pBt, /* The BTree that contains the table */ + Pgno pgno, /* Page number to clear */ + MemPage *pParent, /* Parent page. NULL for the root */ + int freePageFlag /* Deallocate page if true */ +){ + MemPage *pPage = 0; + int rc; + unsigned char *pCell; + int i; + + if( pgno>sqlite3pager_pagecount(pBt->pPager) ){ + return SQLITE_CORRUPT_BKPT; + } + + rc = getAndInitPage(pBt, pgno, &pPage, pParent); + if( rc ) goto cleardatabasepage_out; + rc = sqlite3pager_write(pPage->aData); + if( rc ) goto cleardatabasepage_out; + for(i=0; inCell; i++){ + pCell = findCell(pPage, i); + if( !pPage->leaf ){ + rc = clearDatabasePage(pBt, get4byte(pCell), pPage->pParent, 1); + if( rc ) goto cleardatabasepage_out; + } + rc = clearCell(pPage, pCell); + if( rc ) goto cleardatabasepage_out; + } + if( !pPage->leaf ){ + rc = clearDatabasePage(pBt, get4byte(&pPage->aData[8]), pPage->pParent, 1); + if( rc ) goto cleardatabasepage_out; + } + if( freePageFlag ){ + rc = freePage(pPage); + }else{ + zeroPage(pPage, pPage->aData[0] | PTF_LEAF); + } + +cleardatabasepage_out: + releasePage(pPage); + return rc; +} + +/* +** Delete all information from a single table in the database. iTable is +** the page number of the root of the table. After this routine returns, +** the root page is empty, but still exists. +** +** This routine will fail with SQLITE_LOCKED if there are any open +** read cursors on the table. Open write cursors are moved to the +** root of the table. +*/ +int sqlite3BtreeClearTable(Btree *p, int iTable){ + int rc; + BtCursor *pCur; + BtShared *pBt = p->pBt; + sqlite3 *db = p->pSqlite; + if( p->inTrans!=TRANS_WRITE ){ + return pBt->readOnly ? SQLITE_READONLY : SQLITE_ERROR; + } + + /* If this connection is not in read-uncommitted mode and currently has + ** a read-cursor open on the table being cleared, return SQLITE_LOCKED. + */ + if( 0==db || 0==(db->flags&SQLITE_ReadUncommitted) ){ + for(pCur=pBt->pCursor; pCur; pCur=pCur->pNext){ + if( pCur->pBtree==p && pCur->pgnoRoot==(Pgno)iTable ){ + if( 0==pCur->wrFlag ){ + return SQLITE_LOCKED; + } + moveToRoot(pCur); + } + } + } + + /* Save the position of all cursors open on this table */ + if( SQLITE_OK!=(rc = saveAllCursors(pBt, iTable, 0)) ){ + return rc; + } + + return clearDatabasePage(pBt, (Pgno)iTable, 0, 0); +} + +/* +** Erase all information in a table and add the root of the table to +** the freelist. Except, the root of the principle table (the one on +** page 1) is never added to the freelist. +** +** This routine will fail with SQLITE_LOCKED if there are any open +** cursors on the table. +** +** If AUTOVACUUM is enabled and the page at iTable is not the last +** root page in the database file, then the last root page +** in the database file is moved into the slot formerly occupied by +** iTable and that last slot formerly occupied by the last root page +** is added to the freelist instead of iTable. In this say, all +** root pages are kept at the beginning of the database file, which +** is necessary for AUTOVACUUM to work right. *piMoved is set to the +** page number that used to be the last root page in the file before +** the move. If no page gets moved, *piMoved is set to 0. +** The last root page is recorded in meta[3] and the value of +** meta[3] is updated by this procedure. +*/ +int sqlite3BtreeDropTable(Btree *p, int iTable, int *piMoved){ + int rc; + MemPage *pPage = 0; + BtShared *pBt = p->pBt; + + if( p->inTrans!=TRANS_WRITE ){ + return pBt->readOnly ? SQLITE_READONLY : SQLITE_ERROR; + } + + /* It is illegal to drop a table if any cursors are open on the + ** database. This is because in auto-vacuum mode the backend may + ** need to move another root-page to fill a gap left by the deleted + ** root page. If an open cursor was using this page a problem would + ** occur. + */ + if( pBt->pCursor ){ + return SQLITE_LOCKED; + } + + rc = getPage(pBt, (Pgno)iTable, &pPage); + if( rc ) return rc; + rc = sqlite3BtreeClearTable(p, iTable); + if( rc ){ + releasePage(pPage); + return rc; + } + + *piMoved = 0; + + if( iTable>1 ){ +#ifdef SQLITE_OMIT_AUTOVACUUM + rc = freePage(pPage); + releasePage(pPage); +#else + if( pBt->autoVacuum ){ + Pgno maxRootPgno; + rc = sqlite3BtreeGetMeta(p, 4, &maxRootPgno); + if( rc!=SQLITE_OK ){ + releasePage(pPage); + return rc; + } + + if( iTable==maxRootPgno ){ + /* If the table being dropped is the table with the largest root-page + ** number in the database, put the root page on the free list. + */ + rc = freePage(pPage); + releasePage(pPage); + if( rc!=SQLITE_OK ){ + return rc; + } + }else{ + /* The table being dropped does not have the largest root-page + ** number in the database. So move the page that does into the + ** gap left by the deleted root-page. + */ + MemPage *pMove; + releasePage(pPage); + rc = getPage(pBt, maxRootPgno, &pMove); + if( rc!=SQLITE_OK ){ + return rc; + } + rc = relocatePage(pBt, pMove, PTRMAP_ROOTPAGE, 0, iTable); + releasePage(pMove); + if( rc!=SQLITE_OK ){ + return rc; + } + rc = getPage(pBt, maxRootPgno, &pMove); + if( rc!=SQLITE_OK ){ + return rc; + } + rc = freePage(pMove); + releasePage(pMove); + if( rc!=SQLITE_OK ){ + return rc; + } + *piMoved = maxRootPgno; + } + + /* Set the new 'max-root-page' value in the database header. This + ** is the old value less one, less one more if that happens to + ** be a root-page number, less one again if that is the + ** PENDING_BYTE_PAGE. + */ + maxRootPgno--; + if( maxRootPgno==PENDING_BYTE_PAGE(pBt) ){ + maxRootPgno--; + } + if( maxRootPgno==PTRMAP_PAGENO(pBt, maxRootPgno) ){ + maxRootPgno--; + } + assert( maxRootPgno!=PENDING_BYTE_PAGE(pBt) ); + + rc = sqlite3BtreeUpdateMeta(p, 4, maxRootPgno); + }else{ + rc = freePage(pPage); + releasePage(pPage); + } +#endif + }else{ + /* If sqlite3BtreeDropTable was called on page 1. */ + zeroPage(pPage, PTF_INTKEY|PTF_LEAF ); + releasePage(pPage); + } + return rc; +} + + +/* +** Read the meta-information out of a database file. Meta[0] +** is the number of free pages currently in the database. Meta[1] +** through meta[15] are available for use by higher layers. Meta[0] +** is read-only, the others are read/write. +** +** The schema layer numbers meta values differently. At the schema +** layer (and the SetCookie and ReadCookie opcodes) the number of +** free pages is not visible. So Cookie[0] is the same as Meta[1]. +*/ +int sqlite3BtreeGetMeta(Btree *p, int idx, u32 *pMeta){ + int rc; + unsigned char *pP1; + BtShared *pBt = p->pBt; + + /* Reading a meta-data value requires a read-lock on page 1 (and hence + ** the sqlite_master table. We grab this lock regardless of whether or + ** not the SQLITE_ReadUncommitted flag is set (the table rooted at page + ** 1 is treated as a special case by queryTableLock() and lockTable()). + */ + rc = queryTableLock(p, 1, READ_LOCK); + if( rc!=SQLITE_OK ){ + return rc; + } + + assert( idx>=0 && idx<=15 ); + rc = sqlite3pager_get(pBt->pPager, 1, (void**)&pP1); + if( rc ) return rc; + *pMeta = get4byte(&pP1[36 + idx*4]); + sqlite3pager_unref(pP1); + + /* If autovacuumed is disabled in this build but we are trying to + ** access an autovacuumed database, then make the database readonly. + */ +#ifdef SQLITE_OMIT_AUTOVACUUM + if( idx==4 && *pMeta>0 ) pBt->readOnly = 1; +#endif + + /* Grab the read-lock on page 1. */ + rc = lockTable(p, 1, READ_LOCK); + return rc; +} + +/* +** Write meta-information back into the database. Meta[0] is +** read-only and may not be written. +*/ +int sqlite3BtreeUpdateMeta(Btree *p, int idx, u32 iMeta){ + BtShared *pBt = p->pBt; + unsigned char *pP1; + int rc; + assert( idx>=1 && idx<=15 ); + if( p->inTrans!=TRANS_WRITE ){ + return pBt->readOnly ? SQLITE_READONLY : SQLITE_ERROR; + } + assert( pBt->pPage1!=0 ); + pP1 = pBt->pPage1->aData; + rc = sqlite3pager_write(pP1); + if( rc ) return rc; + put4byte(&pP1[36 + idx*4], iMeta); + return SQLITE_OK; +} + +/* +** Return the flag byte at the beginning of the page that the cursor +** is currently pointing to. +*/ +int sqlite3BtreeFlags(BtCursor *pCur){ + /* TODO: What about CURSOR_REQUIRESEEK state? Probably need to call + ** restoreOrClearCursorPosition() here. + */ + MemPage *pPage = pCur->pPage; + return pPage ? pPage->aData[pPage->hdrOffset] : 0; +} + +#ifdef SQLITE_DEBUG +/* +** Print a disassembly of the given page on standard output. This routine +** is used for debugging and testing only. +*/ +static int btreePageDump(BtShared *pBt, int pgno, int recursive, MemPage *pParent){ + int rc; + MemPage *pPage; + int i, j, c; + int nFree; + u16 idx; + int hdr; + int nCell; + int isInit; + unsigned char *data; + char range[20]; + unsigned char payload[20]; + + rc = getPage(pBt, (Pgno)pgno, &pPage); + isInit = pPage->isInit; + if( pPage->isInit==0 ){ + initPage(pPage, pParent); + } + if( rc ){ + return rc; + } + hdr = pPage->hdrOffset; + data = pPage->aData; + c = data[hdr]; + pPage->intKey = (c & (PTF_INTKEY|PTF_LEAFDATA))!=0; + pPage->zeroData = (c & PTF_ZERODATA)!=0; + pPage->leafData = (c & PTF_LEAFDATA)!=0; + pPage->leaf = (c & PTF_LEAF)!=0; + pPage->hasData = !(pPage->zeroData || (!pPage->leaf && pPage->leafData)); + nCell = get2byte(&data[hdr+3]); + sqlite3DebugPrintf("PAGE %d: flags=0x%02x frag=%d parent=%d\n", pgno, + data[hdr], data[hdr+7], + (pPage->isInit && pPage->pParent) ? pPage->pParent->pgno : 0); + assert( hdr == (pgno==1 ? 100 : 0) ); + idx = hdr + 12 - pPage->leaf*4; + for(i=0; ileaf ){ + child = 0; + }else{ + child = get4byte(pCell); + } + sz = info.nData; + if( !pPage->intKey ) sz += info.nKey; + if( sz>sizeof(payload)-1 ) sz = sizeof(payload)-1; + memcpy(payload, &pCell[info.nHeader], sz); + for(j=0; j0x7f ) payload[j] = '.'; + } + payload[sz] = 0; + sqlite3DebugPrintf( + "cell %2d: i=%-10s chld=%-4d nk=%-4lld nd=%-4d payload=%s\n", + i, range, child, info.nKey, info.nData, payload + ); + } + if( !pPage->leaf ){ + sqlite3DebugPrintf("right_child: %d\n", get4byte(&data[hdr+8])); + } + nFree = 0; + i = 0; + idx = get2byte(&data[hdr+1]); + while( idx>0 && idxpBt->usableSize ){ + int sz = get2byte(&data[idx+2]); + sprintf(range,"%d..%d", idx, idx+sz-1); + nFree += sz; + sqlite3DebugPrintf("freeblock %2d: i=%-10s size=%-4d total=%d\n", + i, range, sz, nFree); + idx = get2byte(&data[idx]); + i++; + } + if( idx!=0 ){ + sqlite3DebugPrintf("ERROR: next freeblock index out of range: %d\n", idx); + } + if( recursive && !pPage->leaf ){ + for(i=0; iisInit = isInit; + sqlite3pager_unref(data); + fflush(stdout); + return SQLITE_OK; +} +int sqlite3BtreePageDump(Btree *p, int pgno, int recursive){ + return btreePageDump(p->pBt, pgno, recursive, 0); +} +#endif + +#if defined(SQLITE_TEST) && defined(SQLITE_DEBUG) +/* +** Fill aResult[] with information about the entry and page that the +** cursor is pointing to. +** +** aResult[0] = The page number +** aResult[1] = The entry number +** aResult[2] = Total number of entries on this page +** aResult[3] = Cell size (local payload + header) +** aResult[4] = Number of free bytes on this page +** aResult[5] = Number of free blocks on the page +** aResult[6] = Total payload size (local + overflow) +** aResult[7] = Header size in bytes +** aResult[8] = Local payload size +** aResult[9] = Parent page number +** +** This routine is used for testing and debugging only. +*/ +int sqlite3BtreeCursorInfo(BtCursor *pCur, int *aResult, int upCnt){ + int cnt, idx; + MemPage *pPage = pCur->pPage; + BtCursor tmpCur; + + int rc = restoreOrClearCursorPosition(pCur, 1); + if( rc!=SQLITE_OK ){ + return rc; + } + + pageIntegrity(pPage); + assert( pPage->isInit ); + getTempCursor(pCur, &tmpCur); + while( upCnt-- ){ + moveToParent(&tmpCur); + } + pPage = tmpCur.pPage; + pageIntegrity(pPage); + aResult[0] = sqlite3pager_pagenumber(pPage->aData); + assert( aResult[0]==pPage->pgno ); + aResult[1] = tmpCur.idx; + aResult[2] = pPage->nCell; + if( tmpCur.idx>=0 && tmpCur.idxnCell ){ + getCellInfo(&tmpCur); + aResult[3] = tmpCur.info.nSize; + aResult[6] = tmpCur.info.nData; + aResult[7] = tmpCur.info.nHeader; + aResult[8] = tmpCur.info.nLocal; + }else{ + aResult[3] = 0; + aResult[6] = 0; + aResult[7] = 0; + aResult[8] = 0; + } + aResult[4] = pPage->nFree; + cnt = 0; + idx = get2byte(&pPage->aData[pPage->hdrOffset+1]); + while( idx>0 && idxpBt->usableSize ){ + cnt++; + idx = get2byte(&pPage->aData[idx]); + } + aResult[5] = cnt; + if( pPage->pParent==0 || isRootPage(pPage) ){ + aResult[9] = 0; + }else{ + aResult[9] = pPage->pParent->pgno; + } + releaseTempCursor(&tmpCur); + return SQLITE_OK; +} +#endif + +/* +** Return the pager associated with a BTree. This routine is used for +** testing and debugging only. +*/ +Pager *sqlite3BtreePager(Btree *p){ + return p->pBt->pPager; +} + +/* +** This structure is passed around through all the sanity checking routines +** in order to keep track of some global state information. +*/ +typedef struct IntegrityCk IntegrityCk; +struct IntegrityCk { + BtShared *pBt; /* The tree being checked out */ + Pager *pPager; /* The associated pager. Also accessible by pBt->pPager */ + int nPage; /* Number of pages in the database */ + int *anRef; /* Number of times each page is referenced */ + char *zErrMsg; /* An error message. NULL of no errors seen. */ +}; + +#ifndef SQLITE_OMIT_INTEGRITY_CHECK +/* +** Append a message to the error message string. +*/ +static void checkAppendMsg( + IntegrityCk *pCheck, + char *zMsg1, + const char *zFormat, + ... +){ + va_list ap; + char *zMsg2; + va_start(ap, zFormat); + zMsg2 = sqlite3VMPrintf(zFormat, ap); + va_end(ap); + if( zMsg1==0 ) zMsg1 = ""; + if( pCheck->zErrMsg ){ + char *zOld = pCheck->zErrMsg; + pCheck->zErrMsg = 0; + sqlite3SetString(&pCheck->zErrMsg, zOld, "\n", zMsg1, zMsg2, (char*)0); + sqliteFree(zOld); + }else{ + sqlite3SetString(&pCheck->zErrMsg, zMsg1, zMsg2, (char*)0); + } + sqliteFree(zMsg2); +} +#endif /* SQLITE_OMIT_INTEGRITY_CHECK */ + +#ifndef SQLITE_OMIT_INTEGRITY_CHECK +/* +** Add 1 to the reference count for page iPage. If this is the second +** reference to the page, add an error message to pCheck->zErrMsg. +** Return 1 if there are 2 ore more references to the page and 0 if +** if this is the first reference to the page. +** +** Also check that the page number is in bounds. +*/ +static int checkRef(IntegrityCk *pCheck, int iPage, char *zContext){ + if( iPage==0 ) return 1; + if( iPage>pCheck->nPage || iPage<0 ){ + checkAppendMsg(pCheck, zContext, "invalid page number %d", iPage); + return 1; + } + if( pCheck->anRef[iPage]==1 ){ + checkAppendMsg(pCheck, zContext, "2nd reference to page %d", iPage); + return 1; + } + return (pCheck->anRef[iPage]++)>1; +} + +#ifndef SQLITE_OMIT_AUTOVACUUM +/* +** Check that the entry in the pointer-map for page iChild maps to +** page iParent, pointer type ptrType. If not, append an error message +** to pCheck. +*/ +static void checkPtrmap( + IntegrityCk *pCheck, /* Integrity check context */ + Pgno iChild, /* Child page number */ + u8 eType, /* Expected pointer map type */ + Pgno iParent, /* Expected pointer map parent page number */ + char *zContext /* Context description (used for error msg) */ +){ + int rc; + u8 ePtrmapType; + Pgno iPtrmapParent; + + rc = ptrmapGet(pCheck->pBt, iChild, &ePtrmapType, &iPtrmapParent); + if( rc!=SQLITE_OK ){ + checkAppendMsg(pCheck, zContext, "Failed to read ptrmap key=%d", iChild); + return; + } + + if( ePtrmapType!=eType || iPtrmapParent!=iParent ){ + checkAppendMsg(pCheck, zContext, + "Bad ptr map entry key=%d expected=(%d,%d) got=(%d,%d)", + iChild, eType, iParent, ePtrmapType, iPtrmapParent); + } +} +#endif + +/* +** Check the integrity of the freelist or of an overflow page list. +** Verify that the number of pages on the list is N. +*/ +static void checkList( + IntegrityCk *pCheck, /* Integrity checking context */ + int isFreeList, /* True for a freelist. False for overflow page list */ + int iPage, /* Page number for first page in the list */ + int N, /* Expected number of pages in the list */ + char *zContext /* Context for error messages */ +){ + int i; + int expected = N; + int iFirst = iPage; + while( N-- > 0 ){ + unsigned char *pOvfl; + if( iPage<1 ){ + checkAppendMsg(pCheck, zContext, + "%d of %d pages missing from overflow list starting at %d", + N+1, expected, iFirst); + break; + } + if( checkRef(pCheck, iPage, zContext) ) break; + if( sqlite3pager_get(pCheck->pPager, (Pgno)iPage, (void**)&pOvfl) ){ + checkAppendMsg(pCheck, zContext, "failed to get page %d", iPage); + break; + } + if( isFreeList ){ + int n = get4byte(&pOvfl[4]); +#ifndef SQLITE_OMIT_AUTOVACUUM + if( pCheck->pBt->autoVacuum ){ + checkPtrmap(pCheck, iPage, PTRMAP_FREEPAGE, 0, zContext); + } +#endif + if( n>pCheck->pBt->usableSize/4-8 ){ + checkAppendMsg(pCheck, zContext, + "freelist leaf count too big on page %d", iPage); + N--; + }else{ + for(i=0; ipBt->autoVacuum ){ + checkPtrmap(pCheck, iFreePage, PTRMAP_FREEPAGE, 0, zContext); + } +#endif + checkRef(pCheck, iFreePage, zContext); + } + N -= n; + } + } +#ifndef SQLITE_OMIT_AUTOVACUUM + else{ + /* If this database supports auto-vacuum and iPage is not the last + ** page in this overflow list, check that the pointer-map entry for + ** the following page matches iPage. + */ + if( pCheck->pBt->autoVacuum && N>0 ){ + i = get4byte(pOvfl); + checkPtrmap(pCheck, i, PTRMAP_OVERFLOW2, iPage, zContext); + } + } +#endif + iPage = get4byte(pOvfl); + sqlite3pager_unref(pOvfl); + } +} +#endif /* SQLITE_OMIT_INTEGRITY_CHECK */ + +#ifndef SQLITE_OMIT_INTEGRITY_CHECK +/* +** Do various sanity checks on a single page of a tree. Return +** the tree depth. Root pages return 0. Parents of root pages +** return 1, and so forth. +** +** These checks are done: +** +** 1. Make sure that cells and freeblocks do not overlap +** but combine to completely cover the page. +** NO 2. Make sure cell keys are in order. +** NO 3. Make sure no key is less than or equal to zLowerBound. +** NO 4. Make sure no key is greater than or equal to zUpperBound. +** 5. Check the integrity of overflow pages. +** 6. Recursively call checkTreePage on all children. +** 7. Verify that the depth of all children is the same. +** 8. Make sure this page is at least 33% full or else it is +** the root of the tree. +*/ +static int checkTreePage( + IntegrityCk *pCheck, /* Context for the sanity check */ + int iPage, /* Page number of the page to check */ + MemPage *pParent, /* Parent page */ + char *zParentContext, /* Parent context */ + char *zLowerBound, /* All keys should be greater than this, if not NULL */ + int nLower, /* Number of characters in zLowerBound */ + char *zUpperBound, /* All keys should be less than this, if not NULL */ + int nUpper /* Number of characters in zUpperBound */ +){ + MemPage *pPage; + int i, rc, depth, d2, pgno, cnt; + int hdr, cellStart; + int nCell; + u8 *data; + BtShared *pBt; + int usableSize; + char zContext[100]; + char *hit; + + sprintf(zContext, "Page %d: ", iPage); + + /* Check that the page exists + */ + pBt = pCheck->pBt; + usableSize = pBt->usableSize; + if( iPage==0 ) return 0; + if( checkRef(pCheck, iPage, zParentContext) ) return 0; + if( (rc = getPage(pBt, (Pgno)iPage, &pPage))!=0 ){ + checkAppendMsg(pCheck, zContext, + "unable to get the page. error code=%d", rc); + return 0; + } + if( (rc = initPage(pPage, pParent))!=0 ){ + checkAppendMsg(pCheck, zContext, "initPage() returns error code %d", rc); + releasePage(pPage); + return 0; + } + + /* Check out all the cells. + */ + depth = 0; + for(i=0; inCell; i++){ + u8 *pCell; + int sz; + CellInfo info; + + /* Check payload overflow pages + */ + sprintf(zContext, "On tree page %d cell %d: ", iPage, i); + pCell = findCell(pPage,i); + parseCellPtr(pPage, pCell, &info); + sz = info.nData; + if( !pPage->intKey ) sz += info.nKey; + if( sz>info.nLocal ){ + int nPage = (sz - info.nLocal + usableSize - 5)/(usableSize - 4); + Pgno pgnoOvfl = get4byte(&pCell[info.iOverflow]); +#ifndef SQLITE_OMIT_AUTOVACUUM + if( pBt->autoVacuum ){ + checkPtrmap(pCheck, pgnoOvfl, PTRMAP_OVERFLOW1, iPage, zContext); + } +#endif + checkList(pCheck, 0, pgnoOvfl, nPage, zContext); + } + + /* Check sanity of left child page. + */ + if( !pPage->leaf ){ + pgno = get4byte(pCell); +#ifndef SQLITE_OMIT_AUTOVACUUM + if( pBt->autoVacuum ){ + checkPtrmap(pCheck, pgno, PTRMAP_BTREE, iPage, zContext); + } +#endif + d2 = checkTreePage(pCheck,pgno,pPage,zContext,0,0,0,0); + if( i>0 && d2!=depth ){ + checkAppendMsg(pCheck, zContext, "Child page depth differs"); + } + depth = d2; + } + } + if( !pPage->leaf ){ + pgno = get4byte(&pPage->aData[pPage->hdrOffset+8]); + sprintf(zContext, "On page %d at right child: ", iPage); +#ifndef SQLITE_OMIT_AUTOVACUUM + if( pBt->autoVacuum ){ + checkPtrmap(pCheck, pgno, PTRMAP_BTREE, iPage, 0); + } +#endif + checkTreePage(pCheck, pgno, pPage, zContext,0,0,0,0); + } + + /* Check for complete coverage of the page + */ + data = pPage->aData; + hdr = pPage->hdrOffset; + hit = sqliteMalloc( usableSize ); + if( hit ){ + memset(hit, 1, get2byte(&data[hdr+5])); + nCell = get2byte(&data[hdr+3]); + cellStart = hdr + 12 - 4*pPage->leaf; + for(i=0; i=usableSize || pc<0 ){ + checkAppendMsg(pCheck, 0, + "Corruption detected in cell %d on page %d",i,iPage,0); + }else{ + for(j=pc+size-1; j>=pc; j--) hit[j]++; + } + } + for(cnt=0, i=get2byte(&data[hdr+1]); i>0 && i=usableSize || i<0 ){ + checkAppendMsg(pCheck, 0, + "Corruption detected in cell %d on page %d",i,iPage,0); + }else{ + for(j=i+size-1; j>=i; j--) hit[j]++; + } + i = get2byte(&data[i]); + } + for(i=cnt=0; i1 ){ + checkAppendMsg(pCheck, 0, + "Multiple uses for byte %d of page %d", i, iPage); + break; + } + } + if( cnt!=data[hdr+7] ){ + checkAppendMsg(pCheck, 0, + "Fragmented space is %d byte reported as %d on page %d", + cnt, data[hdr+7], iPage); + } + } + sqliteFree(hit); + + releasePage(pPage); + return depth+1; +} +#endif /* SQLITE_OMIT_INTEGRITY_CHECK */ + +#ifndef SQLITE_OMIT_INTEGRITY_CHECK +/* +** This routine does a complete check of the given BTree file. aRoot[] is +** an array of pages numbers were each page number is the root page of +** a table. nRoot is the number of entries in aRoot. +** +** If everything checks out, this routine returns NULL. If something is +** amiss, an error message is written into memory obtained from malloc() +** and a pointer to that error message is returned. The calling function +** is responsible for freeing the error message when it is done. +*/ +char *sqlite3BtreeIntegrityCheck(Btree *p, int *aRoot, int nRoot){ + int i; + int nRef; + IntegrityCk sCheck; + BtShared *pBt = p->pBt; + + nRef = *sqlite3pager_stats(pBt->pPager); + if( lockBtreeWithRetry(p)!=SQLITE_OK ){ + return sqliteStrDup("Unable to acquire a read lock on the database"); + } + sCheck.pBt = pBt; + sCheck.pPager = pBt->pPager; + sCheck.nPage = sqlite3pager_pagecount(sCheck.pPager); + if( sCheck.nPage==0 ){ + unlockBtreeIfUnused(pBt); + return 0; + } + sCheck.anRef = sqliteMallocRaw( (sCheck.nPage+1)*sizeof(sCheck.anRef[0]) ); + if( !sCheck.anRef ){ + unlockBtreeIfUnused(pBt); + return sqlite3MPrintf("Unable to malloc %d bytes", + (sCheck.nPage+1)*sizeof(sCheck.anRef[0])); + } + for(i=0; i<=sCheck.nPage; i++){ sCheck.anRef[i] = 0; } + i = PENDING_BYTE_PAGE(pBt); + if( i<=sCheck.nPage ){ + sCheck.anRef[i] = 1; + } + sCheck.zErrMsg = 0; + + /* Check the integrity of the freelist + */ + checkList(&sCheck, 1, get4byte(&pBt->pPage1->aData[32]), + get4byte(&pBt->pPage1->aData[36]), "Main freelist: "); + + /* Check all the tables. + */ + for(i=0; iautoVacuum && aRoot[i]>1 ){ + checkPtrmap(&sCheck, aRoot[i], PTRMAP_ROOTPAGE, 0, 0); + } +#endif + checkTreePage(&sCheck, aRoot[i], 0, "List of tree roots: ", 0,0,0,0); + } + + /* Make sure every page in the file is referenced + */ + for(i=1; i<=sCheck.nPage; i++){ +#ifdef SQLITE_OMIT_AUTOVACUUM + if( sCheck.anRef[i]==0 ){ + checkAppendMsg(&sCheck, 0, "Page %d is never used", i); + } +#else + /* If the database supports auto-vacuum, make sure no tables contain + ** references to pointer-map pages. + */ + if( sCheck.anRef[i]==0 && + (PTRMAP_PAGENO(pBt, i)!=i || !pBt->autoVacuum) ){ + checkAppendMsg(&sCheck, 0, "Page %d is never used", i); + } + if( sCheck.anRef[i]!=0 && + (PTRMAP_PAGENO(pBt, i)==i && pBt->autoVacuum) ){ + checkAppendMsg(&sCheck, 0, "Pointer map page %d is referenced", i); + } +#endif + } + + /* Make sure this analysis did not leave any unref() pages + */ + unlockBtreeIfUnused(pBt); + if( nRef != *sqlite3pager_stats(pBt->pPager) ){ + checkAppendMsg(&sCheck, 0, + "Outstanding page count goes from %d to %d during this analysis", + nRef, *sqlite3pager_stats(pBt->pPager) + ); + } + + /* Clean up and report errors. + */ + sqliteFree(sCheck.anRef); + return sCheck.zErrMsg; +} +#endif /* SQLITE_OMIT_INTEGRITY_CHECK */ + +/* +** Return the full pathname of the underlying database file. +*/ +const char *sqlite3BtreeGetFilename(Btree *p){ + assert( p->pBt->pPager!=0 ); + return sqlite3pager_filename(p->pBt->pPager); +} + +/* +** Return the pathname of the directory that contains the database file. +*/ +const char *sqlite3BtreeGetDirname(Btree *p){ + assert( p->pBt->pPager!=0 ); + return sqlite3pager_dirname(p->pBt->pPager); +} + +/* +** Return the pathname of the journal file for this database. The return +** value of this routine is the same regardless of whether the journal file +** has been created or not. +*/ +const char *sqlite3BtreeGetJournalname(Btree *p){ + assert( p->pBt->pPager!=0 ); + return sqlite3pager_journalname(p->pBt->pPager); +} + +#ifndef SQLITE_OMIT_VACUUM +/* +** Copy the complete content of pBtFrom into pBtTo. A transaction +** must be active for both files. +** +** The size of file pBtFrom may be reduced by this operation. +** If anything goes wrong, the transaction on pBtFrom is rolled back. +*/ +int sqlite3BtreeCopyFile(Btree *pTo, Btree *pFrom){ + int rc = SQLITE_OK; + Pgno i, nPage, nToPage, iSkip; + + BtShared *pBtTo = pTo->pBt; + BtShared *pBtFrom = pFrom->pBt; + + if( pTo->inTrans!=TRANS_WRITE || pFrom->inTrans!=TRANS_WRITE ){ + return SQLITE_ERROR; + } + if( pBtTo->pCursor ) return SQLITE_BUSY; + nToPage = sqlite3pager_pagecount(pBtTo->pPager); + nPage = sqlite3pager_pagecount(pBtFrom->pPager); + iSkip = PENDING_BYTE_PAGE(pBtTo); + for(i=1; rc==SQLITE_OK && i<=nPage; i++){ + void *pPage; + if( i==iSkip ) continue; + rc = sqlite3pager_get(pBtFrom->pPager, i, &pPage); + if( rc ) break; + rc = sqlite3pager_overwrite(pBtTo->pPager, i, pPage); + if( rc ) break; + sqlite3pager_unref(pPage); + } + for(i=nPage+1; rc==SQLITE_OK && i<=nToPage; i++){ + void *pPage; + if( i==iSkip ) continue; + rc = sqlite3pager_get(pBtTo->pPager, i, &pPage); + if( rc ) break; + rc = sqlite3pager_write(pPage); + sqlite3pager_unref(pPage); + sqlite3pager_dont_write(pBtTo->pPager, i); + } + if( !rc && nPagepPager, nPage); + } + if( rc ){ + sqlite3BtreeRollback(pTo); + } + return rc; +} +#endif /* SQLITE_OMIT_VACUUM */ + +/* +** Return non-zero if a transaction is active. +*/ +int sqlite3BtreeIsInTrans(Btree *p){ + return (p && (p->inTrans==TRANS_WRITE)); +} + +/* +** Return non-zero if a statement transaction is active. +*/ +int sqlite3BtreeIsInStmt(Btree *p){ + return (p->pBt && p->pBt->inStmt); +} + +/* +** This call is a no-op if no write-transaction is currently active on pBt. +** +** Otherwise, sync the database file for the btree pBt. zMaster points to +** the name of a master journal file that should be written into the +** individual journal file, or is NULL, indicating no master journal file +** (single database transaction). +** +** When this is called, the master journal should already have been +** created, populated with this journal pointer and synced to disk. +** +** Once this is routine has returned, the only thing required to commit +** the write-transaction for this database file is to delete the journal. +*/ +int sqlite3BtreeSync(Btree *p, const char *zMaster){ + int rc = SQLITE_OK; + if( p->inTrans==TRANS_WRITE ){ + BtShared *pBt = p->pBt; + Pgno nTrunc = 0; +#ifndef SQLITE_OMIT_AUTOVACUUM + if( pBt->autoVacuum ){ + rc = autoVacuumCommit(pBt, &nTrunc); + if( rc!=SQLITE_OK ){ + return rc; + } + } +#endif + rc = sqlite3pager_sync(pBt->pPager, zMaster, nTrunc); + } + return rc; +} + +/* +** This function returns a pointer to a blob of memory associated with +** a single shared-btree. The memory is used by client code for it's own +** purposes (for example, to store a high-level schema associated with +** the shared-btree). The btree layer manages reference counting issues. +** +** The first time this is called on a shared-btree, nBytes bytes of memory +** are allocated, zeroed, and returned to the caller. For each subsequent +** call the nBytes parameter is ignored and a pointer to the same blob +** of memory returned. +** +** Just before the shared-btree is closed, the function passed as the +** xFree argument when the memory allocation was made is invoked on the +** blob of allocated memory. This function should not call sqliteFree() +** on the memory, the btree layer does that. +*/ +void *sqlite3BtreeSchema(Btree *p, int nBytes, void(*xFree)(void *)){ + BtShared *pBt = p->pBt; + if( !pBt->pSchema ){ + pBt->pSchema = sqliteMalloc(nBytes); + pBt->xFreeSchema = xFree; + } + return pBt->pSchema; +} + +/* +** Return true if another user of the same shared btree as the argument +** handle holds an exclusive lock on the sqlite_master table. +*/ +int sqlite3BtreeSchemaLocked(Btree *p){ + return (queryTableLock(p, MASTER_ROOT, READ_LOCK)!=SQLITE_OK); +} + +int sqlite3BtreeLockTable(Btree *p, int iTab, u8 isWriteLock){ + int rc = SQLITE_OK; +#ifndef SQLITE_OMIT_SHARED_CACHE + u8 lockType = (isWriteLock?WRITE_LOCK:READ_LOCK); + rc = queryTableLock(p, iTab, lockType); + if( rc==SQLITE_OK ){ + rc = lockTable(p, iTab, lockType); + } +#endif + return rc; +} + +/* +** The following debugging interface has to be in this file (rather +** than in, for example, test1.c) so that it can get access to +** the definition of BtShared. +*/ +#if defined(SQLITE_DEBUG) && defined(TCLSH) +#include +int sqlite3_shared_cache_report( + void * clientData, + Tcl_Interp *interp, + int objc, + Tcl_Obj *CONST objv[] +){ + const ThreadData *pTd = sqlite3ThreadDataReadOnly(); + if( pTd->useSharedData ){ + BtShared *pBt; + Tcl_Obj *pRet = Tcl_NewObj(); + for(pBt=pTd->pBtree; pBt; pBt=pBt->pNext){ + const char *zFile = sqlite3pager_filename(pBt->pPager); + Tcl_ListObjAppendElement(interp, pRet, Tcl_NewStringObj(zFile, -1)); + Tcl_ListObjAppendElement(interp, pRet, Tcl_NewIntObj(pBt->nRef)); + } + Tcl_SetObjResult(interp, pRet); + } + return TCL_OK; +} +#endif Added: external/sqlite-source-3.3.4/btree.h ============================================================================== --- (empty file) +++ external/sqlite-source-3.3.4/btree.h Mon Apr 3 07:54:59 2006 @@ -0,0 +1,148 @@ +/* +** 2001 September 15 +** +** The author disclaims copyright to this source code. In place of +** a legal notice, here is a blessing: +** +** May you do good and not evil. +** May you find forgiveness for yourself and forgive others. +** May you share freely, never taking more than you give. +** +************************************************************************* +** This header file defines the interface that the sqlite B-Tree file +** subsystem. See comments in the source code for a detailed description +** of what each interface routine does. +** +** @(#) $Id: btree.h,v 1.70 2006/02/11 01:25:51 drh Exp $ +*/ +#ifndef _BTREE_H_ +#define _BTREE_H_ + +/* TODO: This definition is just included so other modules compile. It +** needs to be revisited. +*/ +#define SQLITE_N_BTREE_META 10 + +/* +** If defined as non-zero, auto-vacuum is enabled by default. Otherwise +** it must be turned on for each database using "PRAGMA auto_vacuum = 1". +*/ +#ifndef SQLITE_DEFAULT_AUTOVACUUM + #define SQLITE_DEFAULT_AUTOVACUUM 0 +#endif + +/* +** Forward declarations of structure +*/ +typedef struct Btree Btree; +typedef struct BtCursor BtCursor; +typedef struct BtShared BtShared; + + +int sqlite3BtreeOpen( + const char *zFilename, /* Name of database file to open */ + sqlite3 *db, /* Associated database connection */ + Btree **, /* Return open Btree* here */ + int flags /* Flags */ +); + +/* The flags parameter to sqlite3BtreeOpen can be the bitwise or of the +** following values. +** +** NOTE: These values must match the corresponding PAGER_ values in +** pager.h. +*/ +#define BTREE_OMIT_JOURNAL 1 /* Do not use journal. No argument */ +#define BTREE_NO_READLOCK 2 /* Omit readlocks on readonly files */ +#define BTREE_MEMORY 4 /* In-memory DB. No argument */ + +int sqlite3BtreeClose(Btree*); +int sqlite3BtreeSetBusyHandler(Btree*,BusyHandler*); +int sqlite3BtreeSetCacheSize(Btree*,int); +int sqlite3BtreeSetSafetyLevel(Btree*,int,int); +int sqlite3BtreeSyncDisabled(Btree*); +int sqlite3BtreeSetPageSize(Btree*,int,int); +int sqlite3BtreeGetPageSize(Btree*); +int sqlite3BtreeGetReserve(Btree*); +int sqlite3BtreeSetAutoVacuum(Btree *, int); +int sqlite3BtreeGetAutoVacuum(Btree *); +int sqlite3BtreeBeginTrans(Btree*,int); +int sqlite3BtreeCommit(Btree*); +int sqlite3BtreeRollback(Btree*); +int sqlite3BtreeBeginStmt(Btree*); +int sqlite3BtreeCommitStmt(Btree*); +int sqlite3BtreeRollbackStmt(Btree*); +int sqlite3BtreeCreateTable(Btree*, int*, int flags); +int sqlite3BtreeIsInTrans(Btree*); +int sqlite3BtreeIsInStmt(Btree*); +int sqlite3BtreeSync(Btree*, const char *zMaster); +void *sqlite3BtreeSchema(Btree *, int, void(*)(void *)); +int sqlite3BtreeSchemaLocked(Btree *); +int sqlite3BtreeLockTable(Btree *, int, u8); + +const char *sqlite3BtreeGetFilename(Btree *); +const char *sqlite3BtreeGetDirname(Btree *); +const char *sqlite3BtreeGetJournalname(Btree *); +int sqlite3BtreeCopyFile(Btree *, Btree *); + +/* The flags parameter to sqlite3BtreeCreateTable can be the bitwise OR +** of the following flags: +*/ +#define BTREE_INTKEY 1 /* Table has only 64-bit signed integer keys */ +#define BTREE_ZERODATA 2 /* Table has keys only - no data */ +#define BTREE_LEAFDATA 4 /* Data stored in leaves only. Implies INTKEY */ + +int sqlite3BtreeDropTable(Btree*, int, int*); +int sqlite3BtreeClearTable(Btree*, int); +int sqlite3BtreeGetMeta(Btree*, int idx, u32 *pValue); +int sqlite3BtreeUpdateMeta(Btree*, int idx, u32 value); + +int sqlite3BtreeCursor( + Btree*, /* BTree containing table to open */ + int iTable, /* Index of root page */ + int wrFlag, /* 1 for writing. 0 for read-only */ + int(*)(void*,int,const void*,int,const void*), /* Key comparison function */ + void*, /* First argument to compare function */ + BtCursor **ppCursor /* Returned cursor */ +); + +void sqlite3BtreeSetCompare( + BtCursor *, + int(*)(void*,int,const void*,int,const void*), + void* +); + +int sqlite3BtreeCloseCursor(BtCursor*); +int sqlite3BtreeMoveto(BtCursor*, const void *pKey, i64 nKey, int *pRes); +int sqlite3BtreeDelete(BtCursor*); +int sqlite3BtreeInsert(BtCursor*, const void *pKey, i64 nKey, + const void *pData, int nData); +int sqlite3BtreeFirst(BtCursor*, int *pRes); +int sqlite3BtreeLast(BtCursor*, int *pRes); +int sqlite3BtreeNext(BtCursor*, int *pRes); +int sqlite3BtreeEof(BtCursor*); +int sqlite3BtreeFlags(BtCursor*); +int sqlite3BtreePrevious(BtCursor*, int *pRes); +int sqlite3BtreeKeySize(BtCursor*, i64 *pSize); +int sqlite3BtreeKey(BtCursor*, u32 offset, u32 amt, void*); +const void *sqlite3BtreeKeyFetch(BtCursor*, int *pAmt); +const void *sqlite3BtreeDataFetch(BtCursor*, int *pAmt); +int sqlite3BtreeDataSize(BtCursor*, u32 *pSize); +int sqlite3BtreeData(BtCursor*, u32 offset, u32 amt, void*); + +char *sqlite3BtreeIntegrityCheck(Btree*, int *aRoot, int nRoot); +struct Pager *sqlite3BtreePager(Btree*); + + +#ifdef SQLITE_TEST +int sqlite3BtreeCursorInfo(BtCursor*, int*, int); +void sqlite3BtreeCursorList(Btree*); +#endif + +#ifdef SQLITE_DEBUG +int sqlite3BtreePageDump(Btree*, int, int recursive); +#else +#define sqlite3BtreePageDump(X,Y,Z) SQLITE_OK +#endif + +#endif /* _BTREE_H_ */ Added: external/sqlite-source-3.3.4/build.c ============================================================================== --- (empty file) +++ external/sqlite-source-3.3.4/build.c Mon Apr 3 07:54:59 2006 @@ -0,0 +1,3221 @@ +/* +** 2001 September 15 +** +** The author disclaims copyright to this source code. In place of +** a legal notice, here is a blessing: +** +** May you do good and not evil. +** May you find forgiveness for yourself and forgive others. +** May you share freely, never taking more than you give. +** +************************************************************************* +** This file contains C code routines that are called by the SQLite parser +** when syntax rules are reduced. The routines in this file handle the +** following kinds of SQL syntax: +** +** CREATE TABLE +** DROP TABLE +** CREATE INDEX +** DROP INDEX +** creating ID lists +** BEGIN TRANSACTION +** COMMIT +** ROLLBACK +** +** $Id: build.c,v 1.386 2006/02/10 18:08:09 drh Exp $ +*/ +#include "sqliteInt.h" +#include + +/* +** This routine is called when a new SQL statement is beginning to +** be parsed. Initialize the pParse structure as needed. +*/ +void sqlite3BeginParse(Parse *pParse, int explainFlag){ + pParse->explain = explainFlag; + pParse->nVar = 0; +} + +#ifndef SQLITE_OMIT_SHARED_CACHE +/* +** The TableLock structure is only used by the sqlite3TableLock() and +** codeTableLocks() functions. +*/ +struct TableLock { + int iDb; + int iTab; + u8 isWriteLock; + const char *zName; +}; + +/* +** Have the compiled statement lock the table with rootpage iTab in database +** iDb at the shared-cache level when executed. The isWriteLock argument +** is zero for a read-lock, or non-zero for a write-lock. +** +** The zName parameter should point to the unqualified table name. This is +** used to provide a more informative error message should the lock fail. +*/ +void sqlite3TableLock( + Parse *pParse, + int iDb, + int iTab, + u8 isWriteLock, + const char *zName +){ + int i; + int nBytes; + TableLock *p; + + if( 0==sqlite3ThreadDataReadOnly()->useSharedData || iDb<0 ){ + return; + } + + for(i=0; inTableLock; i++){ + p = &pParse->aTableLock[i]; + if( p->iDb==iDb && p->iTab==iTab ){ + p->isWriteLock = (p->isWriteLock || isWriteLock); + return; + } + } + + nBytes = sizeof(TableLock) * (pParse->nTableLock+1); + sqliteReallocOrFree((void **)&pParse->aTableLock, nBytes); + if( pParse->aTableLock ){ + p = &pParse->aTableLock[pParse->nTableLock++]; + p->iDb = iDb; + p->iTab = iTab; + p->isWriteLock = isWriteLock; + p->zName = zName; + } +} + +/* +** Code an OP_TableLock instruction for each table locked by the +** statement (configured by calls to sqlite3TableLock()). +*/ +static void codeTableLocks(Parse *pParse){ + int i; + Vdbe *pVdbe; + assert( sqlite3ThreadDataReadOnly()->useSharedData || pParse->nTableLock==0 ); + + if( 0==(pVdbe = sqlite3GetVdbe(pParse)) ){ + return; + } + + for(i=0; inTableLock; i++){ + TableLock *p = &pParse->aTableLock[i]; + int p1 = p->iDb; + if( p->isWriteLock ){ + p1 = -1*(p1+1); + } + sqlite3VdbeOp3(pVdbe, OP_TableLock, p1, p->iTab, p->zName, P3_STATIC); + } +} +#else + #define codeTableLocks(x) +#endif + +/* +** This routine is called after a single SQL statement has been +** parsed and a VDBE program to execute that statement has been +** prepared. This routine puts the finishing touches on the +** VDBE program and resets the pParse structure for the next +** parse. +** +** Note that if an error occurred, it might be the case that +** no VDBE code was generated. +*/ +void sqlite3FinishCoding(Parse *pParse){ + sqlite3 *db; + Vdbe *v; + + if( sqlite3MallocFailed() ) return; + if( pParse->nested ) return; + if( !pParse->pVdbe ){ + if( pParse->rc==SQLITE_OK && pParse->nErr ){ + pParse->rc = SQLITE_ERROR; + } + return; + } + + /* Begin by generating some termination code at the end of the + ** vdbe program + */ + db = pParse->db; + v = sqlite3GetVdbe(pParse); + if( v ){ + sqlite3VdbeAddOp(v, OP_Halt, 0, 0); + + /* The cookie mask contains one bit for each database file open. + ** (Bit 0 is for main, bit 1 is for temp, and so forth.) Bits are + ** set for each database that is used. Generate code to start a + ** transaction on each used database and to verify the schema cookie + ** on each used database. + */ + if( pParse->cookieGoto>0 ){ + u32 mask; + int iDb; + sqlite3VdbeJumpHere(v, pParse->cookieGoto-1); + for(iDb=0, mask=1; iDbnDb; mask<<=1, iDb++){ + if( (mask & pParse->cookieMask)==0 ) continue; + sqlite3VdbeAddOp(v, OP_Transaction, iDb, (mask & pParse->writeMask)!=0); + sqlite3VdbeAddOp(v, OP_VerifyCookie, iDb, pParse->cookieValue[iDb]); + } + + /* Once all the cookies have been verified and transactions opened, + ** obtain the required table-locks. This is a no-op unless the + ** shared-cache feature is enabled. + */ + codeTableLocks(pParse); + sqlite3VdbeAddOp(v, OP_Goto, 0, pParse->cookieGoto); + } + +#ifndef SQLITE_OMIT_TRACE + /* Add a No-op that contains the complete text of the compiled SQL + ** statement as its P3 argument. This does not change the functionality + ** of the program. + ** + ** This is used to implement sqlite3_trace(). + */ + sqlite3VdbeOp3(v, OP_Noop, 0, 0, pParse->zSql, pParse->zTail-pParse->zSql); +#endif /* SQLITE_OMIT_TRACE */ + } + + + /* Get the VDBE program ready for execution + */ + if( v && pParse->nErr==0 ){ + FILE *trace = (db->flags & SQLITE_VdbeTrace)!=0 ? stdout : 0; + sqlite3VdbeTrace(v, trace); + sqlite3VdbeMakeReady(v, pParse->nVar, pParse->nMem+3, + pParse->nTab+3, pParse->explain); + pParse->rc = SQLITE_DONE; + pParse->colNamesSet = 0; + }else if( pParse->rc==SQLITE_OK ){ + pParse->rc = SQLITE_ERROR; + } + pParse->nTab = 0; + pParse->nMem = 0; + pParse->nSet = 0; + pParse->nVar = 0; + pParse->cookieMask = 0; + pParse->cookieGoto = 0; +} + +/* +** Run the parser and code generator recursively in order to generate +** code for the SQL statement given onto the end of the pParse context +** currently under construction. When the parser is run recursively +** this way, the final OP_Halt is not appended and other initialization +** and finalization steps are omitted because those are handling by the +** outermost parser. +** +** Not everything is nestable. This facility is designed to permit +** INSERT, UPDATE, and DELETE operations against SQLITE_MASTER. Use +** care if you decide to try to use this routine for some other purposes. +*/ +void sqlite3NestedParse(Parse *pParse, const char *zFormat, ...){ + va_list ap; + char *zSql; +# define SAVE_SZ (sizeof(Parse) - offsetof(Parse,nVar)) + char saveBuf[SAVE_SZ]; + + if( pParse->nErr ) return; + assert( pParse->nested<10 ); /* Nesting should only be of limited depth */ + va_start(ap, zFormat); + zSql = sqlite3VMPrintf(zFormat, ap); + va_end(ap); + if( zSql==0 ){ + return; /* A malloc must have failed */ + } + pParse->nested++; + memcpy(saveBuf, &pParse->nVar, SAVE_SZ); + memset(&pParse->nVar, 0, SAVE_SZ); + sqlite3RunParser(pParse, zSql, 0); + sqliteFree(zSql); + memcpy(&pParse->nVar, saveBuf, SAVE_SZ); + pParse->nested--; +} + +/* +** Locate the in-memory structure that describes a particular database +** table given the name of that table and (optionally) the name of the +** database containing the table. Return NULL if not found. +** +** If zDatabase is 0, all databases are searched for the table and the +** first matching table is returned. (No checking for duplicate table +** names is done.) The search order is TEMP first, then MAIN, then any +** auxiliary databases added using the ATTACH command. +** +** See also sqlite3LocateTable(). +*/ +Table *sqlite3FindTable(sqlite3 *db, const char *zName, const char *zDatabase){ + Table *p = 0; + int i; + assert( zName!=0 ); + for(i=OMIT_TEMPDB; inDb; i++){ + int j = (i<2) ? i^1 : i; /* Search TEMP before MAIN */ + if( zDatabase!=0 && sqlite3StrICmp(zDatabase, db->aDb[j].zName) ) continue; + p = sqlite3HashFind(&db->aDb[j].pSchema->tblHash, zName, strlen(zName)+1); + if( p ) break; + } + return p; +} + +/* +** Locate the in-memory structure that describes a particular database +** table given the name of that table and (optionally) the name of the +** database containing the table. Return NULL if not found. Also leave an +** error message in pParse->zErrMsg. +** +** The difference between this routine and sqlite3FindTable() is that this +** routine leaves an error message in pParse->zErrMsg where +** sqlite3FindTable() does not. +*/ +Table *sqlite3LocateTable(Parse *pParse, const char *zName, const char *zDbase){ + Table *p; + + /* Read the database schema. If an error occurs, leave an error message + ** and code in pParse and return NULL. */ + if( SQLITE_OK!=sqlite3ReadSchema(pParse) ){ + return 0; + } + + p = sqlite3FindTable(pParse->db, zName, zDbase); + if( p==0 ){ + if( zDbase ){ + sqlite3ErrorMsg(pParse, "no such table: %s.%s", zDbase, zName); + }else{ + sqlite3ErrorMsg(pParse, "no such table: %s", zName); + } + pParse->checkSchema = 1; + } + return p; +} + +/* +** Locate the in-memory structure that describes +** a particular index given the name of that index +** and the name of the database that contains the index. +** Return NULL if not found. +** +** If zDatabase is 0, all databases are searched for the +** table and the first matching index is returned. (No checking +** for duplicate index names is done.) The search order is +** TEMP first, then MAIN, then any auxiliary databases added +** using the ATTACH command. +*/ +Index *sqlite3FindIndex(sqlite3 *db, const char *zName, const char *zDb){ + Index *p = 0; + int i; + for(i=OMIT_TEMPDB; inDb; i++){ + int j = (i<2) ? i^1 : i; /* Search TEMP before MAIN */ + Schema *pSchema = db->aDb[j].pSchema; + if( zDb && sqlite3StrICmp(zDb, db->aDb[j].zName) ) continue; + assert( pSchema || (j==1 && !db->aDb[1].pBt) ); + if( pSchema ){ + p = sqlite3HashFind(&pSchema->idxHash, zName, strlen(zName)+1); + } + if( p ) break; + } + return p; +} + +/* +** Reclaim the memory used by an index +*/ +static void freeIndex(Index *p){ + sqliteFree(p->zColAff); + sqliteFree(p); +} + +/* +** Remove the given index from the index hash table, and free +** its memory structures. +** +** The index is removed from the database hash tables but +** it is not unlinked from the Table that it indexes. +** Unlinking from the Table must be done by the calling function. +*/ +static void sqliteDeleteIndex(sqlite3 *db, Index *p){ + Index *pOld; + const char *zName = p->zName; + + pOld = sqlite3HashInsert(&p->pSchema->idxHash, zName, strlen( zName)+1, 0); + assert( pOld==0 || pOld==p ); + freeIndex(p); +} + +/* +** For the index called zIdxName which is found in the database iDb, +** unlike that index from its Table then remove the index from +** the index hash table and free all memory structures associated +** with the index. +*/ +void sqlite3UnlinkAndDeleteIndex(sqlite3 *db, int iDb, const char *zIdxName){ + Index *pIndex; + int len; + Hash *pHash = &db->aDb[iDb].pSchema->idxHash; + + len = strlen(zIdxName); + pIndex = sqlite3HashInsert(pHash, zIdxName, len+1, 0); + if( pIndex ){ + if( pIndex->pTable->pIndex==pIndex ){ + pIndex->pTable->pIndex = pIndex->pNext; + }else{ + Index *p; + for(p=pIndex->pTable->pIndex; p && p->pNext!=pIndex; p=p->pNext){} + if( p && p->pNext==pIndex ){ + p->pNext = pIndex->pNext; + } + } + freeIndex(pIndex); + } + db->flags |= SQLITE_InternChanges; +} + +/* +** Erase all schema information from the in-memory hash tables of +** a single database. This routine is called to reclaim memory +** before the database closes. It is also called during a rollback +** if there were schema changes during the transaction or if a +** schema-cookie mismatch occurs. +** +** If iDb<=0 then reset the internal schema tables for all database +** files. If iDb>=2 then reset the internal schema for only the +** single file indicated. +*/ +void sqlite3ResetInternalSchema(sqlite3 *db, int iDb){ + int i, j; + + assert( iDb>=0 && iDbnDb ); + for(i=iDb; inDb; i++){ + Db *pDb = &db->aDb[i]; + if( pDb->pSchema ){ + sqlite3SchemaFree(pDb->pSchema); + } + if( iDb>0 ) return; + } + assert( iDb==0 ); + db->flags &= ~SQLITE_InternChanges; + + /* If one or more of the auxiliary database files has been closed, + ** then remove them from the auxiliary database list. We take the + ** opportunity to do this here since we have just deleted all of the + ** schema hash tables and therefore do not have to make any changes + ** to any of those tables. + */ + for(i=0; inDb; i++){ + struct Db *pDb = &db->aDb[i]; + if( pDb->pBt==0 ){ + if( pDb->pAux && pDb->xFreeAux ) pDb->xFreeAux(pDb->pAux); + pDb->pAux = 0; + } + } + for(i=j=2; inDb; i++){ + struct Db *pDb = &db->aDb[i]; + if( pDb->pBt==0 ){ + sqliteFree(pDb->zName); + pDb->zName = 0; + continue; + } + if( jaDb[j] = db->aDb[i]; + } + j++; + } + memset(&db->aDb[j], 0, (db->nDb-j)*sizeof(db->aDb[j])); + db->nDb = j; + if( db->nDb<=2 && db->aDb!=db->aDbStatic ){ + memcpy(db->aDbStatic, db->aDb, 2*sizeof(db->aDb[0])); + sqliteFree(db->aDb); + db->aDb = db->aDbStatic; + } +} + +/* +** This routine is called whenever a rollback occurs. If there were +** schema changes during the transaction, then we have to reset the +** internal hash tables and reload them from disk. +*/ +void sqlite3RollbackInternalChanges(sqlite3 *db){ + if( db->flags & SQLITE_InternChanges ){ + sqlite3ResetInternalSchema(db, 0); + } +} + +/* +** This routine is called when a commit occurs. +*/ +void sqlite3CommitInternalChanges(sqlite3 *db){ + db->flags &= ~SQLITE_InternChanges; +} + +/* +** Clear the column names from a table or view. +*/ +static void sqliteResetColumnNames(Table *pTable){ + int i; + Column *pCol; + assert( pTable!=0 ); + if( (pCol = pTable->aCol)!=0 ){ + for(i=0; inCol; i++, pCol++){ + sqliteFree(pCol->zName); + sqlite3ExprDelete(pCol->pDflt); + sqliteFree(pCol->zType); + sqliteFree(pCol->zColl); + } + sqliteFree(pTable->aCol); + } + pTable->aCol = 0; + pTable->nCol = 0; +} + +/* +** Remove the memory data structures associated with the given +** Table. No changes are made to disk by this routine. +** +** This routine just deletes the data structure. It does not unlink +** the table data structure from the hash table. Nor does it remove +** foreign keys from the sqlite.aFKey hash table. But it does destroy +** memory structures of the indices and foreign keys associated with +** the table. +** +** Indices associated with the table are unlinked from the "db" +** data structure if db!=NULL. If db==NULL, indices attached to +** the table are deleted, but it is assumed they have already been +** unlinked. +*/ +void sqlite3DeleteTable(sqlite3 *db, Table *pTable){ + Index *pIndex, *pNext; + FKey *pFKey, *pNextFKey; + + db = 0; + + if( pTable==0 ) return; + + /* Do not delete the table until the reference count reaches zero. */ + pTable->nRef--; + if( pTable->nRef>0 ){ + return; + } + assert( pTable->nRef==0 ); + + /* Delete all indices associated with this table + */ + for(pIndex = pTable->pIndex; pIndex; pIndex=pNext){ + pNext = pIndex->pNext; + assert( pIndex->pSchema==pTable->pSchema ); + sqliteDeleteIndex(db, pIndex); + } + +#ifndef SQLITE_OMIT_FOREIGN_KEY + /* Delete all foreign keys associated with this table. The keys + ** should have already been unlinked from the db->aFKey hash table + */ + for(pFKey=pTable->pFKey; pFKey; pFKey=pNextFKey){ + pNextFKey = pFKey->pNextFrom; + assert( sqlite3HashFind(&pTable->pSchema->aFKey, + pFKey->zTo, strlen(pFKey->zTo)+1)!=pFKey ); + sqliteFree(pFKey); + } +#endif + + /* Delete the Table structure itself. + */ + sqliteResetColumnNames(pTable); + sqliteFree(pTable->zName); + sqliteFree(pTable->zColAff); + sqlite3SelectDelete(pTable->pSelect); +#ifndef SQLITE_OMIT_CHECK + sqlite3ExprDelete(pTable->pCheck); +#endif + sqliteFree(pTable); +} + +/* +** Unlink the given table from the hash tables and the delete the +** table structure with all its indices and foreign keys. +*/ +void sqlite3UnlinkAndDeleteTable(sqlite3 *db, int iDb, const char *zTabName){ + Table *p; + FKey *pF1, *pF2; + Db *pDb; + + assert( db!=0 ); + assert( iDb>=0 && iDbnDb ); + assert( zTabName && zTabName[0] ); + pDb = &db->aDb[iDb]; + p = sqlite3HashInsert(&pDb->pSchema->tblHash, zTabName, strlen(zTabName)+1,0); + if( p ){ +#ifndef SQLITE_OMIT_FOREIGN_KEY + for(pF1=p->pFKey; pF1; pF1=pF1->pNextFrom){ + int nTo = strlen(pF1->zTo) + 1; + pF2 = sqlite3HashFind(&pDb->pSchema->aFKey, pF1->zTo, nTo); + if( pF2==pF1 ){ + sqlite3HashInsert(&pDb->pSchema->aFKey, pF1->zTo, nTo, pF1->pNextTo); + }else{ + while( pF2 && pF2->pNextTo!=pF1 ){ pF2=pF2->pNextTo; } + if( pF2 ){ + pF2->pNextTo = pF1->pNextTo; + } + } + } +#endif + sqlite3DeleteTable(db, p); + } + db->flags |= SQLITE_InternChanges; +} + +/* +** Given a token, return a string that consists of the text of that +** token with any quotations removed. Space to hold the returned string +** is obtained from sqliteMalloc() and must be freed by the calling +** function. +** +** Tokens are often just pointers into the original SQL text and so +** are not \000 terminated and are not persistent. The returned string +** is \000 terminated and is persistent. +*/ +char *sqlite3NameFromToken(Token *pName){ + char *zName; + if( pName ){ + zName = sqliteStrNDup((char*)pName->z, pName->n); + sqlite3Dequote(zName); + }else{ + zName = 0; + } + return zName; +} + +/* +** Open the sqlite_master table stored in database number iDb for +** writing. The table is opened using cursor 0. +*/ +void sqlite3OpenMasterTable(Parse *p, int iDb){ + Vdbe *v = sqlite3GetVdbe(p); + sqlite3TableLock(p, iDb, MASTER_ROOT, 1, SCHEMA_TABLE(iDb)); + sqlite3VdbeAddOp(v, OP_Integer, iDb, 0); + sqlite3VdbeAddOp(v, OP_OpenWrite, 0, MASTER_ROOT); + sqlite3VdbeAddOp(v, OP_SetNumColumns, 0, 5); /* sqlite_master has 5 columns */ +} + +/* +** The token *pName contains the name of a database (either "main" or +** "temp" or the name of an attached db). This routine returns the +** index of the named database in db->aDb[], or -1 if the named db +** does not exist. +*/ +int sqlite3FindDb(sqlite3 *db, Token *pName){ + int i = -1; /* Database number */ + int n; /* Number of characters in the name */ + Db *pDb; /* A database whose name space is being searched */ + char *zName; /* Name we are searching for */ + + zName = sqlite3NameFromToken(pName); + if( zName ){ + n = strlen(zName); + for(i=(db->nDb-1), pDb=&db->aDb[i]; i>=0; i--, pDb--){ + if( (!OMIT_TEMPDB || i!=1 ) && n==strlen(pDb->zName) && + 0==sqlite3StrICmp(pDb->zName, zName) ){ + break; + } + } + sqliteFree(zName); + } + return i; +} + +/* The table or view or trigger name is passed to this routine via tokens +** pName1 and pName2. If the table name was fully qualified, for example: +** +** CREATE TABLE xxx.yyy (...); +** +** Then pName1 is set to "xxx" and pName2 "yyy". On the other hand if +** the table name is not fully qualified, i.e.: +** +** CREATE TABLE yyy(...); +** +** Then pName1 is set to "yyy" and pName2 is "". +** +** This routine sets the *ppUnqual pointer to point at the token (pName1 or +** pName2) that stores the unqualified table name. The index of the +** database "xxx" is returned. +*/ +int sqlite3TwoPartName( + Parse *pParse, /* Parsing and code generating context */ + Token *pName1, /* The "xxx" in the name "xxx.yyy" or "xxx" */ + Token *pName2, /* The "yyy" in the name "xxx.yyy" */ + Token **pUnqual /* Write the unqualified object name here */ +){ + int iDb; /* Database holding the object */ + sqlite3 *db = pParse->db; + + if( pName2 && pName2->n>0 ){ + assert( !db->init.busy ); + *pUnqual = pName2; + iDb = sqlite3FindDb(db, pName1); + if( iDb<0 ){ + sqlite3ErrorMsg(pParse, "unknown database %T", pName1); + pParse->nErr++; + return -1; + } + }else{ + assert( db->init.iDb==0 || db->init.busy ); + iDb = db->init.iDb; + *pUnqual = pName1; + } + return iDb; +} + +/* +** This routine is used to check if the UTF-8 string zName is a legal +** unqualified name for a new schema object (table, index, view or +** trigger). All names are legal except those that begin with the string +** "sqlite_" (in upper, lower or mixed case). This portion of the namespace +** is reserved for internal use. +*/ +int sqlite3CheckObjectName(Parse *pParse, const char *zName){ + if( !pParse->db->init.busy && pParse->nested==0 + && (pParse->db->flags & SQLITE_WriteSchema)==0 + && 0==sqlite3StrNICmp(zName, "sqlite_", 7) ){ + sqlite3ErrorMsg(pParse, "object name reserved for internal use: %s", zName); + return SQLITE_ERROR; + } + return SQLITE_OK; +} + +/* +** Begin constructing a new table representation in memory. This is +** the first of several action routines that get called in response +** to a CREATE TABLE statement. In particular, this routine is called +** after seeing tokens "CREATE" and "TABLE" and the table name. The +** pStart token is the CREATE and pName is the table name. The isTemp +** flag is true if the table should be stored in the auxiliary database +** file instead of in the main database file. This is normally the case +** when the "TEMP" or "TEMPORARY" keyword occurs in between +** CREATE and TABLE. +** +** The new table record is initialized and put in pParse->pNewTable. +** As more of the CREATE TABLE statement is parsed, additional action +** routines will be called to add more information to this record. +** At the end of the CREATE TABLE statement, the sqlite3EndTable() routine +** is called to complete the construction of the new table record. +*/ +void sqlite3StartTable( + Parse *pParse, /* Parser context */ + Token *pStart, /* The "CREATE" token */ + Token *pName1, /* First part of the name of the table or view */ + Token *pName2, /* Second part of the name of the table or view */ + int isTemp, /* True if this is a TEMP table */ + int isView, /* True if this is a VIEW */ + int noErr /* Do nothing if table already exists */ +){ + Table *pTable; + char *zName = 0; /* The name of the new table */ + sqlite3 *db = pParse->db; + Vdbe *v; + int iDb; /* Database number to create the table in */ + Token *pName; /* Unqualified name of the table to create */ + + /* The table or view name to create is passed to this routine via tokens + ** pName1 and pName2. If the table name was fully qualified, for example: + ** + ** CREATE TABLE xxx.yyy (...); + ** + ** Then pName1 is set to "xxx" and pName2 "yyy". On the other hand if + ** the table name is not fully qualified, i.e.: + ** + ** CREATE TABLE yyy(...); + ** + ** Then pName1 is set to "yyy" and pName2 is "". + ** + ** The call below sets the pName pointer to point at the token (pName1 or + ** pName2) that stores the unqualified table name. The variable iDb is + ** set to the index of the database that the table or view is to be + ** created in. + */ + iDb = sqlite3TwoPartName(pParse, pName1, pName2, &pName); + if( iDb<0 ) return; + if( !OMIT_TEMPDB && isTemp && iDb>1 ){ + /* If creating a temp table, the name may not be qualified */ + sqlite3ErrorMsg(pParse, "temporary table name must be unqualified"); + return; + } + if( !OMIT_TEMPDB && isTemp ) iDb = 1; + + pParse->sNameToken = *pName; + zName = sqlite3NameFromToken(pName); + if( zName==0 ) return; + if( SQLITE_OK!=sqlite3CheckObjectName(pParse, zName) ){ + goto begin_table_error; + } + if( db->init.iDb==1 ) isTemp = 1; +#ifndef SQLITE_OMIT_AUTHORIZATION + assert( (isTemp & 1)==isTemp ); + { + int code; + char *zDb = db->aDb[iDb].zName; + if( sqlite3AuthCheck(pParse, SQLITE_INSERT, SCHEMA_TABLE(isTemp), 0, zDb) ){ + goto begin_table_error; + } + if( isView ){ + if( !OMIT_TEMPDB && isTemp ){ + code = SQLITE_CREATE_TEMP_VIEW; + }else{ + code = SQLITE_CREATE_VIEW; + } + }else{ + if( !OMIT_TEMPDB && isTemp ){ + code = SQLITE_CREATE_TEMP_TABLE; + }else{ + code = SQLITE_CREATE_TABLE; + } + } + if( sqlite3AuthCheck(pParse, code, zName, 0, zDb) ){ + goto begin_table_error; + } + } +#endif + + /* Make sure the new table name does not collide with an existing + ** index or table name in the same database. Issue an error message if + ** it does. + */ + if( SQLITE_OK!=sqlite3ReadSchema(pParse) ){ + goto begin_table_error; + } + pTable = sqlite3FindTable(db, zName, db->aDb[iDb].zName); + if( pTable ){ + if( !noErr ){ + sqlite3ErrorMsg(pParse, "table %T already exists", pName); + } + goto begin_table_error; + } + if( sqlite3FindIndex(db, zName, 0)!=0 && (iDb==0 || !db->init.busy) ){ + sqlite3ErrorMsg(pParse, "there is already an index named %s", zName); + goto begin_table_error; + } + pTable = sqliteMalloc( sizeof(Table) ); + if( pTable==0 ){ + pParse->rc = SQLITE_NOMEM; + pParse->nErr++; + goto begin_table_error; + } + pTable->zName = zName; + pTable->nCol = 0; + pTable->aCol = 0; + pTable->iPKey = -1; + pTable->pIndex = 0; + pTable->pSchema = db->aDb[iDb].pSchema; + pTable->nRef = 1; + if( pParse->pNewTable ) sqlite3DeleteTable(db, pParse->pNewTable); + pParse->pNewTable = pTable; + + /* If this is the magic sqlite_sequence table used by autoincrement, + ** then record a pointer to this table in the main database structure + ** so that INSERT can find the table easily. + */ +#ifndef SQLITE_OMIT_AUTOINCREMENT + if( !pParse->nested && strcmp(zName, "sqlite_sequence")==0 ){ + pTable->pSchema->pSeqTab = pTable; + } +#endif + + /* Begin generating the code that will insert the table record into + ** the SQLITE_MASTER table. Note in particular that we must go ahead + ** and allocate the record number for the table entry now. Before any + ** PRIMARY KEY or UNIQUE keywords are parsed. Those keywords will cause + ** indices to be created and the table record must come before the + ** indices. Hence, the record number for the table must be allocated + ** now. + */ + if( !db->init.busy && (v = sqlite3GetVdbe(pParse))!=0 ){ + int lbl; + int fileFormat; + sqlite3BeginWriteOperation(pParse, 0, iDb); + + /* If the file format and encoding in the database have not been set, + ** set them now. + */ + sqlite3VdbeAddOp(v, OP_ReadCookie, iDb, 1); /* file_format */ + lbl = sqlite3VdbeMakeLabel(v); + sqlite3VdbeAddOp(v, OP_If, 0, lbl); + fileFormat = (db->flags & SQLITE_LegacyFileFmt)!=0 ? + 1 : SQLITE_DEFAULT_FILE_FORMAT; + sqlite3VdbeAddOp(v, OP_Integer, fileFormat, 0); + sqlite3VdbeAddOp(v, OP_SetCookie, iDb, 1); + sqlite3VdbeAddOp(v, OP_Integer, ENC(db), 0); + sqlite3VdbeAddOp(v, OP_SetCookie, iDb, 4); + sqlite3VdbeResolveLabel(v, lbl); + + /* This just creates a place-holder record in the sqlite_master table. + ** The record created does not contain anything yet. It will be replaced + ** by the real entry in code generated at sqlite3EndTable(). + ** + ** The rowid for the new entry is left on the top of the stack. + ** The rowid value is needed by the code that sqlite3EndTable will + ** generate. + */ +#ifndef SQLITE_OMIT_VIEW + if( isView ){ + sqlite3VdbeAddOp(v, OP_Integer, 0, 0); + }else +#endif + { + sqlite3VdbeAddOp(v, OP_CreateTable, iDb, 0); + } + sqlite3OpenMasterTable(pParse, iDb); + sqlite3VdbeAddOp(v, OP_NewRowid, 0, 0); + sqlite3VdbeAddOp(v, OP_Dup, 0, 0); + sqlite3VdbeAddOp(v, OP_Null, 0, 0); + sqlite3VdbeAddOp(v, OP_Insert, 0, 0); + sqlite3VdbeAddOp(v, OP_Close, 0, 0); + sqlite3VdbeAddOp(v, OP_Pull, 1, 0); + } + + /* Normal (non-error) return. */ + return; + + /* If an error occurs, we jump here */ +begin_table_error: + sqliteFree(zName); + return; +} + +/* +** This macro is used to compare two strings in a case-insensitive manner. +** It is slightly faster than calling sqlite3StrICmp() directly, but +** produces larger code. +** +** WARNING: This macro is not compatible with the strcmp() family. It +** returns true if the two strings are equal, otherwise false. +*/ +#define STRICMP(x, y) (\ +sqlite3UpperToLower[*(unsigned char *)(x)]== \ +sqlite3UpperToLower[*(unsigned char *)(y)] \ +&& sqlite3StrICmp((x)+1,(y)+1)==0 ) + +/* +** Add a new column to the table currently being constructed. +** +** The parser calls this routine once for each column declaration +** in a CREATE TABLE statement. sqlite3StartTable() gets called +** first to get things going. Then this routine is called for each +** column. +*/ +void sqlite3AddColumn(Parse *pParse, Token *pName){ + Table *p; + int i; + char *z; + Column *pCol; + if( (p = pParse->pNewTable)==0 ) return; + z = sqlite3NameFromToken(pName); + if( z==0 ) return; + for(i=0; inCol; i++){ + if( STRICMP(z, p->aCol[i].zName) ){ + sqlite3ErrorMsg(pParse, "duplicate column name: %s", z); + sqliteFree(z); + return; + } + } + if( (p->nCol & 0x7)==0 ){ + Column *aNew; + aNew = sqliteRealloc( p->aCol, (p->nCol+8)*sizeof(p->aCol[0])); + if( aNew==0 ){ + sqliteFree(z); + return; + } + p->aCol = aNew; + } + pCol = &p->aCol[p->nCol]; + memset(pCol, 0, sizeof(p->aCol[0])); + pCol->zName = z; + + /* If there is no type specified, columns have the default affinity + ** 'NONE'. If there is a type specified, then sqlite3AddColumnType() will + ** be called next to set pCol->affinity correctly. + */ + pCol->affinity = SQLITE_AFF_NONE; + p->nCol++; +} + +/* +** This routine is called by the parser while in the middle of +** parsing a CREATE TABLE statement. A "NOT NULL" constraint has +** been seen on a column. This routine sets the notNull flag on +** the column currently under construction. +*/ +void sqlite3AddNotNull(Parse *pParse, int onError){ + Table *p; + int i; + if( (p = pParse->pNewTable)==0 ) return; + i = p->nCol-1; + if( i>=0 ) p->aCol[i].notNull = onError; +} + +/* +** Scan the column type name zType (length nType) and return the +** associated affinity type. +** +** This routine does a case-independent search of zType for the +** substrings in the following table. If one of the substrings is +** found, the corresponding affinity is returned. If zType contains +** more than one of the substrings, entries toward the top of +** the table take priority. For example, if zType is 'BLOBINT', +** SQLITE_AFF_INTEGER is returned. +** +** Substring | Affinity +** -------------------------------- +** 'INT' | SQLITE_AFF_INTEGER +** 'CHAR' | SQLITE_AFF_TEXT +** 'CLOB' | SQLITE_AFF_TEXT +** 'TEXT' | SQLITE_AFF_TEXT +** 'BLOB' | SQLITE_AFF_NONE +** 'REAL' | SQLITE_AFF_REAL +** 'FLOA' | SQLITE_AFF_REAL +** 'DOUB' | SQLITE_AFF_REAL +** +** If none of the substrings in the above table are found, +** SQLITE_AFF_NUMERIC is returned. +*/ +char sqlite3AffinityType(const Token *pType){ + u32 h = 0; + char aff = SQLITE_AFF_NUMERIC; + const unsigned char *zIn = pType->z; + const unsigned char *zEnd = &pType->z[pType->n]; + + while( zIn!=zEnd ){ + h = (h<<8) + sqlite3UpperToLower[*zIn]; + zIn++; + if( h==(('c'<<24)+('h'<<16)+('a'<<8)+'r') ){ /* CHAR */ + aff = SQLITE_AFF_TEXT; + }else if( h==(('c'<<24)+('l'<<16)+('o'<<8)+'b') ){ /* CLOB */ + aff = SQLITE_AFF_TEXT; + }else if( h==(('t'<<24)+('e'<<16)+('x'<<8)+'t') ){ /* TEXT */ + aff = SQLITE_AFF_TEXT; + }else if( h==(('b'<<24)+('l'<<16)+('o'<<8)+'b') /* BLOB */ + && (aff==SQLITE_AFF_NUMERIC || aff==SQLITE_AFF_REAL) ){ + aff = SQLITE_AFF_NONE; +#ifndef SQLITE_OMIT_FLOATING_POINT + }else if( h==(('r'<<24)+('e'<<16)+('a'<<8)+'l') /* REAL */ + && aff==SQLITE_AFF_NUMERIC ){ + aff = SQLITE_AFF_REAL; + }else if( h==(('f'<<24)+('l'<<16)+('o'<<8)+'a') /* FLOA */ + && aff==SQLITE_AFF_NUMERIC ){ + aff = SQLITE_AFF_REAL; + }else if( h==(('d'<<24)+('o'<<16)+('u'<<8)+'b') /* DOUB */ + && aff==SQLITE_AFF_NUMERIC ){ + aff = SQLITE_AFF_REAL; +#endif + }else if( (h&0x00FFFFFF)==(('i'<<16)+('n'<<8)+'t') ){ /* INT */ + aff = SQLITE_AFF_INTEGER; + break; + } + } + + return aff; +} + +/* +** This routine is called by the parser while in the middle of +** parsing a CREATE TABLE statement. The pFirst token is the first +** token in the sequence of tokens that describe the type of the +** column currently under construction. pLast is the last token +** in the sequence. Use this information to construct a string +** that contains the typename of the column and store that string +** in zType. +*/ +void sqlite3AddColumnType(Parse *pParse, Token *pType){ + Table *p; + int i; + Column *pCol; + + if( (p = pParse->pNewTable)==0 ) return; + i = p->nCol-1; + if( i<0 ) return; + pCol = &p->aCol[i]; + sqliteFree(pCol->zType); + pCol->zType = sqlite3NameFromToken(pType); + pCol->affinity = sqlite3AffinityType(pType); +} + +/* +** The expression is the default value for the most recently added column +** of the table currently under construction. +** +** Default value expressions must be constant. Raise an exception if this +** is not the case. +** +** This routine is called by the parser while in the middle of +** parsing a CREATE TABLE statement. +*/ +void sqlite3AddDefaultValue(Parse *pParse, Expr *pExpr){ + Table *p; + Column *pCol; + if( (p = pParse->pNewTable)!=0 ){ + pCol = &(p->aCol[p->nCol-1]); + if( !sqlite3ExprIsConstantOrFunction(pExpr) ){ + sqlite3ErrorMsg(pParse, "default value of column [%s] is not constant", + pCol->zName); + }else{ + sqlite3ExprDelete(pCol->pDflt); + pCol->pDflt = sqlite3ExprDup(pExpr); + } + } + sqlite3ExprDelete(pExpr); +} + +/* +** Designate the PRIMARY KEY for the table. pList is a list of names +** of columns that form the primary key. If pList is NULL, then the +** most recently added column of the table is the primary key. +** +** A table can have at most one primary key. If the table already has +** a primary key (and this is the second primary key) then create an +** error. +** +** If the PRIMARY KEY is on a single column whose datatype is INTEGER, +** then we will try to use that column as the rowid. Set the Table.iPKey +** field of the table under construction to be the index of the +** INTEGER PRIMARY KEY column. Table.iPKey is set to -1 if there is +** no INTEGER PRIMARY KEY. +** +** If the key is not an INTEGER PRIMARY KEY, then create a unique +** index for the key. No index is created for INTEGER PRIMARY KEYs. +*/ +void sqlite3AddPrimaryKey( + Parse *pParse, /* Parsing context */ + ExprList *pList, /* List of field names to be indexed */ + int onError, /* What to do with a uniqueness conflict */ + int autoInc, /* True if the AUTOINCREMENT keyword is present */ + int sortOrder /* SQLITE_SO_ASC or SQLITE_SO_DESC */ +){ + Table *pTab = pParse->pNewTable; + char *zType = 0; + int iCol = -1, i; + if( pTab==0 ) goto primary_key_exit; + if( pTab->hasPrimKey ){ + sqlite3ErrorMsg(pParse, + "table \"%s\" has more than one primary key", pTab->zName); + goto primary_key_exit; + } + pTab->hasPrimKey = 1; + if( pList==0 ){ + iCol = pTab->nCol - 1; + pTab->aCol[iCol].isPrimKey = 1; + }else{ + for(i=0; inExpr; i++){ + for(iCol=0; iColnCol; iCol++){ + if( sqlite3StrICmp(pList->a[i].zName, pTab->aCol[iCol].zName)==0 ){ + break; + } + } + if( iColnCol ){ + pTab->aCol[iCol].isPrimKey = 1; + } + } + if( pList->nExpr>1 ) iCol = -1; + } + if( iCol>=0 && iColnCol ){ + zType = pTab->aCol[iCol].zType; + } + if( zType && sqlite3StrICmp(zType, "INTEGER")==0 + && sortOrder==SQLITE_SO_ASC ){ + pTab->iPKey = iCol; + pTab->keyConf = onError; + pTab->autoInc = autoInc; + }else if( autoInc ){ +#ifndef SQLITE_OMIT_AUTOINCREMENT + sqlite3ErrorMsg(pParse, "AUTOINCREMENT is only allowed on an " + "INTEGER PRIMARY KEY"); +#endif + }else{ + sqlite3CreateIndex(pParse, 0, 0, 0, pList, onError, 0, 0, sortOrder, 0); + pList = 0; + } + +primary_key_exit: + sqlite3ExprListDelete(pList); + return; +} + +/* +** Add a new CHECK constraint to the table currently under construction. +*/ +void sqlite3AddCheckConstraint( + Parse *pParse, /* Parsing context */ + Expr *pCheckExpr /* The check expression */ +){ +#ifndef SQLITE_OMIT_CHECK + Table *pTab = pParse->pNewTable; + if( pTab ){ + /* The CHECK expression must be duplicated so that tokens refer + ** to malloced space and not the (ephemeral) text of the CREATE TABLE + ** statement */ + pTab->pCheck = sqlite3ExprAnd(pTab->pCheck, sqlite3ExprDup(pCheckExpr)); + } +#endif + sqlite3ExprDelete(pCheckExpr); +} + +/* +** Set the collation function of the most recently parsed table column +** to the CollSeq given. +*/ +void sqlite3AddCollateType(Parse *pParse, const char *zType, int nType){ + Table *p; + int i; + + if( (p = pParse->pNewTable)==0 ) return; + i = p->nCol-1; + + if( sqlite3LocateCollSeq(pParse, zType, nType) ){ + Index *pIdx; + p->aCol[i].zColl = sqliteStrNDup(zType, nType); + + /* If the column is declared as " PRIMARY KEY COLLATE ", + ** then an index may have been created on this column before the + ** collation type was added. Correct this if it is the case. + */ + for(pIdx=p->pIndex; pIdx; pIdx=pIdx->pNext){ + assert( pIdx->nColumn==1 ); + if( pIdx->aiColumn[0]==i ){ + pIdx->azColl[0] = p->aCol[i].zColl; + } + } + } +} + +/* +** This function returns the collation sequence for database native text +** encoding identified by the string zName, length nName. +** +** If the requested collation sequence is not available, or not available +** in the database native encoding, the collation factory is invoked to +** request it. If the collation factory does not supply such a sequence, +** and the sequence is available in another text encoding, then that is +** returned instead. +** +** If no versions of the requested collations sequence are available, or +** another error occurs, NULL is returned and an error message written into +** pParse. +*/ +CollSeq *sqlite3LocateCollSeq(Parse *pParse, const char *zName, int nName){ + sqlite3 *db = pParse->db; + u8 enc = ENC(db); + u8 initbusy = db->init.busy; + CollSeq *pColl; + + pColl = sqlite3FindCollSeq(db, enc, zName, nName, initbusy); + if( !initbusy && (!pColl || !pColl->xCmp) ){ + pColl = sqlite3GetCollSeq(db, pColl, zName, nName); + if( !pColl ){ + if( nName<0 ){ + nName = strlen(zName); + } + sqlite3ErrorMsg(pParse, "no such collation sequence: %.*s", nName, zName); + pColl = 0; + } + } + + return pColl; +} + + +/* +** Generate code that will increment the schema cookie. +** +** The schema cookie is used to determine when the schema for the +** database changes. After each schema change, the cookie value +** changes. When a process first reads the schema it records the +** cookie. Thereafter, whenever it goes to access the database, +** it checks the cookie to make sure the schema has not changed +** since it was last read. +** +** This plan is not completely bullet-proof. It is possible for +** the schema to change multiple times and for the cookie to be +** set back to prior value. But schema changes are infrequent +** and the probability of hitting the same cookie value is only +** 1 chance in 2^32. So we're safe enough. +*/ +void sqlite3ChangeCookie(sqlite3 *db, Vdbe *v, int iDb){ + sqlite3VdbeAddOp(v, OP_Integer, db->aDb[iDb].pSchema->schema_cookie+1, 0); + sqlite3VdbeAddOp(v, OP_SetCookie, iDb, 0); +} + +/* +** Measure the number of characters needed to output the given +** identifier. The number returned includes any quotes used +** but does not include the null terminator. +** +** The estimate is conservative. It might be larger that what is +** really needed. +*/ +static int identLength(const char *z){ + int n; + for(n=0; *z; n++, z++){ + if( *z=='"' ){ n++; } + } + return n + 2; +} + +/* +** Write an identifier onto the end of the given string. Add +** quote characters as needed. +*/ +static void identPut(char *z, int *pIdx, char *zSignedIdent){ + unsigned char *zIdent = (unsigned char*)zSignedIdent; + int i, j, needQuote; + i = *pIdx; + for(j=0; zIdent[j]; j++){ + if( !isalnum(zIdent[j]) && zIdent[j]!='_' ) break; + } + needQuote = zIdent[j]!=0 || isdigit(zIdent[0]) + || sqlite3KeywordCode(zIdent, j)!=TK_ID; + if( needQuote ) z[i++] = '"'; + for(j=0; zIdent[j]; j++){ + z[i++] = zIdent[j]; + if( zIdent[j]=='"' ) z[i++] = '"'; + } + if( needQuote ) z[i++] = '"'; + z[i] = 0; + *pIdx = i; +} + +/* +** Generate a CREATE TABLE statement appropriate for the given +** table. Memory to hold the text of the statement is obtained +** from sqliteMalloc() and must be freed by the calling function. +*/ +static char *createTableStmt(Table *p, int isTemp){ + int i, k, n; + char *zStmt; + char *zSep, *zSep2, *zEnd, *z; + Column *pCol; + n = 0; + for(pCol = p->aCol, i=0; inCol; i++, pCol++){ + n += identLength(pCol->zName); + z = pCol->zType; + if( z ){ + n += (strlen(z) + 1); + } + } + n += identLength(p->zName); + if( n<50 ){ + zSep = ""; + zSep2 = ","; + zEnd = ")"; + }else{ + zSep = "\n "; + zSep2 = ",\n "; + zEnd = "\n)"; + } + n += 35 + 6*p->nCol; + zStmt = sqliteMallocRaw( n ); + if( zStmt==0 ) return 0; + strcpy(zStmt, !OMIT_TEMPDB&&isTemp ? "CREATE TEMP TABLE ":"CREATE TABLE "); + k = strlen(zStmt); + identPut(zStmt, &k, p->zName); + zStmt[k++] = '('; + for(pCol=p->aCol, i=0; inCol; i++, pCol++){ + strcpy(&zStmt[k], zSep); + k += strlen(&zStmt[k]); + zSep = zSep2; + identPut(zStmt, &k, pCol->zName); + if( (z = pCol->zType)!=0 ){ + zStmt[k++] = ' '; + strcpy(&zStmt[k], z); + k += strlen(z); + } + } + strcpy(&zStmt[k], zEnd); + return zStmt; +} + +/* +** This routine is called to report the final ")" that terminates +** a CREATE TABLE statement. +** +** The table structure that other action routines have been building +** is added to the internal hash tables, assuming no errors have +** occurred. +** +** An entry for the table is made in the master table on disk, unless +** this is a temporary table or db->init.busy==1. When db->init.busy==1 +** it means we are reading the sqlite_master table because we just +** connected to the database or because the sqlite_master table has +** recently changed, so the entry for this table already exists in +** the sqlite_master table. We do not want to create it again. +** +** If the pSelect argument is not NULL, it means that this routine +** was called to create a table generated from a +** "CREATE TABLE ... AS SELECT ..." statement. The column names of +** the new table will match the result set of the SELECT. +*/ +void sqlite3EndTable( + Parse *pParse, /* Parse context */ + Token *pCons, /* The ',' token after the last column defn. */ + Token *pEnd, /* The final ')' token in the CREATE TABLE */ + Select *pSelect /* Select from a "CREATE ... AS SELECT" */ +){ + Table *p; + sqlite3 *db = pParse->db; + int iDb; + + if( (pEnd==0 && pSelect==0) || pParse->nErr || sqlite3MallocFailed() ) { + return; + } + p = pParse->pNewTable; + if( p==0 ) return; + + assert( !db->init.busy || !pSelect ); + + iDb = sqlite3SchemaToIndex(pParse->db, p->pSchema); + +#ifndef SQLITE_OMIT_CHECK + /* Resolve names in all CHECK constraint expressions. + */ + if( p->pCheck ){ + SrcList sSrc; /* Fake SrcList for pParse->pNewTable */ + NameContext sNC; /* Name context for pParse->pNewTable */ + + memset(&sNC, 0, sizeof(sNC)); + memset(&sSrc, 0, sizeof(sSrc)); + sSrc.nSrc = 1; + sSrc.a[0].zName = p->zName; + sSrc.a[0].pTab = p; + sSrc.a[0].iCursor = -1; + sNC.pParse = pParse; + sNC.pSrcList = &sSrc; + sNC.isCheck = 1; + if( sqlite3ExprResolveNames(&sNC, p->pCheck) ){ + return; + } + } +#endif /* !defined(SQLITE_OMIT_CHECK) */ + + /* If the db->init.busy is 1 it means we are reading the SQL off the + ** "sqlite_master" or "sqlite_temp_master" table on the disk. + ** So do not write to the disk again. Extract the root page number + ** for the table from the db->init.newTnum field. (The page number + ** should have been put there by the sqliteOpenCb routine.) + */ + if( db->init.busy ){ + p->tnum = db->init.newTnum; + } + + /* If not initializing, then create a record for the new table + ** in the SQLITE_MASTER table of the database. The record number + ** for the new table entry should already be on the stack. + ** + ** If this is a TEMPORARY table, write the entry into the auxiliary + ** file instead of into the main database file. + */ + if( !db->init.busy ){ + int n; + Vdbe *v; + char *zType; /* "view" or "table" */ + char *zType2; /* "VIEW" or "TABLE" */ + char *zStmt; /* Text of the CREATE TABLE or CREATE VIEW statement */ + + v = sqlite3GetVdbe(pParse); + if( v==0 ) return; + + sqlite3VdbeAddOp(v, OP_Close, 0, 0); + + /* Create the rootpage for the new table and push it onto the stack. + ** A view has no rootpage, so just push a zero onto the stack for + ** views. Initialize zType at the same time. + */ + if( p->pSelect==0 ){ + /* A regular table */ + zType = "table"; + zType2 = "TABLE"; +#ifndef SQLITE_OMIT_VIEW + }else{ + /* A view */ + zType = "view"; + zType2 = "VIEW"; +#endif + } + + /* If this is a CREATE TABLE xx AS SELECT ..., execute the SELECT + ** statement to populate the new table. The root-page number for the + ** new table is on the top of the vdbe stack. + ** + ** Once the SELECT has been coded by sqlite3Select(), it is in a + ** suitable state to query for the column names and types to be used + ** by the new table. + ** + ** A shared-cache write-lock is not required to write to the new table, + ** as a schema-lock must have already been obtained to create it. Since + ** a schema-lock excludes all other database users, the write-lock would + ** be redundant. + */ + if( pSelect ){ + Table *pSelTab; + sqlite3VdbeAddOp(v, OP_Dup, 0, 0); + sqlite3VdbeAddOp(v, OP_Integer, iDb, 0); + sqlite3VdbeAddOp(v, OP_OpenWrite, 1, 0); + pParse->nTab = 2; + sqlite3Select(pParse, pSelect, SRT_Table, 1, 0, 0, 0, 0); + sqlite3VdbeAddOp(v, OP_Close, 1, 0); + if( pParse->nErr==0 ){ + pSelTab = sqlite3ResultSetOfSelect(pParse, 0, pSelect); + if( pSelTab==0 ) return; + assert( p->aCol==0 ); + p->nCol = pSelTab->nCol; + p->aCol = pSelTab->aCol; + pSelTab->nCol = 0; + pSelTab->aCol = 0; + sqlite3DeleteTable(0, pSelTab); + } + } + + /* Compute the complete text of the CREATE statement */ + if( pSelect ){ + zStmt = createTableStmt(p, p->pSchema==pParse->db->aDb[1].pSchema); + }else{ + n = pEnd->z - pParse->sNameToken.z + 1; + zStmt = sqlite3MPrintf("CREATE %s %.*s", zType2, n, pParse->sNameToken.z); + } + + /* A slot for the record has already been allocated in the + ** SQLITE_MASTER table. We just need to update that slot with all + ** the information we've collected. The rowid for the preallocated + ** slot is the 2nd item on the stack. The top of the stack is the + ** root page for the new table (or a 0 if this is a view). + */ + sqlite3NestedParse(pParse, + "UPDATE %Q.%s " + "SET type='%s', name=%Q, tbl_name=%Q, rootpage=#0, sql=%Q " + "WHERE rowid=#1", + db->aDb[iDb].zName, SCHEMA_TABLE(iDb), + zType, + p->zName, + p->zName, + zStmt + ); + sqliteFree(zStmt); + sqlite3ChangeCookie(db, v, iDb); + +#ifndef SQLITE_OMIT_AUTOINCREMENT + /* Check to see if we need to create an sqlite_sequence table for + ** keeping track of autoincrement keys. + */ + if( p->autoInc ){ + Db *pDb = &db->aDb[iDb]; + if( pDb->pSchema->pSeqTab==0 ){ + sqlite3NestedParse(pParse, + "CREATE TABLE %Q.sqlite_sequence(name,seq)", + pDb->zName + ); + } + } +#endif + + /* Reparse everything to update our internal data structures */ + sqlite3VdbeOp3(v, OP_ParseSchema, iDb, 0, + sqlite3MPrintf("tbl_name='%q'",p->zName), P3_DYNAMIC); + } + + + /* Add the table to the in-memory representation of the database. + */ + if( db->init.busy && pParse->nErr==0 ){ + Table *pOld; + FKey *pFKey; + Schema *pSchema = p->pSchema; + pOld = sqlite3HashInsert(&pSchema->tblHash, p->zName, strlen(p->zName)+1,p); + if( pOld ){ + assert( p==pOld ); /* Malloc must have failed inside HashInsert() */ + return; + } +#ifndef SQLITE_OMIT_FOREIGN_KEY + for(pFKey=p->pFKey; pFKey; pFKey=pFKey->pNextFrom){ + int nTo = strlen(pFKey->zTo) + 1; + pFKey->pNextTo = sqlite3HashFind(&pSchema->aFKey, pFKey->zTo, nTo); + sqlite3HashInsert(&pSchema->aFKey, pFKey->zTo, nTo, pFKey); + } +#endif + pParse->pNewTable = 0; + db->nTable++; + db->flags |= SQLITE_InternChanges; + +#ifndef SQLITE_OMIT_ALTERTABLE + if( !p->pSelect ){ + const char *zName = (const char *)pParse->sNameToken.z; + int nName; + assert( !pSelect && pCons && pEnd ); + if( pCons->z==0 ){ + pCons = pEnd; + } + nName = (const char *)pCons->z - zName; + p->addColOffset = 13 + sqlite3utf8CharLen(zName, nName); + } +#endif + } +} + +#ifndef SQLITE_OMIT_VIEW +/* +** The parser calls this routine in order to create a new VIEW +*/ +void sqlite3CreateView( + Parse *pParse, /* The parsing context */ + Token *pBegin, /* The CREATE token that begins the statement */ + Token *pName1, /* The token that holds the name of the view */ + Token *pName2, /* The token that holds the name of the view */ + Select *pSelect, /* A SELECT statement that will become the new view */ + int isTemp /* TRUE for a TEMPORARY view */ +){ + Table *p; + int n; + const unsigned char *z; + Token sEnd; + DbFixer sFix; + Token *pName; + int iDb; + + if( pParse->nVar>0 ){ + sqlite3ErrorMsg(pParse, "parameters are not allowed in views"); + sqlite3SelectDelete(pSelect); + return; + } + sqlite3StartTable(pParse, pBegin, pName1, pName2, isTemp, 1, 0); + p = pParse->pNewTable; + if( p==0 || pParse->nErr ){ + sqlite3SelectDelete(pSelect); + return; + } + sqlite3TwoPartName(pParse, pName1, pName2, &pName); + iDb = sqlite3SchemaToIndex(pParse->db, p->pSchema); + if( sqlite3FixInit(&sFix, pParse, iDb, "view", pName) + && sqlite3FixSelect(&sFix, pSelect) + ){ + sqlite3SelectDelete(pSelect); + return; + } + + /* Make a copy of the entire SELECT statement that defines the view. + ** This will force all the Expr.token.z values to be dynamically + ** allocated rather than point to the input string - which means that + ** they will persist after the current sqlite3_exec() call returns. + */ + p->pSelect = sqlite3SelectDup(pSelect); + sqlite3SelectDelete(pSelect); + if( sqlite3MallocFailed() ){ + return; + } + if( !pParse->db->init.busy ){ + sqlite3ViewGetColumnNames(pParse, p); + } + + /* Locate the end of the CREATE VIEW statement. Make sEnd point to + ** the end. + */ + sEnd = pParse->sLastToken; + if( sEnd.z[0]!=0 && sEnd.z[0]!=';' ){ + sEnd.z += sEnd.n; + } + sEnd.n = 0; + n = sEnd.z - pBegin->z; + z = (const unsigned char*)pBegin->z; + while( n>0 && (z[n-1]==';' || isspace(z[n-1])) ){ n--; } + sEnd.z = &z[n-1]; + sEnd.n = 1; + + /* Use sqlite3EndTable() to add the view to the SQLITE_MASTER table */ + sqlite3EndTable(pParse, 0, &sEnd, 0); + return; +} +#endif /* SQLITE_OMIT_VIEW */ + +#ifndef SQLITE_OMIT_VIEW +/* +** The Table structure pTable is really a VIEW. Fill in the names of +** the columns of the view in the pTable structure. Return the number +** of errors. If an error is seen leave an error message in pParse->zErrMsg. +*/ +int sqlite3ViewGetColumnNames(Parse *pParse, Table *pTable){ + Table *pSelTab; /* A fake table from which we get the result set */ + Select *pSel; /* Copy of the SELECT that implements the view */ + int nErr = 0; /* Number of errors encountered */ + int n; /* Temporarily holds the number of cursors assigned */ + + assert( pTable ); + + /* A positive nCol means the columns names for this view are + ** already known. + */ + if( pTable->nCol>0 ) return 0; + + /* A negative nCol is a special marker meaning that we are currently + ** trying to compute the column names. If we enter this routine with + ** a negative nCol, it means two or more views form a loop, like this: + ** + ** CREATE VIEW one AS SELECT * FROM two; + ** CREATE VIEW two AS SELECT * FROM one; + ** + ** Actually, this error is caught previously and so the following test + ** should always fail. But we will leave it in place just to be safe. + */ + if( pTable->nCol<0 ){ + sqlite3ErrorMsg(pParse, "view %s is circularly defined", pTable->zName); + return 1; + } + assert( pTable->nCol>=0 ); + + /* If we get this far, it means we need to compute the table names. + ** Note that the call to sqlite3ResultSetOfSelect() will expand any + ** "*" elements in the results set of the view and will assign cursors + ** to the elements of the FROM clause. But we do not want these changes + ** to be permanent. So the computation is done on a copy of the SELECT + ** statement that defines the view. + */ + assert( pTable->pSelect ); + pSel = sqlite3SelectDup(pTable->pSelect); + if( pSel ){ + n = pParse->nTab; + sqlite3SrcListAssignCursors(pParse, pSel->pSrc); + pTable->nCol = -1; + pSelTab = sqlite3ResultSetOfSelect(pParse, 0, pSel); + pParse->nTab = n; + if( pSelTab ){ + assert( pTable->aCol==0 ); + pTable->nCol = pSelTab->nCol; + pTable->aCol = pSelTab->aCol; + pSelTab->nCol = 0; + pSelTab->aCol = 0; + sqlite3DeleteTable(0, pSelTab); + pTable->pSchema->flags |= DB_UnresetViews; + }else{ + pTable->nCol = 0; + nErr++; + } + sqlite3SelectDelete(pSel); + } else { + nErr++; + } + return nErr; +} +#endif /* SQLITE_OMIT_VIEW */ + +#ifndef SQLITE_OMIT_VIEW +/* +** Clear the column names from every VIEW in database idx. +*/ +static void sqliteViewResetAll(sqlite3 *db, int idx){ + HashElem *i; + if( !DbHasProperty(db, idx, DB_UnresetViews) ) return; + for(i=sqliteHashFirst(&db->aDb[idx].pSchema->tblHash); i;i=sqliteHashNext(i)){ + Table *pTab = sqliteHashData(i); + if( pTab->pSelect ){ + sqliteResetColumnNames(pTab); + } + } + DbClearProperty(db, idx, DB_UnresetViews); +} +#else +# define sqliteViewResetAll(A,B) +#endif /* SQLITE_OMIT_VIEW */ + +/* +** This function is called by the VDBE to adjust the internal schema +** used by SQLite when the btree layer moves a table root page. The +** root-page of a table or index in database iDb has changed from iFrom +** to iTo. +*/ +#ifndef SQLITE_OMIT_AUTOVACUUM +void sqlite3RootPageMoved(Db *pDb, int iFrom, int iTo){ + HashElem *pElem; + Hash *pHash; + + pHash = &pDb->pSchema->tblHash; + for(pElem=sqliteHashFirst(pHash); pElem; pElem=sqliteHashNext(pElem)){ + Table *pTab = sqliteHashData(pElem); + if( pTab->tnum==iFrom ){ + pTab->tnum = iTo; + return; + } + } + pHash = &pDb->pSchema->idxHash; + for(pElem=sqliteHashFirst(pHash); pElem; pElem=sqliteHashNext(pElem)){ + Index *pIdx = sqliteHashData(pElem); + if( pIdx->tnum==iFrom ){ + pIdx->tnum = iTo; + return; + } + } + assert(0); +} +#endif + +/* +** Write code to erase the table with root-page iTable from database iDb. +** Also write code to modify the sqlite_master table and internal schema +** if a root-page of another table is moved by the btree-layer whilst +** erasing iTable (this can happen with an auto-vacuum database). +*/ +static void destroyRootPage(Parse *pParse, int iTable, int iDb){ + Vdbe *v = sqlite3GetVdbe(pParse); + sqlite3VdbeAddOp(v, OP_Destroy, iTable, iDb); +#ifndef SQLITE_OMIT_AUTOVACUUM + /* OP_Destroy pushes an integer onto the stack. If this integer + ** is non-zero, then it is the root page number of a table moved to + ** location iTable. The following code modifies the sqlite_master table to + ** reflect this. + ** + ** The "#0" in the SQL is a special constant that means whatever value + ** is on the top of the stack. See sqlite3RegisterExpr(). + */ + sqlite3NestedParse(pParse, + "UPDATE %Q.%s SET rootpage=%d WHERE #0 AND rootpage=#0", + pParse->db->aDb[iDb].zName, SCHEMA_TABLE(iDb), iTable); +#endif +} + +/* +** Write VDBE code to erase table pTab and all associated indices on disk. +** Code to update the sqlite_master tables and internal schema definitions +** in case a root-page belonging to another table is moved by the btree layer +** is also added (this can happen with an auto-vacuum database). +*/ +static void destroyTable(Parse *pParse, Table *pTab){ +#ifdef SQLITE_OMIT_AUTOVACUUM + Index *pIdx; + int iDb = sqlite3SchemaToIndex(pParse->db, pTab->pSchema); + destroyRootPage(pParse, pTab->tnum, iDb); + for(pIdx=pTab->pIndex; pIdx; pIdx=pIdx->pNext){ + destroyRootPage(pParse, pIdx->tnum, iDb); + } +#else + /* If the database may be auto-vacuum capable (if SQLITE_OMIT_AUTOVACUUM + ** is not defined), then it is important to call OP_Destroy on the + ** table and index root-pages in order, starting with the numerically + ** largest root-page number. This guarantees that none of the root-pages + ** to be destroyed is relocated by an earlier OP_Destroy. i.e. if the + ** following were coded: + ** + ** OP_Destroy 4 0 + ** ... + ** OP_Destroy 5 0 + ** + ** and root page 5 happened to be the largest root-page number in the + ** database, then root page 5 would be moved to page 4 by the + ** "OP_Destroy 4 0" opcode. The subsequent "OP_Destroy 5 0" would hit + ** a free-list page. + */ + int iTab = pTab->tnum; + int iDestroyed = 0; + + while( 1 ){ + Index *pIdx; + int iLargest = 0; + + if( iDestroyed==0 || iTabpIndex; pIdx; pIdx=pIdx->pNext){ + int iIdx = pIdx->tnum; + assert( pIdx->pSchema==pTab->pSchema ); + if( (iDestroyed==0 || (iIdxiLargest ){ + iLargest = iIdx; + } + } + if( iLargest==0 ){ + return; + }else{ + int iDb = sqlite3SchemaToIndex(pParse->db, pTab->pSchema); + destroyRootPage(pParse, iLargest, iDb); + iDestroyed = iLargest; + } + } +#endif +} + +/* +** This routine is called to do the work of a DROP TABLE statement. +** pName is the name of the table to be dropped. +*/ +void sqlite3DropTable(Parse *pParse, SrcList *pName, int isView, int noErr){ + Table *pTab; + Vdbe *v; + sqlite3 *db = pParse->db; + int iDb; + + if( pParse->nErr || sqlite3MallocFailed() ){ + goto exit_drop_table; + } + assert( pName->nSrc==1 ); + pTab = sqlite3LocateTable(pParse, pName->a[0].zName, pName->a[0].zDatabase); + + if( pTab==0 ){ + if( noErr ){ + sqlite3ErrorClear(pParse); + } + goto exit_drop_table; + } + iDb = sqlite3SchemaToIndex(db, pTab->pSchema); + assert( iDb>=0 && iDbnDb ); +#ifndef SQLITE_OMIT_AUTHORIZATION + { + int code; + const char *zTab = SCHEMA_TABLE(iDb); + const char *zDb = db->aDb[iDb].zName; + if( sqlite3AuthCheck(pParse, SQLITE_DELETE, zTab, 0, zDb)){ + goto exit_drop_table; + } + if( isView ){ + if( !OMIT_TEMPDB && iDb==1 ){ + code = SQLITE_DROP_TEMP_VIEW; + }else{ + code = SQLITE_DROP_VIEW; + } + }else{ + if( !OMIT_TEMPDB && iDb==1 ){ + code = SQLITE_DROP_TEMP_TABLE; + }else{ + code = SQLITE_DROP_TABLE; + } + } + if( sqlite3AuthCheck(pParse, code, pTab->zName, 0, zDb) ){ + goto exit_drop_table; + } + if( sqlite3AuthCheck(pParse, SQLITE_DELETE, pTab->zName, 0, zDb) ){ + goto exit_drop_table; + } + } +#endif + if( pTab->readOnly || pTab==db->aDb[iDb].pSchema->pSeqTab ){ + sqlite3ErrorMsg(pParse, "table %s may not be dropped", pTab->zName); + goto exit_drop_table; + } + +#ifndef SQLITE_OMIT_VIEW + /* Ensure DROP TABLE is not used on a view, and DROP VIEW is not used + ** on a table. + */ + if( isView && pTab->pSelect==0 ){ + sqlite3ErrorMsg(pParse, "use DROP TABLE to delete table %s", pTab->zName); + goto exit_drop_table; + } + if( !isView && pTab->pSelect ){ + sqlite3ErrorMsg(pParse, "use DROP VIEW to delete view %s", pTab->zName); + goto exit_drop_table; + } +#endif + + /* Generate code to remove the table from the master table + ** on disk. + */ + v = sqlite3GetVdbe(pParse); + if( v ){ + Trigger *pTrigger; + Db *pDb = &db->aDb[iDb]; + sqlite3BeginWriteOperation(pParse, 0, iDb); + + /* Drop all triggers associated with the table being dropped. Code + ** is generated to remove entries from sqlite_master and/or + ** sqlite_temp_master if required. + */ + pTrigger = pTab->pTrigger; + while( pTrigger ){ + assert( pTrigger->pSchema==pTab->pSchema || + pTrigger->pSchema==db->aDb[1].pSchema ); + sqlite3DropTriggerPtr(pParse, pTrigger, 1); + pTrigger = pTrigger->pNext; + } + +#ifndef SQLITE_OMIT_AUTOINCREMENT + /* Remove any entries of the sqlite_sequence table associated with + ** the table being dropped. This is done before the table is dropped + ** at the btree level, in case the sqlite_sequence table needs to + ** move as a result of the drop (can happen in auto-vacuum mode). + */ + if( pTab->autoInc ){ + sqlite3NestedParse(pParse, + "DELETE FROM %s.sqlite_sequence WHERE name=%Q", + pDb->zName, pTab->zName + ); + } +#endif + + /* Drop all SQLITE_MASTER table and index entries that refer to the + ** table. The program name loops through the master table and deletes + ** every row that refers to a table of the same name as the one being + ** dropped. Triggers are handled seperately because a trigger can be + ** created in the temp database that refers to a table in another + ** database. + */ + sqlite3NestedParse(pParse, + "DELETE FROM %Q.%s WHERE tbl_name=%Q and type!='trigger'", + pDb->zName, SCHEMA_TABLE(iDb), pTab->zName); + if( !isView ){ + destroyTable(pParse, pTab); + } + + /* Remove the table entry from SQLite's internal schema and modify + ** the schema cookie. + */ + sqlite3VdbeOp3(v, OP_DropTable, iDb, 0, pTab->zName, 0); + sqlite3ChangeCookie(db, v, iDb); + } + sqliteViewResetAll(db, iDb); + +exit_drop_table: + sqlite3SrcListDelete(pName); +} + +/* +** This routine is called to create a new foreign key on the table +** currently under construction. pFromCol determines which columns +** in the current table point to the foreign key. If pFromCol==0 then +** connect the key to the last column inserted. pTo is the name of +** the table referred to. pToCol is a list of tables in the other +** pTo table that the foreign key points to. flags contains all +** information about the conflict resolution algorithms specified +** in the ON DELETE, ON UPDATE and ON INSERT clauses. +** +** An FKey structure is created and added to the table currently +** under construction in the pParse->pNewTable field. The new FKey +** is not linked into db->aFKey at this point - that does not happen +** until sqlite3EndTable(). +** +** The foreign key is set for IMMEDIATE processing. A subsequent call +** to sqlite3DeferForeignKey() might change this to DEFERRED. +*/ +void sqlite3CreateForeignKey( + Parse *pParse, /* Parsing context */ + ExprList *pFromCol, /* Columns in this table that point to other table */ + Token *pTo, /* Name of the other table */ + ExprList *pToCol, /* Columns in the other table */ + int flags /* Conflict resolution algorithms. */ +){ +#ifndef SQLITE_OMIT_FOREIGN_KEY + FKey *pFKey = 0; + Table *p = pParse->pNewTable; + int nByte; + int i; + int nCol; + char *z; + + assert( pTo!=0 ); + if( p==0 || pParse->nErr ) goto fk_end; + if( pFromCol==0 ){ + int iCol = p->nCol-1; + if( iCol<0 ) goto fk_end; + if( pToCol && pToCol->nExpr!=1 ){ + sqlite3ErrorMsg(pParse, "foreign key on %s" + " should reference only one column of table %T", + p->aCol[iCol].zName, pTo); + goto fk_end; + } + nCol = 1; + }else if( pToCol && pToCol->nExpr!=pFromCol->nExpr ){ + sqlite3ErrorMsg(pParse, + "number of columns in foreign key does not match the number of " + "columns in the referenced table"); + goto fk_end; + }else{ + nCol = pFromCol->nExpr; + } + nByte = sizeof(*pFKey) + nCol*sizeof(pFKey->aCol[0]) + pTo->n + 1; + if( pToCol ){ + for(i=0; inExpr; i++){ + nByte += strlen(pToCol->a[i].zName) + 1; + } + } + pFKey = sqliteMalloc( nByte ); + if( pFKey==0 ) goto fk_end; + pFKey->pFrom = p; + pFKey->pNextFrom = p->pFKey; + z = (char*)&pFKey[1]; + pFKey->aCol = (struct sColMap*)z; + z += sizeof(struct sColMap)*nCol; + pFKey->zTo = z; + memcpy(z, pTo->z, pTo->n); + z[pTo->n] = 0; + z += pTo->n+1; + pFKey->pNextTo = 0; + pFKey->nCol = nCol; + if( pFromCol==0 ){ + pFKey->aCol[0].iFrom = p->nCol-1; + }else{ + for(i=0; inCol; j++){ + if( sqlite3StrICmp(p->aCol[j].zName, pFromCol->a[i].zName)==0 ){ + pFKey->aCol[i].iFrom = j; + break; + } + } + if( j>=p->nCol ){ + sqlite3ErrorMsg(pParse, + "unknown column \"%s\" in foreign key definition", + pFromCol->a[i].zName); + goto fk_end; + } + } + } + if( pToCol ){ + for(i=0; ia[i].zName); + pFKey->aCol[i].zCol = z; + memcpy(z, pToCol->a[i].zName, n); + z[n] = 0; + z += n+1; + } + } + pFKey->isDeferred = 0; + pFKey->deleteConf = flags & 0xff; + pFKey->updateConf = (flags >> 8 ) & 0xff; + pFKey->insertConf = (flags >> 16 ) & 0xff; + + /* Link the foreign key to the table as the last step. + */ + p->pFKey = pFKey; + pFKey = 0; + +fk_end: + sqliteFree(pFKey); +#endif /* !defined(SQLITE_OMIT_FOREIGN_KEY) */ + sqlite3ExprListDelete(pFromCol); + sqlite3ExprListDelete(pToCol); +} + +/* +** This routine is called when an INITIALLY IMMEDIATE or INITIALLY DEFERRED +** clause is seen as part of a foreign key definition. The isDeferred +** parameter is 1 for INITIALLY DEFERRED and 0 for INITIALLY IMMEDIATE. +** The behavior of the most recently created foreign key is adjusted +** accordingly. +*/ +void sqlite3DeferForeignKey(Parse *pParse, int isDeferred){ +#ifndef SQLITE_OMIT_FOREIGN_KEY + Table *pTab; + FKey *pFKey; + if( (pTab = pParse->pNewTable)==0 || (pFKey = pTab->pFKey)==0 ) return; + pFKey->isDeferred = isDeferred; +#endif +} + +/* +** Generate code that will erase and refill index *pIdx. This is +** used to initialize a newly created index or to recompute the +** content of an index in response to a REINDEX command. +** +** if memRootPage is not negative, it means that the index is newly +** created. The memory cell specified by memRootPage contains the +** root page number of the index. If memRootPage is negative, then +** the index already exists and must be cleared before being refilled and +** the root page number of the index is taken from pIndex->tnum. +*/ +static void sqlite3RefillIndex(Parse *pParse, Index *pIndex, int memRootPage){ + Table *pTab = pIndex->pTable; /* The table that is indexed */ + int iTab = pParse->nTab; /* Btree cursor used for pTab */ + int iIdx = pParse->nTab+1; /* Btree cursor used for pIndex */ + int addr1; /* Address of top of loop */ + int tnum; /* Root page of index */ + Vdbe *v; /* Generate code into this virtual machine */ + KeyInfo *pKey; /* KeyInfo for index */ + int iDb = sqlite3SchemaToIndex(pParse->db, pIndex->pSchema); + +#ifndef SQLITE_OMIT_AUTHORIZATION + if( sqlite3AuthCheck(pParse, SQLITE_REINDEX, pIndex->zName, 0, + pParse->db->aDb[iDb].zName ) ){ + return; + } +#endif + + /* Require a write-lock on the table to perform this operation */ + sqlite3TableLock(pParse, iDb, pTab->tnum, 1, pTab->zName); + + v = sqlite3GetVdbe(pParse); + if( v==0 ) return; + if( memRootPage>=0 ){ + sqlite3VdbeAddOp(v, OP_MemLoad, memRootPage, 0); + tnum = 0; + }else{ + tnum = pIndex->tnum; + sqlite3VdbeAddOp(v, OP_Clear, tnum, iDb); + } + sqlite3VdbeAddOp(v, OP_Integer, iDb, 0); + pKey = sqlite3IndexKeyinfo(pParse, pIndex); + sqlite3VdbeOp3(v, OP_OpenWrite, iIdx, tnum, (char *)pKey, P3_KEYINFO_HANDOFF); + sqlite3OpenTable(pParse, iTab, iDb, pTab, OP_OpenRead); + addr1 = sqlite3VdbeAddOp(v, OP_Rewind, iTab, 0); + sqlite3GenerateIndexKey(v, pIndex, iTab); + if( pIndex->onError!=OE_None ){ + int curaddr = sqlite3VdbeCurrentAddr(v); + int addr2 = curaddr+4; + sqlite3VdbeChangeP2(v, curaddr-1, addr2); + sqlite3VdbeAddOp(v, OP_Rowid, iTab, 0); + sqlite3VdbeAddOp(v, OP_AddImm, 1, 0); + sqlite3VdbeAddOp(v, OP_IsUnique, iIdx, addr2); + sqlite3VdbeOp3(v, OP_Halt, SQLITE_CONSTRAINT, OE_Abort, + "indexed columns are not unique", P3_STATIC); + assert( addr2==sqlite3VdbeCurrentAddr(v) ); + } + sqlite3VdbeAddOp(v, OP_IdxInsert, iIdx, 0); + sqlite3VdbeAddOp(v, OP_Next, iTab, addr1+1); + sqlite3VdbeJumpHere(v, addr1); + sqlite3VdbeAddOp(v, OP_Close, iTab, 0); + sqlite3VdbeAddOp(v, OP_Close, iIdx, 0); +} + +/* +** Create a new index for an SQL table. pName1.pName2 is the name of the index +** and pTblList is the name of the table that is to be indexed. Both will +** be NULL for a primary key or an index that is created to satisfy a +** UNIQUE constraint. If pTable and pIndex are NULL, use pParse->pNewTable +** as the table to be indexed. pParse->pNewTable is a table that is +** currently being constructed by a CREATE TABLE statement. +** +** pList is a list of columns to be indexed. pList will be NULL if this +** is a primary key or unique-constraint on the most recent column added +** to the table currently under construction. +*/ +void sqlite3CreateIndex( + Parse *pParse, /* All information about this parse */ + Token *pName1, /* First part of index name. May be NULL */ + Token *pName2, /* Second part of index name. May be NULL */ + SrcList *pTblName, /* Table to index. Use pParse->pNewTable if 0 */ + ExprList *pList, /* A list of columns to be indexed */ + int onError, /* OE_Abort, OE_Ignore, OE_Replace, or OE_None */ + Token *pStart, /* The CREATE token that begins a CREATE TABLE statement */ + Token *pEnd, /* The ")" that closes the CREATE INDEX statement */ + int sortOrder, /* Sort order of primary key when pList==NULL */ + int ifNotExist /* Omit error if index already exists */ +){ + Table *pTab = 0; /* Table to be indexed */ + Index *pIndex = 0; /* The index to be created */ + char *zName = 0; /* Name of the index */ + int nName; /* Number of characters in zName */ + int i, j; + Token nullId; /* Fake token for an empty ID list */ + DbFixer sFix; /* For assigning database names to pTable */ + int sortOrderMask; /* 1 to honor DESC in index. 0 to ignore. */ + sqlite3 *db = pParse->db; + Db *pDb; /* The specific table containing the indexed database */ + int iDb; /* Index of the database that is being written */ + Token *pName = 0; /* Unqualified name of the index to create */ + struct ExprList_item *pListItem; /* For looping over pList */ + int nCol; + int nExtra = 0; + char *zExtra; + + if( pParse->nErr || sqlite3MallocFailed() ){ + goto exit_create_index; + } + + /* + ** Find the table that is to be indexed. Return early if not found. + */ + if( pTblName!=0 ){ + + /* Use the two-part index name to determine the database + ** to search for the table. 'Fix' the table name to this db + ** before looking up the table. + */ + assert( pName1 && pName2 ); + iDb = sqlite3TwoPartName(pParse, pName1, pName2, &pName); + if( iDb<0 ) goto exit_create_index; + +#ifndef SQLITE_OMIT_TEMPDB + /* If the index name was unqualified, check if the the table + ** is a temp table. If so, set the database to 1. + */ + pTab = sqlite3SrcListLookup(pParse, pTblName); + if( pName2 && pName2->n==0 && pTab && pTab->pSchema==db->aDb[1].pSchema ){ + iDb = 1; + } +#endif + + if( sqlite3FixInit(&sFix, pParse, iDb, "index", pName) && + sqlite3FixSrcList(&sFix, pTblName) + ){ + /* Because the parser constructs pTblName from a single identifier, + ** sqlite3FixSrcList can never fail. */ + assert(0); + } + pTab = sqlite3LocateTable(pParse, pTblName->a[0].zName, + pTblName->a[0].zDatabase); + if( !pTab ) goto exit_create_index; + assert( db->aDb[iDb].pSchema==pTab->pSchema ); + }else{ + assert( pName==0 ); + pTab = pParse->pNewTable; + if( !pTab ) goto exit_create_index; + iDb = sqlite3SchemaToIndex(db, pTab->pSchema); + } + pDb = &db->aDb[iDb]; + + if( pTab==0 || pParse->nErr ) goto exit_create_index; + if( pTab->readOnly ){ + sqlite3ErrorMsg(pParse, "table %s may not be indexed", pTab->zName); + goto exit_create_index; + } +#ifndef SQLITE_OMIT_VIEW + if( pTab->pSelect ){ + sqlite3ErrorMsg(pParse, "views may not be indexed"); + goto exit_create_index; + } +#endif + + /* + ** Find the name of the index. Make sure there is not already another + ** index or table with the same name. + ** + ** Exception: If we are reading the names of permanent indices from the + ** sqlite_master table (because some other process changed the schema) and + ** one of the index names collides with the name of a temporary table or + ** index, then we will continue to process this index. + ** + ** If pName==0 it means that we are + ** dealing with a primary key or UNIQUE constraint. We have to invent our + ** own name. + */ + if( pName ){ + zName = sqlite3NameFromToken(pName); + if( SQLITE_OK!=sqlite3ReadSchema(pParse) ) goto exit_create_index; + if( zName==0 ) goto exit_create_index; + if( SQLITE_OK!=sqlite3CheckObjectName(pParse, zName) ){ + goto exit_create_index; + } + if( !db->init.busy ){ + if( SQLITE_OK!=sqlite3ReadSchema(pParse) ) goto exit_create_index; + if( sqlite3FindIndex(db, zName, pDb->zName)!=0 ){ + if( !ifNotExist ){ + sqlite3ErrorMsg(pParse, "index %s already exists", zName); + } + goto exit_create_index; + } + if( sqlite3FindTable(db, zName, 0)!=0 ){ + sqlite3ErrorMsg(pParse, "there is already a table named %s", zName); + goto exit_create_index; + } + } + }else{ + char zBuf[30]; + int n; + Index *pLoop; + for(pLoop=pTab->pIndex, n=1; pLoop; pLoop=pLoop->pNext, n++){} + sprintf(zBuf,"_%d",n); + zName = 0; + sqlite3SetString(&zName, "sqlite_autoindex_", pTab->zName, zBuf, (char*)0); + if( zName==0 ) goto exit_create_index; + } + + /* Check for authorization to create an index. + */ +#ifndef SQLITE_OMIT_AUTHORIZATION + { + const char *zDb = pDb->zName; + if( sqlite3AuthCheck(pParse, SQLITE_INSERT, SCHEMA_TABLE(iDb), 0, zDb) ){ + goto exit_create_index; + } + i = SQLITE_CREATE_INDEX; + if( !OMIT_TEMPDB && iDb==1 ) i = SQLITE_CREATE_TEMP_INDEX; + if( sqlite3AuthCheck(pParse, i, zName, pTab->zName, zDb) ){ + goto exit_create_index; + } + } +#endif + + /* If pList==0, it means this routine was called to make a primary + ** key out of the last column added to the table under construction. + ** So create a fake list to simulate this. + */ + if( pList==0 ){ + nullId.z = (u8*)pTab->aCol[pTab->nCol-1].zName; + nullId.n = strlen((char*)nullId.z); + pList = sqlite3ExprListAppend(0, 0, &nullId); + if( pList==0 ) goto exit_create_index; + pList->a[0].sortOrder = sortOrder; + } + + /* Figure out how many bytes of space are required to store explicitly + ** specified collation sequence names. + */ + for(i=0; inExpr; i++){ + Expr *pExpr = pList->a[i].pExpr; + if( pExpr ){ + nExtra += (1 + strlen(pExpr->pColl->zName)); + } + } + + /* + ** Allocate the index structure. + */ + nName = strlen(zName); + nCol = pList->nExpr; + pIndex = sqliteMalloc( + sizeof(Index) + /* Index structure */ + sizeof(int)*nCol + /* Index.aiColumn */ + sizeof(int)*(nCol+1) + /* Index.aiRowEst */ + sizeof(char *)*nCol + /* Index.azColl */ + sizeof(u8)*nCol + /* Index.aSortOrder */ + nName + 1 + /* Index.zName */ + nExtra /* Collation sequence names */ + ); + if( sqlite3MallocFailed() ) goto exit_create_index; + pIndex->azColl = (char**)(&pIndex[1]); + pIndex->aiColumn = (int *)(&pIndex->azColl[nCol]); + pIndex->aiRowEst = (unsigned *)(&pIndex->aiColumn[nCol]); + pIndex->aSortOrder = (u8 *)(&pIndex->aiRowEst[nCol+1]); + pIndex->zName = (char *)(&pIndex->aSortOrder[nCol]); + zExtra = (char *)(&pIndex->zName[nName+1]); + strcpy(pIndex->zName, zName); + pIndex->pTable = pTab; + pIndex->nColumn = pList->nExpr; + pIndex->onError = onError; + pIndex->autoIndex = pName==0; + pIndex->pSchema = db->aDb[iDb].pSchema; + + /* Check to see if we should honor DESC requests on index columns + */ + if( pDb->pSchema->file_format>=4 ){ + sortOrderMask = -1; /* Honor DESC */ + }else{ + sortOrderMask = 0; /* Ignore DESC */ + } + + /* Scan the names of the columns of the table to be indexed and + ** load the column indices into the Index structure. Report an error + ** if any column is not found. + */ + for(i=0, pListItem=pList->a; inExpr; i++, pListItem++){ + const char *zColName = pListItem->zName; + Column *pTabCol; + int requestedSortOrder; + char *zColl; /* Collation sequence */ + + for(j=0, pTabCol=pTab->aCol; jnCol; j++, pTabCol++){ + if( sqlite3StrICmp(zColName, pTabCol->zName)==0 ) break; + } + if( j>=pTab->nCol ){ + sqlite3ErrorMsg(pParse, "table %s has no column named %s", + pTab->zName, zColName); + goto exit_create_index; + } + pIndex->aiColumn[i] = j; + if( pListItem->pExpr ){ + assert( pListItem->pExpr->pColl ); + zColl = zExtra; + strcpy(zExtra, pListItem->pExpr->pColl->zName); + zExtra += (strlen(zColl) + 1); + }else{ + zColl = pTab->aCol[j].zColl; + if( !zColl ){ + zColl = db->pDfltColl->zName; + } + } + if( !db->init.busy && !sqlite3LocateCollSeq(pParse, zColl, -1) ){ + goto exit_create_index; + } + pIndex->azColl[i] = zColl; + requestedSortOrder = pListItem->sortOrder & sortOrderMask; + pIndex->aSortOrder[i] = requestedSortOrder; + } + sqlite3DefaultRowEst(pIndex); + + if( pTab==pParse->pNewTable ){ + /* This routine has been called to create an automatic index as a + ** result of a PRIMARY KEY or UNIQUE clause on a column definition, or + ** a PRIMARY KEY or UNIQUE clause following the column definitions. + ** i.e. one of: + ** + ** CREATE TABLE t(x PRIMARY KEY, y); + ** CREATE TABLE t(x, y, UNIQUE(x, y)); + ** + ** Either way, check to see if the table already has such an index. If + ** so, don't bother creating this one. This only applies to + ** automatically created indices. Users can do as they wish with + ** explicit indices. + */ + Index *pIdx; + for(pIdx=pTab->pIndex; pIdx; pIdx=pIdx->pNext){ + int k; + assert( pIdx->onError!=OE_None ); + assert( pIdx->autoIndex ); + assert( pIndex->onError!=OE_None ); + + if( pIdx->nColumn!=pIndex->nColumn ) continue; + for(k=0; knColumn; k++){ + const char *z1 = pIdx->azColl[k]; + const char *z2 = pIndex->azColl[k]; + if( pIdx->aiColumn[k]!=pIndex->aiColumn[k] ) break; + if( pIdx->aSortOrder[k]!=pIndex->aSortOrder[k] ) break; + if( z1!=z2 && sqlite3StrICmp(z1, z2) ) break; + } + if( k==pIdx->nColumn ){ + if( pIdx->onError!=pIndex->onError ){ + /* This constraint creates the same index as a previous + ** constraint specified somewhere in the CREATE TABLE statement. + ** However the ON CONFLICT clauses are different. If both this + ** constraint and the previous equivalent constraint have explicit + ** ON CONFLICT clauses this is an error. Otherwise, use the + ** explicitly specified behaviour for the index. + */ + if( !(pIdx->onError==OE_Default || pIndex->onError==OE_Default) ){ + sqlite3ErrorMsg(pParse, + "conflicting ON CONFLICT clauses specified", 0); + } + if( pIdx->onError==OE_Default ){ + pIdx->onError = pIndex->onError; + } + } + goto exit_create_index; + } + } + } + + /* Link the new Index structure to its table and to the other + ** in-memory database structures. + */ + if( db->init.busy ){ + Index *p; + p = sqlite3HashInsert(&pIndex->pSchema->idxHash, + pIndex->zName, strlen(pIndex->zName)+1, pIndex); + if( p ){ + assert( p==pIndex ); /* Malloc must have failed */ + goto exit_create_index; + } + db->flags |= SQLITE_InternChanges; + if( pTblName!=0 ){ + pIndex->tnum = db->init.newTnum; + } + } + + /* If the db->init.busy is 0 then create the index on disk. This + ** involves writing the index into the master table and filling in the + ** index with the current table contents. + ** + ** The db->init.busy is 0 when the user first enters a CREATE INDEX + ** command. db->init.busy is 1 when a database is opened and + ** CREATE INDEX statements are read out of the master table. In + ** the latter case the index already exists on disk, which is why + ** we don't want to recreate it. + ** + ** If pTblName==0 it means this index is generated as a primary key + ** or UNIQUE constraint of a CREATE TABLE statement. Since the table + ** has just been created, it contains no data and the index initialization + ** step can be skipped. + */ + else if( db->init.busy==0 ){ + Vdbe *v; + char *zStmt; + int iMem = pParse->nMem++; + + v = sqlite3GetVdbe(pParse); + if( v==0 ) goto exit_create_index; + + + /* Create the rootpage for the index + */ + sqlite3BeginWriteOperation(pParse, 1, iDb); + sqlite3VdbeAddOp(v, OP_CreateIndex, iDb, 0); + sqlite3VdbeAddOp(v, OP_MemStore, iMem, 0); + + /* Gather the complete text of the CREATE INDEX statement into + ** the zStmt variable + */ + if( pStart && pEnd ){ + /* A named index with an explicit CREATE INDEX statement */ + zStmt = sqlite3MPrintf("CREATE%s INDEX %.*s", + onError==OE_None ? "" : " UNIQUE", + pEnd->z - pName->z + 1, + pName->z); + }else{ + /* An automatic index created by a PRIMARY KEY or UNIQUE constraint */ + /* zStmt = sqlite3MPrintf(""); */ + zStmt = 0; + } + + /* Add an entry in sqlite_master for this index + */ + sqlite3NestedParse(pParse, + "INSERT INTO %Q.%s VALUES('index',%Q,%Q,#0,%Q);", + db->aDb[iDb].zName, SCHEMA_TABLE(iDb), + pIndex->zName, + pTab->zName, + zStmt + ); + sqlite3VdbeAddOp(v, OP_Pop, 1, 0); + sqliteFree(zStmt); + + /* Fill the index with data and reparse the schema. Code an OP_Expire + ** to invalidate all pre-compiled statements. + */ + if( pTblName ){ + sqlite3RefillIndex(pParse, pIndex, iMem); + sqlite3ChangeCookie(db, v, iDb); + sqlite3VdbeOp3(v, OP_ParseSchema, iDb, 0, + sqlite3MPrintf("name='%q'", pIndex->zName), P3_DYNAMIC); + sqlite3VdbeAddOp(v, OP_Expire, 0, 0); + } + } + + /* When adding an index to the list of indices for a table, make + ** sure all indices labeled OE_Replace come after all those labeled + ** OE_Ignore. This is necessary for the correct operation of UPDATE + ** and INSERT. + */ + if( db->init.busy || pTblName==0 ){ + if( onError!=OE_Replace || pTab->pIndex==0 + || pTab->pIndex->onError==OE_Replace){ + pIndex->pNext = pTab->pIndex; + pTab->pIndex = pIndex; + }else{ + Index *pOther = pTab->pIndex; + while( pOther->pNext && pOther->pNext->onError!=OE_Replace ){ + pOther = pOther->pNext; + } + pIndex->pNext = pOther->pNext; + pOther->pNext = pIndex; + } + pIndex = 0; + } + + /* Clean up before exiting */ +exit_create_index: + if( pIndex ){ + freeIndex(pIndex); + } + sqlite3ExprListDelete(pList); + sqlite3SrcListDelete(pTblName); + sqliteFree(zName); + return; +} + +/* +** Generate code to make sure the file format number is at least minFormat. +** The generated code will increase the file format number if necessary. +*/ +void sqlite3MinimumFileFormat(Parse *pParse, int iDb, int minFormat){ + Vdbe *v; + v = sqlite3GetVdbe(pParse); + if( v ){ + sqlite3VdbeAddOp(v, OP_ReadCookie, iDb, 1); + sqlite3VdbeAddOp(v, OP_Integer, minFormat, 0); + sqlite3VdbeAddOp(v, OP_Ge, 0, sqlite3VdbeCurrentAddr(v)+3); + sqlite3VdbeAddOp(v, OP_Integer, minFormat, 0); + sqlite3VdbeAddOp(v, OP_SetCookie, iDb, 1); + } +} + +/* +** Fill the Index.aiRowEst[] array with default information - information +** to be used when we have not run the ANALYZE command. +** +** aiRowEst[0] is suppose to contain the number of elements in the index. +** Since we do not know, guess 1 million. aiRowEst[1] is an estimate of the +** number of rows in the table that match any particular value of the +** first column of the index. aiRowEst[2] is an estimate of the number +** of rows that match any particular combiniation of the first 2 columns +** of the index. And so forth. It must always be the case that +* +** aiRowEst[N]<=aiRowEst[N-1] +** aiRowEst[N]>=1 +** +** Apart from that, we have little to go on besides intuition as to +** how aiRowEst[] should be initialized. The numbers generated here +** are based on typical values found in actual indices. +*/ +void sqlite3DefaultRowEst(Index *pIdx){ + unsigned *a = pIdx->aiRowEst; + int i; + assert( a!=0 ); + a[0] = 1000000; + for(i=pIdx->nColumn; i>=1; i--){ + a[i] = 10; + } + if( pIdx->onError!=OE_None ){ + a[pIdx->nColumn] = 1; + } +} + +/* +** This routine will drop an existing named index. This routine +** implements the DROP INDEX statement. +*/ +void sqlite3DropIndex(Parse *pParse, SrcList *pName, int ifExists){ + Index *pIndex; + Vdbe *v; + sqlite3 *db = pParse->db; + int iDb; + + if( pParse->nErr || sqlite3MallocFailed() ){ + goto exit_drop_index; + } + assert( pName->nSrc==1 ); + if( SQLITE_OK!=sqlite3ReadSchema(pParse) ){ + goto exit_drop_index; + } + pIndex = sqlite3FindIndex(db, pName->a[0].zName, pName->a[0].zDatabase); + if( pIndex==0 ){ + if( !ifExists ){ + sqlite3ErrorMsg(pParse, "no such index: %S", pName, 0); + } + pParse->checkSchema = 1; + goto exit_drop_index; + } + if( pIndex->autoIndex ){ + sqlite3ErrorMsg(pParse, "index associated with UNIQUE " + "or PRIMARY KEY constraint cannot be dropped", 0); + goto exit_drop_index; + } + iDb = sqlite3SchemaToIndex(db, pIndex->pSchema); +#ifndef SQLITE_OMIT_AUTHORIZATION + { + int code = SQLITE_DROP_INDEX; + Table *pTab = pIndex->pTable; + const char *zDb = db->aDb[iDb].zName; + const char *zTab = SCHEMA_TABLE(iDb); + if( sqlite3AuthCheck(pParse, SQLITE_DELETE, zTab, 0, zDb) ){ + goto exit_drop_index; + } + if( !OMIT_TEMPDB && iDb ) code = SQLITE_DROP_TEMP_INDEX; + if( sqlite3AuthCheck(pParse, code, pIndex->zName, pTab->zName, zDb) ){ + goto exit_drop_index; + } + } +#endif + + /* Generate code to remove the index and from the master table */ + v = sqlite3GetVdbe(pParse); + if( v ){ + sqlite3NestedParse(pParse, + "DELETE FROM %Q.%s WHERE name=%Q", + db->aDb[iDb].zName, SCHEMA_TABLE(iDb), + pIndex->zName + ); + sqlite3ChangeCookie(db, v, iDb); + destroyRootPage(pParse, pIndex->tnum, iDb); + sqlite3VdbeOp3(v, OP_DropIndex, iDb, 0, pIndex->zName, 0); + } + +exit_drop_index: + sqlite3SrcListDelete(pName); +} + +/* +** ppArray points into a structure where there is an array pointer +** followed by two integers. The first integer is the +** number of elements in the structure array. The second integer +** is the number of allocated slots in the array. +** +** In other words, the structure looks something like this: +** +** struct Example1 { +** struct subElem *aEntry; +** int nEntry; +** int nAlloc; +** } +** +** The pnEntry parameter points to the equivalent of Example1.nEntry. +** +** This routine allocates a new slot in the array, zeros it out, +** and returns its index. If malloc fails a negative number is returned. +** +** szEntry is the sizeof of a single array entry. initSize is the +** number of array entries allocated on the initial allocation. +*/ +int sqlite3ArrayAllocate(void **ppArray, int szEntry, int initSize){ + char *p; + int *an = (int*)&ppArray[1]; + if( an[0]>=an[1] ){ + void *pNew; + int newSize; + newSize = an[1]*2 + initSize; + pNew = sqliteRealloc(*ppArray, newSize*szEntry); + if( pNew==0 ){ + return -1; + } + an[1] = newSize; + *ppArray = pNew; + } + p = *ppArray; + memset(&p[an[0]*szEntry], 0, szEntry); + return an[0]++; +} + +/* +** Append a new element to the given IdList. Create a new IdList if +** need be. +** +** A new IdList is returned, or NULL if malloc() fails. +*/ +IdList *sqlite3IdListAppend(IdList *pList, Token *pToken){ + int i; + if( pList==0 ){ + pList = sqliteMalloc( sizeof(IdList) ); + if( pList==0 ) return 0; + pList->nAlloc = 0; + } + i = sqlite3ArrayAllocate((void**)&pList->a, sizeof(pList->a[0]), 5); + if( i<0 ){ + sqlite3IdListDelete(pList); + return 0; + } + pList->a[i].zName = sqlite3NameFromToken(pToken); + return pList; +} + +/* +** Delete an IdList. +*/ +void sqlite3IdListDelete(IdList *pList){ + int i; + if( pList==0 ) return; + for(i=0; inId; i++){ + sqliteFree(pList->a[i].zName); + } + sqliteFree(pList->a); + sqliteFree(pList); +} + +/* +** Return the index in pList of the identifier named zId. Return -1 +** if not found. +*/ +int sqlite3IdListIndex(IdList *pList, const char *zName){ + int i; + if( pList==0 ) return -1; + for(i=0; inId; i++){ + if( sqlite3StrICmp(pList->a[i].zName, zName)==0 ) return i; + } + return -1; +} + +/* +** Append a new table name to the given SrcList. Create a new SrcList if +** need be. A new entry is created in the SrcList even if pToken is NULL. +** +** A new SrcList is returned, or NULL if malloc() fails. +** +** If pDatabase is not null, it means that the table has an optional +** database name prefix. Like this: "database.table". The pDatabase +** points to the table name and the pTable points to the database name. +** The SrcList.a[].zName field is filled with the table name which might +** come from pTable (if pDatabase is NULL) or from pDatabase. +** SrcList.a[].zDatabase is filled with the database name from pTable, +** or with NULL if no database is specified. +** +** In other words, if call like this: +** +** sqlite3SrcListAppend(A,B,0); +** +** Then B is a table name and the database name is unspecified. If called +** like this: +** +** sqlite3SrcListAppend(A,B,C); +** +** Then C is the table name and B is the database name. +*/ +SrcList *sqlite3SrcListAppend(SrcList *pList, Token *pTable, Token *pDatabase){ + struct SrcList_item *pItem; + if( pList==0 ){ + pList = sqliteMalloc( sizeof(SrcList) ); + if( pList==0 ) return 0; + pList->nAlloc = 1; + } + if( pList->nSrc>=pList->nAlloc ){ + SrcList *pNew; + pList->nAlloc *= 2; + pNew = sqliteRealloc(pList, + sizeof(*pList) + (pList->nAlloc-1)*sizeof(pList->a[0]) ); + if( pNew==0 ){ + sqlite3SrcListDelete(pList); + return 0; + } + pList = pNew; + } + pItem = &pList->a[pList->nSrc]; + memset(pItem, 0, sizeof(pList->a[0])); + if( pDatabase && pDatabase->z==0 ){ + pDatabase = 0; + } + if( pDatabase && pTable ){ + Token *pTemp = pDatabase; + pDatabase = pTable; + pTable = pTemp; + } + pItem->zName = sqlite3NameFromToken(pTable); + pItem->zDatabase = sqlite3NameFromToken(pDatabase); + pItem->iCursor = -1; + pItem->isPopulated = 0; + pList->nSrc++; + return pList; +} + +/* +** Assign cursors to all tables in a SrcList +*/ +void sqlite3SrcListAssignCursors(Parse *pParse, SrcList *pList){ + int i; + struct SrcList_item *pItem; + assert(pList || sqlite3MallocFailed() ); + if( pList ){ + for(i=0, pItem=pList->a; inSrc; i++, pItem++){ + if( pItem->iCursor>=0 ) break; + pItem->iCursor = pParse->nTab++; + if( pItem->pSelect ){ + sqlite3SrcListAssignCursors(pParse, pItem->pSelect->pSrc); + } + } + } +} + +/* +** Add an alias to the last identifier on the given identifier list. +*/ +void sqlite3SrcListAddAlias(SrcList *pList, Token *pToken){ + if( pList && pList->nSrc>0 ){ + pList->a[pList->nSrc-1].zAlias = sqlite3NameFromToken(pToken); + } +} + +/* +** Delete an entire SrcList including all its substructure. +*/ +void sqlite3SrcListDelete(SrcList *pList){ + int i; + struct SrcList_item *pItem; + if( pList==0 ) return; + for(pItem=pList->a, i=0; inSrc; i++, pItem++){ + sqliteFree(pItem->zDatabase); + sqliteFree(pItem->zName); + sqliteFree(pItem->zAlias); + sqlite3DeleteTable(0, pItem->pTab); + sqlite3SelectDelete(pItem->pSelect); + sqlite3ExprDelete(pItem->pOn); + sqlite3IdListDelete(pItem->pUsing); + } + sqliteFree(pList); +} + +/* +** Begin a transaction +*/ +void sqlite3BeginTransaction(Parse *pParse, int type){ + sqlite3 *db; + Vdbe *v; + int i; + + if( pParse==0 || (db=pParse->db)==0 || db->aDb[0].pBt==0 ) return; + if( pParse->nErr || sqlite3MallocFailed() ) return; + if( sqlite3AuthCheck(pParse, SQLITE_TRANSACTION, "BEGIN", 0, 0) ) return; + + v = sqlite3GetVdbe(pParse); + if( !v ) return; + if( type!=TK_DEFERRED ){ + for(i=0; inDb; i++){ + sqlite3VdbeAddOp(v, OP_Transaction, i, (type==TK_EXCLUSIVE)+1); + } + } + sqlite3VdbeAddOp(v, OP_AutoCommit, 0, 0); +} + +/* +** Commit a transaction +*/ +void sqlite3CommitTransaction(Parse *pParse){ + sqlite3 *db; + Vdbe *v; + + if( pParse==0 || (db=pParse->db)==0 || db->aDb[0].pBt==0 ) return; + if( pParse->nErr || sqlite3MallocFailed() ) return; + if( sqlite3AuthCheck(pParse, SQLITE_TRANSACTION, "COMMIT", 0, 0) ) return; + + v = sqlite3GetVdbe(pParse); + if( v ){ + sqlite3VdbeAddOp(v, OP_AutoCommit, 1, 0); + } +} + +/* +** Rollback a transaction +*/ +void sqlite3RollbackTransaction(Parse *pParse){ + sqlite3 *db; + Vdbe *v; + + if( pParse==0 || (db=pParse->db)==0 || db->aDb[0].pBt==0 ) return; + if( pParse->nErr || sqlite3MallocFailed() ) return; + if( sqlite3AuthCheck(pParse, SQLITE_TRANSACTION, "ROLLBACK", 0, 0) ) return; + + v = sqlite3GetVdbe(pParse); + if( v ){ + sqlite3VdbeAddOp(v, OP_AutoCommit, 1, 1); + } +} + +/* +** Make sure the TEMP database is open and available for use. Return +** the number of errors. Leave any error messages in the pParse structure. +*/ +static int sqlite3OpenTempDatabase(Parse *pParse){ + sqlite3 *db = pParse->db; + if( db->aDb[1].pBt==0 && !pParse->explain ){ + int rc = sqlite3BtreeFactory(db, 0, 0, MAX_PAGES, &db->aDb[1].pBt); + if( rc!=SQLITE_OK ){ + sqlite3ErrorMsg(pParse, "unable to open a temporary database " + "file for storing temporary tables"); + pParse->rc = rc; + return 1; + } + if( db->flags & !db->autoCommit ){ + rc = sqlite3BtreeBeginTrans(db->aDb[1].pBt, 1); + if( rc!=SQLITE_OK ){ + sqlite3ErrorMsg(pParse, "unable to get a write lock on " + "the temporary database file"); + pParse->rc = rc; + return 1; + } + } + assert( db->aDb[1].pSchema ); + } + return 0; +} + +/* +** Generate VDBE code that will verify the schema cookie and start +** a read-transaction for all named database files. +** +** It is important that all schema cookies be verified and all +** read transactions be started before anything else happens in +** the VDBE program. But this routine can be called after much other +** code has been generated. So here is what we do: +** +** The first time this routine is called, we code an OP_Goto that +** will jump to a subroutine at the end of the program. Then we +** record every database that needs its schema verified in the +** pParse->cookieMask field. Later, after all other code has been +** generated, the subroutine that does the cookie verifications and +** starts the transactions will be coded and the OP_Goto P2 value +** will be made to point to that subroutine. The generation of the +** cookie verification subroutine code happens in sqlite3FinishCoding(). +** +** If iDb<0 then code the OP_Goto only - don't set flag to verify the +** schema on any databases. This can be used to position the OP_Goto +** early in the code, before we know if any database tables will be used. +*/ +void sqlite3CodeVerifySchema(Parse *pParse, int iDb){ + sqlite3 *db; + Vdbe *v; + int mask; + + v = sqlite3GetVdbe(pParse); + if( v==0 ) return; /* This only happens if there was a prior error */ + db = pParse->db; + if( pParse->cookieGoto==0 ){ + pParse->cookieGoto = sqlite3VdbeAddOp(v, OP_Goto, 0, 0)+1; + } + if( iDb>=0 ){ + assert( iDbnDb ); + assert( db->aDb[iDb].pBt!=0 || iDb==1 ); + assert( iDb<32 ); + mask = 1<cookieMask & mask)==0 ){ + pParse->cookieMask |= mask; + pParse->cookieValue[iDb] = db->aDb[iDb].pSchema->schema_cookie; + if( !OMIT_TEMPDB && iDb==1 ){ + sqlite3OpenTempDatabase(pParse); + } + } + } +} + +/* +** Generate VDBE code that prepares for doing an operation that +** might change the database. +** +** This routine starts a new transaction if we are not already within +** a transaction. If we are already within a transaction, then a checkpoint +** is set if the setStatement parameter is true. A checkpoint should +** be set for operations that might fail (due to a constraint) part of +** the way through and which will need to undo some writes without having to +** rollback the whole transaction. For operations where all constraints +** can be checked before any changes are made to the database, it is never +** necessary to undo a write and the checkpoint should not be set. +** +** Only database iDb and the temp database are made writable by this call. +** If iDb==0, then the main and temp databases are made writable. If +** iDb==1 then only the temp database is made writable. If iDb>1 then the +** specified auxiliary database and the temp database are made writable. +*/ +void sqlite3BeginWriteOperation(Parse *pParse, int setStatement, int iDb){ + Vdbe *v = sqlite3GetVdbe(pParse); + if( v==0 ) return; + sqlite3CodeVerifySchema(pParse, iDb); + pParse->writeMask |= 1<nested==0 ){ + sqlite3VdbeAddOp(v, OP_Statement, iDb, 0); + } + if( (OMIT_TEMPDB || iDb!=1) && pParse->db->aDb[1].pBt!=0 ){ + sqlite3BeginWriteOperation(pParse, setStatement, 1); + } +} + +/* +** Check to see if pIndex uses the collating sequence pColl. Return +** true if it does and false if it does not. +*/ +#ifndef SQLITE_OMIT_REINDEX +static int collationMatch(const char *zColl, Index *pIndex){ + int i; + for(i=0; inColumn; i++){ + const char *z = pIndex->azColl[i]; + if( z==zColl || (z && zColl && 0==sqlite3StrICmp(z, zColl)) ){ + return 1; + } + } + return 0; +} +#endif + +/* +** Recompute all indices of pTab that use the collating sequence pColl. +** If pColl==0 then recompute all indices of pTab. +*/ +#ifndef SQLITE_OMIT_REINDEX +static void reindexTable(Parse *pParse, Table *pTab, char const *zColl){ + Index *pIndex; /* An index associated with pTab */ + + for(pIndex=pTab->pIndex; pIndex; pIndex=pIndex->pNext){ + if( zColl==0 || collationMatch(zColl, pIndex) ){ + int iDb = sqlite3SchemaToIndex(pParse->db, pTab->pSchema); + sqlite3BeginWriteOperation(pParse, 0, iDb); + sqlite3RefillIndex(pParse, pIndex, -1); + } + } +} +#endif + +/* +** Recompute all indices of all tables in all databases where the +** indices use the collating sequence pColl. If pColl==0 then recompute +** all indices everywhere. +*/ +#ifndef SQLITE_OMIT_REINDEX +static void reindexDatabases(Parse *pParse, char const *zColl){ + Db *pDb; /* A single database */ + int iDb; /* The database index number */ + sqlite3 *db = pParse->db; /* The database connection */ + HashElem *k; /* For looping over tables in pDb */ + Table *pTab; /* A table in the database */ + + for(iDb=0, pDb=db->aDb; iDbnDb; iDb++, pDb++){ + if( pDb==0 ) continue; + for(k=sqliteHashFirst(&pDb->pSchema->tblHash); k; k=sqliteHashNext(k)){ + pTab = (Table*)sqliteHashData(k); + reindexTable(pParse, pTab, zColl); + } + } +} +#endif + +/* +** Generate code for the REINDEX command. +** +** REINDEX -- 1 +** REINDEX -- 2 +** REINDEX ?.? -- 3 +** REINDEX ?.? -- 4 +** +** Form 1 causes all indices in all attached databases to be rebuilt. +** Form 2 rebuilds all indices in all databases that use the named +** collating function. Forms 3 and 4 rebuild the named index or all +** indices associated with the named table. +*/ +#ifndef SQLITE_OMIT_REINDEX +void sqlite3Reindex(Parse *pParse, Token *pName1, Token *pName2){ + CollSeq *pColl; /* Collating sequence to be reindexed, or NULL */ + char *z; /* Name of a table or index */ + const char *zDb; /* Name of the database */ + Table *pTab; /* A table in the database */ + Index *pIndex; /* An index associated with pTab */ + int iDb; /* The database index number */ + sqlite3 *db = pParse->db; /* The database connection */ + Token *pObjName; /* Name of the table or index to be reindexed */ + + /* Read the database schema. If an error occurs, leave an error message + ** and code in pParse and return NULL. */ + if( SQLITE_OK!=sqlite3ReadSchema(pParse) ){ + return; + } + + if( pName1==0 || pName1->z==0 ){ + reindexDatabases(pParse, 0); + return; + }else if( pName2==0 || pName2->z==0 ){ + assert( pName1->z ); + pColl = sqlite3FindCollSeq(db, ENC(db), (char*)pName1->z, pName1->n, 0); + if( pColl ){ + char *zColl = sqliteStrNDup((const char *)pName1->z, pName1->n); + if( zColl ){ + reindexDatabases(pParse, zColl); + sqliteFree(zColl); + } + return; + } + } + iDb = sqlite3TwoPartName(pParse, pName1, pName2, &pObjName); + if( iDb<0 ) return; + z = sqlite3NameFromToken(pObjName); + zDb = db->aDb[iDb].zName; + pTab = sqlite3FindTable(db, z, zDb); + if( pTab ){ + reindexTable(pParse, pTab, 0); + sqliteFree(z); + return; + } + pIndex = sqlite3FindIndex(db, z, zDb); + sqliteFree(z); + if( pIndex ){ + sqlite3BeginWriteOperation(pParse, 0, iDb); + sqlite3RefillIndex(pParse, pIndex, -1); + return; + } + sqlite3ErrorMsg(pParse, "unable to identify the object to be reindexed"); +} +#endif + +/* +** Return a dynamicly allocated KeyInfo structure that can be used +** with OP_OpenRead or OP_OpenWrite to access database index pIdx. +** +** If successful, a pointer to the new structure is returned. In this case +** the caller is responsible for calling sqliteFree() on the returned +** pointer. If an error occurs (out of memory or missing collation +** sequence), NULL is returned and the state of pParse updated to reflect +** the error. +*/ +KeyInfo *sqlite3IndexKeyinfo(Parse *pParse, Index *pIdx){ + int i; + int nCol = pIdx->nColumn; + int nBytes = sizeof(KeyInfo) + (nCol-1)*sizeof(CollSeq*) + nCol; + KeyInfo *pKey = (KeyInfo *)sqliteMalloc(nBytes); + + if( pKey ){ + pKey->aSortOrder = (u8 *)&(pKey->aColl[nCol]); + assert( &pKey->aSortOrder[nCol]==&(((u8 *)pKey)[nBytes]) ); + for(i=0; iazColl[i]; + assert( zColl ); + pKey->aColl[i] = sqlite3LocateCollSeq(pParse, zColl, -1); + pKey->aSortOrder[i] = pIdx->aSortOrder[i]; + } + pKey->nField = nCol; + } + + if( pParse->nErr ){ + sqliteFree(pKey); + pKey = 0; + } + return pKey; +} Added: external/sqlite-source-3.3.4/callback.c ============================================================================== --- (empty file) +++ external/sqlite-source-3.3.4/callback.c Mon Apr 3 07:54:59 2006 @@ -0,0 +1,314 @@ +/* +** 2005 May 23 +** +** The author disclaims copyright to this source code. In place of +** a legal notice, here is a blessing: +** +** May you do good and not evil. +** May you find forgiveness for yourself and forgive others. +** May you share freely, never taking more than you give. +** +************************************************************************* +** +** This file contains functions used to access the internal hash tables +** of user defined functions and collation sequences. +** +** $Id: callback.c,v 1.12 2006/01/18 16:51:35 danielk1977 Exp $ +*/ + +#include "sqliteInt.h" + +/* +** Invoke the 'collation needed' callback to request a collation sequence +** in the database text encoding of name zName, length nName. +** If the collation sequence +*/ +static void callCollNeeded(sqlite3 *db, const char *zName, int nName){ + assert( !db->xCollNeeded || !db->xCollNeeded16 ); + if( nName<0 ) nName = strlen(zName); + if( db->xCollNeeded ){ + char *zExternal = sqliteStrNDup(zName, nName); + if( !zExternal ) return; + db->xCollNeeded(db->pCollNeededArg, db, (int)ENC(db), zExternal); + sqliteFree(zExternal); + } +#ifndef SQLITE_OMIT_UTF16 + if( db->xCollNeeded16 ){ + char const *zExternal; + sqlite3_value *pTmp = sqlite3ValueNew(); + sqlite3ValueSetStr(pTmp, nName, zName, SQLITE_UTF8, SQLITE_STATIC); + zExternal = sqlite3ValueText(pTmp, SQLITE_UTF16NATIVE); + if( zExternal ){ + db->xCollNeeded16(db->pCollNeededArg, db, (int)ENC(db), zExternal); + } + sqlite3ValueFree(pTmp); + } +#endif +} + +/* +** This routine is called if the collation factory fails to deliver a +** collation function in the best encoding but there may be other versions +** of this collation function (for other text encodings) available. Use one +** of these instead if they exist. Avoid a UTF-8 <-> UTF-16 conversion if +** possible. +*/ +static int synthCollSeq(sqlite3 *db, CollSeq *pColl){ + CollSeq *pColl2; + char *z = pColl->zName; + int n = strlen(z); + int i; + static const u8 aEnc[] = { SQLITE_UTF16BE, SQLITE_UTF16LE, SQLITE_UTF8 }; + for(i=0; i<3; i++){ + pColl2 = sqlite3FindCollSeq(db, aEnc[i], z, n, 0); + if( pColl2->xCmp!=0 ){ + memcpy(pColl, pColl2, sizeof(CollSeq)); + return SQLITE_OK; + } + } + return SQLITE_ERROR; +} + +/* +** This function is responsible for invoking the collation factory callback +** or substituting a collation sequence of a different encoding when the +** requested collation sequence is not available in the database native +** encoding. +** +** If it is not NULL, then pColl must point to the database native encoding +** collation sequence with name zName, length nName. +** +** The return value is either the collation sequence to be used in database +** db for collation type name zName, length nName, or NULL, if no collation +** sequence can be found. +*/ +CollSeq *sqlite3GetCollSeq( + sqlite3* db, + CollSeq *pColl, + const char *zName, + int nName +){ + CollSeq *p; + + p = pColl; + if( !p ){ + p = sqlite3FindCollSeq(db, ENC(db), zName, nName, 0); + } + if( !p || !p->xCmp ){ + /* No collation sequence of this type for this encoding is registered. + ** Call the collation factory to see if it can supply us with one. + */ + callCollNeeded(db, zName, nName); + p = sqlite3FindCollSeq(db, ENC(db), zName, nName, 0); + } + if( p && !p->xCmp && synthCollSeq(db, p) ){ + p = 0; + } + assert( !p || p->xCmp ); + return p; +} + +/* +** This routine is called on a collation sequence before it is used to +** check that it is defined. An undefined collation sequence exists when +** a database is loaded that contains references to collation sequences +** that have not been defined by sqlite3_create_collation() etc. +** +** If required, this routine calls the 'collation needed' callback to +** request a definition of the collating sequence. If this doesn't work, +** an equivalent collating sequence that uses a text encoding different +** from the main database is substituted, if one is available. +*/ +int sqlite3CheckCollSeq(Parse *pParse, CollSeq *pColl){ + if( pColl ){ + const char *zName = pColl->zName; + CollSeq *p = sqlite3GetCollSeq(pParse->db, pColl, zName, -1); + if( !p ){ + if( pParse->nErr==0 ){ + sqlite3ErrorMsg(pParse, "no such collation sequence: %s", zName); + } + pParse->nErr++; + return SQLITE_ERROR; + } + assert( p==pColl ); + } + return SQLITE_OK; +} + + + +/* +** Locate and return an entry from the db.aCollSeq hash table. If the entry +** specified by zName and nName is not found and parameter 'create' is +** true, then create a new entry. Otherwise return NULL. +** +** Each pointer stored in the sqlite3.aCollSeq hash table contains an +** array of three CollSeq structures. The first is the collation sequence +** prefferred for UTF-8, the second UTF-16le, and the third UTF-16be. +** +** Stored immediately after the three collation sequences is a copy of +** the collation sequence name. A pointer to this string is stored in +** each collation sequence structure. +*/ +static CollSeq *findCollSeqEntry( + sqlite3 *db, + const char *zName, + int nName, + int create +){ + CollSeq *pColl; + if( nName<0 ) nName = strlen(zName); + pColl = sqlite3HashFind(&db->aCollSeq, zName, nName); + + if( 0==pColl && create ){ + pColl = sqliteMalloc( 3*sizeof(*pColl) + nName + 1 ); + if( pColl ){ + CollSeq *pDel = 0; + pColl[0].zName = (char*)&pColl[3]; + pColl[0].enc = SQLITE_UTF8; + pColl[1].zName = (char*)&pColl[3]; + pColl[1].enc = SQLITE_UTF16LE; + pColl[2].zName = (char*)&pColl[3]; + pColl[2].enc = SQLITE_UTF16BE; + memcpy(pColl[0].zName, zName, nName); + pColl[0].zName[nName] = 0; + pDel = sqlite3HashInsert(&db->aCollSeq, pColl[0].zName, nName, pColl); + + /* If a malloc() failure occured in sqlite3HashInsert(), it will + ** return the pColl pointer to be deleted (because it wasn't added + ** to the hash table). + */ + assert( !pDel || + (sqlite3MallocFailed() && pDel==pColl) ); + sqliteFree(pDel); + } + } + return pColl; +} + +/* +** Parameter zName points to a UTF-8 encoded string nName bytes long. +** Return the CollSeq* pointer for the collation sequence named zName +** for the encoding 'enc' from the database 'db'. +** +** If the entry specified is not found and 'create' is true, then create a +** new entry. Otherwise return NULL. +*/ +CollSeq *sqlite3FindCollSeq( + sqlite3 *db, + u8 enc, + const char *zName, + int nName, + int create +){ + CollSeq *pColl; + if( zName ){ + pColl = findCollSeqEntry(db, zName, nName, create); + }else{ + pColl = db->pDfltColl; + } + assert( SQLITE_UTF8==1 && SQLITE_UTF16LE==2 && SQLITE_UTF16BE==3 ); + assert( enc>=SQLITE_UTF8 && enc<=SQLITE_UTF16BE ); + if( pColl ) pColl += enc-1; + return pColl; +} + +/* +** Locate a user function given a name, a number of arguments and a flag +** indicating whether the function prefers UTF-16 over UTF-8. Return a +** pointer to the FuncDef structure that defines that function, or return +** NULL if the function does not exist. +** +** If the createFlag argument is true, then a new (blank) FuncDef +** structure is created and liked into the "db" structure if a +** no matching function previously existed. When createFlag is true +** and the nArg parameter is -1, then only a function that accepts +** any number of arguments will be returned. +** +** If createFlag is false and nArg is -1, then the first valid +** function found is returned. A function is valid if either xFunc +** or xStep is non-zero. +** +** If createFlag is false, then a function with the required name and +** number of arguments may be returned even if the eTextRep flag does not +** match that requested. +*/ +FuncDef *sqlite3FindFunction( + sqlite3 *db, /* An open database */ + const char *zName, /* Name of the function. Not null-terminated */ + int nName, /* Number of characters in the name */ + int nArg, /* Number of arguments. -1 means any number */ + u8 enc, /* Preferred text encoding */ + int createFlag /* Create new entry if true and does not otherwise exist */ +){ + FuncDef *p; /* Iterator variable */ + FuncDef *pFirst; /* First function with this name */ + FuncDef *pBest = 0; /* Best match found so far */ + int bestmatch = 0; + + + assert( enc==SQLITE_UTF8 || enc==SQLITE_UTF16LE || enc==SQLITE_UTF16BE ); + if( nArg<-1 ) nArg = -1; + + pFirst = (FuncDef*)sqlite3HashFind(&db->aFunc, zName, nName); + for(p=pFirst; p; p=p->pNext){ + /* During the search for the best function definition, bestmatch is set + ** as follows to indicate the quality of the match with the definition + ** pointed to by pBest: + ** + ** 0: pBest is NULL. No match has been found. + ** 1: A variable arguments function that prefers UTF-8 when a UTF-16 + ** encoding is requested, or vice versa. + ** 2: A variable arguments function that uses UTF-16BE when UTF-16LE is + ** requested, or vice versa. + ** 3: A variable arguments function using the same text encoding. + ** 4: A function with the exact number of arguments requested that + ** prefers UTF-8 when a UTF-16 encoding is requested, or vice versa. + ** 5: A function with the exact number of arguments requested that + ** prefers UTF-16LE when UTF-16BE is requested, or vice versa. + ** 6: An exact match. + ** + ** A larger value of 'matchqual' indicates a more desirable match. + */ + if( p->nArg==-1 || p->nArg==nArg || nArg==-1 ){ + int match = 1; /* Quality of this match */ + if( p->nArg==nArg || nArg==-1 ){ + match = 4; + } + if( enc==p->iPrefEnc ){ + match += 2; + } + else if( (enc==SQLITE_UTF16LE && p->iPrefEnc==SQLITE_UTF16BE) || + (enc==SQLITE_UTF16BE && p->iPrefEnc==SQLITE_UTF16LE) ){ + match += 1; + } + + if( match>bestmatch ){ + pBest = p; + bestmatch = match; + } + } + } + + /* If the createFlag parameter is true, and the seach did not reveal an + ** exact match for the name, number of arguments and encoding, then add a + ** new entry to the hash table and return it. + */ + if( createFlag && bestmatch<6 && + (pBest = sqliteMalloc(sizeof(*pBest)+nName))!=0 ){ + pBest->nArg = nArg; + pBest->pNext = pFirst; + pBest->iPrefEnc = enc; + memcpy(pBest->zName, zName, nName); + pBest->zName[nName] = 0; + if( pBest==sqlite3HashInsert(&db->aFunc,pBest->zName,nName,(void*)pBest) ){ + sqliteFree(pBest); + return 0; + } + } + + if( pBest && (pBest->xStep || pBest->xFunc || createFlag) ){ + return pBest; + } + return 0; +} Added: external/sqlite-source-3.3.4/complete.c ============================================================================== --- (empty file) +++ external/sqlite-source-3.3.4/complete.c Mon Apr 3 07:54:59 2006 @@ -0,0 +1,263 @@ +/* +** 2001 September 15 +** +** The author disclaims copyright to this source code. In place of +** a legal notice, here is a blessing: +** +** May you do good and not evil. +** May you find forgiveness for yourself and forgive others. +** May you share freely, never taking more than you give. +** +************************************************************************* +** An tokenizer for SQL +** +** This file contains C code that implements the sqlite3_complete() API. +** This code used to be part of the tokenizer.c source file. But by +** separating it out, the code will be automatically omitted from +** static links that do not use it. +** +** $Id: complete.c,v 1.3 2006/01/18 15:25:17 danielk1977 Exp $ +*/ +#include "sqliteInt.h" +#ifndef SQLITE_OMIT_COMPLETE + +/* +** This is defined in tokenize.c. We just have to import the definition. +*/ +extern const char sqlite3IsIdChar[]; +#define IdChar(C) (((c=C)&0x80)!=0 || (c>0x1f && sqlite3IsIdChar[c-0x20])) + + +/* +** Token types used by the sqlite3_complete() routine. See the header +** comments on that procedure for additional information. +*/ +#define tkSEMI 0 +#define tkWS 1 +#define tkOTHER 2 +#define tkEXPLAIN 3 +#define tkCREATE 4 +#define tkTEMP 5 +#define tkTRIGGER 6 +#define tkEND 7 + +/* +** Return TRUE if the given SQL string ends in a semicolon. +** +** Special handling is require for CREATE TRIGGER statements. +** Whenever the CREATE TRIGGER keywords are seen, the statement +** must end with ";END;". +** +** This implementation uses a state machine with 7 states: +** +** (0) START At the beginning or end of an SQL statement. This routine +** returns 1 if it ends in the START state and 0 if it ends +** in any other state. +** +** (1) NORMAL We are in the middle of statement which ends with a single +** semicolon. +** +** (2) EXPLAIN The keyword EXPLAIN has been seen at the beginning of +** a statement. +** +** (3) CREATE The keyword CREATE has been seen at the beginning of a +** statement, possibly preceeded by EXPLAIN and/or followed by +** TEMP or TEMPORARY +** +** (4) TRIGGER We are in the middle of a trigger definition that must be +** ended by a semicolon, the keyword END, and another semicolon. +** +** (5) SEMI We've seen the first semicolon in the ";END;" that occurs at +** the end of a trigger definition. +** +** (6) END We've seen the ";END" of the ";END;" that occurs at the end +** of a trigger difinition. +** +** Transitions between states above are determined by tokens extracted +** from the input. The following tokens are significant: +** +** (0) tkSEMI A semicolon. +** (1) tkWS Whitespace +** (2) tkOTHER Any other SQL token. +** (3) tkEXPLAIN The "explain" keyword. +** (4) tkCREATE The "create" keyword. +** (5) tkTEMP The "temp" or "temporary" keyword. +** (6) tkTRIGGER The "trigger" keyword. +** (7) tkEND The "end" keyword. +** +** Whitespace never causes a state transition and is always ignored. +** +** If we compile with SQLITE_OMIT_TRIGGER, all of the computation needed +** to recognize the end of a trigger can be omitted. All we have to do +** is look for a semicolon that is not part of an string or comment. +*/ +int sqlite3_complete(const char *zSql){ + u8 state = 0; /* Current state, using numbers defined in header comment */ + u8 token; /* Value of the next token */ + +#ifndef SQLITE_OMIT_TRIGGER + /* A complex statement machine used to detect the end of a CREATE TRIGGER + ** statement. This is the normal case. + */ + static const u8 trans[7][8] = { + /* Token: */ + /* State: ** SEMI WS OTHER EXPLAIN CREATE TEMP TRIGGER END */ + /* 0 START: */ { 0, 0, 1, 2, 3, 1, 1, 1, }, + /* 1 NORMAL: */ { 0, 1, 1, 1, 1, 1, 1, 1, }, + /* 2 EXPLAIN: */ { 0, 2, 1, 1, 3, 1, 1, 1, }, + /* 3 CREATE: */ { 0, 3, 1, 1, 1, 3, 4, 1, }, + /* 4 TRIGGER: */ { 5, 4, 4, 4, 4, 4, 4, 4, }, + /* 5 SEMI: */ { 5, 5, 4, 4, 4, 4, 4, 6, }, + /* 6 END: */ { 0, 6, 4, 4, 4, 4, 4, 4, }, + }; +#else + /* If triggers are not suppored by this compile then the statement machine + ** used to detect the end of a statement is much simplier + */ + static const u8 trans[2][3] = { + /* Token: */ + /* State: ** SEMI WS OTHER */ + /* 0 START: */ { 0, 0, 1, }, + /* 1 NORMAL: */ { 0, 1, 1, }, + }; +#endif /* SQLITE_OMIT_TRIGGER */ + + while( *zSql ){ + switch( *zSql ){ + case ';': { /* A semicolon */ + token = tkSEMI; + break; + } + case ' ': + case '\r': + case '\t': + case '\n': + case '\f': { /* White space is ignored */ + token = tkWS; + break; + } + case '/': { /* C-style comments */ + if( zSql[1]!='*' ){ + token = tkOTHER; + break; + } + zSql += 2; + while( zSql[0] && (zSql[0]!='*' || zSql[1]!='/') ){ zSql++; } + if( zSql[0]==0 ) return 0; + zSql++; + token = tkWS; + break; + } + case '-': { /* SQL-style comments from "--" to end of line */ + if( zSql[1]!='-' ){ + token = tkOTHER; + break; + } + while( *zSql && *zSql!='\n' ){ zSql++; } + if( *zSql==0 ) return state==0; + token = tkWS; + break; + } + case '[': { /* Microsoft-style identifiers in [...] */ + zSql++; + while( *zSql && *zSql!=']' ){ zSql++; } + if( *zSql==0 ) return 0; + token = tkOTHER; + break; + } + case '`': /* Grave-accent quoted symbols used by MySQL */ + case '"': /* single- and double-quoted strings */ + case '\'': { + int c = *zSql; + zSql++; + while( *zSql && *zSql!=c ){ zSql++; } + if( *zSql==0 ) return 0; + token = tkOTHER; + break; + } + default: { + int c; + if( IdChar((u8)*zSql) ){ + /* Keywords and unquoted identifiers */ + int nId; + for(nId=1; IdChar(zSql[nId]); nId++){} +#ifdef SQLITE_OMIT_TRIGGER + token = tkOTHER; +#else + switch( *zSql ){ + case 'c': case 'C': { + if( nId==6 && sqlite3StrNICmp(zSql, "create", 6)==0 ){ + token = tkCREATE; + }else{ + token = tkOTHER; + } + break; + } + case 't': case 'T': { + if( nId==7 && sqlite3StrNICmp(zSql, "trigger", 7)==0 ){ + token = tkTRIGGER; + }else if( nId==4 && sqlite3StrNICmp(zSql, "temp", 4)==0 ){ + token = tkTEMP; + }else if( nId==9 && sqlite3StrNICmp(zSql, "temporary", 9)==0 ){ + token = tkTEMP; + }else{ + token = tkOTHER; + } + break; + } + case 'e': case 'E': { + if( nId==3 && sqlite3StrNICmp(zSql, "end", 3)==0 ){ + token = tkEND; + }else +#ifndef SQLITE_OMIT_EXPLAIN + if( nId==7 && sqlite3StrNICmp(zSql, "explain", 7)==0 ){ + token = tkEXPLAIN; + }else +#endif + { + token = tkOTHER; + } + break; + } + default: { + token = tkOTHER; + break; + } + } +#endif /* SQLITE_OMIT_TRIGGER */ + zSql += nId-1; + }else{ + /* Operators and special symbols */ + token = tkOTHER; + } + break; + } + } + state = trans[state][token]; + zSql++; + } + return state==0; +} + +#ifndef SQLITE_OMIT_UTF16 +/* +** This routine is the same as the sqlite3_complete() routine described +** above, except that the parameter is required to be UTF-16 encoded, not +** UTF-8. +*/ +int sqlite3_complete16(const void *zSql){ + sqlite3_value *pVal; + char const *zSql8; + int rc = 0; + + pVal = sqlite3ValueNew(); + sqlite3ValueSetStr(pVal, -1, zSql, SQLITE_UTF16NATIVE, SQLITE_STATIC); + zSql8 = sqlite3ValueText(pVal, SQLITE_UTF8); + if( zSql8 ){ + rc = sqlite3_complete(zSql8); + } + sqlite3ValueFree(pVal); + return sqlite3ApiExit(0, rc); +} +#endif /* SQLITE_OMIT_UTF16 */ +#endif /* SQLITE_OMIT_COMPLETE */ Added: external/sqlite-source-3.3.4/date.c ============================================================================== --- (empty file) +++ external/sqlite-source-3.3.4/date.c Mon Apr 3 07:54:59 2006 @@ -0,0 +1,998 @@ +/* +** 2003 October 31 +** +** The author disclaims copyright to this source code. In place of +** a legal notice, here is a blessing: +** +** May you do good and not evil. +** May you find forgiveness for yourself and forgive others. +** May you share freely, never taking more than you give. +** +************************************************************************* +** This file contains the C functions that implement date and time +** functions for SQLite. +** +** There is only one exported symbol in this file - the function +** sqlite3RegisterDateTimeFunctions() found at the bottom of the file. +** All other code has file scope. +** +** $Id: date.c,v 1.54 2006/01/31 20:49:13 drh Exp $ +** +** NOTES: +** +** SQLite processes all times and dates as Julian Day numbers. The +** dates and times are stored as the number of days since noon +** in Greenwich on November 24, 4714 B.C. according to the Gregorian +** calendar system. +** +** 1970-01-01 00:00:00 is JD 2440587.5 +** 2000-01-01 00:00:00 is JD 2451544.5 +** +** This implemention requires years to be expressed as a 4-digit number +** which means that only dates between 0000-01-01 and 9999-12-31 can +** be represented, even though julian day numbers allow a much wider +** range of dates. +** +** The Gregorian calendar system is used for all dates and times, +** even those that predate the Gregorian calendar. Historians usually +** use the Julian calendar for dates prior to 1582-10-15 and for some +** dates afterwards, depending on locale. Beware of this difference. +** +** The conversion algorithms are implemented based on descriptions +** in the following text: +** +** Jean Meeus +** Astronomical Algorithms, 2nd Edition, 1998 +** ISBM 0-943396-61-1 +** Willmann-Bell, Inc +** Richmond, Virginia (USA) +*/ +#include "sqliteInt.h" +#include "os.h" +#include +#include +#include +#include + +#ifndef SQLITE_OMIT_DATETIME_FUNCS + +/* +** A structure for holding a single date and time. +*/ +typedef struct DateTime DateTime; +struct DateTime { + double rJD; /* The julian day number */ + int Y, M, D; /* Year, month, and day */ + int h, m; /* Hour and minutes */ + int tz; /* Timezone offset in minutes */ + double s; /* Seconds */ + char validYMD; /* True if Y,M,D are valid */ + char validHMS; /* True if h,m,s are valid */ + char validJD; /* True if rJD is valid */ + char validTZ; /* True if tz is valid */ +}; + + +/* +** Convert zDate into one or more integers. Additional arguments +** come in groups of 5 as follows: +** +** N number of digits in the integer +** min minimum allowed value of the integer +** max maximum allowed value of the integer +** nextC first character after the integer +** pVal where to write the integers value. +** +** Conversions continue until one with nextC==0 is encountered. +** The function returns the number of successful conversions. +*/ +static int getDigits(const char *zDate, ...){ + va_list ap; + int val; + int N; + int min; + int max; + int nextC; + int *pVal; + int cnt = 0; + va_start(ap, zDate); + do{ + N = va_arg(ap, int); + min = va_arg(ap, int); + max = va_arg(ap, int); + nextC = va_arg(ap, int); + pVal = va_arg(ap, int*); + val = 0; + while( N-- ){ + if( !isdigit(*(u8*)zDate) ){ + goto end_getDigits; + } + val = val*10 + *zDate - '0'; + zDate++; + } + if( valmax || (nextC!=0 && nextC!=*zDate) ){ + goto end_getDigits; + } + *pVal = val; + zDate++; + cnt++; + }while( nextC ); +end_getDigits: + va_end(ap); + return cnt; +} + +/* +** Read text from z[] and convert into a floating point number. Return +** the number of digits converted. +*/ +#define getValue sqlite3AtoF + +/* +** Parse a timezone extension on the end of a date-time. +** The extension is of the form: +** +** (+/-)HH:MM +** +** If the parse is successful, write the number of minutes +** of change in *pnMin and return 0. If a parser error occurs, +** return 0. +** +** A missing specifier is not considered an error. +*/ +static int parseTimezone(const char *zDate, DateTime *p){ + int sgn = 0; + int nHr, nMn; + while( isspace(*(u8*)zDate) ){ zDate++; } + p->tz = 0; + if( *zDate=='-' ){ + sgn = -1; + }else if( *zDate=='+' ){ + sgn = +1; + }else{ + return *zDate!=0; + } + zDate++; + if( getDigits(zDate, 2, 0, 14, ':', &nHr, 2, 0, 59, 0, &nMn)!=2 ){ + return 1; + } + zDate += 5; + p->tz = sgn*(nMn + nHr*60); + while( isspace(*(u8*)zDate) ){ zDate++; } + return *zDate!=0; +} + +/* +** Parse times of the form HH:MM or HH:MM:SS or HH:MM:SS.FFFF. +** The HH, MM, and SS must each be exactly 2 digits. The +** fractional seconds FFFF can be one or more digits. +** +** Return 1 if there is a parsing error and 0 on success. +*/ +static int parseHhMmSs(const char *zDate, DateTime *p){ + int h, m, s; + double ms = 0.0; + if( getDigits(zDate, 2, 0, 24, ':', &h, 2, 0, 59, 0, &m)!=2 ){ + return 1; + } + zDate += 5; + if( *zDate==':' ){ + zDate++; + if( getDigits(zDate, 2, 0, 59, 0, &s)!=1 ){ + return 1; + } + zDate += 2; + if( *zDate=='.' && isdigit((u8)zDate[1]) ){ + double rScale = 1.0; + zDate++; + while( isdigit(*(u8*)zDate) ){ + ms = ms*10.0 + *zDate - '0'; + rScale *= 10.0; + zDate++; + } + ms /= rScale; + } + }else{ + s = 0; + } + p->validJD = 0; + p->validHMS = 1; + p->h = h; + p->m = m; + p->s = s + ms; + if( parseTimezone(zDate, p) ) return 1; + p->validTZ = p->tz!=0; + return 0; +} + +/* +** Convert from YYYY-MM-DD HH:MM:SS to julian day. We always assume +** that the YYYY-MM-DD is according to the Gregorian calendar. +** +** Reference: Meeus page 61 +*/ +static void computeJD(DateTime *p){ + int Y, M, D, A, B, X1, X2; + + if( p->validJD ) return; + if( p->validYMD ){ + Y = p->Y; + M = p->M; + D = p->D; + }else{ + Y = 2000; /* If no YMD specified, assume 2000-Jan-01 */ + M = 1; + D = 1; + } + if( M<=2 ){ + Y--; + M += 12; + } + A = Y/100; + B = 2 - A + (A/4); + X1 = 365.25*(Y+4716); + X2 = 30.6001*(M+1); + p->rJD = X1 + X2 + D + B - 1524.5; + p->validJD = 1; + p->validYMD = 0; + if( p->validHMS ){ + p->rJD += (p->h*3600.0 + p->m*60.0 + p->s)/86400.0; + if( p->validTZ ){ + p->rJD -= p->tz*60/86400.0; + p->validHMS = 0; + p->validTZ = 0; + } + } +} + +/* +** Parse dates of the form +** +** YYYY-MM-DD HH:MM:SS.FFF +** YYYY-MM-DD HH:MM:SS +** YYYY-MM-DD HH:MM +** YYYY-MM-DD +** +** Write the result into the DateTime structure and return 0 +** on success and 1 if the input string is not a well-formed +** date. +*/ +static int parseYyyyMmDd(const char *zDate, DateTime *p){ + int Y, M, D, neg; + + if( zDate[0]=='-' ){ + zDate++; + neg = 1; + }else{ + neg = 0; + } + if( getDigits(zDate,4,0,9999,'-',&Y,2,1,12,'-',&M,2,1,31,0,&D)!=3 ){ + return 1; + } + zDate += 10; + while( isspace(*(u8*)zDate) || 'T'==*(u8*)zDate ){ zDate++; } + if( parseHhMmSs(zDate, p)==0 ){ + /* We got the time */ + }else if( *zDate==0 ){ + p->validHMS = 0; + }else{ + return 1; + } + p->validJD = 0; + p->validYMD = 1; + p->Y = neg ? -Y : Y; + p->M = M; + p->D = D; + if( p->validTZ ){ + computeJD(p); + } + return 0; +} + +/* +** Attempt to parse the given string into a Julian Day Number. Return +** the number of errors. +** +** The following are acceptable forms for the input string: +** +** YYYY-MM-DD HH:MM:SS.FFF +/-HH:MM +** DDDD.DD +** now +** +** In the first form, the +/-HH:MM is always optional. The fractional +** seconds extension (the ".FFF") is optional. The seconds portion +** (":SS.FFF") is option. The year and date can be omitted as long +** as there is a time string. The time string can be omitted as long +** as there is a year and date. +*/ +static int parseDateOrTime(const char *zDate, DateTime *p){ + memset(p, 0, sizeof(*p)); + if( parseYyyyMmDd(zDate,p)==0 ){ + return 0; + }else if( parseHhMmSs(zDate, p)==0 ){ + return 0; + }else if( sqlite3StrICmp(zDate,"now")==0){ + double r; + sqlite3OsCurrentTime(&r); + p->rJD = r; + p->validJD = 1; + return 0; + }else if( sqlite3IsNumber(zDate, 0, SQLITE_UTF8) ){ + getValue(zDate, &p->rJD); + p->validJD = 1; + return 0; + } + return 1; +} + +/* +** Compute the Year, Month, and Day from the julian day number. +*/ +static void computeYMD(DateTime *p){ + int Z, A, B, C, D, E, X1; + if( p->validYMD ) return; + if( !p->validJD ){ + p->Y = 2000; + p->M = 1; + p->D = 1; + }else{ + Z = p->rJD + 0.5; + A = (Z - 1867216.25)/36524.25; + A = Z + 1 + A - (A/4); + B = A + 1524; + C = (B - 122.1)/365.25; + D = 365.25*C; + E = (B-D)/30.6001; + X1 = 30.6001*E; + p->D = B - D - X1; + p->M = E<14 ? E-1 : E-13; + p->Y = p->M>2 ? C - 4716 : C - 4715; + } + p->validYMD = 1; +} + +/* +** Compute the Hour, Minute, and Seconds from the julian day number. +*/ +static void computeHMS(DateTime *p){ + int Z, s; + if( p->validHMS ) return; + Z = p->rJD + 0.5; + s = (p->rJD + 0.5 - Z)*86400000.0 + 0.5; + p->s = 0.001*s; + s = p->s; + p->s -= s; + p->h = s/3600; + s -= p->h*3600; + p->m = s/60; + p->s += s - p->m*60; + p->validHMS = 1; +} + +/* +** Compute both YMD and HMS +*/ +static void computeYMD_HMS(DateTime *p){ + computeYMD(p); + computeHMS(p); +} + +/* +** Clear the YMD and HMS and the TZ +*/ +static void clearYMD_HMS_TZ(DateTime *p){ + p->validYMD = 0; + p->validHMS = 0; + p->validTZ = 0; +} + +/* +** Compute the difference (in days) between localtime and UTC (a.k.a. GMT) +** for the time value p where p is in UTC. +*/ +static double localtimeOffset(DateTime *p){ + DateTime x, y; + time_t t; + struct tm *pTm; + x = *p; + computeYMD_HMS(&x); + if( x.Y<1971 || x.Y>=2038 ){ + x.Y = 2000; + x.M = 1; + x.D = 1; + x.h = 0; + x.m = 0; + x.s = 0.0; + } else { + int s = x.s + 0.5; + x.s = s; + } + x.tz = 0; + x.validJD = 0; + computeJD(&x); + t = (x.rJD-2440587.5)*86400.0 + 0.5; + sqlite3OsEnterMutex(); + pTm = localtime(&t); + y.Y = pTm->tm_year + 1900; + y.M = pTm->tm_mon + 1; + y.D = pTm->tm_mday; + y.h = pTm->tm_hour; + y.m = pTm->tm_min; + y.s = pTm->tm_sec; + sqlite3OsLeaveMutex(); + y.validYMD = 1; + y.validHMS = 1; + y.validJD = 0; + y.validTZ = 0; + computeJD(&y); + return y.rJD - x.rJD; +} + +/* +** Process a modifier to a date-time stamp. The modifiers are +** as follows: +** +** NNN days +** NNN hours +** NNN minutes +** NNN.NNNN seconds +** NNN months +** NNN years +** start of month +** start of year +** start of week +** start of day +** weekday N +** unixepoch +** localtime +** utc +** +** Return 0 on success and 1 if there is any kind of error. +*/ +static int parseModifier(const char *zMod, DateTime *p){ + int rc = 1; + int n; + double r; + char *z, zBuf[30]; + z = zBuf; + for(n=0; nrJD += localtimeOffset(p); + clearYMD_HMS_TZ(p); + rc = 0; + } + break; + } + case 'u': { + /* + ** unixepoch + ** + ** Treat the current value of p->rJD as the number of + ** seconds since 1970. Convert to a real julian day number. + */ + if( strcmp(z, "unixepoch")==0 && p->validJD ){ + p->rJD = p->rJD/86400.0 + 2440587.5; + clearYMD_HMS_TZ(p); + rc = 0; + }else if( strcmp(z, "utc")==0 ){ + double c1; + computeJD(p); + c1 = localtimeOffset(p); + p->rJD -= c1; + clearYMD_HMS_TZ(p); + p->rJD += c1 - localtimeOffset(p); + rc = 0; + } + break; + } + case 'w': { + /* + ** weekday N + ** + ** Move the date to the same time on the next occurrence of + ** weekday N where 0==Sunday, 1==Monday, and so forth. If the + ** date is already on the appropriate weekday, this is a no-op. + */ + if( strncmp(z, "weekday ", 8)==0 && getValue(&z[8],&r)>0 + && (n=r)==r && n>=0 && r<7 ){ + int Z; + computeYMD_HMS(p); + p->validTZ = 0; + p->validJD = 0; + computeJD(p); + Z = p->rJD + 1.5; + Z %= 7; + if( Z>n ) Z -= 7; + p->rJD += n - Z; + clearYMD_HMS_TZ(p); + rc = 0; + } + break; + } + case 's': { + /* + ** start of TTTTT + ** + ** Move the date backwards to the beginning of the current day, + ** or month or year. + */ + if( strncmp(z, "start of ", 9)!=0 ) break; + z += 9; + computeYMD(p); + p->validHMS = 1; + p->h = p->m = 0; + p->s = 0.0; + p->validTZ = 0; + p->validJD = 0; + if( strcmp(z,"month")==0 ){ + p->D = 1; + rc = 0; + }else if( strcmp(z,"year")==0 ){ + computeYMD(p); + p->M = 1; + p->D = 1; + rc = 0; + }else if( strcmp(z,"day")==0 ){ + rc = 0; + } + break; + } + case '+': + case '-': + case '0': + case '1': + case '2': + case '3': + case '4': + case '5': + case '6': + case '7': + case '8': + case '9': { + n = getValue(z, &r); + if( n<=0 ) break; + if( z[n]==':' ){ + /* A modifier of the form (+|-)HH:MM:SS.FFF adds (or subtracts) the + ** specified number of hours, minutes, seconds, and fractional seconds + ** to the time. The ".FFF" may be omitted. The ":SS.FFF" may be + ** omitted. + */ + const char *z2 = z; + DateTime tx; + int day; + if( !isdigit(*(u8*)z2) ) z2++; + memset(&tx, 0, sizeof(tx)); + if( parseHhMmSs(z2, &tx) ) break; + computeJD(&tx); + tx.rJD -= 0.5; + day = (int)tx.rJD; + tx.rJD -= day; + if( z[0]=='-' ) tx.rJD = -tx.rJD; + computeJD(p); + clearYMD_HMS_TZ(p); + p->rJD += tx.rJD; + rc = 0; + break; + } + z += n; + while( isspace(*(u8*)z) ) z++; + n = strlen(z); + if( n>10 || n<3 ) break; + if( z[n-1]=='s' ){ z[n-1] = 0; n--; } + computeJD(p); + rc = 0; + if( n==3 && strcmp(z,"day")==0 ){ + p->rJD += r; + }else if( n==4 && strcmp(z,"hour")==0 ){ + p->rJD += r/24.0; + }else if( n==6 && strcmp(z,"minute")==0 ){ + p->rJD += r/(24.0*60.0); + }else if( n==6 && strcmp(z,"second")==0 ){ + p->rJD += r/(24.0*60.0*60.0); + }else if( n==5 && strcmp(z,"month")==0 ){ + int x, y; + computeYMD_HMS(p); + p->M += r; + x = p->M>0 ? (p->M-1)/12 : (p->M-12)/12; + p->Y += x; + p->M -= x*12; + p->validJD = 0; + computeJD(p); + y = r; + if( y!=r ){ + p->rJD += (r - y)*30.0; + } + }else if( n==4 && strcmp(z,"year")==0 ){ + computeYMD_HMS(p); + p->Y += r; + p->validJD = 0; + computeJD(p); + }else{ + rc = 1; + } + clearYMD_HMS_TZ(p); + break; + } + default: { + break; + } + } + return rc; +} + +/* +** Process time function arguments. argv[0] is a date-time stamp. +** argv[1] and following are modifiers. Parse them all and write +** the resulting time into the DateTime structure p. Return 0 +** on success and 1 if there are any errors. +*/ +static int isDate(int argc, sqlite3_value **argv, DateTime *p){ + int i; + if( argc==0 ) return 1; + if( SQLITE_NULL==sqlite3_value_type(argv[0]) || + parseDateOrTime((char*)sqlite3_value_text(argv[0]), p) ) return 1; + for(i=1; izErrMsg and return NULL. If all tables +** are found, return a pointer to the last table. +*/ +Table *sqlite3SrcListLookup(Parse *pParse, SrcList *pSrc){ + Table *pTab = 0; + int i; + struct SrcList_item *pItem; + for(i=0, pItem=pSrc->a; inSrc; i++, pItem++){ + pTab = sqlite3LocateTable(pParse, pItem->zName, pItem->zDatabase); + sqlite3DeleteTable(pParse->db, pItem->pTab); + pItem->pTab = pTab; + if( pTab ){ + pTab->nRef++; + } + } + return pTab; +} + +/* +** Check to make sure the given table is writable. If it is not +** writable, generate an error message and return 1. If it is +** writable return 0; +*/ +int sqlite3IsReadOnly(Parse *pParse, Table *pTab, int viewOk){ + if( pTab->readOnly && (pParse->db->flags & SQLITE_WriteSchema)==0 + && pParse->nested==0 ){ + sqlite3ErrorMsg(pParse, "table %s may not be modified", pTab->zName); + return 1; + } +#ifndef SQLITE_OMIT_VIEW + if( !viewOk && pTab->pSelect ){ + sqlite3ErrorMsg(pParse,"cannot modify %s because it is a view",pTab->zName); + return 1; + } +#endif + return 0; +} + +/* +** Generate code that will open a table for reading. +*/ +void sqlite3OpenTable( + Parse *p, /* Generate code into this VDBE */ + int iCur, /* The cursor number of the table */ + int iDb, /* The database index in sqlite3.aDb[] */ + Table *pTab, /* The table to be opened */ + int opcode /* OP_OpenRead or OP_OpenWrite */ +){ + Vdbe *v = sqlite3GetVdbe(p); + assert( opcode==OP_OpenWrite || opcode==OP_OpenRead ); + sqlite3TableLock(p, iDb, pTab->tnum, (opcode==OP_OpenWrite), pTab->zName); + sqlite3VdbeAddOp(v, OP_Integer, iDb, 0); + VdbeComment((v, "# %s", pTab->zName)); + sqlite3VdbeAddOp(v, opcode, iCur, pTab->tnum); + sqlite3VdbeAddOp(v, OP_SetNumColumns, iCur, pTab->nCol); +} + + +/* +** Generate code for a DELETE FROM statement. +** +** DELETE FROM table_wxyz WHERE a<5 AND b NOT NULL; +** \________/ \________________/ +** pTabList pWhere +*/ +void sqlite3DeleteFrom( + Parse *pParse, /* The parser context */ + SrcList *pTabList, /* The table from which we should delete things */ + Expr *pWhere /* The WHERE clause. May be null */ +){ + Vdbe *v; /* The virtual database engine */ + Table *pTab; /* The table from which records will be deleted */ + const char *zDb; /* Name of database holding pTab */ + int end, addr = 0; /* A couple addresses of generated code */ + int i; /* Loop counter */ + WhereInfo *pWInfo; /* Information about the WHERE clause */ + Index *pIdx; /* For looping over indices of the table */ + int iCur; /* VDBE Cursor number for pTab */ + sqlite3 *db; /* Main database structure */ + AuthContext sContext; /* Authorization context */ + int oldIdx = -1; /* Cursor for the OLD table of AFTER triggers */ + NameContext sNC; /* Name context to resolve expressions in */ + int iDb; + +#ifndef SQLITE_OMIT_TRIGGER + int isView; /* True if attempting to delete from a view */ + int triggers_exist = 0; /* True if any triggers exist */ +#endif + + sContext.pParse = 0; + if( pParse->nErr || sqlite3MallocFailed() ){ + goto delete_from_cleanup; + } + db = pParse->db; + assert( pTabList->nSrc==1 ); + + /* Locate the table which we want to delete. This table has to be + ** put in an SrcList structure because some of the subroutines we + ** will be calling are designed to work with multiple tables and expect + ** an SrcList* parameter instead of just a Table* parameter. + */ + pTab = sqlite3SrcListLookup(pParse, pTabList); + if( pTab==0 ) goto delete_from_cleanup; + + /* Figure out if we have any triggers and if the table being + ** deleted from is a view + */ +#ifndef SQLITE_OMIT_TRIGGER + triggers_exist = sqlite3TriggersExist(pParse, pTab, TK_DELETE, 0); + isView = pTab->pSelect!=0; +#else +# define triggers_exist 0 +# define isView 0 +#endif +#ifdef SQLITE_OMIT_VIEW +# undef isView +# define isView 0 +#endif + + if( sqlite3IsReadOnly(pParse, pTab, triggers_exist) ){ + goto delete_from_cleanup; + } + iDb = sqlite3SchemaToIndex(db, pTab->pSchema); + assert( iDbnDb ); + zDb = db->aDb[iDb].zName; + if( sqlite3AuthCheck(pParse, SQLITE_DELETE, pTab->zName, 0, zDb) ){ + goto delete_from_cleanup; + } + + /* If pTab is really a view, make sure it has been initialized. + */ + if( isView && sqlite3ViewGetColumnNames(pParse, pTab) ){ + goto delete_from_cleanup; + } + + /* Allocate a cursor used to store the old.* data for a trigger. + */ + if( triggers_exist ){ + oldIdx = pParse->nTab++; + } + + /* Resolve the column names in the WHERE clause. + */ + assert( pTabList->nSrc==1 ); + iCur = pTabList->a[0].iCursor = pParse->nTab++; + memset(&sNC, 0, sizeof(sNC)); + sNC.pParse = pParse; + sNC.pSrcList = pTabList; + if( sqlite3ExprResolveNames(&sNC, pWhere) ){ + goto delete_from_cleanup; + } + + /* Start the view context + */ + if( isView ){ + sqlite3AuthContextPush(pParse, &sContext, pTab->zName); + } + + /* Begin generating code. + */ + v = sqlite3GetVdbe(pParse); + if( v==0 ){ + goto delete_from_cleanup; + } + if( pParse->nested==0 ) sqlite3VdbeCountChanges(v); + sqlite3BeginWriteOperation(pParse, triggers_exist, iDb); + + /* If we are trying to delete from a view, realize that view into + ** a ephemeral table. + */ + if( isView ){ + Select *pView = sqlite3SelectDup(pTab->pSelect); + sqlite3Select(pParse, pView, SRT_VirtualTab, iCur, 0, 0, 0, 0); + sqlite3SelectDelete(pView); + } + + /* Initialize the counter of the number of rows deleted, if + ** we are counting rows. + */ + if( db->flags & SQLITE_CountRows ){ + sqlite3VdbeAddOp(v, OP_Integer, 0, 0); + } + + /* Special case: A DELETE without a WHERE clause deletes everything. + ** It is easier just to erase the whole table. Note, however, that + ** this means that the row change count will be incorrect. + */ + if( pWhere==0 && !triggers_exist ){ + if( db->flags & SQLITE_CountRows ){ + /* If counting rows deleted, just count the total number of + ** entries in the table. */ + int endOfLoop = sqlite3VdbeMakeLabel(v); + int addr2; + if( !isView ){ + sqlite3OpenTable(pParse, iCur, iDb, pTab, OP_OpenRead); + } + sqlite3VdbeAddOp(v, OP_Rewind, iCur, sqlite3VdbeCurrentAddr(v)+2); + addr2 = sqlite3VdbeAddOp(v, OP_AddImm, 1, 0); + sqlite3VdbeAddOp(v, OP_Next, iCur, addr2); + sqlite3VdbeResolveLabel(v, endOfLoop); + sqlite3VdbeAddOp(v, OP_Close, iCur, 0); + } + if( !isView ){ + sqlite3VdbeAddOp(v, OP_Clear, pTab->tnum, iDb); + if( !pParse->nested ){ + sqlite3VdbeChangeP3(v, -1, pTab->zName, P3_STATIC); + } + for(pIdx=pTab->pIndex; pIdx; pIdx=pIdx->pNext){ + assert( pIdx->pSchema==pTab->pSchema ); + sqlite3VdbeAddOp(v, OP_Clear, pIdx->tnum, iDb); + } + } + } + + /* The usual case: There is a WHERE clause so we have to scan through + ** the table and pick which records to delete. + */ + else{ + /* Begin the database scan + */ + pWInfo = sqlite3WhereBegin(pParse, pTabList, pWhere, 0); + if( pWInfo==0 ) goto delete_from_cleanup; + + /* Remember the rowid of every item to be deleted. + */ + sqlite3VdbeAddOp(v, OP_Rowid, iCur, 0); + sqlite3VdbeAddOp(v, OP_FifoWrite, 0, 0); + if( db->flags & SQLITE_CountRows ){ + sqlite3VdbeAddOp(v, OP_AddImm, 1, 0); + } + + /* End the database scan loop. + */ + sqlite3WhereEnd(pWInfo); + + /* Open the pseudo-table used to store OLD if there are triggers. + */ + if( triggers_exist ){ + sqlite3VdbeAddOp(v, OP_OpenPseudo, oldIdx, 0); + sqlite3VdbeAddOp(v, OP_SetNumColumns, oldIdx, pTab->nCol); + } + + /* Delete every item whose key was written to the list during the + ** database scan. We have to delete items after the scan is complete + ** because deleting an item can change the scan order. + */ + end = sqlite3VdbeMakeLabel(v); + + /* This is the beginning of the delete loop when there are + ** row triggers. + */ + if( triggers_exist ){ + addr = sqlite3VdbeAddOp(v, OP_FifoRead, 0, end); + if( !isView ){ + sqlite3VdbeAddOp(v, OP_Dup, 0, 0); + sqlite3OpenTable(pParse, iCur, iDb, pTab, OP_OpenRead); + } + sqlite3VdbeAddOp(v, OP_MoveGe, iCur, 0); + sqlite3VdbeAddOp(v, OP_Rowid, iCur, 0); + sqlite3VdbeAddOp(v, OP_RowData, iCur, 0); + sqlite3VdbeAddOp(v, OP_Insert, oldIdx, 0); + if( !isView ){ + sqlite3VdbeAddOp(v, OP_Close, iCur, 0); + } + + (void)sqlite3CodeRowTrigger(pParse, TK_DELETE, 0, TRIGGER_BEFORE, pTab, + -1, oldIdx, (pParse->trigStack)?pParse->trigStack->orconf:OE_Default, + addr); + } + + if( !isView ){ + /* Open cursors for the table we are deleting from and all its + ** indices. If there are row triggers, this happens inside the + ** OP_FifoRead loop because the cursor have to all be closed + ** before the trigger fires. If there are no row triggers, the + ** cursors are opened only once on the outside the loop. + */ + sqlite3OpenTableAndIndices(pParse, pTab, iCur, OP_OpenWrite); + + /* This is the beginning of the delete loop when there are no + ** row triggers */ + if( !triggers_exist ){ + addr = sqlite3VdbeAddOp(v, OP_FifoRead, 0, end); + } + + /* Delete the row */ + sqlite3GenerateRowDelete(db, v, pTab, iCur, pParse->nested==0); + } + + /* If there are row triggers, close all cursors then invoke + ** the AFTER triggers + */ + if( triggers_exist ){ + if( !isView ){ + for(i=1, pIdx=pTab->pIndex; pIdx; i++, pIdx=pIdx->pNext){ + sqlite3VdbeAddOp(v, OP_Close, iCur + i, pIdx->tnum); + } + sqlite3VdbeAddOp(v, OP_Close, iCur, 0); + } + (void)sqlite3CodeRowTrigger(pParse, TK_DELETE, 0, TRIGGER_AFTER, pTab, -1, + oldIdx, (pParse->trigStack)?pParse->trigStack->orconf:OE_Default, + addr); + } + + /* End of the delete loop */ + sqlite3VdbeAddOp(v, OP_Goto, 0, addr); + sqlite3VdbeResolveLabel(v, end); + + /* Close the cursors after the loop if there are no row triggers */ + if( !triggers_exist ){ + for(i=1, pIdx=pTab->pIndex; pIdx; i++, pIdx=pIdx->pNext){ + sqlite3VdbeAddOp(v, OP_Close, iCur + i, pIdx->tnum); + } + sqlite3VdbeAddOp(v, OP_Close, iCur, 0); + } + } + + /* + ** Return the number of rows that were deleted. If this routine is + ** generating code because of a call to sqlite3NestedParse(), do not + ** invoke the callback function. + */ + if( db->flags & SQLITE_CountRows && pParse->nested==0 && !pParse->trigStack ){ + sqlite3VdbeAddOp(v, OP_Callback, 1, 0); + sqlite3VdbeSetNumCols(v, 1); + sqlite3VdbeSetColName(v, 0, COLNAME_NAME, "rows deleted", P3_STATIC); + } + +delete_from_cleanup: + sqlite3AuthContextPop(&sContext); + sqlite3SrcListDelete(pTabList); + sqlite3ExprDelete(pWhere); + return; +} + +/* +** This routine generates VDBE code that causes a single row of a +** single table to be deleted. +** +** The VDBE must be in a particular state when this routine is called. +** These are the requirements: +** +** 1. A read/write cursor pointing to pTab, the table containing the row +** to be deleted, must be opened as cursor number "base". +** +** 2. Read/write cursors for all indices of pTab must be open as +** cursor number base+i for the i-th index. +** +** 3. The record number of the row to be deleted must be on the top +** of the stack. +** +** This routine pops the top of the stack to remove the record number +** and then generates code to remove both the table record and all index +** entries that point to that record. +*/ +void sqlite3GenerateRowDelete( + sqlite3 *db, /* The database containing the index */ + Vdbe *v, /* Generate code into this VDBE */ + Table *pTab, /* Table containing the row to be deleted */ + int iCur, /* Cursor number for the table */ + int count /* Increment the row change counter */ +){ + int addr; + addr = sqlite3VdbeAddOp(v, OP_NotExists, iCur, 0); + sqlite3GenerateRowIndexDelete(db, v, pTab, iCur, 0); + sqlite3VdbeAddOp(v, OP_Delete, iCur, (count?OPFLAG_NCHANGE:0)); + if( count ){ + sqlite3VdbeChangeP3(v, -1, pTab->zName, P3_STATIC); + } + sqlite3VdbeJumpHere(v, addr); +} + +/* +** This routine generates VDBE code that causes the deletion of all +** index entries associated with a single row of a single table. +** +** The VDBE must be in a particular state when this routine is called. +** These are the requirements: +** +** 1. A read/write cursor pointing to pTab, the table containing the row +** to be deleted, must be opened as cursor number "iCur". +** +** 2. Read/write cursors for all indices of pTab must be open as +** cursor number iCur+i for the i-th index. +** +** 3. The "iCur" cursor must be pointing to the row that is to be +** deleted. +*/ +void sqlite3GenerateRowIndexDelete( + sqlite3 *db, /* The database containing the index */ + Vdbe *v, /* Generate code into this VDBE */ + Table *pTab, /* Table containing the row to be deleted */ + int iCur, /* Cursor number for the table */ + char *aIdxUsed /* Only delete if aIdxUsed!=0 && aIdxUsed[i]!=0 */ +){ + int i; + Index *pIdx; + + for(i=1, pIdx=pTab->pIndex; pIdx; i++, pIdx=pIdx->pNext){ + if( aIdxUsed!=0 && aIdxUsed[i-1]==0 ) continue; + sqlite3GenerateIndexKey(v, pIdx, iCur); + sqlite3VdbeAddOp(v, OP_IdxDelete, iCur+i, 0); + } +} + +/* +** Generate code that will assemble an index key and put it on the top +** of the tack. The key with be for index pIdx which is an index on pTab. +** iCur is the index of a cursor open on the pTab table and pointing to +** the entry that needs indexing. +*/ +void sqlite3GenerateIndexKey( + Vdbe *v, /* Generate code into this VDBE */ + Index *pIdx, /* The index for which to generate a key */ + int iCur /* Cursor number for the pIdx->pTable table */ +){ + int j; + Table *pTab = pIdx->pTable; + + sqlite3VdbeAddOp(v, OP_Rowid, iCur, 0); + for(j=0; jnColumn; j++){ + int idx = pIdx->aiColumn[j]; + if( idx==pTab->iPKey ){ + sqlite3VdbeAddOp(v, OP_Dup, j, 0); + }else{ + sqlite3VdbeAddOp(v, OP_Column, iCur, idx); + sqlite3ColumnDefault(v, pTab, idx); + } + } + sqlite3VdbeAddOp(v, OP_MakeIdxRec, pIdx->nColumn, 0); + sqlite3IndexAffinityStr(v, pIdx); +} Added: external/sqlite-source-3.3.4/expr.c ============================================================================== --- (empty file) +++ external/sqlite-source-3.3.4/expr.c Mon Apr 3 07:54:59 2006 @@ -0,0 +1,2306 @@ +/* +** 2001 September 15 +** +** The author disclaims copyright to this source code. In place of +** a legal notice, here is a blessing: +** +** May you do good and not evil. +** May you find forgiveness for yourself and forgive others. +** May you share freely, never taking more than you give. +** +************************************************************************* +** This file contains routines used for analyzing expressions and +** for generating VDBE code that evaluates expressions in SQLite. +** +** $Id: expr.c,v 1.254 2006/02/10 07:07:15 danielk1977 Exp $ +*/ +#include "sqliteInt.h" +#include + +/* +** Return the 'affinity' of the expression pExpr if any. +** +** If pExpr is a column, a reference to a column via an 'AS' alias, +** or a sub-select with a column as the return value, then the +** affinity of that column is returned. Otherwise, 0x00 is returned, +** indicating no affinity for the expression. +** +** i.e. the WHERE clause expresssions in the following statements all +** have an affinity: +** +** CREATE TABLE t1(a); +** SELECT * FROM t1 WHERE a; +** SELECT a AS b FROM t1 WHERE b; +** SELECT * FROM t1 WHERE (select a from t1); +*/ +char sqlite3ExprAffinity(Expr *pExpr){ + int op = pExpr->op; + if( op==TK_AS ){ + return sqlite3ExprAffinity(pExpr->pLeft); + } + if( op==TK_SELECT ){ + return sqlite3ExprAffinity(pExpr->pSelect->pEList->a[0].pExpr); + } +#ifndef SQLITE_OMIT_CAST + if( op==TK_CAST ){ + return sqlite3AffinityType(&pExpr->token); + } +#endif + return pExpr->affinity; +} + +/* +** Return the default collation sequence for the expression pExpr. If +** there is no default collation type, return 0. +*/ +CollSeq *sqlite3ExprCollSeq(Parse *pParse, Expr *pExpr){ + CollSeq *pColl = 0; + if( pExpr ){ + pColl = pExpr->pColl; + if( (pExpr->op==TK_AS || pExpr->op==TK_CAST) && !pColl ){ + return sqlite3ExprCollSeq(pParse, pExpr->pLeft); + } + } + if( sqlite3CheckCollSeq(pParse, pColl) ){ + pColl = 0; + } + return pColl; +} + +/* +** pExpr is an operand of a comparison operator. aff2 is the +** type affinity of the other operand. This routine returns the +** type affinity that should be used for the comparison operator. +*/ +char sqlite3CompareAffinity(Expr *pExpr, char aff2){ + char aff1 = sqlite3ExprAffinity(pExpr); + if( aff1 && aff2 ){ + /* Both sides of the comparison are columns. If one has numeric + ** affinity, use that. Otherwise use no affinity. + */ + if( sqlite3IsNumericAffinity(aff1) || sqlite3IsNumericAffinity(aff2) ){ + return SQLITE_AFF_NUMERIC; + }else{ + return SQLITE_AFF_NONE; + } + }else if( !aff1 && !aff2 ){ + /* Neither side of the comparison is a column. Compare the + ** results directly. + */ + return SQLITE_AFF_NONE; + }else{ + /* One side is a column, the other is not. Use the columns affinity. */ + assert( aff1==0 || aff2==0 ); + return (aff1 + aff2); + } +} + +/* +** pExpr is a comparison operator. Return the type affinity that should +** be applied to both operands prior to doing the comparison. +*/ +static char comparisonAffinity(Expr *pExpr){ + char aff; + assert( pExpr->op==TK_EQ || pExpr->op==TK_IN || pExpr->op==TK_LT || + pExpr->op==TK_GT || pExpr->op==TK_GE || pExpr->op==TK_LE || + pExpr->op==TK_NE ); + assert( pExpr->pLeft ); + aff = sqlite3ExprAffinity(pExpr->pLeft); + if( pExpr->pRight ){ + aff = sqlite3CompareAffinity(pExpr->pRight, aff); + } + else if( pExpr->pSelect ){ + aff = sqlite3CompareAffinity(pExpr->pSelect->pEList->a[0].pExpr, aff); + } + else if( !aff ){ + aff = SQLITE_AFF_NUMERIC; + } + return aff; +} + +/* +** pExpr is a comparison expression, eg. '=', '<', IN(...) etc. +** idx_affinity is the affinity of an indexed column. Return true +** if the index with affinity idx_affinity may be used to implement +** the comparison in pExpr. +*/ +int sqlite3IndexAffinityOk(Expr *pExpr, char idx_affinity){ + char aff = comparisonAffinity(pExpr); + switch( aff ){ + case SQLITE_AFF_NONE: + return 1; + case SQLITE_AFF_TEXT: + return idx_affinity==SQLITE_AFF_TEXT; + default: + return sqlite3IsNumericAffinity(idx_affinity); + } +} + +/* +** Return the P1 value that should be used for a binary comparison +** opcode (OP_Eq, OP_Ge etc.) used to compare pExpr1 and pExpr2. +** If jumpIfNull is true, then set the low byte of the returned +** P1 value to tell the opcode to jump if either expression +** evaluates to NULL. +*/ +static int binaryCompareP1(Expr *pExpr1, Expr *pExpr2, int jumpIfNull){ + char aff = sqlite3ExprAffinity(pExpr2); + return ((int)sqlite3CompareAffinity(pExpr1, aff))+(jumpIfNull?0x100:0); +} + +/* +** Return a pointer to the collation sequence that should be used by +** a binary comparison operator comparing pLeft and pRight. +** +** If the left hand expression has a collating sequence type, then it is +** used. Otherwise the collation sequence for the right hand expression +** is used, or the default (BINARY) if neither expression has a collating +** type. +*/ +static CollSeq* binaryCompareCollSeq(Parse *pParse, Expr *pLeft, Expr *pRight){ + CollSeq *pColl = sqlite3ExprCollSeq(pParse, pLeft); + if( !pColl ){ + pColl = sqlite3ExprCollSeq(pParse, pRight); + } + return pColl; +} + +/* +** Generate code for a comparison operator. +*/ +static int codeCompare( + Parse *pParse, /* The parsing (and code generating) context */ + Expr *pLeft, /* The left operand */ + Expr *pRight, /* The right operand */ + int opcode, /* The comparison opcode */ + int dest, /* Jump here if true. */ + int jumpIfNull /* If true, jump if either operand is NULL */ +){ + int p1 = binaryCompareP1(pLeft, pRight, jumpIfNull); + CollSeq *p3 = binaryCompareCollSeq(pParse, pLeft, pRight); + return sqlite3VdbeOp3(pParse->pVdbe, opcode, p1, dest, (void*)p3, P3_COLLSEQ); +} + +/* +** Construct a new expression node and return a pointer to it. Memory +** for this node is obtained from sqliteMalloc(). The calling function +** is responsible for making sure the node eventually gets freed. +*/ +Expr *sqlite3Expr(int op, Expr *pLeft, Expr *pRight, const Token *pToken){ + Expr *pNew; + pNew = sqliteMalloc( sizeof(Expr) ); + if( pNew==0 ){ + /* When malloc fails, delete pLeft and pRight. Expressions passed to + ** this function must always be allocated with sqlite3Expr() for this + ** reason. + */ + sqlite3ExprDelete(pLeft); + sqlite3ExprDelete(pRight); + return 0; + } + pNew->op = op; + pNew->pLeft = pLeft; + pNew->pRight = pRight; + pNew->iAgg = -1; + if( pToken ){ + assert( pToken->dyn==0 ); + pNew->span = pNew->token = *pToken; + }else if( pLeft && pRight ){ + sqlite3ExprSpan(pNew, &pLeft->span, &pRight->span); + } + return pNew; +} + +/* +** When doing a nested parse, you can include terms in an expression +** that look like this: #0 #1 #2 ... These terms refer to elements +** on the stack. "#0" means the top of the stack. +** "#1" means the next down on the stack. And so forth. +** +** This routine is called by the parser to deal with on of those terms. +** It immediately generates code to store the value in a memory location. +** The returns an expression that will code to extract the value from +** that memory location as needed. +*/ +Expr *sqlite3RegisterExpr(Parse *pParse, Token *pToken){ + Vdbe *v = pParse->pVdbe; + Expr *p; + int depth; + if( pParse->nested==0 ){ + sqlite3ErrorMsg(pParse, "near \"%T\": syntax error", pToken); + return 0; + } + if( v==0 ) return 0; + p = sqlite3Expr(TK_REGISTER, 0, 0, pToken); + if( p==0 ){ + return 0; /* Malloc failed */ + } + depth = atoi((char*)&pToken->z[1]); + p->iTable = pParse->nMem++; + sqlite3VdbeAddOp(v, OP_Dup, depth, 0); + sqlite3VdbeAddOp(v, OP_MemStore, p->iTable, 1); + return p; +} + +/* +** Join two expressions using an AND operator. If either expression is +** NULL, then just return the other expression. +*/ +Expr *sqlite3ExprAnd(Expr *pLeft, Expr *pRight){ + if( pLeft==0 ){ + return pRight; + }else if( pRight==0 ){ + return pLeft; + }else{ + return sqlite3Expr(TK_AND, pLeft, pRight, 0); + } +} + +/* +** Set the Expr.span field of the given expression to span all +** text between the two given tokens. +*/ +void sqlite3ExprSpan(Expr *pExpr, Token *pLeft, Token *pRight){ + assert( pRight!=0 ); + assert( pLeft!=0 ); + if( !sqlite3MallocFailed() && pRight->z && pLeft->z ){ + assert( pLeft->dyn==0 || pLeft->z[pLeft->n]==0 ); + if( pLeft->dyn==0 && pRight->dyn==0 ){ + pExpr->span.z = pLeft->z; + pExpr->span.n = pRight->n + (pRight->z - pLeft->z); + }else{ + pExpr->span.z = 0; + } + } +} + +/* +** Construct a new expression node for a function with multiple +** arguments. +*/ +Expr *sqlite3ExprFunction(ExprList *pList, Token *pToken){ + Expr *pNew; + assert( pToken ); + pNew = sqliteMalloc( sizeof(Expr) ); + if( pNew==0 ){ + sqlite3ExprListDelete(pList); /* Avoid leaking memory when malloc fails */ + return 0; + } + pNew->op = TK_FUNCTION; + pNew->pList = pList; + assert( pToken->dyn==0 ); + pNew->token = *pToken; + pNew->span = pNew->token; + return pNew; +} + +/* +** Assign a variable number to an expression that encodes a wildcard +** in the original SQL statement. +** +** Wildcards consisting of a single "?" are assigned the next sequential +** variable number. +** +** Wildcards of the form "?nnn" are assigned the number "nnn". We make +** sure "nnn" is not too be to avoid a denial of service attack when +** the SQL statement comes from an external source. +** +** Wildcards of the form ":aaa" or "$aaa" are assigned the same number +** as the previous instance of the same wildcard. Or if this is the first +** instance of the wildcard, the next sequenial variable number is +** assigned. +*/ +void sqlite3ExprAssignVarNumber(Parse *pParse, Expr *pExpr){ + Token *pToken; + if( pExpr==0 ) return; + pToken = &pExpr->token; + assert( pToken->n>=1 ); + assert( pToken->z!=0 ); + assert( pToken->z[0]!=0 ); + if( pToken->n==1 ){ + /* Wildcard of the form "?". Assign the next variable number */ + pExpr->iTable = ++pParse->nVar; + }else if( pToken->z[0]=='?' ){ + /* Wildcard of the form "?nnn". Convert "nnn" to an integer and + ** use it as the variable number */ + int i; + pExpr->iTable = i = atoi((char*)&pToken->z[1]); + if( i<1 || i>SQLITE_MAX_VARIABLE_NUMBER ){ + sqlite3ErrorMsg(pParse, "variable number must be between ?1 and ?%d", + SQLITE_MAX_VARIABLE_NUMBER); + } + if( i>pParse->nVar ){ + pParse->nVar = i; + } + }else{ + /* Wildcards of the form ":aaa" or "$aaa". Reuse the same variable + ** number as the prior appearance of the same name, or if the name + ** has never appeared before, reuse the same variable number + */ + int i, n; + n = pToken->n; + for(i=0; inVarExpr; i++){ + Expr *pE; + if( (pE = pParse->apVarExpr[i])!=0 + && pE->token.n==n + && memcmp(pE->token.z, pToken->z, n)==0 ){ + pExpr->iTable = pE->iTable; + break; + } + } + if( i>=pParse->nVarExpr ){ + pExpr->iTable = ++pParse->nVar; + if( pParse->nVarExpr>=pParse->nVarExprAlloc-1 ){ + pParse->nVarExprAlloc += pParse->nVarExprAlloc + 10; + sqliteReallocOrFree((void**)&pParse->apVarExpr, + pParse->nVarExprAlloc*sizeof(pParse->apVarExpr[0]) ); + } + if( !sqlite3MallocFailed() ){ + assert( pParse->apVarExpr!=0 ); + pParse->apVarExpr[pParse->nVarExpr++] = pExpr; + } + } + } +} + +/* +** Recursively delete an expression tree. +*/ +void sqlite3ExprDelete(Expr *p){ + if( p==0 ) return; + if( p->span.dyn ) sqliteFree((char*)p->span.z); + if( p->token.dyn ) sqliteFree((char*)p->token.z); + sqlite3ExprDelete(p->pLeft); + sqlite3ExprDelete(p->pRight); + sqlite3ExprListDelete(p->pList); + sqlite3SelectDelete(p->pSelect); + sqliteFree(p); +} + +/* +** The Expr.token field might be a string literal that is quoted. +** If so, remove the quotation marks. +*/ +void sqlite3DequoteExpr(Expr *p){ + if( ExprHasAnyProperty(p, EP_Dequoted) ){ + return; + } + ExprSetProperty(p, EP_Dequoted); + if( p->token.dyn==0 ){ + sqlite3TokenCopy(&p->token, &p->token); + } + sqlite3Dequote((char*)p->token.z); +} + + +/* +** The following group of routines make deep copies of expressions, +** expression lists, ID lists, and select statements. The copies can +** be deleted (by being passed to their respective ...Delete() routines) +** without effecting the originals. +** +** The expression list, ID, and source lists return by sqlite3ExprListDup(), +** sqlite3IdListDup(), and sqlite3SrcListDup() can not be further expanded +** by subsequent calls to sqlite*ListAppend() routines. +** +** Any tables that the SrcList might point to are not duplicated. +*/ +Expr *sqlite3ExprDup(Expr *p){ + Expr *pNew; + if( p==0 ) return 0; + pNew = sqliteMallocRaw( sizeof(*p) ); + if( pNew==0 ) return 0; + memcpy(pNew, p, sizeof(*pNew)); + if( p->token.z!=0 ){ + pNew->token.z = (u8*)sqliteStrNDup((char*)p->token.z, p->token.n); + pNew->token.dyn = 1; + }else{ + assert( pNew->token.z==0 ); + } + pNew->span.z = 0; + pNew->pLeft = sqlite3ExprDup(p->pLeft); + pNew->pRight = sqlite3ExprDup(p->pRight); + pNew->pList = sqlite3ExprListDup(p->pList); + pNew->pSelect = sqlite3SelectDup(p->pSelect); + pNew->pTab = p->pTab; + return pNew; +} +void sqlite3TokenCopy(Token *pTo, Token *pFrom){ + if( pTo->dyn ) sqliteFree((char*)pTo->z); + if( pFrom->z ){ + pTo->n = pFrom->n; + pTo->z = (u8*)sqliteStrNDup((char*)pFrom->z, pFrom->n); + pTo->dyn = 1; + }else{ + pTo->z = 0; + } +} +ExprList *sqlite3ExprListDup(ExprList *p){ + ExprList *pNew; + struct ExprList_item *pItem, *pOldItem; + int i; + if( p==0 ) return 0; + pNew = sqliteMalloc( sizeof(*pNew) ); + if( pNew==0 ) return 0; + pNew->nExpr = pNew->nAlloc = p->nExpr; + pNew->a = pItem = sqliteMalloc( p->nExpr*sizeof(p->a[0]) ); + if( pItem==0 ){ + sqliteFree(pNew); + return 0; + } + pOldItem = p->a; + for(i=0; inExpr; i++, pItem++, pOldItem++){ + Expr *pNewExpr, *pOldExpr; + pItem->pExpr = pNewExpr = sqlite3ExprDup(pOldExpr = pOldItem->pExpr); + if( pOldExpr->span.z!=0 && pNewExpr ){ + /* Always make a copy of the span for top-level expressions in the + ** expression list. The logic in SELECT processing that determines + ** the names of columns in the result set needs this information */ + sqlite3TokenCopy(&pNewExpr->span, &pOldExpr->span); + } + assert( pNewExpr==0 || pNewExpr->span.z!=0 + || pOldExpr->span.z==0 + || sqlite3MallocFailed() ); + pItem->zName = sqliteStrDup(pOldItem->zName); + pItem->sortOrder = pOldItem->sortOrder; + pItem->isAgg = pOldItem->isAgg; + pItem->done = 0; + } + return pNew; +} + +/* +** If cursors, triggers, views and subqueries are all omitted from +** the build, then none of the following routines, except for +** sqlite3SelectDup(), can be called. sqlite3SelectDup() is sometimes +** called with a NULL argument. +*/ +#if !defined(SQLITE_OMIT_VIEW) || !defined(SQLITE_OMIT_TRIGGER) \ + || !defined(SQLITE_OMIT_SUBQUERY) +SrcList *sqlite3SrcListDup(SrcList *p){ + SrcList *pNew; + int i; + int nByte; + if( p==0 ) return 0; + nByte = sizeof(*p) + (p->nSrc>0 ? sizeof(p->a[0]) * (p->nSrc-1) : 0); + pNew = sqliteMallocRaw( nByte ); + if( pNew==0 ) return 0; + pNew->nSrc = pNew->nAlloc = p->nSrc; + for(i=0; inSrc; i++){ + struct SrcList_item *pNewItem = &pNew->a[i]; + struct SrcList_item *pOldItem = &p->a[i]; + Table *pTab; + pNewItem->zDatabase = sqliteStrDup(pOldItem->zDatabase); + pNewItem->zName = sqliteStrDup(pOldItem->zName); + pNewItem->zAlias = sqliteStrDup(pOldItem->zAlias); + pNewItem->jointype = pOldItem->jointype; + pNewItem->iCursor = pOldItem->iCursor; + pNewItem->isPopulated = pOldItem->isPopulated; + pTab = pNewItem->pTab = pOldItem->pTab; + if( pTab ){ + pTab->nRef++; + } + pNewItem->pSelect = sqlite3SelectDup(pOldItem->pSelect); + pNewItem->pOn = sqlite3ExprDup(pOldItem->pOn); + pNewItem->pUsing = sqlite3IdListDup(pOldItem->pUsing); + pNewItem->colUsed = pOldItem->colUsed; + } + return pNew; +} +IdList *sqlite3IdListDup(IdList *p){ + IdList *pNew; + int i; + if( p==0 ) return 0; + pNew = sqliteMallocRaw( sizeof(*pNew) ); + if( pNew==0 ) return 0; + pNew->nId = pNew->nAlloc = p->nId; + pNew->a = sqliteMallocRaw( p->nId*sizeof(p->a[0]) ); + if( pNew->a==0 ){ + sqliteFree(pNew); + return 0; + } + for(i=0; inId; i++){ + struct IdList_item *pNewItem = &pNew->a[i]; + struct IdList_item *pOldItem = &p->a[i]; + pNewItem->zName = sqliteStrDup(pOldItem->zName); + pNewItem->idx = pOldItem->idx; + } + return pNew; +} +Select *sqlite3SelectDup(Select *p){ + Select *pNew; + if( p==0 ) return 0; + pNew = sqliteMallocRaw( sizeof(*p) ); + if( pNew==0 ) return 0; + pNew->isDistinct = p->isDistinct; + pNew->pEList = sqlite3ExprListDup(p->pEList); + pNew->pSrc = sqlite3SrcListDup(p->pSrc); + pNew->pWhere = sqlite3ExprDup(p->pWhere); + pNew->pGroupBy = sqlite3ExprListDup(p->pGroupBy); + pNew->pHaving = sqlite3ExprDup(p->pHaving); + pNew->pOrderBy = sqlite3ExprListDup(p->pOrderBy); + pNew->op = p->op; + pNew->pPrior = sqlite3SelectDup(p->pPrior); + pNew->pLimit = sqlite3ExprDup(p->pLimit); + pNew->pOffset = sqlite3ExprDup(p->pOffset); + pNew->iLimit = -1; + pNew->iOffset = -1; + pNew->isResolved = p->isResolved; + pNew->isAgg = p->isAgg; + pNew->usesVirt = 0; + pNew->disallowOrderBy = 0; + pNew->pRightmost = 0; + pNew->addrOpenVirt[0] = -1; + pNew->addrOpenVirt[1] = -1; + pNew->addrOpenVirt[2] = -1; + return pNew; +} +#else +Select *sqlite3SelectDup(Select *p){ + assert( p==0 ); + return 0; +} +#endif + + +/* +** Add a new element to the end of an expression list. If pList is +** initially NULL, then create a new expression list. +*/ +ExprList *sqlite3ExprListAppend(ExprList *pList, Expr *pExpr, Token *pName){ + if( pList==0 ){ + pList = sqliteMalloc( sizeof(ExprList) ); + if( pList==0 ){ + goto no_mem; + } + assert( pList->nAlloc==0 ); + } + if( pList->nAlloc<=pList->nExpr ){ + struct ExprList_item *a; + int n = pList->nAlloc*2 + 4; + a = sqliteRealloc(pList->a, n*sizeof(pList->a[0])); + if( a==0 ){ + goto no_mem; + } + pList->a = a; + pList->nAlloc = n; + } + assert( pList->a!=0 ); + if( pExpr || pName ){ + struct ExprList_item *pItem = &pList->a[pList->nExpr++]; + memset(pItem, 0, sizeof(*pItem)); + pItem->zName = sqlite3NameFromToken(pName); + pItem->pExpr = pExpr; + } + return pList; + +no_mem: + /* Avoid leaking memory if malloc has failed. */ + sqlite3ExprDelete(pExpr); + sqlite3ExprListDelete(pList); + return 0; +} + +/* +** Delete an entire expression list. +*/ +void sqlite3ExprListDelete(ExprList *pList){ + int i; + struct ExprList_item *pItem; + if( pList==0 ) return; + assert( pList->a!=0 || (pList->nExpr==0 && pList->nAlloc==0) ); + assert( pList->nExpr<=pList->nAlloc ); + for(pItem=pList->a, i=0; inExpr; i++, pItem++){ + sqlite3ExprDelete(pItem->pExpr); + sqliteFree(pItem->zName); + } + sqliteFree(pList->a); + sqliteFree(pList); +} + +/* +** Walk an expression tree. Call xFunc for each node visited. +** +** The return value from xFunc determines whether the tree walk continues. +** 0 means continue walking the tree. 1 means do not walk children +** of the current node but continue with siblings. 2 means abandon +** the tree walk completely. +** +** The return value from this routine is 1 to abandon the tree walk +** and 0 to continue. +** +** NOTICE: This routine does *not* descend into subqueries. +*/ +static int walkExprList(ExprList *, int (*)(void *, Expr*), void *); +static int walkExprTree(Expr *pExpr, int (*xFunc)(void*,Expr*), void *pArg){ + int rc; + if( pExpr==0 ) return 0; + rc = (*xFunc)(pArg, pExpr); + if( rc==0 ){ + if( walkExprTree(pExpr->pLeft, xFunc, pArg) ) return 1; + if( walkExprTree(pExpr->pRight, xFunc, pArg) ) return 1; + if( walkExprList(pExpr->pList, xFunc, pArg) ) return 1; + } + return rc>1; +} + +/* +** Call walkExprTree() for every expression in list p. +*/ +static int walkExprList(ExprList *p, int (*xFunc)(void *, Expr*), void *pArg){ + int i; + struct ExprList_item *pItem; + if( !p ) return 0; + for(i=p->nExpr, pItem=p->a; i>0; i--, pItem++){ + if( walkExprTree(pItem->pExpr, xFunc, pArg) ) return 1; + } + return 0; +} + +/* +** Call walkExprTree() for every expression in Select p, not including +** expressions that are part of sub-selects in any FROM clause or the LIMIT +** or OFFSET expressions.. +*/ +static int walkSelectExpr(Select *p, int (*xFunc)(void *, Expr*), void *pArg){ + walkExprList(p->pEList, xFunc, pArg); + walkExprTree(p->pWhere, xFunc, pArg); + walkExprList(p->pGroupBy, xFunc, pArg); + walkExprTree(p->pHaving, xFunc, pArg); + walkExprList(p->pOrderBy, xFunc, pArg); + return 0; +} + + +/* +** This routine is designed as an xFunc for walkExprTree(). +** +** pArg is really a pointer to an integer. If we can tell by looking +** at pExpr that the expression that contains pExpr is not a constant +** expression, then set *pArg to 0 and return 2 to abandon the tree walk. +** If pExpr does does not disqualify the expression from being a constant +** then do nothing. +** +** After walking the whole tree, if no nodes are found that disqualify +** the expression as constant, then we assume the whole expression +** is constant. See sqlite3ExprIsConstant() for additional information. +*/ +static int exprNodeIsConstant(void *pArg, Expr *pExpr){ + switch( pExpr->op ){ + /* Consider functions to be constant if all their arguments are constant + ** and *pArg==2 */ + case TK_FUNCTION: + if( *((int*)pArg)==2 ) return 0; + /* Fall through */ + case TK_ID: + case TK_COLUMN: + case TK_DOT: + case TK_AGG_FUNCTION: + case TK_AGG_COLUMN: +#ifndef SQLITE_OMIT_SUBQUERY + case TK_SELECT: + case TK_EXISTS: +#endif + *((int*)pArg) = 0; + return 2; + case TK_IN: + if( pExpr->pSelect ){ + *((int*)pArg) = 0; + return 2; + } + default: + return 0; + } +} + +/* +** Walk an expression tree. Return 1 if the expression is constant +** and 0 if it involves variables or function calls. +** +** For the purposes of this function, a double-quoted string (ex: "abc") +** is considered a variable but a single-quoted string (ex: 'abc') is +** a constant. +*/ +int sqlite3ExprIsConstant(Expr *p){ + int isConst = 1; + walkExprTree(p, exprNodeIsConstant, &isConst); + return isConst; +} + +/* +** Walk an expression tree. Return 1 if the expression is constant +** or a function call with constant arguments. Return and 0 if there +** are any variables. +** +** For the purposes of this function, a double-quoted string (ex: "abc") +** is considered a variable but a single-quoted string (ex: 'abc') is +** a constant. +*/ +int sqlite3ExprIsConstantOrFunction(Expr *p){ + int isConst = 2; + walkExprTree(p, exprNodeIsConstant, &isConst); + return isConst!=0; +} + +/* +** If the expression p codes a constant integer that is small enough +** to fit in a 32-bit integer, return 1 and put the value of the integer +** in *pValue. If the expression is not an integer or if it is too big +** to fit in a signed 32-bit integer, return 0 and leave *pValue unchanged. +*/ +int sqlite3ExprIsInteger(Expr *p, int *pValue){ + switch( p->op ){ + case TK_INTEGER: { + if( sqlite3GetInt32((char*)p->token.z, pValue) ){ + return 1; + } + break; + } + case TK_UPLUS: { + return sqlite3ExprIsInteger(p->pLeft, pValue); + } + case TK_UMINUS: { + int v; + if( sqlite3ExprIsInteger(p->pLeft, &v) ){ + *pValue = -v; + return 1; + } + break; + } + default: break; + } + return 0; +} + +/* +** Return TRUE if the given string is a row-id column name. +*/ +int sqlite3IsRowid(const char *z){ + if( sqlite3StrICmp(z, "_ROWID_")==0 ) return 1; + if( sqlite3StrICmp(z, "ROWID")==0 ) return 1; + if( sqlite3StrICmp(z, "OID")==0 ) return 1; + return 0; +} + +/* +** Given the name of a column of the form X.Y.Z or Y.Z or just Z, look up +** that name in the set of source tables in pSrcList and make the pExpr +** expression node refer back to that source column. The following changes +** are made to pExpr: +** +** pExpr->iDb Set the index in db->aDb[] of the database holding +** the table. +** pExpr->iTable Set to the cursor number for the table obtained +** from pSrcList. +** pExpr->iColumn Set to the column number within the table. +** pExpr->op Set to TK_COLUMN. +** pExpr->pLeft Any expression this points to is deleted +** pExpr->pRight Any expression this points to is deleted. +** +** The pDbToken is the name of the database (the "X"). This value may be +** NULL meaning that name is of the form Y.Z or Z. Any available database +** can be used. The pTableToken is the name of the table (the "Y"). This +** value can be NULL if pDbToken is also NULL. If pTableToken is NULL it +** means that the form of the name is Z and that columns from any table +** can be used. +** +** If the name cannot be resolved unambiguously, leave an error message +** in pParse and return non-zero. Return zero on success. +*/ +static int lookupName( + Parse *pParse, /* The parsing context */ + Token *pDbToken, /* Name of the database containing table, or NULL */ + Token *pTableToken, /* Name of table containing column, or NULL */ + Token *pColumnToken, /* Name of the column. */ + NameContext *pNC, /* The name context used to resolve the name */ + Expr *pExpr /* Make this EXPR node point to the selected column */ +){ + char *zDb = 0; /* Name of the database. The "X" in X.Y.Z */ + char *zTab = 0; /* Name of the table. The "Y" in X.Y.Z or Y.Z */ + char *zCol = 0; /* Name of the column. The "Z" */ + int i, j; /* Loop counters */ + int cnt = 0; /* Number of matching column names */ + int cntTab = 0; /* Number of matching table names */ + sqlite3 *db = pParse->db; /* The database */ + struct SrcList_item *pItem; /* Use for looping over pSrcList items */ + struct SrcList_item *pMatch = 0; /* The matching pSrcList item */ + NameContext *pTopNC = pNC; /* First namecontext in the list */ + + assert( pColumnToken && pColumnToken->z ); /* The Z in X.Y.Z cannot be NULL */ + zDb = sqlite3NameFromToken(pDbToken); + zTab = sqlite3NameFromToken(pTableToken); + zCol = sqlite3NameFromToken(pColumnToken); + if( sqlite3MallocFailed() ){ + goto lookupname_end; + } + + pExpr->iTable = -1; + while( pNC && cnt==0 ){ + ExprList *pEList; + SrcList *pSrcList = pNC->pSrcList; + + if( pSrcList ){ + for(i=0, pItem=pSrcList->a; inSrc; i++, pItem++){ + Table *pTab = pItem->pTab; + int iDb = sqlite3SchemaToIndex(db, pTab->pSchema); + Column *pCol; + + if( pTab==0 ) continue; + assert( pTab->nCol>0 ); + if( zTab ){ + if( pItem->zAlias ){ + char *zTabName = pItem->zAlias; + if( sqlite3StrICmp(zTabName, zTab)!=0 ) continue; + }else{ + char *zTabName = pTab->zName; + if( zTabName==0 || sqlite3StrICmp(zTabName, zTab)!=0 ) continue; + if( zDb!=0 && sqlite3StrICmp(db->aDb[iDb].zName, zDb)!=0 ){ + continue; + } + } + } + if( 0==(cntTab++) ){ + pExpr->iTable = pItem->iCursor; + pExpr->pSchema = pTab->pSchema; + pMatch = pItem; + } + for(j=0, pCol=pTab->aCol; jnCol; j++, pCol++){ + if( sqlite3StrICmp(pCol->zName, zCol)==0 ){ + const char *zColl = pTab->aCol[j].zColl; + IdList *pUsing; + cnt++; + pExpr->iTable = pItem->iCursor; + pMatch = pItem; + pExpr->pSchema = pTab->pSchema; + /* Substitute the rowid (column -1) for the INTEGER PRIMARY KEY */ + pExpr->iColumn = j==pTab->iPKey ? -1 : j; + pExpr->affinity = pTab->aCol[j].affinity; + pExpr->pColl = sqlite3FindCollSeq(db, ENC(db), zColl,-1, 0); + if( pItem->jointype & JT_NATURAL ){ + /* If this match occurred in the left table of a natural join, + ** then skip the right table to avoid a duplicate match */ + pItem++; + i++; + } + if( (pUsing = pItem->pUsing)!=0 ){ + /* If this match occurs on a column that is in the USING clause + ** of a join, skip the search of the right table of the join + ** to avoid a duplicate match there. */ + int k; + for(k=0; knId; k++){ + if( sqlite3StrICmp(pUsing->a[k].zName, zCol)==0 ){ + pItem++; + i++; + break; + } + } + } + break; + } + } + } + } + +#ifndef SQLITE_OMIT_TRIGGER + /* If we have not already resolved the name, then maybe + ** it is a new.* or old.* trigger argument reference + */ + if( zDb==0 && zTab!=0 && cnt==0 && pParse->trigStack!=0 ){ + TriggerStack *pTriggerStack = pParse->trigStack; + Table *pTab = 0; + if( pTriggerStack->newIdx != -1 && sqlite3StrICmp("new", zTab) == 0 ){ + pExpr->iTable = pTriggerStack->newIdx; + assert( pTriggerStack->pTab ); + pTab = pTriggerStack->pTab; + }else if( pTriggerStack->oldIdx != -1 && sqlite3StrICmp("old", zTab)==0 ){ + pExpr->iTable = pTriggerStack->oldIdx; + assert( pTriggerStack->pTab ); + pTab = pTriggerStack->pTab; + } + + if( pTab ){ + int iCol; + Column *pCol = pTab->aCol; + + pExpr->pSchema = pTab->pSchema; + cntTab++; + for(iCol=0; iCol < pTab->nCol; iCol++, pCol++) { + if( sqlite3StrICmp(pCol->zName, zCol)==0 ){ + const char *zColl = pTab->aCol[iCol].zColl; + cnt++; + pExpr->iColumn = iCol==pTab->iPKey ? -1 : iCol; + pExpr->affinity = pTab->aCol[iCol].affinity; + pExpr->pColl = sqlite3FindCollSeq(db, ENC(db), zColl,-1, 0); + pExpr->pTab = pTab; + break; + } + } + } + } +#endif /* !defined(SQLITE_OMIT_TRIGGER) */ + + /* + ** Perhaps the name is a reference to the ROWID + */ + if( cnt==0 && cntTab==1 && sqlite3IsRowid(zCol) ){ + cnt = 1; + pExpr->iColumn = -1; + pExpr->affinity = SQLITE_AFF_INTEGER; + } + + /* + ** If the input is of the form Z (not Y.Z or X.Y.Z) then the name Z + ** might refer to an result-set alias. This happens, for example, when + ** we are resolving names in the WHERE clause of the following command: + ** + ** SELECT a+b AS x FROM table WHERE x<10; + ** + ** In cases like this, replace pExpr with a copy of the expression that + ** forms the result set entry ("a+b" in the example) and return immediately. + ** Note that the expression in the result set should have already been + ** resolved by the time the WHERE clause is resolved. + */ + if( cnt==0 && (pEList = pNC->pEList)!=0 && zTab==0 ){ + for(j=0; jnExpr; j++){ + char *zAs = pEList->a[j].zName; + if( zAs!=0 && sqlite3StrICmp(zAs, zCol)==0 ){ + assert( pExpr->pLeft==0 && pExpr->pRight==0 ); + pExpr->op = TK_AS; + pExpr->iColumn = j; + pExpr->pLeft = sqlite3ExprDup(pEList->a[j].pExpr); + cnt = 1; + assert( zTab==0 && zDb==0 ); + goto lookupname_end_2; + } + } + } + + /* Advance to the next name context. The loop will exit when either + ** we have a match (cnt>0) or when we run out of name contexts. + */ + if( cnt==0 ){ + pNC = pNC->pNext; + } + } + + /* + ** If X and Y are NULL (in other words if only the column name Z is + ** supplied) and the value of Z is enclosed in double-quotes, then + ** Z is a string literal if it doesn't match any column names. In that + ** case, we need to return right away and not make any changes to + ** pExpr. + ** + ** Because no reference was made to outer contexts, the pNC->nRef + ** fields are not changed in any context. + */ + if( cnt==0 && zTab==0 && pColumnToken->z[0]=='"' ){ + sqliteFree(zCol); + return 0; + } + + /* + ** cnt==0 means there was not match. cnt>1 means there were two or + ** more matches. Either way, we have an error. + */ + if( cnt!=1 ){ + char *z = 0; + char *zErr; + zErr = cnt==0 ? "no such column: %s" : "ambiguous column name: %s"; + if( zDb ){ + sqlite3SetString(&z, zDb, ".", zTab, ".", zCol, (char*)0); + }else if( zTab ){ + sqlite3SetString(&z, zTab, ".", zCol, (char*)0); + }else{ + z = sqliteStrDup(zCol); + } + sqlite3ErrorMsg(pParse, zErr, z); + sqliteFree(z); + pTopNC->nErr++; + } + + /* If a column from a table in pSrcList is referenced, then record + ** this fact in the pSrcList.a[].colUsed bitmask. Column 0 causes + ** bit 0 to be set. Column 1 sets bit 1. And so forth. If the + ** column number is greater than the number of bits in the bitmask + ** then set the high-order bit of the bitmask. + */ + if( pExpr->iColumn>=0 && pMatch!=0 ){ + int n = pExpr->iColumn; + if( n>=sizeof(Bitmask)*8 ){ + n = sizeof(Bitmask)*8-1; + } + assert( pMatch->iCursor==pExpr->iTable ); + pMatch->colUsed |= 1<pLeft); + pExpr->pLeft = 0; + sqlite3ExprDelete(pExpr->pRight); + pExpr->pRight = 0; + pExpr->op = TK_COLUMN; +lookupname_end_2: + sqliteFree(zCol); + if( cnt==1 ){ + assert( pNC!=0 ); + sqlite3AuthRead(pParse, pExpr, pNC->pSrcList); + if( pMatch && !pMatch->pSelect ){ + pExpr->pTab = pMatch->pTab; + } + /* Increment the nRef value on all name contexts from TopNC up to + ** the point where the name matched. */ + for(;;){ + assert( pTopNC!=0 ); + pTopNC->nRef++; + if( pTopNC==pNC ) break; + pTopNC = pTopNC->pNext; + } + return 0; + } else { + return 1; + } +} + +/* +** This routine is designed as an xFunc for walkExprTree(). +** +** Resolve symbolic names into TK_COLUMN operators for the current +** node in the expression tree. Return 0 to continue the search down +** the tree or 2 to abort the tree walk. +** +** This routine also does error checking and name resolution for +** function names. The operator for aggregate functions is changed +** to TK_AGG_FUNCTION. +*/ +static int nameResolverStep(void *pArg, Expr *pExpr){ + NameContext *pNC = (NameContext*)pArg; + Parse *pParse; + + if( pExpr==0 ) return 1; + assert( pNC!=0 ); + pParse = pNC->pParse; + + if( ExprHasAnyProperty(pExpr, EP_Resolved) ) return 1; + ExprSetProperty(pExpr, EP_Resolved); +#ifndef NDEBUG + if( pNC->pSrcList && pNC->pSrcList->nAlloc>0 ){ + SrcList *pSrcList = pNC->pSrcList; + int i; + for(i=0; ipSrcList->nSrc; i++){ + assert( pSrcList->a[i].iCursor>=0 && pSrcList->a[i].iCursornTab); + } + } +#endif + switch( pExpr->op ){ + /* Double-quoted strings (ex: "abc") are used as identifiers if + ** possible. Otherwise they remain as strings. Single-quoted + ** strings (ex: 'abc') are always string literals. + */ + case TK_STRING: { + if( pExpr->token.z[0]=='\'' ) break; + /* Fall thru into the TK_ID case if this is a double-quoted string */ + } + /* A lone identifier is the name of a column. + */ + case TK_ID: { + lookupName(pParse, 0, 0, &pExpr->token, pNC, pExpr); + return 1; + } + + /* A table name and column name: ID.ID + ** Or a database, table and column: ID.ID.ID + */ + case TK_DOT: { + Token *pColumn; + Token *pTable; + Token *pDb; + Expr *pRight; + + /* if( pSrcList==0 ) break; */ + pRight = pExpr->pRight; + if( pRight->op==TK_ID ){ + pDb = 0; + pTable = &pExpr->pLeft->token; + pColumn = &pRight->token; + }else{ + assert( pRight->op==TK_DOT ); + pDb = &pExpr->pLeft->token; + pTable = &pRight->pLeft->token; + pColumn = &pRight->pRight->token; + } + lookupName(pParse, pDb, pTable, pColumn, pNC, pExpr); + return 1; + } + + /* Resolve function names + */ + case TK_CONST_FUNC: + case TK_FUNCTION: { + ExprList *pList = pExpr->pList; /* The argument list */ + int n = pList ? pList->nExpr : 0; /* Number of arguments */ + int no_such_func = 0; /* True if no such function exists */ + int wrong_num_args = 0; /* True if wrong number of arguments */ + int is_agg = 0; /* True if is an aggregate function */ + int i; + int nId; /* Number of characters in function name */ + const char *zId; /* The function name. */ + FuncDef *pDef; /* Information about the function */ + int enc = ENC(pParse->db); /* The database encoding */ + + zId = (char*)pExpr->token.z; + nId = pExpr->token.n; + pDef = sqlite3FindFunction(pParse->db, zId, nId, n, enc, 0); + if( pDef==0 ){ + pDef = sqlite3FindFunction(pParse->db, zId, nId, -1, enc, 0); + if( pDef==0 ){ + no_such_func = 1; + }else{ + wrong_num_args = 1; + } + }else{ + is_agg = pDef->xFunc==0; + } + if( is_agg && !pNC->allowAgg ){ + sqlite3ErrorMsg(pParse, "misuse of aggregate function %.*s()", nId,zId); + pNC->nErr++; + is_agg = 0; + }else if( no_such_func ){ + sqlite3ErrorMsg(pParse, "no such function: %.*s", nId, zId); + pNC->nErr++; + }else if( wrong_num_args ){ + sqlite3ErrorMsg(pParse,"wrong number of arguments to function %.*s()", + nId, zId); + pNC->nErr++; + } + if( is_agg ){ + pExpr->op = TK_AGG_FUNCTION; + pNC->hasAgg = 1; + } + if( is_agg ) pNC->allowAgg = 0; + for(i=0; pNC->nErr==0 && ia[i].pExpr, nameResolverStep, pNC); + } + if( is_agg ) pNC->allowAgg = 1; + /* FIX ME: Compute pExpr->affinity based on the expected return + ** type of the function + */ + return is_agg; + } +#ifndef SQLITE_OMIT_SUBQUERY + case TK_SELECT: + case TK_EXISTS: +#endif + case TK_IN: { + if( pExpr->pSelect ){ + int nRef = pNC->nRef; +#ifndef SQLITE_OMIT_CHECK + if( pNC->isCheck ){ + sqlite3ErrorMsg(pParse,"subqueries prohibited in CHECK constraints"); + } +#endif + sqlite3SelectResolve(pParse, pExpr->pSelect, pNC); + assert( pNC->nRef>=nRef ); + if( nRef!=pNC->nRef ){ + ExprSetProperty(pExpr, EP_VarSelect); + } + } + break; + } +#ifndef SQLITE_OMIT_CHECK + case TK_VARIABLE: { + if( pNC->isCheck ){ + sqlite3ErrorMsg(pParse,"parameters prohibited in CHECK constraints"); + } + break; + } +#endif + } + return 0; +} + +/* +** This routine walks an expression tree and resolves references to +** table columns. Nodes of the form ID.ID or ID resolve into an +** index to the table in the table list and a column offset. The +** Expr.opcode for such nodes is changed to TK_COLUMN. The Expr.iTable +** value is changed to the index of the referenced table in pTabList +** plus the "base" value. The base value will ultimately become the +** VDBE cursor number for a cursor that is pointing into the referenced +** table. The Expr.iColumn value is changed to the index of the column +** of the referenced table. The Expr.iColumn value for the special +** ROWID column is -1. Any INTEGER PRIMARY KEY column is tried as an +** alias for ROWID. +** +** Also resolve function names and check the functions for proper +** usage. Make sure all function names are recognized and all functions +** have the correct number of arguments. Leave an error message +** in pParse->zErrMsg if anything is amiss. Return the number of errors. +** +** If the expression contains aggregate functions then set the EP_Agg +** property on the expression. +*/ +int sqlite3ExprResolveNames( + NameContext *pNC, /* Namespace to resolve expressions in. */ + Expr *pExpr /* The expression to be analyzed. */ +){ + int savedHasAgg; + if( pExpr==0 ) return 0; + savedHasAgg = pNC->hasAgg; + pNC->hasAgg = 0; + walkExprTree(pExpr, nameResolverStep, pNC); + if( pNC->nErr>0 ){ + ExprSetProperty(pExpr, EP_Error); + } + if( pNC->hasAgg ){ + ExprSetProperty(pExpr, EP_Agg); + }else if( savedHasAgg ){ + pNC->hasAgg = 1; + } + return ExprHasProperty(pExpr, EP_Error); +} + +/* +** A pointer instance of this structure is used to pass information +** through walkExprTree into codeSubqueryStep(). +*/ +typedef struct QueryCoder QueryCoder; +struct QueryCoder { + Parse *pParse; /* The parsing context */ + NameContext *pNC; /* Namespace of first enclosing query */ +}; + + +/* +** Generate code for scalar subqueries used as an expression +** and IN operators. Examples: +** +** (SELECT a FROM b) -- subquery +** EXISTS (SELECT a FROM b) -- EXISTS subquery +** x IN (4,5,11) -- IN operator with list on right-hand side +** x IN (SELECT a FROM b) -- IN operator with subquery on the right +** +** The pExpr parameter describes the expression that contains the IN +** operator or subquery. +*/ +#ifndef SQLITE_OMIT_SUBQUERY +void sqlite3CodeSubselect(Parse *pParse, Expr *pExpr){ + int testAddr = 0; /* One-time test address */ + Vdbe *v = sqlite3GetVdbe(pParse); + if( v==0 ) return; + + /* This code must be run in its entirety every time it is encountered + ** if any of the following is true: + ** + ** * The right-hand side is a correlated subquery + ** * The right-hand side is an expression list containing variables + ** * We are inside a trigger + ** + ** If all of the above are false, then we can run this code just once + ** save the results, and reuse the same result on subsequent invocations. + */ + if( !ExprHasAnyProperty(pExpr, EP_VarSelect) && !pParse->trigStack ){ + int mem = pParse->nMem++; + sqlite3VdbeAddOp(v, OP_MemLoad, mem, 0); + testAddr = sqlite3VdbeAddOp(v, OP_If, 0, 0); + assert( testAddr>0 || sqlite3MallocFailed() ); + sqlite3VdbeAddOp(v, OP_MemInt, 1, mem); + } + + switch( pExpr->op ){ + case TK_IN: { + char affinity; + KeyInfo keyInfo; + int addr; /* Address of OP_OpenVirtual instruction */ + + affinity = sqlite3ExprAffinity(pExpr->pLeft); + + /* Whether this is an 'x IN(SELECT...)' or an 'x IN()' + ** expression it is handled the same way. A virtual table is + ** filled with single-field index keys representing the results + ** from the SELECT or the . + ** + ** If the 'x' expression is a column value, or the SELECT... + ** statement returns a column value, then the affinity of that + ** column is used to build the index keys. If both 'x' and the + ** SELECT... statement are columns, then numeric affinity is used + ** if either column has NUMERIC or INTEGER affinity. If neither + ** 'x' nor the SELECT... statement are columns, then numeric affinity + ** is used. + */ + pExpr->iTable = pParse->nTab++; + addr = sqlite3VdbeAddOp(v, OP_OpenVirtual, pExpr->iTable, 0); + memset(&keyInfo, 0, sizeof(keyInfo)); + keyInfo.nField = 1; + sqlite3VdbeAddOp(v, OP_SetNumColumns, pExpr->iTable, 1); + + if( pExpr->pSelect ){ + /* Case 1: expr IN (SELECT ...) + ** + ** Generate code to write the results of the select into the temporary + ** table allocated and opened above. + */ + int iParm = pExpr->iTable + (((int)affinity)<<16); + ExprList *pEList; + assert( (pExpr->iTable&0x0000FFFF)==pExpr->iTable ); + sqlite3Select(pParse, pExpr->pSelect, SRT_Set, iParm, 0, 0, 0, 0); + pEList = pExpr->pSelect->pEList; + if( pEList && pEList->nExpr>0 ){ + keyInfo.aColl[0] = binaryCompareCollSeq(pParse, pExpr->pLeft, + pEList->a[0].pExpr); + } + }else if( pExpr->pList ){ + /* Case 2: expr IN (exprlist) + ** + ** For each expression, build an index key from the evaluation and + ** store it in the temporary table. If is a column, then use + ** that columns affinity when building index keys. If is not + ** a column, use numeric affinity. + */ + int i; + ExprList *pList = pExpr->pList; + struct ExprList_item *pItem; + + if( !affinity ){ + affinity = SQLITE_AFF_NUMERIC; + } + keyInfo.aColl[0] = pExpr->pLeft->pColl; + + /* Loop through each expression in . */ + for(i=pList->nExpr, pItem=pList->a; i>0; i--, pItem++){ + Expr *pE2 = pItem->pExpr; + + /* If the expression is not constant then we will need to + ** disable the test that was generated above that makes sure + ** this code only executes once. Because for a non-constant + ** expression we need to rerun this code each time. + */ + if( testAddr>0 && !sqlite3ExprIsConstant(pE2) ){ + VdbeOp *aOp = sqlite3VdbeGetOp(v, testAddr-1); + int j; + for(j=0; j<3; j++){ + aOp[j].opcode = OP_Noop; + } + testAddr = 0; + } + + /* Evaluate the expression and insert it into the temp table */ + sqlite3ExprCode(pParse, pE2); + sqlite3VdbeOp3(v, OP_MakeRecord, 1, 0, &affinity, 1); + sqlite3VdbeAddOp(v, OP_IdxInsert, pExpr->iTable, 0); + } + } + sqlite3VdbeChangeP3(v, addr, (void *)&keyInfo, P3_KEYINFO); + break; + } + + case TK_EXISTS: + case TK_SELECT: { + /* This has to be a scalar SELECT. Generate code to put the + ** value of this select in a memory cell and record the number + ** of the memory cell in iColumn. + */ + static const Token one = { (u8*)"1", 0, 1 }; + Select *pSel; + int iMem; + int sop; + + pExpr->iColumn = iMem = pParse->nMem++; + pSel = pExpr->pSelect; + if( pExpr->op==TK_SELECT ){ + sop = SRT_Mem; + sqlite3VdbeAddOp(v, OP_MemNull, iMem, 0); + VdbeComment((v, "# Init subquery result")); + }else{ + sop = SRT_Exists; + sqlite3VdbeAddOp(v, OP_MemInt, 0, iMem); + VdbeComment((v, "# Init EXISTS result")); + } + sqlite3ExprDelete(pSel->pLimit); + pSel->pLimit = sqlite3Expr(TK_INTEGER, 0, 0, &one); + sqlite3Select(pParse, pSel, sop, iMem, 0, 0, 0, 0); + break; + } + } + + if( testAddr ){ + sqlite3VdbeJumpHere(v, testAddr); + } + return; +} +#endif /* SQLITE_OMIT_SUBQUERY */ + +/* +** Generate an instruction that will put the integer describe by +** text z[0..n-1] on the stack. +*/ +static void codeInteger(Vdbe *v, const char *z, int n){ + int i; + if( sqlite3GetInt32(z, &i) ){ + sqlite3VdbeAddOp(v, OP_Integer, i, 0); + }else if( sqlite3FitsIn64Bits(z) ){ + sqlite3VdbeOp3(v, OP_Int64, 0, 0, z, n); + }else{ + sqlite3VdbeOp3(v, OP_Real, 0, 0, z, n); + } +} + +/* +** Generate code into the current Vdbe to evaluate the given +** expression and leave the result on the top of stack. +** +** This code depends on the fact that certain token values (ex: TK_EQ) +** are the same as opcode values (ex: OP_Eq) that implement the corresponding +** operation. Special comments in vdbe.c and the mkopcodeh.awk script in +** the make process cause these values to align. Assert()s in the code +** below verify that the numbers are aligned correctly. +*/ +void sqlite3ExprCode(Parse *pParse, Expr *pExpr){ + Vdbe *v = pParse->pVdbe; + int op; + int stackChng = 1; /* Amount of change to stack depth */ + + if( v==0 ) return; + if( pExpr==0 ){ + sqlite3VdbeAddOp(v, OP_Null, 0, 0); + return; + } + op = pExpr->op; + switch( op ){ + case TK_AGG_COLUMN: { + AggInfo *pAggInfo = pExpr->pAggInfo; + struct AggInfo_col *pCol = &pAggInfo->aCol[pExpr->iAgg]; + if( !pAggInfo->directMode ){ + sqlite3VdbeAddOp(v, OP_MemLoad, pCol->iMem, 0); + break; + }else if( pAggInfo->useSortingIdx ){ + sqlite3VdbeAddOp(v, OP_Column, pAggInfo->sortingIdx, + pCol->iSorterColumn); + break; + } + /* Otherwise, fall thru into the TK_COLUMN case */ + } + case TK_COLUMN: { + if( pExpr->iTable<0 ){ + /* This only happens when coding check constraints */ + assert( pParse->ckOffset>0 ); + sqlite3VdbeAddOp(v, OP_Dup, pParse->ckOffset-pExpr->iColumn-1, 1); + }else if( pExpr->iColumn>=0 ){ + Table *pTab = pExpr->pTab; + int iCol = pExpr->iColumn; + sqlite3VdbeAddOp(v, OP_Column, pExpr->iTable, iCol); + sqlite3ColumnDefault(v, pTab, iCol); +#ifndef SQLITE_OMIT_FLOATING_POINT + if( pTab && pTab->aCol[iCol].affinity==SQLITE_AFF_REAL ){ + sqlite3VdbeAddOp(v, OP_RealAffinity, 0, 0); + } +#endif + }else{ + sqlite3VdbeAddOp(v, OP_Rowid, pExpr->iTable, 0); + } + break; + } + case TK_INTEGER: { + codeInteger(v, (char*)pExpr->token.z, pExpr->token.n); + break; + } + case TK_FLOAT: + case TK_STRING: { + assert( TK_FLOAT==OP_Real ); + assert( TK_STRING==OP_String8 ); + sqlite3DequoteExpr(pExpr); + sqlite3VdbeOp3(v, op, 0, 0, (char*)pExpr->token.z, pExpr->token.n); + break; + } + case TK_NULL: { + sqlite3VdbeAddOp(v, OP_Null, 0, 0); + break; + } +#ifndef SQLITE_OMIT_BLOB_LITERAL + case TK_BLOB: { + int n; + const char *z; + assert( TK_BLOB==OP_HexBlob ); + n = pExpr->token.n - 3; + z = (char*)pExpr->token.z + 2; + assert( n>=0 ); + if( n==0 ){ + z = ""; + } + sqlite3VdbeOp3(v, op, 0, 0, z, n); + break; + } +#endif + case TK_VARIABLE: { + sqlite3VdbeAddOp(v, OP_Variable, pExpr->iTable, 0); + if( pExpr->token.n>1 ){ + sqlite3VdbeChangeP3(v, -1, (char*)pExpr->token.z, pExpr->token.n); + } + break; + } + case TK_REGISTER: { + sqlite3VdbeAddOp(v, OP_MemLoad, pExpr->iTable, 0); + break; + } +#ifndef SQLITE_OMIT_CAST + case TK_CAST: { + /* Expressions of the form: CAST(pLeft AS token) */ + int aff, to_op; + sqlite3ExprCode(pParse, pExpr->pLeft); + aff = sqlite3AffinityType(&pExpr->token); + to_op = aff - SQLITE_AFF_TEXT + OP_ToText; + assert( to_op==OP_ToText || aff!=SQLITE_AFF_TEXT ); + assert( to_op==OP_ToBlob || aff!=SQLITE_AFF_NONE ); + assert( to_op==OP_ToNumeric || aff!=SQLITE_AFF_NUMERIC ); + assert( to_op==OP_ToInt || aff!=SQLITE_AFF_INTEGER ); + assert( to_op==OP_ToReal || aff!=SQLITE_AFF_REAL ); + sqlite3VdbeAddOp(v, to_op, 0, 0); + stackChng = 0; + break; + } +#endif /* SQLITE_OMIT_CAST */ + case TK_LT: + case TK_LE: + case TK_GT: + case TK_GE: + case TK_NE: + case TK_EQ: { + assert( TK_LT==OP_Lt ); + assert( TK_LE==OP_Le ); + assert( TK_GT==OP_Gt ); + assert( TK_GE==OP_Ge ); + assert( TK_EQ==OP_Eq ); + assert( TK_NE==OP_Ne ); + sqlite3ExprCode(pParse, pExpr->pLeft); + sqlite3ExprCode(pParse, pExpr->pRight); + codeCompare(pParse, pExpr->pLeft, pExpr->pRight, op, 0, 0); + stackChng = -1; + break; + } + case TK_AND: + case TK_OR: + case TK_PLUS: + case TK_STAR: + case TK_MINUS: + case TK_REM: + case TK_BITAND: + case TK_BITOR: + case TK_SLASH: + case TK_LSHIFT: + case TK_RSHIFT: + case TK_CONCAT: { + assert( TK_AND==OP_And ); + assert( TK_OR==OP_Or ); + assert( TK_PLUS==OP_Add ); + assert( TK_MINUS==OP_Subtract ); + assert( TK_REM==OP_Remainder ); + assert( TK_BITAND==OP_BitAnd ); + assert( TK_BITOR==OP_BitOr ); + assert( TK_SLASH==OP_Divide ); + assert( TK_LSHIFT==OP_ShiftLeft ); + assert( TK_RSHIFT==OP_ShiftRight ); + assert( TK_CONCAT==OP_Concat ); + sqlite3ExprCode(pParse, pExpr->pLeft); + sqlite3ExprCode(pParse, pExpr->pRight); + sqlite3VdbeAddOp(v, op, 0, 0); + stackChng = -1; + break; + } + case TK_UMINUS: { + Expr *pLeft = pExpr->pLeft; + assert( pLeft ); + if( pLeft->op==TK_FLOAT || pLeft->op==TK_INTEGER ){ + Token *p = &pLeft->token; + char *z = sqlite3MPrintf("-%.*s", p->n, p->z); + if( pLeft->op==TK_FLOAT ){ + sqlite3VdbeOp3(v, OP_Real, 0, 0, z, p->n+1); + }else{ + codeInteger(v, z, p->n+1); + } + sqliteFree(z); + break; + } + /* Fall through into TK_NOT */ + } + case TK_BITNOT: + case TK_NOT: { + assert( TK_BITNOT==OP_BitNot ); + assert( TK_NOT==OP_Not ); + sqlite3ExprCode(pParse, pExpr->pLeft); + sqlite3VdbeAddOp(v, op, 0, 0); + stackChng = 0; + break; + } + case TK_ISNULL: + case TK_NOTNULL: { + int dest; + assert( TK_ISNULL==OP_IsNull ); + assert( TK_NOTNULL==OP_NotNull ); + sqlite3VdbeAddOp(v, OP_Integer, 1, 0); + sqlite3ExprCode(pParse, pExpr->pLeft); + dest = sqlite3VdbeCurrentAddr(v) + 2; + sqlite3VdbeAddOp(v, op, 1, dest); + sqlite3VdbeAddOp(v, OP_AddImm, -1, 0); + stackChng = 0; + break; + } + case TK_AGG_FUNCTION: { + AggInfo *pInfo = pExpr->pAggInfo; + if( pInfo==0 ){ + sqlite3ErrorMsg(pParse, "misuse of aggregate: %T", + &pExpr->span); + }else{ + sqlite3VdbeAddOp(v, OP_MemLoad, pInfo->aFunc[pExpr->iAgg].iMem, 0); + } + break; + } + case TK_CONST_FUNC: + case TK_FUNCTION: { + ExprList *pList = pExpr->pList; + int nExpr = pList ? pList->nExpr : 0; + FuncDef *pDef; + int nId; + const char *zId; + int constMask = 0; + int i; + u8 enc = ENC(pParse->db); + CollSeq *pColl = 0; + zId = (char*)pExpr->token.z; + nId = pExpr->token.n; + pDef = sqlite3FindFunction(pParse->db, zId, nId, nExpr, enc, 0); + assert( pDef!=0 ); + nExpr = sqlite3ExprCodeExprList(pParse, pList); + for(i=0; ia[i].pExpr) ){ + constMask |= (1<needCollSeq && !pColl ){ + pColl = sqlite3ExprCollSeq(pParse, pList->a[i].pExpr); + } + } + if( pDef->needCollSeq ){ + if( !pColl ) pColl = pParse->db->pDfltColl; + sqlite3VdbeOp3(v, OP_CollSeq, 0, 0, (char *)pColl, P3_COLLSEQ); + } + sqlite3VdbeOp3(v, OP_Function, constMask, nExpr, (char*)pDef, P3_FUNCDEF); + stackChng = 1-nExpr; + break; + } +#ifndef SQLITE_OMIT_SUBQUERY + case TK_EXISTS: + case TK_SELECT: { + sqlite3CodeSubselect(pParse, pExpr); + sqlite3VdbeAddOp(v, OP_MemLoad, pExpr->iColumn, 0); + VdbeComment((v, "# load subquery result")); + break; + } + case TK_IN: { + int addr; + char affinity; + int ckOffset = pParse->ckOffset; + sqlite3CodeSubselect(pParse, pExpr); + + /* Figure out the affinity to use to create a key from the results + ** of the expression. affinityStr stores a static string suitable for + ** P3 of OP_MakeRecord. + */ + affinity = comparisonAffinity(pExpr); + + sqlite3VdbeAddOp(v, OP_Integer, 1, 0); + pParse->ckOffset = ckOffset+1; + + /* Code the from " IN (...)". The temporary table + ** pExpr->iTable contains the values that make up the (...) set. + */ + sqlite3ExprCode(pParse, pExpr->pLeft); + addr = sqlite3VdbeCurrentAddr(v); + sqlite3VdbeAddOp(v, OP_NotNull, -1, addr+4); /* addr + 0 */ + sqlite3VdbeAddOp(v, OP_Pop, 2, 0); + sqlite3VdbeAddOp(v, OP_Null, 0, 0); + sqlite3VdbeAddOp(v, OP_Goto, 0, addr+7); + sqlite3VdbeOp3(v, OP_MakeRecord, 1, 0, &affinity, 1); /* addr + 4 */ + sqlite3VdbeAddOp(v, OP_Found, pExpr->iTable, addr+7); + sqlite3VdbeAddOp(v, OP_AddImm, -1, 0); /* addr + 6 */ + + break; + } +#endif + case TK_BETWEEN: { + Expr *pLeft = pExpr->pLeft; + struct ExprList_item *pLItem = pExpr->pList->a; + Expr *pRight = pLItem->pExpr; + sqlite3ExprCode(pParse, pLeft); + sqlite3VdbeAddOp(v, OP_Dup, 0, 0); + sqlite3ExprCode(pParse, pRight); + codeCompare(pParse, pLeft, pRight, OP_Ge, 0, 0); + sqlite3VdbeAddOp(v, OP_Pull, 1, 0); + pLItem++; + pRight = pLItem->pExpr; + sqlite3ExprCode(pParse, pRight); + codeCompare(pParse, pLeft, pRight, OP_Le, 0, 0); + sqlite3VdbeAddOp(v, OP_And, 0, 0); + break; + } + case TK_UPLUS: + case TK_AS: { + sqlite3ExprCode(pParse, pExpr->pLeft); + stackChng = 0; + break; + } + case TK_CASE: { + int expr_end_label; + int jumpInst; + int nExpr; + int i; + ExprList *pEList; + struct ExprList_item *aListelem; + + assert(pExpr->pList); + assert((pExpr->pList->nExpr % 2) == 0); + assert(pExpr->pList->nExpr > 0); + pEList = pExpr->pList; + aListelem = pEList->a; + nExpr = pEList->nExpr; + expr_end_label = sqlite3VdbeMakeLabel(v); + if( pExpr->pLeft ){ + sqlite3ExprCode(pParse, pExpr->pLeft); + } + for(i=0; ipLeft ){ + sqlite3VdbeAddOp(v, OP_Dup, 1, 1); + jumpInst = codeCompare(pParse, pExpr->pLeft, aListelem[i].pExpr, + OP_Ne, 0, 1); + sqlite3VdbeAddOp(v, OP_Pop, 1, 0); + }else{ + jumpInst = sqlite3VdbeAddOp(v, OP_IfNot, 1, 0); + } + sqlite3ExprCode(pParse, aListelem[i+1].pExpr); + sqlite3VdbeAddOp(v, OP_Goto, 0, expr_end_label); + sqlite3VdbeJumpHere(v, jumpInst); + } + if( pExpr->pLeft ){ + sqlite3VdbeAddOp(v, OP_Pop, 1, 0); + } + if( pExpr->pRight ){ + sqlite3ExprCode(pParse, pExpr->pRight); + }else{ + sqlite3VdbeAddOp(v, OP_Null, 0, 0); + } + sqlite3VdbeResolveLabel(v, expr_end_label); + break; + } +#ifndef SQLITE_OMIT_TRIGGER + case TK_RAISE: { + if( !pParse->trigStack ){ + sqlite3ErrorMsg(pParse, + "RAISE() may only be used within a trigger-program"); + return; + } + if( pExpr->iColumn!=OE_Ignore ){ + assert( pExpr->iColumn==OE_Rollback || + pExpr->iColumn == OE_Abort || + pExpr->iColumn == OE_Fail ); + sqlite3DequoteExpr(pExpr); + sqlite3VdbeOp3(v, OP_Halt, SQLITE_CONSTRAINT, pExpr->iColumn, + (char*)pExpr->token.z, pExpr->token.n); + } else { + assert( pExpr->iColumn == OE_Ignore ); + sqlite3VdbeAddOp(v, OP_ContextPop, 0, 0); + sqlite3VdbeAddOp(v, OP_Goto, 0, pParse->trigStack->ignoreJump); + VdbeComment((v, "# raise(IGNORE)")); + } + stackChng = 0; + break; + } +#endif + } + + if( pParse->ckOffset ){ + pParse->ckOffset += stackChng; + assert( pParse->ckOffset ); + } +} + +#ifndef SQLITE_OMIT_TRIGGER +/* +** Generate code that evalutes the given expression and leaves the result +** on the stack. See also sqlite3ExprCode(). +** +** This routine might also cache the result and modify the pExpr tree +** so that it will make use of the cached result on subsequent evaluations +** rather than evaluate the whole expression again. Trivial expressions are +** not cached. If the expression is cached, its result is stored in a +** memory location. +*/ +void sqlite3ExprCodeAndCache(Parse *pParse, Expr *pExpr){ + Vdbe *v = pParse->pVdbe; + int iMem; + int addr1, addr2; + if( v==0 ) return; + addr1 = sqlite3VdbeCurrentAddr(v); + sqlite3ExprCode(pParse, pExpr); + addr2 = sqlite3VdbeCurrentAddr(v); + if( addr2>addr1+1 || sqlite3VdbeGetOp(v, addr1)->opcode==OP_Function ){ + iMem = pExpr->iTable = pParse->nMem++; + sqlite3VdbeAddOp(v, OP_MemStore, iMem, 0); + pExpr->op = TK_REGISTER; + } +} +#endif + +/* +** Generate code that pushes the value of every element of the given +** expression list onto the stack. +** +** Return the number of elements pushed onto the stack. +*/ +int sqlite3ExprCodeExprList( + Parse *pParse, /* Parsing context */ + ExprList *pList /* The expression list to be coded */ +){ + struct ExprList_item *pItem; + int i, n; + if( pList==0 ) return 0; + n = pList->nExpr; + for(pItem=pList->a, i=n; i>0; i--, pItem++){ + sqlite3ExprCode(pParse, pItem->pExpr); + } + return n; +} + +/* +** Generate code for a boolean expression such that a jump is made +** to the label "dest" if the expression is true but execution +** continues straight thru if the expression is false. +** +** If the expression evaluates to NULL (neither true nor false), then +** take the jump if the jumpIfNull flag is true. +** +** This code depends on the fact that certain token values (ex: TK_EQ) +** are the same as opcode values (ex: OP_Eq) that implement the corresponding +** operation. Special comments in vdbe.c and the mkopcodeh.awk script in +** the make process cause these values to align. Assert()s in the code +** below verify that the numbers are aligned correctly. +*/ +void sqlite3ExprIfTrue(Parse *pParse, Expr *pExpr, int dest, int jumpIfNull){ + Vdbe *v = pParse->pVdbe; + int op = 0; + int ckOffset = pParse->ckOffset; + if( v==0 || pExpr==0 ) return; + op = pExpr->op; + switch( op ){ + case TK_AND: { + int d2 = sqlite3VdbeMakeLabel(v); + sqlite3ExprIfFalse(pParse, pExpr->pLeft, d2, !jumpIfNull); + sqlite3ExprIfTrue(pParse, pExpr->pRight, dest, jumpIfNull); + sqlite3VdbeResolveLabel(v, d2); + break; + } + case TK_OR: { + sqlite3ExprIfTrue(pParse, pExpr->pLeft, dest, jumpIfNull); + sqlite3ExprIfTrue(pParse, pExpr->pRight, dest, jumpIfNull); + break; + } + case TK_NOT: { + sqlite3ExprIfFalse(pParse, pExpr->pLeft, dest, jumpIfNull); + break; + } + case TK_LT: + case TK_LE: + case TK_GT: + case TK_GE: + case TK_NE: + case TK_EQ: { + assert( TK_LT==OP_Lt ); + assert( TK_LE==OP_Le ); + assert( TK_GT==OP_Gt ); + assert( TK_GE==OP_Ge ); + assert( TK_EQ==OP_Eq ); + assert( TK_NE==OP_Ne ); + sqlite3ExprCode(pParse, pExpr->pLeft); + sqlite3ExprCode(pParse, pExpr->pRight); + codeCompare(pParse, pExpr->pLeft, pExpr->pRight, op, dest, jumpIfNull); + break; + } + case TK_ISNULL: + case TK_NOTNULL: { + assert( TK_ISNULL==OP_IsNull ); + assert( TK_NOTNULL==OP_NotNull ); + sqlite3ExprCode(pParse, pExpr->pLeft); + sqlite3VdbeAddOp(v, op, 1, dest); + break; + } + case TK_BETWEEN: { + /* The expression "x BETWEEN y AND z" is implemented as: + ** + ** 1 IF (x < y) GOTO 3 + ** 2 IF (x <= z) GOTO + ** 3 ... + */ + int addr; + Expr *pLeft = pExpr->pLeft; + Expr *pRight = pExpr->pList->a[0].pExpr; + sqlite3ExprCode(pParse, pLeft); + sqlite3VdbeAddOp(v, OP_Dup, 0, 0); + sqlite3ExprCode(pParse, pRight); + addr = codeCompare(pParse, pLeft, pRight, OP_Lt, 0, !jumpIfNull); + + pRight = pExpr->pList->a[1].pExpr; + sqlite3ExprCode(pParse, pRight); + codeCompare(pParse, pLeft, pRight, OP_Le, dest, jumpIfNull); + + sqlite3VdbeAddOp(v, OP_Integer, 0, 0); + sqlite3VdbeJumpHere(v, addr); + sqlite3VdbeAddOp(v, OP_Pop, 1, 0); + break; + } + default: { + sqlite3ExprCode(pParse, pExpr); + sqlite3VdbeAddOp(v, OP_If, jumpIfNull, dest); + break; + } + } + pParse->ckOffset = ckOffset; +} + +/* +** Generate code for a boolean expression such that a jump is made +** to the label "dest" if the expression is false but execution +** continues straight thru if the expression is true. +** +** If the expression evaluates to NULL (neither true nor false) then +** jump if jumpIfNull is true or fall through if jumpIfNull is false. +*/ +void sqlite3ExprIfFalse(Parse *pParse, Expr *pExpr, int dest, int jumpIfNull){ + Vdbe *v = pParse->pVdbe; + int op = 0; + int ckOffset = pParse->ckOffset; + if( v==0 || pExpr==0 ) return; + + /* The value of pExpr->op and op are related as follows: + ** + ** pExpr->op op + ** --------- ---------- + ** TK_ISNULL OP_NotNull + ** TK_NOTNULL OP_IsNull + ** TK_NE OP_Eq + ** TK_EQ OP_Ne + ** TK_GT OP_Le + ** TK_LE OP_Gt + ** TK_GE OP_Lt + ** TK_LT OP_Ge + ** + ** For other values of pExpr->op, op is undefined and unused. + ** The value of TK_ and OP_ constants are arranged such that we + ** can compute the mapping above using the following expression. + ** Assert()s verify that the computation is correct. + */ + op = ((pExpr->op+(TK_ISNULL&1))^1)-(TK_ISNULL&1); + + /* Verify correct alignment of TK_ and OP_ constants + */ + assert( pExpr->op!=TK_ISNULL || op==OP_NotNull ); + assert( pExpr->op!=TK_NOTNULL || op==OP_IsNull ); + assert( pExpr->op!=TK_NE || op==OP_Eq ); + assert( pExpr->op!=TK_EQ || op==OP_Ne ); + assert( pExpr->op!=TK_LT || op==OP_Ge ); + assert( pExpr->op!=TK_LE || op==OP_Gt ); + assert( pExpr->op!=TK_GT || op==OP_Le ); + assert( pExpr->op!=TK_GE || op==OP_Lt ); + + switch( pExpr->op ){ + case TK_AND: { + sqlite3ExprIfFalse(pParse, pExpr->pLeft, dest, jumpIfNull); + sqlite3ExprIfFalse(pParse, pExpr->pRight, dest, jumpIfNull); + break; + } + case TK_OR: { + int d2 = sqlite3VdbeMakeLabel(v); + sqlite3ExprIfTrue(pParse, pExpr->pLeft, d2, !jumpIfNull); + sqlite3ExprIfFalse(pParse, pExpr->pRight, dest, jumpIfNull); + sqlite3VdbeResolveLabel(v, d2); + break; + } + case TK_NOT: { + sqlite3ExprIfTrue(pParse, pExpr->pLeft, dest, jumpIfNull); + break; + } + case TK_LT: + case TK_LE: + case TK_GT: + case TK_GE: + case TK_NE: + case TK_EQ: { + sqlite3ExprCode(pParse, pExpr->pLeft); + sqlite3ExprCode(pParse, pExpr->pRight); + codeCompare(pParse, pExpr->pLeft, pExpr->pRight, op, dest, jumpIfNull); + break; + } + case TK_ISNULL: + case TK_NOTNULL: { + sqlite3ExprCode(pParse, pExpr->pLeft); + sqlite3VdbeAddOp(v, op, 1, dest); + break; + } + case TK_BETWEEN: { + /* The expression is "x BETWEEN y AND z". It is implemented as: + ** + ** 1 IF (x >= y) GOTO 3 + ** 2 GOTO + ** 3 IF (x > z) GOTO + */ + int addr; + Expr *pLeft = pExpr->pLeft; + Expr *pRight = pExpr->pList->a[0].pExpr; + sqlite3ExprCode(pParse, pLeft); + sqlite3VdbeAddOp(v, OP_Dup, 0, 0); + sqlite3ExprCode(pParse, pRight); + addr = sqlite3VdbeCurrentAddr(v); + codeCompare(pParse, pLeft, pRight, OP_Ge, addr+3, !jumpIfNull); + + sqlite3VdbeAddOp(v, OP_Pop, 1, 0); + sqlite3VdbeAddOp(v, OP_Goto, 0, dest); + pRight = pExpr->pList->a[1].pExpr; + sqlite3ExprCode(pParse, pRight); + codeCompare(pParse, pLeft, pRight, OP_Gt, dest, jumpIfNull); + break; + } + default: { + sqlite3ExprCode(pParse, pExpr); + sqlite3VdbeAddOp(v, OP_IfNot, jumpIfNull, dest); + break; + } + } + pParse->ckOffset = ckOffset; +} + +/* +** Do a deep comparison of two expression trees. Return TRUE (non-zero) +** if they are identical and return FALSE if they differ in any way. +*/ +int sqlite3ExprCompare(Expr *pA, Expr *pB){ + int i; + if( pA==0||pB==0 ){ + return pB==pA; + } + if( pA->op!=pB->op ) return 0; + if( (pA->flags & EP_Distinct)!=(pB->flags & EP_Distinct) ) return 0; + if( !sqlite3ExprCompare(pA->pLeft, pB->pLeft) ) return 0; + if( !sqlite3ExprCompare(pA->pRight, pB->pRight) ) return 0; + if( pA->pList ){ + if( pB->pList==0 ) return 0; + if( pA->pList->nExpr!=pB->pList->nExpr ) return 0; + for(i=0; ipList->nExpr; i++){ + if( !sqlite3ExprCompare(pA->pList->a[i].pExpr, pB->pList->a[i].pExpr) ){ + return 0; + } + } + }else if( pB->pList ){ + return 0; + } + if( pA->pSelect || pB->pSelect ) return 0; + if( pA->iTable!=pB->iTable || pA->iColumn!=pB->iColumn ) return 0; + if( pA->token.z ){ + if( pB->token.z==0 ) return 0; + if( pB->token.n!=pA->token.n ) return 0; + if( sqlite3StrNICmp((char*)pA->token.z,(char*)pB->token.z,pB->token.n)!=0 ){ + return 0; + } + } + return 1; +} + + +/* +** Add a new element to the pAggInfo->aCol[] array. Return the index of +** the new element. Return a negative number if malloc fails. +*/ +static int addAggInfoColumn(AggInfo *pInfo){ + int i; + i = sqlite3ArrayAllocate((void**)&pInfo->aCol, sizeof(pInfo->aCol[0]), 3); + if( i<0 ){ + return -1; + } + return i; +} + +/* +** Add a new element to the pAggInfo->aFunc[] array. Return the index of +** the new element. Return a negative number if malloc fails. +*/ +static int addAggInfoFunc(AggInfo *pInfo){ + int i; + i = sqlite3ArrayAllocate((void**)&pInfo->aFunc, sizeof(pInfo->aFunc[0]), 2); + if( i<0 ){ + return -1; + } + return i; +} + +/* +** This is an xFunc for walkExprTree() used to implement +** sqlite3ExprAnalyzeAggregates(). See sqlite3ExprAnalyzeAggregates +** for additional information. +** +** This routine analyzes the aggregate function at pExpr. +*/ +static int analyzeAggregate(void *pArg, Expr *pExpr){ + int i; + NameContext *pNC = (NameContext *)pArg; + Parse *pParse = pNC->pParse; + SrcList *pSrcList = pNC->pSrcList; + AggInfo *pAggInfo = pNC->pAggInfo; + + + switch( pExpr->op ){ + case TK_COLUMN: { + /* Check to see if the column is in one of the tables in the FROM + ** clause of the aggregate query */ + if( pSrcList ){ + struct SrcList_item *pItem = pSrcList->a; + for(i=0; inSrc; i++, pItem++){ + struct AggInfo_col *pCol; + if( pExpr->iTable==pItem->iCursor ){ + /* If we reach this point, it means that pExpr refers to a table + ** that is in the FROM clause of the aggregate query. + ** + ** Make an entry for the column in pAggInfo->aCol[] if there + ** is not an entry there already. + */ + pCol = pAggInfo->aCol; + for(i=0; inColumn; i++, pCol++){ + if( pCol->iTable==pExpr->iTable && + pCol->iColumn==pExpr->iColumn ){ + break; + } + } + if( i>=pAggInfo->nColumn && (i = addAggInfoColumn(pAggInfo))>=0 ){ + pCol = &pAggInfo->aCol[i]; + pCol->iTable = pExpr->iTable; + pCol->iColumn = pExpr->iColumn; + pCol->iMem = pParse->nMem++; + pCol->iSorterColumn = -1; + pCol->pExpr = pExpr; + if( pAggInfo->pGroupBy ){ + int j, n; + ExprList *pGB = pAggInfo->pGroupBy; + struct ExprList_item *pTerm = pGB->a; + n = pGB->nExpr; + for(j=0; jpExpr; + if( pE->op==TK_COLUMN && pE->iTable==pExpr->iTable && + pE->iColumn==pExpr->iColumn ){ + pCol->iSorterColumn = j; + break; + } + } + } + if( pCol->iSorterColumn<0 ){ + pCol->iSorterColumn = pAggInfo->nSortingColumn++; + } + } + /* There is now an entry for pExpr in pAggInfo->aCol[] (either + ** because it was there before or because we just created it). + ** Convert the pExpr to be a TK_AGG_COLUMN referring to that + ** pAggInfo->aCol[] entry. + */ + pExpr->pAggInfo = pAggInfo; + pExpr->op = TK_AGG_COLUMN; + pExpr->iAgg = i; + break; + } /* endif pExpr->iTable==pItem->iCursor */ + } /* end loop over pSrcList */ + } + return 1; + } + case TK_AGG_FUNCTION: { + /* The pNC->nDepth==0 test causes aggregate functions in subqueries + ** to be ignored */ + if( pNC->nDepth==0 ){ + /* Check to see if pExpr is a duplicate of another aggregate + ** function that is already in the pAggInfo structure + */ + struct AggInfo_func *pItem = pAggInfo->aFunc; + for(i=0; inFunc; i++, pItem++){ + if( sqlite3ExprCompare(pItem->pExpr, pExpr) ){ + break; + } + } + if( i>=pAggInfo->nFunc ){ + /* pExpr is original. Make a new entry in pAggInfo->aFunc[] + */ + u8 enc = ENC(pParse->db); + i = addAggInfoFunc(pAggInfo); + if( i>=0 ){ + pItem = &pAggInfo->aFunc[i]; + pItem->pExpr = pExpr; + pItem->iMem = pParse->nMem++; + pItem->pFunc = sqlite3FindFunction(pParse->db, + (char*)pExpr->token.z, pExpr->token.n, + pExpr->pList ? pExpr->pList->nExpr : 0, enc, 0); + if( pExpr->flags & EP_Distinct ){ + pItem->iDistinct = pParse->nTab++; + }else{ + pItem->iDistinct = -1; + } + } + } + /* Make pExpr point to the appropriate pAggInfo->aFunc[] entry + */ + pExpr->iAgg = i; + pExpr->pAggInfo = pAggInfo; + return 1; + } + } + } + + /* Recursively walk subqueries looking for TK_COLUMN nodes that need + ** to be changed to TK_AGG_COLUMN. But increment nDepth so that + ** TK_AGG_FUNCTION nodes in subqueries will be unchanged. + */ + if( pExpr->pSelect ){ + pNC->nDepth++; + walkSelectExpr(pExpr->pSelect, analyzeAggregate, pNC); + pNC->nDepth--; + } + return 0; +} + +/* +** Analyze the given expression looking for aggregate functions and +** for variables that need to be added to the pParse->aAgg[] array. +** Make additional entries to the pParse->aAgg[] array as necessary. +** +** This routine should only be called after the expression has been +** analyzed by sqlite3ExprResolveNames(). +** +** If errors are seen, leave an error message in zErrMsg and return +** the number of errors. +*/ +int sqlite3ExprAnalyzeAggregates(NameContext *pNC, Expr *pExpr){ + int nErr = pNC->pParse->nErr; + walkExprTree(pExpr, analyzeAggregate, pNC); + return pNC->pParse->nErr - nErr; +} + +/* +** Call sqlite3ExprAnalyzeAggregates() for every expression in an +** expression list. Return the number of errors. +** +** If an error is found, the analysis is cut short. +*/ +int sqlite3ExprAnalyzeAggList(NameContext *pNC, ExprList *pList){ + struct ExprList_item *pItem; + int i; + int nErr = 0; + if( pList ){ + for(pItem=pList->a, i=0; nErr==0 && inExpr; i++, pItem++){ + nErr += sqlite3ExprAnalyzeAggregates(pNC, pItem->pExpr); + } + } + return nErr; +} Added: external/sqlite-source-3.3.4/func.c ============================================================================== --- (empty file) +++ external/sqlite-source-3.3.4/func.c Mon Apr 3 07:54:59 2006 @@ -0,0 +1,1137 @@ +/* +** 2002 February 23 +** +** The author disclaims copyright to this source code. In place of +** a legal notice, here is a blessing: +** +** May you do good and not evil. +** May you find forgiveness for yourself and forgive others. +** May you share freely, never taking more than you give. +** +************************************************************************* +** This file contains the C functions that implement various SQL +** functions of SQLite. +** +** There is only one exported symbol in this file - the function +** sqliteRegisterBuildinFunctions() found at the bottom of the file. +** All other code has file scope. +** +** $Id: func.c,v 1.121 2006/02/09 22:13:42 drh Exp $ +*/ +#include "sqliteInt.h" +#include +/* #include */ +#include +#include +#include "vdbeInt.h" +#include "os.h" + +/* +** Return the collating function associated with a function. +*/ +static CollSeq *sqlite3GetFuncCollSeq(sqlite3_context *context){ + return context->pColl; +} + +/* +** Implementation of the non-aggregate min() and max() functions +*/ +static void minmaxFunc( + sqlite3_context *context, + int argc, + sqlite3_value **argv +){ + int i; + int mask; /* 0 for min() or 0xffffffff for max() */ + int iBest; + CollSeq *pColl; + + if( argc==0 ) return; + mask = sqlite3_user_data(context)==0 ? 0 : -1; + pColl = sqlite3GetFuncCollSeq(context); + assert( pColl ); + assert( mask==-1 || mask==0 ); + iBest = 0; + if( sqlite3_value_type(argv[0])==SQLITE_NULL ) return; + for(i=1; i=0 ){ + iBest = i; + } + } + sqlite3_result_value(context, argv[iBest]); +} + +/* +** Return the type of the argument. +*/ +static void typeofFunc( + sqlite3_context *context, + int argc, + sqlite3_value **argv +){ + const char *z = 0; + switch( sqlite3_value_type(argv[0]) ){ + case SQLITE_NULL: z = "null"; break; + case SQLITE_INTEGER: z = "integer"; break; + case SQLITE_TEXT: z = "text"; break; + case SQLITE_FLOAT: z = "real"; break; + case SQLITE_BLOB: z = "blob"; break; + } + sqlite3_result_text(context, z, -1, SQLITE_STATIC); +} + + +/* +** Implementation of the length() function +*/ +static void lengthFunc( + sqlite3_context *context, + int argc, + sqlite3_value **argv +){ + int len; + + assert( argc==1 ); + switch( sqlite3_value_type(argv[0]) ){ + case SQLITE_BLOB: + case SQLITE_INTEGER: + case SQLITE_FLOAT: { + sqlite3_result_int(context, sqlite3_value_bytes(argv[0])); + break; + } + case SQLITE_TEXT: { + const unsigned char *z = sqlite3_value_text(argv[0]); + for(len=0; *z; z++){ if( (0xc0&*z)!=0x80 ) len++; } + sqlite3_result_int(context, len); + break; + } + default: { + sqlite3_result_null(context); + break; + } + } +} + +/* +** Implementation of the abs() function +*/ +static void absFunc(sqlite3_context *context, int argc, sqlite3_value **argv){ + assert( argc==1 ); + switch( sqlite3_value_type(argv[0]) ){ + case SQLITE_INTEGER: { + i64 iVal = sqlite3_value_int64(argv[0]); + if( iVal<0 ) iVal = iVal * -1; + sqlite3_result_int64(context, iVal); + break; + } + case SQLITE_NULL: { + sqlite3_result_null(context); + break; + } + default: { + double rVal = sqlite3_value_double(argv[0]); + if( rVal<0 ) rVal = rVal * -1.0; + sqlite3_result_double(context, rVal); + break; + } + } +} + +/* +** Implementation of the substr() function +*/ +static void substrFunc( + sqlite3_context *context, + int argc, + sqlite3_value **argv +){ + const unsigned char *z; + const unsigned char *z2; + int i; + int p1, p2, len; + + assert( argc==3 ); + z = sqlite3_value_text(argv[0]); + if( z==0 ) return; + p1 = sqlite3_value_int(argv[1]); + p2 = sqlite3_value_int(argv[2]); + for(len=0, z2=z; *z2; z2++){ if( (0xc0&*z2)!=0x80 ) len++; } + if( p1<0 ){ + p1 += len; + if( p1<0 ){ + p2 += p1; + p1 = 0; + } + }else if( p1>0 ){ + p1--; + } + if( p1+p2>len ){ + p2 = len-p1; + } + for(i=0; i30 ) n = 30; + if( n<0 ) n = 0; + } + if( SQLITE_NULL==sqlite3_value_type(argv[0]) ) return; + r = sqlite3_value_double(argv[0]); + sqlite3_snprintf(sizeof(zBuf),zBuf,"%.*f",n,r); + sqlite3_result_text(context, zBuf, -1, SQLITE_TRANSIENT); +} + +/* +** Implementation of the upper() and lower() SQL functions. +*/ +static void upperFunc(sqlite3_context *context, int argc, sqlite3_value **argv){ + unsigned char *z; + int i; + if( argc<1 || SQLITE_NULL==sqlite3_value_type(argv[0]) ) return; + z = sqliteMalloc(sqlite3_value_bytes(argv[0])+1); + if( z==0 ) return; + strcpy((char*)z, (char*)sqlite3_value_text(argv[0])); + for(i=0; z[i]; i++){ + z[i] = toupper(z[i]); + } + sqlite3_result_text(context, (char*)z, -1, SQLITE_TRANSIENT); + sqliteFree(z); +} +static void lowerFunc(sqlite3_context *context, int argc, sqlite3_value **argv){ + unsigned char *z; + int i; + if( argc<1 || SQLITE_NULL==sqlite3_value_type(argv[0]) ) return; + z = sqliteMalloc(sqlite3_value_bytes(argv[0])+1); + if( z==0 ) return; + strcpy((char*)z, (char*)sqlite3_value_text(argv[0])); + for(i=0; z[i]; i++){ + z[i] = tolower(z[i]); + } + sqlite3_result_text(context, (char*)z, -1, SQLITE_TRANSIENT); + sqliteFree(z); +} + +/* +** Implementation of the IFNULL(), NVL(), and COALESCE() functions. +** All three do the same thing. They return the first non-NULL +** argument. +*/ +static void ifnullFunc( + sqlite3_context *context, + int argc, + sqlite3_value **argv +){ + int i; + for(i=0; imatchOne; + u8 matchAll = pInfo->matchAll; + u8 matchSet = pInfo->matchSet; + u8 noCase = pInfo->noCase; + int prevEscape = 0; /* True if the previous character was 'escape' */ + + while( (c = *zPattern)!=0 ){ + if( !prevEscape && c==matchAll ){ + while( (c=zPattern[1]) == matchAll || c == matchOne ){ + if( c==matchOne ){ + if( *zString==0 ) return 0; + sqliteNextChar(zString); + } + zPattern++; + } + if( c && esc && sqlite3ReadUtf8(&zPattern[1])==esc ){ + u8 const *zTemp = &zPattern[1]; + sqliteNextChar(zTemp); + c = *zTemp; + } + if( c==0 ) return 1; + if( c==matchSet ){ + assert( esc==0 ); /* This is GLOB, not LIKE */ + while( *zString && patternCompare(&zPattern[1],zString,pInfo,esc)==0 ){ + sqliteNextChar(zString); + } + return *zString!=0; + }else{ + while( (c2 = *zString)!=0 ){ + if( noCase ){ + c2 = sqlite3UpperToLower[c2]; + c = sqlite3UpperToLower[c]; + while( c2 != 0 && c2 != c ){ c2 = sqlite3UpperToLower[*++zString]; } + }else{ + while( c2 != 0 && c2 != c ){ c2 = *++zString; } + } + if( c2==0 ) return 0; + if( patternCompare(&zPattern[1],zString,pInfo,esc) ) return 1; + sqliteNextChar(zString); + } + return 0; + } + }else if( !prevEscape && c==matchOne ){ + if( *zString==0 ) return 0; + sqliteNextChar(zString); + zPattern++; + }else if( c==matchSet ){ + int prior_c = 0; + assert( esc==0 ); /* This only occurs for GLOB, not LIKE */ + seen = 0; + invert = 0; + c = sqliteCharVal(zString); + if( c==0 ) return 0; + c2 = *++zPattern; + if( c2=='^' ){ invert = 1; c2 = *++zPattern; } + if( c2==']' ){ + if( c==']' ) seen = 1; + c2 = *++zPattern; + } + while( (c2 = sqliteCharVal(zPattern))!=0 && c2!=']' ){ + if( c2=='-' && zPattern[1]!=']' && zPattern[1]!=0 && prior_c>0 ){ + zPattern++; + c2 = sqliteCharVal(zPattern); + if( c>=prior_c && c<=c2 ) seen = 1; + prior_c = 0; + }else if( c==c2 ){ + seen = 1; + prior_c = c2; + }else{ + prior_c = c2; + } + sqliteNextChar(zPattern); + } + if( c2==0 || (seen ^ invert)==0 ) return 0; + sqliteNextChar(zString); + zPattern++; + }else if( esc && !prevEscape && sqlite3ReadUtf8(zPattern)==esc){ + prevEscape = 1; + sqliteNextChar(zPattern); + }else{ + if( noCase ){ + if( sqlite3UpperToLower[c] != sqlite3UpperToLower[*zString] ) return 0; + }else{ + if( c != *zString ) return 0; + } + zPattern++; + zString++; + prevEscape = 0; + } + } + return *zString==0; +} + +/* +** Count the number of times that the LIKE operator (or GLOB which is +** just a variation of LIKE) gets called. This is used for testing +** only. +*/ +#ifdef SQLITE_TEST +int sqlite3_like_count = 0; +#endif + + +/* +** Implementation of the like() SQL function. This function implements +** the build-in LIKE operator. The first argument to the function is the +** pattern and the second argument is the string. So, the SQL statements: +** +** A LIKE B +** +** is implemented as like(B,A). +** +** This same function (with a different compareInfo structure) computes +** the GLOB operator. +*/ +static void likeFunc( + sqlite3_context *context, + int argc, + sqlite3_value **argv +){ + const unsigned char *zA = sqlite3_value_text(argv[0]); + const unsigned char *zB = sqlite3_value_text(argv[1]); + int escape = 0; + if( argc==3 ){ + /* The escape character string must consist of a single UTF-8 character. + ** Otherwise, return an error. + */ + const unsigned char *zEsc = sqlite3_value_text(argv[2]); + if( sqlite3utf8CharLen((char*)zEsc, -1)!=1 ){ + sqlite3_result_error(context, + "ESCAPE expression must be a single character", -1); + return; + } + escape = sqlite3ReadUtf8(zEsc); + } + if( zA && zB ){ + struct compareInfo *pInfo = sqlite3_user_data(context); +#ifdef SQLITE_TEST + sqlite3_like_count++; +#endif + sqlite3_result_int(context, patternCompare(zA, zB, pInfo, escape)); + } +} + +/* +** Implementation of the NULLIF(x,y) function. The result is the first +** argument if the arguments are different. The result is NULL if the +** arguments are equal to each other. +*/ +static void nullifFunc( + sqlite3_context *context, + int argc, + sqlite3_value **argv +){ + CollSeq *pColl = sqlite3GetFuncCollSeq(context); + if( sqlite3MemCompare(argv[0], argv[1], pColl)!=0 ){ + sqlite3_result_value(context, argv[0]); + } +} + +/* +** Implementation of the VERSION(*) function. The result is the version +** of the SQLite library that is running. +*/ +static void versionFunc( + sqlite3_context *context, + int argc, + sqlite3_value **argv +){ + sqlite3_result_text(context, sqlite3_version, -1, SQLITE_STATIC); +} + + +/* +** EXPERIMENTAL - This is not an official function. The interface may +** change. This function may disappear. Do not write code that depends +** on this function. +** +** Implementation of the QUOTE() function. This function takes a single +** argument. If the argument is numeric, the return value is the same as +** the argument. If the argument is NULL, the return value is the string +** "NULL". Otherwise, the argument is enclosed in single quotes with +** single-quote escapes. +*/ +static void quoteFunc(sqlite3_context *context, int argc, sqlite3_value **argv){ + if( argc<1 ) return; + switch( sqlite3_value_type(argv[0]) ){ + case SQLITE_NULL: { + sqlite3_result_text(context, "NULL", 4, SQLITE_STATIC); + break; + } + case SQLITE_INTEGER: + case SQLITE_FLOAT: { + sqlite3_result_value(context, argv[0]); + break; + } + case SQLITE_BLOB: { + static const char hexdigits[] = { + '0', '1', '2', '3', '4', '5', '6', '7', + '8', '9', 'A', 'B', 'C', 'D', 'E', 'F' + }; + char *zText = 0; + int nBlob = sqlite3_value_bytes(argv[0]); + char const *zBlob = sqlite3_value_blob(argv[0]); + + zText = (char *)sqliteMalloc((2*nBlob)+4); + if( !zText ){ + sqlite3_result_error(context, "out of memory", -1); + }else{ + int i; + for(i=0; i>4)&0x0F]; + zText[(i*2)+3] = hexdigits[(zBlob[i])&0x0F]; + } + zText[(nBlob*2)+2] = '\''; + zText[(nBlob*2)+3] = '\0'; + zText[0] = 'X'; + zText[1] = '\''; + sqlite3_result_text(context, zText, -1, SQLITE_TRANSIENT); + sqliteFree(zText); + } + break; + } + case SQLITE_TEXT: { + int i,j,n; + const unsigned char *zArg = sqlite3_value_text(argv[0]); + char *z; + + for(i=n=0; zArg[i]; i++){ if( zArg[i]=='\'' ) n++; } + z = sqliteMalloc( i+n+3 ); + if( z==0 ) return; + z[0] = '\''; + for(i=0, j=1; zArg[i]; i++){ + z[j++] = zArg[i]; + if( zArg[i]=='\'' ){ + z[j++] = '\''; + } + } + z[j++] = '\''; + z[j] = 0; + sqlite3_result_text(context, z, j, SQLITE_TRANSIENT); + sqliteFree(z); + } + } +} + +#ifdef SQLITE_SOUNDEX +/* +** Compute the soundex encoding of a word. +*/ +static void soundexFunc(sqlite3_context *context, int argc, sqlite3_value **argv){ + char zResult[8]; + const u8 *zIn; + int i, j; + static const unsigned char iCode[] = { + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 1, 2, 3, 0, 1, 2, 0, 0, 2, 2, 4, 5, 5, 0, + 1, 2, 6, 2, 3, 0, 1, 0, 2, 0, 2, 0, 0, 0, 0, 0, + 0, 0, 1, 2, 3, 0, 1, 2, 0, 0, 2, 2, 4, 5, 5, 0, + 1, 2, 6, 2, 3, 0, 1, 0, 2, 0, 2, 0, 0, 0, 0, 0, + }; + assert( argc==1 ); + zIn = (u8*)sqlite3_value_text(argv[0]); + for(i=0; zIn[i] && !isalpha(zIn[i]); i++){} + if( zIn[i] ){ + zResult[0] = toupper(zIn[i]); + for(j=1; j<4 && zIn[i]; i++){ + int code = iCode[zIn[i]&0x7f]; + if( code>0 ){ + zResult[j++] = code + '0'; + } + } + while( j<4 ){ + zResult[j++] = '0'; + } + zResult[j] = 0; + sqlite3_result_text(context, zResult, 4, SQLITE_TRANSIENT); + }else{ + sqlite3_result_text(context, "?000", 4, SQLITE_STATIC); + } +} +#endif + +#ifdef SQLITE_TEST +/* +** This function generates a string of random characters. Used for +** generating test data. +*/ +static void randStr(sqlite3_context *context, int argc, sqlite3_value **argv){ + static const unsigned char zSrc[] = + "abcdefghijklmnopqrstuvwxyz" + "ABCDEFGHIJKLMNOPQRSTUVWXYZ" + "0123456789" + ".-!,:*^+=_|?/<> "; + int iMin, iMax, n, r, i; + unsigned char zBuf[1000]; + if( argc>=1 ){ + iMin = sqlite3_value_int(argv[0]); + if( iMin<0 ) iMin = 0; + if( iMin>=sizeof(zBuf) ) iMin = sizeof(zBuf)-1; + }else{ + iMin = 1; + } + if( argc>=2 ){ + iMax = sqlite3_value_int(argv[1]); + if( iMax=sizeof(zBuf) ) iMax = sizeof(zBuf)-1; + }else{ + iMax = 50; + } + n = iMin; + if( iMax>iMin ){ + sqlite3Randomness(sizeof(r), &r); + r &= 0x7fffffff; + n += r%(iMax + 1 - iMin); + } + assert( ncnt++; + if( type==SQLITE_INTEGER ){ + p->sum += sqlite3_value_int64(argv[0]); + if( !p->approx ){ + i64 iVal; + p->approx = p->sum!=(LONGDOUBLE_TYPE)(iVal = (i64)p->sum); + } + }else{ + p->sum += sqlite3_value_double(argv[0]); + p->approx = 1; + } + } +} +static void sumFinalize(sqlite3_context *context){ + SumCtx *p; + p = sqlite3_aggregate_context(context, 0); + if( p && p->cnt>0 ){ + if( p->approx ){ + sqlite3_result_double(context, p->sum); + }else{ + sqlite3_result_int64(context, (i64)p->sum); + } + } +} +static void avgFinalize(sqlite3_context *context){ + SumCtx *p; + p = sqlite3_aggregate_context(context, 0); + if( p && p->cnt>0 ){ + sqlite3_result_double(context, p->sum/(double)p->cnt); + } +} +static void totalFinalize(sqlite3_context *context){ + SumCtx *p; + p = sqlite3_aggregate_context(context, 0); + sqlite3_result_double(context, p ? p->sum : 0.0); +} + +/* +** The following structure keeps track of state information for the +** count() aggregate function. +*/ +typedef struct CountCtx CountCtx; +struct CountCtx { + i64 n; +}; + +/* +** Routines to implement the count() aggregate function. +*/ +static void countStep(sqlite3_context *context, int argc, sqlite3_value **argv){ + CountCtx *p; + p = sqlite3_aggregate_context(context, sizeof(*p)); + if( (argc==0 || SQLITE_NULL!=sqlite3_value_type(argv[0])) && p ){ + p->n++; + } +} +static void countFinalize(sqlite3_context *context){ + CountCtx *p; + p = sqlite3_aggregate_context(context, 0); + sqlite3_result_int64(context, p ? p->n : 0); +} + +/* +** Routines to implement min() and max() aggregate functions. +*/ +static void minmaxStep(sqlite3_context *context, int argc, sqlite3_value **argv){ + Mem *pArg = (Mem *)argv[0]; + Mem *pBest; + + if( sqlite3_value_type(argv[0])==SQLITE_NULL ) return; + pBest = (Mem *)sqlite3_aggregate_context(context, sizeof(*pBest)); + if( !pBest ) return; + + if( pBest->flags ){ + int max; + int cmp; + CollSeq *pColl = sqlite3GetFuncCollSeq(context); + /* This step function is used for both the min() and max() aggregates, + ** the only difference between the two being that the sense of the + ** comparison is inverted. For the max() aggregate, the + ** sqlite3_user_data() function returns (void *)-1. For min() it + ** returns (void *)db, where db is the sqlite3* database pointer. + ** Therefore the next statement sets variable 'max' to 1 for the max() + ** aggregate, or 0 for min(). + */ + max = ((sqlite3_user_data(context)==(void *)-1)?1:0); + cmp = sqlite3MemCompare(pBest, pArg, pColl); + if( (max && cmp<0) || (!max && cmp>0) ){ + sqlite3VdbeMemCopy(pBest, pArg); + } + }else{ + sqlite3VdbeMemCopy(pBest, pArg); + } +} +static void minMaxFinalize(sqlite3_context *context){ + sqlite3_value *pRes; + pRes = (sqlite3_value *)sqlite3_aggregate_context(context, 0); + if( pRes ){ + if( pRes->flags ){ + sqlite3_result_value(context, pRes); + } + sqlite3VdbeMemRelease(pRes); + } +} + + +/* +** This function registered all of the above C functions as SQL +** functions. This should be the only routine in this file with +** external linkage. +*/ +void sqlite3RegisterBuiltinFunctions(sqlite3 *db){ + static const struct { + char *zName; + signed char nArg; + u8 argType; /* 0: none. 1: db 2: (-1) */ + u8 eTextRep; /* 1: UTF-16. 0: UTF-8 */ + u8 needCollSeq; + void (*xFunc)(sqlite3_context*,int,sqlite3_value **); + } aFuncs[] = { + { "min", -1, 0, SQLITE_UTF8, 1, minmaxFunc }, + { "min", 0, 0, SQLITE_UTF8, 1, 0 }, + { "max", -1, 2, SQLITE_UTF8, 1, minmaxFunc }, + { "max", 0, 2, SQLITE_UTF8, 1, 0 }, + { "typeof", 1, 0, SQLITE_UTF8, 0, typeofFunc }, + { "length", 1, 0, SQLITE_UTF8, 0, lengthFunc }, + { "substr", 3, 0, SQLITE_UTF8, 0, substrFunc }, +#ifndef SQLITE_OMIT_UTF16 + { "substr", 3, 0, SQLITE_UTF16LE, 0, sqlite3utf16Substr }, +#endif + { "abs", 1, 0, SQLITE_UTF8, 0, absFunc }, + { "round", 1, 0, SQLITE_UTF8, 0, roundFunc }, + { "round", 2, 0, SQLITE_UTF8, 0, roundFunc }, + { "upper", 1, 0, SQLITE_UTF8, 0, upperFunc }, + { "lower", 1, 0, SQLITE_UTF8, 0, lowerFunc }, + { "coalesce", -1, 0, SQLITE_UTF8, 0, ifnullFunc }, + { "coalesce", 0, 0, SQLITE_UTF8, 0, 0 }, + { "coalesce", 1, 0, SQLITE_UTF8, 0, 0 }, + { "ifnull", 2, 0, SQLITE_UTF8, 1, ifnullFunc }, + { "random", -1, 0, SQLITE_UTF8, 0, randomFunc }, + { "nullif", 2, 0, SQLITE_UTF8, 1, nullifFunc }, + { "sqlite_version", 0, 0, SQLITE_UTF8, 0, versionFunc}, + { "quote", 1, 0, SQLITE_UTF8, 0, quoteFunc }, + { "last_insert_rowid", 0, 1, SQLITE_UTF8, 0, last_insert_rowid }, + { "changes", 0, 1, SQLITE_UTF8, 0, changes }, + { "total_changes", 0, 1, SQLITE_UTF8, 0, total_changes }, +#ifdef SQLITE_SOUNDEX + { "soundex", 1, 0, SQLITE_UTF8, 0, soundexFunc}, +#endif +#ifdef SQLITE_TEST + { "randstr", 2, 0, SQLITE_UTF8, 0, randStr }, + { "test_destructor", 1, 1, SQLITE_UTF8, 0, test_destructor}, + { "test_destructor_count", 0, 0, SQLITE_UTF8, 0, test_destructor_count}, + { "test_auxdata", -1, 0, SQLITE_UTF8, 0, test_auxdata}, + { "test_error", 1, 0, SQLITE_UTF8, 0, test_error}, +#endif + }; + static const struct { + char *zName; + signed char nArg; + u8 argType; + u8 needCollSeq; + void (*xStep)(sqlite3_context*,int,sqlite3_value**); + void (*xFinalize)(sqlite3_context*); + } aAggs[] = { + { "min", 1, 0, 1, minmaxStep, minMaxFinalize }, + { "max", 1, 2, 1, minmaxStep, minMaxFinalize }, + { "sum", 1, 0, 0, sumStep, sumFinalize }, + { "total", 1, 0, 0, sumStep, totalFinalize }, + { "avg", 1, 0, 0, sumStep, avgFinalize }, + { "count", 0, 0, 0, countStep, countFinalize }, + { "count", 1, 0, 0, countStep, countFinalize }, + }; + int i; + + for(i=0; ineedCollSeq = 1; + } + } + } +#ifndef SQLITE_OMIT_ALTERTABLE + sqlite3AlterFunctions(db); +#endif +#ifndef SQLITE_OMIT_PARSER + sqlite3AttachFunctions(db); +#endif + for(i=0; ineedCollSeq = 1; + } + } + } + sqlite3RegisterDateTimeFunctions(db); +#ifdef SQLITE_SSE + sqlite3SseFunctions(db); +#endif +#ifdef SQLITE_CASE_SENSITIVE_LIKE + sqlite3RegisterLikeFunctions(db, 1); +#else + sqlite3RegisterLikeFunctions(db, 0); +#endif +} + +/* +** Set the LIKEOPT flag on the 2-argument function with the given name. +*/ +static void setLikeOptFlag(sqlite3 *db, const char *zName, int flagVal){ + FuncDef *pDef; + pDef = sqlite3FindFunction(db, zName, strlen(zName), 2, SQLITE_UTF8, 0); + if( pDef ){ + pDef->flags = flagVal; + } +} + +/* +** Register the built-in LIKE and GLOB functions. The caseSensitive +** parameter determines whether or not the LIKE operator is case +** sensitive. GLOB is always case sensitive. +*/ +void sqlite3RegisterLikeFunctions(sqlite3 *db, int caseSensitive){ + struct compareInfo *pInfo; + if( caseSensitive ){ + pInfo = (struct compareInfo*)&likeInfoAlt; + }else{ + pInfo = (struct compareInfo*)&likeInfoNorm; + } + sqlite3CreateFunc(db, "like", 2, SQLITE_UTF8, pInfo, likeFunc, 0, 0); + sqlite3CreateFunc(db, "like", 3, SQLITE_UTF8, pInfo, likeFunc, 0, 0); + sqlite3CreateFunc(db, "glob", 2, SQLITE_UTF8, + (struct compareInfo*)&globInfo, likeFunc, 0,0); + setLikeOptFlag(db, "glob", SQLITE_FUNC_LIKE | SQLITE_FUNC_CASE); + setLikeOptFlag(db, "like", + caseSensitive ? (SQLITE_FUNC_LIKE | SQLITE_FUNC_CASE) : SQLITE_FUNC_LIKE); +} + +/* +** pExpr points to an expression which implements a function. If +** it is appropriate to apply the LIKE optimization to that function +** then set aWc[0] through aWc[2] to the wildcard characters and +** return TRUE. If the function is not a LIKE-style function then +** return FALSE. +*/ +int sqlite3IsLikeFunction(sqlite3 *db, Expr *pExpr, int *pIsNocase, char *aWc){ + FuncDef *pDef; + if( pExpr->op!=TK_FUNCTION ){ + return 0; + } + if( pExpr->pList->nExpr!=2 ){ + return 0; + } + pDef = sqlite3FindFunction(db, (char*)pExpr->token.z, pExpr->token.n, 2, + SQLITE_UTF8, 0); + if( pDef==0 || (pDef->flags & SQLITE_FUNC_LIKE)==0 ){ + return 0; + } + + /* The memcpy() statement assumes that the wildcard characters are + ** the first three statements in the compareInfo structure. The + ** asserts() that follow verify that assumption + */ + memcpy(aWc, pDef->pUserData, 3); + assert( (char*)&likeInfoAlt == (char*)&likeInfoAlt.matchAll ); + assert( &((char*)&likeInfoAlt)[1] == (char*)&likeInfoAlt.matchOne ); + assert( &((char*)&likeInfoAlt)[2] == (char*)&likeInfoAlt.matchSet ); + *pIsNocase = (pDef->flags & SQLITE_FUNC_CASE)==0; + return 1; +} Added: external/sqlite-source-3.3.4/hash.c ============================================================================== --- (empty file) +++ external/sqlite-source-3.3.4/hash.c Mon Apr 3 07:54:59 2006 @@ -0,0 +1,392 @@ +/* +** 2001 September 22 +** +** The author disclaims copyright to this source code. In place of +** a legal notice, here is a blessing: +** +** May you do good and not evil. +** May you find forgiveness for yourself and forgive others. +** May you share freely, never taking more than you give. +** +************************************************************************* +** This is the implementation of generic hash-tables +** used in SQLite. +** +** $Id: hash.c,v 1.17 2005/10/03 15:11:09 drh Exp $ +*/ +#include "sqliteInt.h" +#include + +/* Turn bulk memory into a hash table object by initializing the +** fields of the Hash structure. +** +** "pNew" is a pointer to the hash table that is to be initialized. +** keyClass is one of the constants SQLITE_HASH_INT, SQLITE_HASH_POINTER, +** SQLITE_HASH_BINARY, or SQLITE_HASH_STRING. The value of keyClass +** determines what kind of key the hash table will use. "copyKey" is +** true if the hash table should make its own private copy of keys and +** false if it should just use the supplied pointer. CopyKey only makes +** sense for SQLITE_HASH_STRING and SQLITE_HASH_BINARY and is ignored +** for other key classes. +*/ +void sqlite3HashInit(Hash *pNew, int keyClass, int copyKey){ + assert( pNew!=0 ); + assert( keyClass>=SQLITE_HASH_STRING && keyClass<=SQLITE_HASH_BINARY ); + pNew->keyClass = keyClass; +#if 0 + if( keyClass==SQLITE_HASH_POINTER || keyClass==SQLITE_HASH_INT ) copyKey = 0; +#endif + pNew->copyKey = copyKey; + pNew->first = 0; + pNew->count = 0; + pNew->htsize = 0; + pNew->ht = 0; +} + +/* Remove all entries from a hash table. Reclaim all memory. +** Call this routine to delete a hash table or to reset a hash table +** to the empty state. +*/ +void sqlite3HashClear(Hash *pH){ + HashElem *elem; /* For looping over all elements of the table */ + + assert( pH!=0 ); + elem = pH->first; + pH->first = 0; + if( pH->ht ) sqliteFree(pH->ht); + pH->ht = 0; + pH->htsize = 0; + while( elem ){ + HashElem *next_elem = elem->next; + if( pH->copyKey && elem->pKey ){ + sqliteFree(elem->pKey); + } + sqliteFree(elem); + elem = next_elem; + } + pH->count = 0; +} + +#if 0 /* NOT USED */ +/* +** Hash and comparison functions when the mode is SQLITE_HASH_INT +*/ +static int intHash(const void *pKey, int nKey){ + return nKey ^ (nKey<<8) ^ (nKey>>8); +} +static int intCompare(const void *pKey1, int n1, const void *pKey2, int n2){ + return n2 - n1; +} +#endif + +#if 0 /* NOT USED */ +/* +** Hash and comparison functions when the mode is SQLITE_HASH_POINTER +*/ +static int ptrHash(const void *pKey, int nKey){ + uptr x = Addr(pKey); + return x ^ (x<<8) ^ (x>>8); +} +static int ptrCompare(const void *pKey1, int n1, const void *pKey2, int n2){ + if( pKey1==pKey2 ) return 0; + if( pKey1 0 ){ + h = (h<<3) ^ h ^ sqlite3UpperToLower[(unsigned char)*z++]; + nKey--; + } + return h & 0x7fffffff; +} +static int strCompare(const void *pKey1, int n1, const void *pKey2, int n2){ + if( n1!=n2 ) return 1; + return sqlite3StrNICmp((const char*)pKey1,(const char*)pKey2,n1); +} + +/* +** Hash and comparison functions when the mode is SQLITE_HASH_BINARY +*/ +static int binHash(const void *pKey, int nKey){ + int h = 0; + const char *z = (const char *)pKey; + while( nKey-- > 0 ){ + h = (h<<3) ^ h ^ *(z++); + } + return h & 0x7fffffff; +} +static int binCompare(const void *pKey1, int n1, const void *pKey2, int n2){ + if( n1!=n2 ) return 1; + return memcmp(pKey1,pKey2,n1); +} + +/* +** Return a pointer to the appropriate hash function given the key class. +** +** The C syntax in this function definition may be unfamilar to some +** programmers, so we provide the following additional explanation: +** +** The name of the function is "hashFunction". The function takes a +** single parameter "keyClass". The return value of hashFunction() +** is a pointer to another function. Specifically, the return value +** of hashFunction() is a pointer to a function that takes two parameters +** with types "const void*" and "int" and returns an "int". +*/ +static int (*hashFunction(int keyClass))(const void*,int){ +#if 0 /* HASH_INT and HASH_POINTER are never used */ + switch( keyClass ){ + case SQLITE_HASH_INT: return &intHash; + case SQLITE_HASH_POINTER: return &ptrHash; + case SQLITE_HASH_STRING: return &strHash; + case SQLITE_HASH_BINARY: return &binHash;; + default: break; + } + return 0; +#else + if( keyClass==SQLITE_HASH_STRING ){ + return &strHash; + }else{ + assert( keyClass==SQLITE_HASH_BINARY ); + return &binHash; + } +#endif +} + +/* +** Return a pointer to the appropriate hash function given the key class. +** +** For help in interpreted the obscure C code in the function definition, +** see the header comment on the previous function. +*/ +static int (*compareFunction(int keyClass))(const void*,int,const void*,int){ +#if 0 /* HASH_INT and HASH_POINTER are never used */ + switch( keyClass ){ + case SQLITE_HASH_INT: return &intCompare; + case SQLITE_HASH_POINTER: return &ptrCompare; + case SQLITE_HASH_STRING: return &strCompare; + case SQLITE_HASH_BINARY: return &binCompare; + default: break; + } + return 0; +#else + if( keyClass==SQLITE_HASH_STRING ){ + return &strCompare; + }else{ + assert( keyClass==SQLITE_HASH_BINARY ); + return &binCompare; + } +#endif +} + +/* Link an element into the hash table +*/ +static void insertElement( + Hash *pH, /* The complete hash table */ + struct _ht *pEntry, /* The entry into which pNew is inserted */ + HashElem *pNew /* The element to be inserted */ +){ + HashElem *pHead; /* First element already in pEntry */ + pHead = pEntry->chain; + if( pHead ){ + pNew->next = pHead; + pNew->prev = pHead->prev; + if( pHead->prev ){ pHead->prev->next = pNew; } + else { pH->first = pNew; } + pHead->prev = pNew; + }else{ + pNew->next = pH->first; + if( pH->first ){ pH->first->prev = pNew; } + pNew->prev = 0; + pH->first = pNew; + } + pEntry->count++; + pEntry->chain = pNew; +} + + +/* Resize the hash table so that it cantains "new_size" buckets. +** "new_size" must be a power of 2. The hash table might fail +** to resize if sqliteMalloc() fails. +*/ +static void rehash(Hash *pH, int new_size){ + struct _ht *new_ht; /* The new hash table */ + HashElem *elem, *next_elem; /* For looping over existing elements */ + int (*xHash)(const void*,int); /* The hash function */ + + assert( (new_size & (new_size-1))==0 ); + new_ht = (struct _ht *)sqliteMalloc( new_size*sizeof(struct _ht) ); + if( new_ht==0 ) return; + if( pH->ht ) sqliteFree(pH->ht); + pH->ht = new_ht; + pH->htsize = new_size; + xHash = hashFunction(pH->keyClass); + for(elem=pH->first, pH->first=0; elem; elem = next_elem){ + int h = (*xHash)(elem->pKey, elem->nKey) & (new_size-1); + next_elem = elem->next; + insertElement(pH, &new_ht[h], elem); + } +} + +/* This function (for internal use only) locates an element in an +** hash table that matches the given key. The hash for this key has +** already been computed and is passed as the 4th parameter. +*/ +static HashElem *findElementGivenHash( + const Hash *pH, /* The pH to be searched */ + const void *pKey, /* The key we are searching for */ + int nKey, + int h /* The hash for this key. */ +){ + HashElem *elem; /* Used to loop thru the element list */ + int count; /* Number of elements left to test */ + int (*xCompare)(const void*,int,const void*,int); /* comparison function */ + + if( pH->ht ){ + struct _ht *pEntry = &pH->ht[h]; + elem = pEntry->chain; + count = pEntry->count; + xCompare = compareFunction(pH->keyClass); + while( count-- && elem ){ + if( (*xCompare)(elem->pKey,elem->nKey,pKey,nKey)==0 ){ + return elem; + } + elem = elem->next; + } + } + return 0; +} + +/* Remove a single entry from the hash table given a pointer to that +** element and a hash on the element's key. +*/ +static void removeElementGivenHash( + Hash *pH, /* The pH containing "elem" */ + HashElem* elem, /* The element to be removed from the pH */ + int h /* Hash value for the element */ +){ + struct _ht *pEntry; + if( elem->prev ){ + elem->prev->next = elem->next; + }else{ + pH->first = elem->next; + } + if( elem->next ){ + elem->next->prev = elem->prev; + } + pEntry = &pH->ht[h]; + if( pEntry->chain==elem ){ + pEntry->chain = elem->next; + } + pEntry->count--; + if( pEntry->count<=0 ){ + pEntry->chain = 0; + } + if( pH->copyKey && elem->pKey ){ + sqliteFree(elem->pKey); + } + sqliteFree( elem ); + pH->count--; + if( pH->count<=0 ){ + assert( pH->first==0 ); + assert( pH->count==0 ); + sqlite3HashClear(pH); + } +} + +/* Attempt to locate an element of the hash table pH with a key +** that matches pKey,nKey. Return the data for this element if it is +** found, or NULL if there is no match. +*/ +void *sqlite3HashFind(const Hash *pH, const void *pKey, int nKey){ + int h; /* A hash on key */ + HashElem *elem; /* The element that matches key */ + int (*xHash)(const void*,int); /* The hash function */ + + if( pH==0 || pH->ht==0 ) return 0; + xHash = hashFunction(pH->keyClass); + assert( xHash!=0 ); + h = (*xHash)(pKey,nKey); + assert( (pH->htsize & (pH->htsize-1))==0 ); + elem = findElementGivenHash(pH,pKey,nKey, h & (pH->htsize-1)); + return elem ? elem->data : 0; +} + +/* Insert an element into the hash table pH. The key is pKey,nKey +** and the data is "data". +** +** If no element exists with a matching key, then a new +** element is created. A copy of the key is made if the copyKey +** flag is set. NULL is returned. +** +** If another element already exists with the same key, then the +** new data replaces the old data and the old data is returned. +** The key is not copied in this instance. If a malloc fails, then +** the new data is returned and the hash table is unchanged. +** +** If the "data" parameter to this function is NULL, then the +** element corresponding to "key" is removed from the hash table. +*/ +void *sqlite3HashInsert(Hash *pH, const void *pKey, int nKey, void *data){ + int hraw; /* Raw hash value of the key */ + int h; /* the hash of the key modulo hash table size */ + HashElem *elem; /* Used to loop thru the element list */ + HashElem *new_elem; /* New element added to the pH */ + int (*xHash)(const void*,int); /* The hash function */ + + assert( pH!=0 ); + xHash = hashFunction(pH->keyClass); + assert( xHash!=0 ); + hraw = (*xHash)(pKey, nKey); + assert( (pH->htsize & (pH->htsize-1))==0 ); + h = hraw & (pH->htsize-1); + elem = findElementGivenHash(pH,pKey,nKey,h); + if( elem ){ + void *old_data = elem->data; + if( data==0 ){ + removeElementGivenHash(pH,elem,h); + }else{ + elem->data = data; + } + return old_data; + } + if( data==0 ) return 0; + new_elem = (HashElem*)sqliteMalloc( sizeof(HashElem) ); + if( new_elem==0 ) return data; + if( pH->copyKey && pKey!=0 ){ + new_elem->pKey = sqliteMallocRaw( nKey ); + if( new_elem->pKey==0 ){ + sqliteFree(new_elem); + return data; + } + memcpy((void*)new_elem->pKey, pKey, nKey); + }else{ + new_elem->pKey = (void*)pKey; + } + new_elem->nKey = nKey; + pH->count++; + if( pH->htsize==0 ){ + rehash(pH,8); + if( pH->htsize==0 ){ + pH->count = 0; + sqliteFree(new_elem); + return data; + } + } + if( pH->count > pH->htsize ){ + rehash(pH,pH->htsize*2); + } + assert( pH->htsize>0 ); + assert( (pH->htsize & (pH->htsize-1))==0 ); + h = hraw & (pH->htsize-1); + insertElement(pH, &pH->ht[h], new_elem); + new_elem->data = data; + return 0; +} Added: external/sqlite-source-3.3.4/hash.h ============================================================================== --- (empty file) +++ external/sqlite-source-3.3.4/hash.h Mon Apr 3 07:54:59 2006 @@ -0,0 +1,109 @@ +/* +** 2001 September 22 +** +** The author disclaims copyright to this source code. In place of +** a legal notice, here is a blessing: +** +** May you do good and not evil. +** May you find forgiveness for yourself and forgive others. +** May you share freely, never taking more than you give. +** +************************************************************************* +** This is the header file for the generic hash-table implemenation +** used in SQLite. +** +** $Id: hash.h,v 1.8 2004/08/20 14:08:51 drh Exp $ +*/ +#ifndef _SQLITE_HASH_H_ +#define _SQLITE_HASH_H_ + +/* Forward declarations of structures. */ +typedef struct Hash Hash; +typedef struct HashElem HashElem; + +/* A complete hash table is an instance of the following structure. +** The internals of this structure are intended to be opaque -- client +** code should not attempt to access or modify the fields of this structure +** directly. Change this structure only by using the routines below. +** However, many of the "procedures" and "functions" for modifying and +** accessing this structure are really macros, so we can't really make +** this structure opaque. +*/ +struct Hash { + char keyClass; /* SQLITE_HASH_INT, _POINTER, _STRING, _BINARY */ + char copyKey; /* True if copy of key made on insert */ + int count; /* Number of entries in this table */ + HashElem *first; /* The first element of the array */ + int htsize; /* Number of buckets in the hash table */ + struct _ht { /* the hash table */ + int count; /* Number of entries with this hash */ + HashElem *chain; /* Pointer to first entry with this hash */ + } *ht; +}; + +/* Each element in the hash table is an instance of the following +** structure. All elements are stored on a single doubly-linked list. +** +** Again, this structure is intended to be opaque, but it can't really +** be opaque because it is used by macros. +*/ +struct HashElem { + HashElem *next, *prev; /* Next and previous elements in the table */ + void *data; /* Data associated with this element */ + void *pKey; int nKey; /* Key associated with this element */ +}; + +/* +** There are 4 different modes of operation for a hash table: +** +** SQLITE_HASH_INT nKey is used as the key and pKey is ignored. +** +** SQLITE_HASH_POINTER pKey is used as the key and nKey is ignored. +** +** SQLITE_HASH_STRING pKey points to a string that is nKey bytes long +** (including the null-terminator, if any). Case +** is ignored in comparisons. +** +** SQLITE_HASH_BINARY pKey points to binary data nKey bytes long. +** memcmp() is used to compare keys. +** +** A copy of the key is made for SQLITE_HASH_STRING and SQLITE_HASH_BINARY +** if the copyKey parameter to HashInit is 1. +*/ +/* #define SQLITE_HASH_INT 1 // NOT USED */ +/* #define SQLITE_HASH_POINTER 2 // NOT USED */ +#define SQLITE_HASH_STRING 3 +#define SQLITE_HASH_BINARY 4 + +/* +** Access routines. To delete, insert a NULL pointer. +*/ +void sqlite3HashInit(Hash*, int keytype, int copyKey); +void *sqlite3HashInsert(Hash*, const void *pKey, int nKey, void *pData); +void *sqlite3HashFind(const Hash*, const void *pKey, int nKey); +void sqlite3HashClear(Hash*); + +/* +** Macros for looping over all elements of a hash table. The idiom is +** like this: +** +** Hash h; +** HashElem *p; +** ... +** for(p=sqliteHashFirst(&h); p; p=sqliteHashNext(p)){ +** SomeStructure *pData = sqliteHashData(p); +** // do something with pData +** } +*/ +#define sqliteHashFirst(H) ((H)->first) +#define sqliteHashNext(E) ((E)->next) +#define sqliteHashData(E) ((E)->data) +#define sqliteHashKey(E) ((E)->pKey) +#define sqliteHashKeysize(E) ((E)->nKey) + +/* +** Number of entries in a hash table +*/ +#define sqliteHashCount(H) ((H)->count) + +#endif /* _SQLITE_HASH_H_ */ Added: external/sqlite-source-3.3.4/insert.c ============================================================================== --- (empty file) +++ external/sqlite-source-3.3.4/insert.c Mon Apr 3 07:54:59 2006 @@ -0,0 +1,1117 @@ +/* +** 2001 September 15 +** +** The author disclaims copyright to this source code. In place of +** a legal notice, here is a blessing: +** +** May you do good and not evil. +** May you find forgiveness for yourself and forgive others. +** May you share freely, never taking more than you give. +** +************************************************************************* +** This file contains C code routines that are called by the parser +** to handle INSERT statements in SQLite. +** +** $Id: insert.c,v 1.161 2006/02/10 02:27:43 danielk1977 Exp $ +*/ +#include "sqliteInt.h" + +/* +** Set P3 of the most recently inserted opcode to a column affinity +** string for index pIdx. A column affinity string has one character +** for each column in the table, according to the affinity of the column: +** +** Character Column affinity +** ------------------------------ +** 'a' TEXT +** 'b' NONE +** 'c' NUMERIC +** 'd' INTEGER +** 'e' REAL +*/ +void sqlite3IndexAffinityStr(Vdbe *v, Index *pIdx){ + if( !pIdx->zColAff ){ + /* The first time a column affinity string for a particular index is + ** required, it is allocated and populated here. It is then stored as + ** a member of the Index structure for subsequent use. + ** + ** The column affinity string will eventually be deleted by + ** sqliteDeleteIndex() when the Index structure itself is cleaned + ** up. + */ + int n; + Table *pTab = pIdx->pTable; + pIdx->zColAff = (char *)sqliteMalloc(pIdx->nColumn+1); + if( !pIdx->zColAff ){ + return; + } + for(n=0; nnColumn; n++){ + pIdx->zColAff[n] = pTab->aCol[pIdx->aiColumn[n]].affinity; + } + pIdx->zColAff[pIdx->nColumn] = '\0'; + } + + sqlite3VdbeChangeP3(v, -1, pIdx->zColAff, 0); +} + +/* +** Set P3 of the most recently inserted opcode to a column affinity +** string for table pTab. A column affinity string has one character +** for each column indexed by the index, according to the affinity of the +** column: +** +** Character Column affinity +** ------------------------------ +** 'a' TEXT +** 'b' NONE +** 'c' NUMERIC +** 'd' INTEGER +** 'e' REAL +*/ +void sqlite3TableAffinityStr(Vdbe *v, Table *pTab){ + /* The first time a column affinity string for a particular table + ** is required, it is allocated and populated here. It is then + ** stored as a member of the Table structure for subsequent use. + ** + ** The column affinity string will eventually be deleted by + ** sqlite3DeleteTable() when the Table structure itself is cleaned up. + */ + if( !pTab->zColAff ){ + char *zColAff; + int i; + + zColAff = (char *)sqliteMalloc(pTab->nCol+1); + if( !zColAff ){ + return; + } + + for(i=0; inCol; i++){ + zColAff[i] = pTab->aCol[i].affinity; + } + zColAff[pTab->nCol] = '\0'; + + pTab->zColAff = zColAff; + } + + sqlite3VdbeChangeP3(v, -1, pTab->zColAff, 0); +} + +/* +** Return non-zero if SELECT statement p opens the table with rootpage +** iTab in database iDb. This is used to see if a statement of the form +** "INSERT INTO SELECT ..." can run without using temporary +** table for the results of the SELECT. +** +** No checking is done for sub-selects that are part of expressions. +*/ +static int selectReadsTable(Select *p, Schema *pSchema, int iTab){ + int i; + struct SrcList_item *pItem; + if( p->pSrc==0 ) return 0; + for(i=0, pItem=p->pSrc->a; ipSrc->nSrc; i++, pItem++){ + if( pItem->pSelect ){ + if( selectReadsTable(pItem->pSelect, pSchema, iTab) ) return 1; + }else{ + if( pItem->pTab->pSchema==pSchema && pItem->pTab->tnum==iTab ) return 1; + } + } + return 0; +} + +/* +** This routine is call to handle SQL of the following forms: +** +** insert into TABLE (IDLIST) values(EXPRLIST) +** insert into TABLE (IDLIST) select +** +** The IDLIST following the table name is always optional. If omitted, +** then a list of all columns for the table is substituted. The IDLIST +** appears in the pColumn parameter. pColumn is NULL if IDLIST is omitted. +** +** The pList parameter holds EXPRLIST in the first form of the INSERT +** statement above, and pSelect is NULL. For the second form, pList is +** NULL and pSelect is a pointer to the select statement used to generate +** data for the insert. +** +** The code generated follows one of three templates. For a simple +** select with data coming from a VALUES clause, the code executes +** once straight down through. The template looks like this: +** +** open write cursor to and its indices +** puts VALUES clause expressions onto the stack +** write the resulting record into
+** cleanup +** +** If the statement is of the form +** +** INSERT INTO
SELECT ... +** +** And the SELECT clause does not read from
at any time, then +** the generated code follows this template: +** +** goto B +** A: setup for the SELECT +** loop over the tables in the SELECT +** gosub C +** end loop +** cleanup after the SELECT +** goto D +** B: open write cursor to
and its indices +** goto A +** C: insert the select result into
+** return +** D: cleanup +** +** The third template is used if the insert statement takes its +** values from a SELECT but the data is being inserted into a table +** that is also read as part of the SELECT. In the third form, +** we have to use a intermediate table to store the results of +** the select. The template is like this: +** +** goto B +** A: setup for the SELECT +** loop over the tables in the SELECT +** gosub C +** end loop +** cleanup after the SELECT +** goto D +** C: insert the select result into the intermediate table +** return +** B: open a cursor to an intermediate table +** goto A +** D: open write cursor to
and its indices +** loop over the intermediate table +** transfer values form intermediate table into
+** end the loop +** cleanup +*/ +void sqlite3Insert( + Parse *pParse, /* Parser context */ + SrcList *pTabList, /* Name of table into which we are inserting */ + ExprList *pList, /* List of values to be inserted */ + Select *pSelect, /* A SELECT statement to use as the data source */ + IdList *pColumn, /* Column names corresponding to IDLIST. */ + int onError /* How to handle constraint errors */ +){ + Table *pTab; /* The table to insert into */ + char *zTab; /* Name of the table into which we are inserting */ + const char *zDb; /* Name of the database holding this table */ + int i, j, idx; /* Loop counters */ + Vdbe *v; /* Generate code into this virtual machine */ + Index *pIdx; /* For looping over indices of the table */ + int nColumn; /* Number of columns in the data */ + int base = 0; /* VDBE Cursor number for pTab */ + int iCont=0,iBreak=0; /* Beginning and end of the loop over srcTab */ + sqlite3 *db; /* The main database structure */ + int keyColumn = -1; /* Column that is the INTEGER PRIMARY KEY */ + int endOfLoop; /* Label for the end of the insertion loop */ + int useTempTable = 0; /* Store SELECT results in intermediate table */ + int srcTab = 0; /* Data comes from this temporary cursor if >=0 */ + int iSelectLoop = 0; /* Address of code that implements the SELECT */ + int iCleanup = 0; /* Address of the cleanup code */ + int iInsertBlock = 0; /* Address of the subroutine used to insert data */ + int iCntMem = 0; /* Memory cell used for the row counter */ + int newIdx = -1; /* Cursor for the NEW table */ + Db *pDb; /* The database containing table being inserted into */ + int counterMem = 0; /* Memory cell holding AUTOINCREMENT counter */ + int iDb; + +#ifndef SQLITE_OMIT_TRIGGER + int isView; /* True if attempting to insert into a view */ + int triggers_exist = 0; /* True if there are FOR EACH ROW triggers */ +#endif + +#ifndef SQLITE_OMIT_AUTOINCREMENT + int counterRowid = 0; /* Memory cell holding rowid of autoinc counter */ +#endif + + if( pParse->nErr || sqlite3MallocFailed() ){ + goto insert_cleanup; + } + db = pParse->db; + + /* Locate the table into which we will be inserting new information. + */ + assert( pTabList->nSrc==1 ); + zTab = pTabList->a[0].zName; + if( zTab==0 ) goto insert_cleanup; + pTab = sqlite3SrcListLookup(pParse, pTabList); + if( pTab==0 ){ + goto insert_cleanup; + } + iDb = sqlite3SchemaToIndex(db, pTab->pSchema); + assert( iDbnDb ); + pDb = &db->aDb[iDb]; + zDb = pDb->zName; + if( sqlite3AuthCheck(pParse, SQLITE_INSERT, pTab->zName, 0, zDb) ){ + goto insert_cleanup; + } + + /* Figure out if we have any triggers and if the table being + ** inserted into is a view + */ +#ifndef SQLITE_OMIT_TRIGGER + triggers_exist = sqlite3TriggersExist(pParse, pTab, TK_INSERT, 0); + isView = pTab->pSelect!=0; +#else +# define triggers_exist 0 +# define isView 0 +#endif +#ifdef SQLITE_OMIT_VIEW +# undef isView +# define isView 0 +#endif + + /* Ensure that: + * (a) the table is not read-only, + * (b) that if it is a view then ON INSERT triggers exist + */ + if( sqlite3IsReadOnly(pParse, pTab, triggers_exist) ){ + goto insert_cleanup; + } + if( pTab==0 ) goto insert_cleanup; + + /* If pTab is really a view, make sure it has been initialized. + */ + if( isView && sqlite3ViewGetColumnNames(pParse, pTab) ){ + goto insert_cleanup; + } + + /* Allocate a VDBE + */ + v = sqlite3GetVdbe(pParse); + if( v==0 ) goto insert_cleanup; + if( pParse->nested==0 ) sqlite3VdbeCountChanges(v); + sqlite3BeginWriteOperation(pParse, pSelect || triggers_exist, iDb); + + /* if there are row triggers, allocate a temp table for new.* references. */ + if( triggers_exist ){ + newIdx = pParse->nTab++; + } + +#ifndef SQLITE_OMIT_AUTOINCREMENT + /* If this is an AUTOINCREMENT table, look up the sequence number in the + ** sqlite_sequence table and store it in memory cell counterMem. Also + ** remember the rowid of the sqlite_sequence table entry in memory cell + ** counterRowid. + */ + if( pTab->autoInc ){ + int iCur = pParse->nTab; + int addr = sqlite3VdbeCurrentAddr(v); + counterRowid = pParse->nMem++; + counterMem = pParse->nMem++; + sqlite3OpenTable(pParse, iCur, iDb, pDb->pSchema->pSeqTab, OP_OpenRead); + sqlite3VdbeAddOp(v, OP_Rewind, iCur, addr+13); + sqlite3VdbeAddOp(v, OP_Column, iCur, 0); + sqlite3VdbeOp3(v, OP_String8, 0, 0, pTab->zName, 0); + sqlite3VdbeAddOp(v, OP_Ne, 0x100, addr+12); + sqlite3VdbeAddOp(v, OP_Rowid, iCur, 0); + sqlite3VdbeAddOp(v, OP_MemStore, counterRowid, 1); + sqlite3VdbeAddOp(v, OP_Column, iCur, 1); + sqlite3VdbeAddOp(v, OP_MemStore, counterMem, 1); + sqlite3VdbeAddOp(v, OP_Goto, 0, addr+13); + sqlite3VdbeAddOp(v, OP_Next, iCur, addr+4); + sqlite3VdbeAddOp(v, OP_Close, iCur, 0); + } +#endif /* SQLITE_OMIT_AUTOINCREMENT */ + + /* Figure out how many columns of data are supplied. If the data + ** is coming from a SELECT statement, then this step also generates + ** all the code to implement the SELECT statement and invoke a subroutine + ** to process each row of the result. (Template 2.) If the SELECT + ** statement uses the the table that is being inserted into, then the + ** subroutine is also coded here. That subroutine stores the SELECT + ** results in a temporary table. (Template 3.) + */ + if( pSelect ){ + /* Data is coming from a SELECT. Generate code to implement that SELECT + */ + int rc, iInitCode; + iInitCode = sqlite3VdbeAddOp(v, OP_Goto, 0, 0); + iSelectLoop = sqlite3VdbeCurrentAddr(v); + iInsertBlock = sqlite3VdbeMakeLabel(v); + + /* Resolve the expressions in the SELECT statement and execute it. */ + rc = sqlite3Select(pParse, pSelect, SRT_Subroutine, iInsertBlock,0,0,0,0); + if( rc || pParse->nErr || sqlite3MallocFailed() ){ + goto insert_cleanup; + } + + iCleanup = sqlite3VdbeMakeLabel(v); + sqlite3VdbeAddOp(v, OP_Goto, 0, iCleanup); + assert( pSelect->pEList ); + nColumn = pSelect->pEList->nExpr; + + /* Set useTempTable to TRUE if the result of the SELECT statement + ** should be written into a temporary table. Set to FALSE if each + ** row of the SELECT can be written directly into the result table. + ** + ** A temp table must be used if the table being updated is also one + ** of the tables being read by the SELECT statement. Also use a + ** temp table in the case of row triggers. + */ + if( triggers_exist || selectReadsTable(pSelect,pTab->pSchema,pTab->tnum) ){ + useTempTable = 1; + } + + if( useTempTable ){ + /* Generate the subroutine that SELECT calls to process each row of + ** the result. Store the result in a temporary table + */ + srcTab = pParse->nTab++; + sqlite3VdbeResolveLabel(v, iInsertBlock); + sqlite3VdbeAddOp(v, OP_MakeRecord, nColumn, 0); + sqlite3VdbeAddOp(v, OP_NewRowid, srcTab, 0); + sqlite3VdbeAddOp(v, OP_Pull, 1, 0); + sqlite3VdbeAddOp(v, OP_Insert, srcTab, 0); + sqlite3VdbeAddOp(v, OP_Return, 0, 0); + + /* The following code runs first because the GOTO at the very top + ** of the program jumps to it. Create the temporary table, then jump + ** back up and execute the SELECT code above. + */ + sqlite3VdbeJumpHere(v, iInitCode); + sqlite3VdbeAddOp(v, OP_OpenVirtual, srcTab, 0); + sqlite3VdbeAddOp(v, OP_SetNumColumns, srcTab, nColumn); + sqlite3VdbeAddOp(v, OP_Goto, 0, iSelectLoop); + sqlite3VdbeResolveLabel(v, iCleanup); + }else{ + sqlite3VdbeJumpHere(v, iInitCode); + } + }else{ + /* This is the case if the data for the INSERT is coming from a VALUES + ** clause + */ + NameContext sNC; + memset(&sNC, 0, sizeof(sNC)); + sNC.pParse = pParse; + assert( pList!=0 ); + srcTab = -1; + useTempTable = 0; + assert( pList ); + nColumn = pList->nExpr; + for(i=0; ia[i].pExpr) ){ + goto insert_cleanup; + } + } + } + + /* Make sure the number of columns in the source data matches the number + ** of columns to be inserted into the table. + */ + if( pColumn==0 && nColumn!=pTab->nCol ){ + sqlite3ErrorMsg(pParse, + "table %S has %d columns but %d values were supplied", + pTabList, 0, pTab->nCol, nColumn); + goto insert_cleanup; + } + if( pColumn!=0 && nColumn!=pColumn->nId ){ + sqlite3ErrorMsg(pParse, "%d values for %d columns", nColumn, pColumn->nId); + goto insert_cleanup; + } + + /* If the INSERT statement included an IDLIST term, then make sure + ** all elements of the IDLIST really are columns of the table and + ** remember the column indices. + ** + ** If the table has an INTEGER PRIMARY KEY column and that column + ** is named in the IDLIST, then record in the keyColumn variable + ** the index into IDLIST of the primary key column. keyColumn is + ** the index of the primary key as it appears in IDLIST, not as + ** is appears in the original table. (The index of the primary + ** key in the original table is pTab->iPKey.) + */ + if( pColumn ){ + for(i=0; inId; i++){ + pColumn->a[i].idx = -1; + } + for(i=0; inId; i++){ + for(j=0; jnCol; j++){ + if( sqlite3StrICmp(pColumn->a[i].zName, pTab->aCol[j].zName)==0 ){ + pColumn->a[i].idx = j; + if( j==pTab->iPKey ){ + keyColumn = i; + } + break; + } + } + if( j>=pTab->nCol ){ + if( sqlite3IsRowid(pColumn->a[i].zName) ){ + keyColumn = i; + }else{ + sqlite3ErrorMsg(pParse, "table %S has no column named %s", + pTabList, 0, pColumn->a[i].zName); + pParse->nErr++; + goto insert_cleanup; + } + } + } + } + + /* If there is no IDLIST term but the table has an integer primary + ** key, the set the keyColumn variable to the primary key column index + ** in the original table definition. + */ + if( pColumn==0 ){ + keyColumn = pTab->iPKey; + } + + /* Open the temp table for FOR EACH ROW triggers + */ + if( triggers_exist ){ + sqlite3VdbeAddOp(v, OP_OpenPseudo, newIdx, 0); + sqlite3VdbeAddOp(v, OP_SetNumColumns, newIdx, pTab->nCol); + } + + /* Initialize the count of rows to be inserted + */ + if( db->flags & SQLITE_CountRows ){ + iCntMem = pParse->nMem++; + sqlite3VdbeAddOp(v, OP_MemInt, 0, iCntMem); + } + + /* Open tables and indices if there are no row triggers */ + if( !triggers_exist ){ + base = pParse->nTab; + sqlite3OpenTableAndIndices(pParse, pTab, base, OP_OpenWrite); + } + + /* If the data source is a temporary table, then we have to create + ** a loop because there might be multiple rows of data. If the data + ** source is a subroutine call from the SELECT statement, then we need + ** to launch the SELECT statement processing. + */ + if( useTempTable ){ + iBreak = sqlite3VdbeMakeLabel(v); + sqlite3VdbeAddOp(v, OP_Rewind, srcTab, iBreak); + iCont = sqlite3VdbeCurrentAddr(v); + }else if( pSelect ){ + sqlite3VdbeAddOp(v, OP_Goto, 0, iSelectLoop); + sqlite3VdbeResolveLabel(v, iInsertBlock); + } + + /* Run the BEFORE and INSTEAD OF triggers, if there are any + */ + endOfLoop = sqlite3VdbeMakeLabel(v); + if( triggers_exist & TRIGGER_BEFORE ){ + + /* build the NEW.* reference row. Note that if there is an INTEGER + ** PRIMARY KEY into which a NULL is being inserted, that NULL will be + ** translated into a unique ID for the row. But on a BEFORE trigger, + ** we do not know what the unique ID will be (because the insert has + ** not happened yet) so we substitute a rowid of -1 + */ + if( keyColumn<0 ){ + sqlite3VdbeAddOp(v, OP_Integer, -1, 0); + }else if( useTempTable ){ + sqlite3VdbeAddOp(v, OP_Column, srcTab, keyColumn); + }else{ + assert( pSelect==0 ); /* Otherwise useTempTable is true */ + sqlite3ExprCode(pParse, pList->a[keyColumn].pExpr); + sqlite3VdbeAddOp(v, OP_NotNull, -1, sqlite3VdbeCurrentAddr(v)+3); + sqlite3VdbeAddOp(v, OP_Pop, 1, 0); + sqlite3VdbeAddOp(v, OP_Integer, -1, 0); + sqlite3VdbeAddOp(v, OP_MustBeInt, 0, 0); + } + + /* Create the new column data + */ + for(i=0; inCol; i++){ + if( pColumn==0 ){ + j = i; + }else{ + for(j=0; jnId; j++){ + if( pColumn->a[j].idx==i ) break; + } + } + if( pColumn && j>=pColumn->nId ){ + sqlite3ExprCode(pParse, pTab->aCol[i].pDflt); + }else if( useTempTable ){ + sqlite3VdbeAddOp(v, OP_Column, srcTab, j); + }else{ + assert( pSelect==0 ); /* Otherwise useTempTable is true */ + sqlite3ExprCodeAndCache(pParse, pList->a[j].pExpr); + } + } + sqlite3VdbeAddOp(v, OP_MakeRecord, pTab->nCol, 0); + + /* If this is an INSERT on a view with an INSTEAD OF INSERT trigger, + ** do not attempt any conversions before assembling the record. + ** If this is a real table, attempt conversions as required by the + ** table column affinities. + */ + if( !isView ){ + sqlite3TableAffinityStr(v, pTab); + } + sqlite3VdbeAddOp(v, OP_Insert, newIdx, 0); + + /* Fire BEFORE or INSTEAD OF triggers */ + if( sqlite3CodeRowTrigger(pParse, TK_INSERT, 0, TRIGGER_BEFORE, pTab, + newIdx, -1, onError, endOfLoop) ){ + goto insert_cleanup; + } + } + + /* If any triggers exists, the opening of tables and indices is deferred + ** until now. + */ + if( triggers_exist && !isView ){ + base = pParse->nTab; + sqlite3OpenTableAndIndices(pParse, pTab, base, OP_OpenWrite); + } + + /* Push the record number for the new entry onto the stack. The + ** record number is a randomly generate integer created by NewRowid + ** except when the table has an INTEGER PRIMARY KEY column, in which + ** case the record number is the same as that column. + */ + if( !isView ){ + if( keyColumn>=0 ){ + if( useTempTable ){ + sqlite3VdbeAddOp(v, OP_Column, srcTab, keyColumn); + }else if( pSelect ){ + sqlite3VdbeAddOp(v, OP_Dup, nColumn - keyColumn - 1, 1); + }else{ + sqlite3ExprCode(pParse, pList->a[keyColumn].pExpr); + } + /* If the PRIMARY KEY expression is NULL, then use OP_NewRowid + ** to generate a unique primary key value. + */ + sqlite3VdbeAddOp(v, OP_NotNull, -1, sqlite3VdbeCurrentAddr(v)+3); + sqlite3VdbeAddOp(v, OP_Pop, 1, 0); + sqlite3VdbeAddOp(v, OP_NewRowid, base, counterMem); + sqlite3VdbeAddOp(v, OP_MustBeInt, 0, 0); + }else{ + sqlite3VdbeAddOp(v, OP_NewRowid, base, counterMem); + } +#ifndef SQLITE_OMIT_AUTOINCREMENT + if( pTab->autoInc ){ + sqlite3VdbeAddOp(v, OP_MemMax, counterMem, 0); + } +#endif /* SQLITE_OMIT_AUTOINCREMENT */ + + /* Push onto the stack, data for all columns of the new entry, beginning + ** with the first column. + */ + for(i=0; inCol; i++){ + if( i==pTab->iPKey ){ + /* The value of the INTEGER PRIMARY KEY column is always a NULL. + ** Whenever this column is read, the record number will be substituted + ** in its place. So will fill this column with a NULL to avoid + ** taking up data space with information that will never be used. */ + sqlite3VdbeAddOp(v, OP_Null, 0, 0); + continue; + } + if( pColumn==0 ){ + j = i; + }else{ + for(j=0; jnId; j++){ + if( pColumn->a[j].idx==i ) break; + } + } + if( pColumn && j>=pColumn->nId ){ + sqlite3ExprCode(pParse, pTab->aCol[i].pDflt); + }else if( useTempTable ){ + sqlite3VdbeAddOp(v, OP_Column, srcTab, j); + }else if( pSelect ){ + sqlite3VdbeAddOp(v, OP_Dup, i+nColumn-j, 1); + }else{ + sqlite3ExprCode(pParse, pList->a[j].pExpr); + } + } + + /* Generate code to check constraints and generate index keys and + ** do the insertion. + */ + sqlite3GenerateConstraintChecks(pParse, pTab, base, 0, keyColumn>=0, + 0, onError, endOfLoop); + sqlite3CompleteInsertion(pParse, pTab, base, 0,0,0, + (triggers_exist & TRIGGER_AFTER)!=0 ? newIdx : -1); + } + + /* Update the count of rows that are inserted + */ + if( (db->flags & SQLITE_CountRows)!=0 ){ + sqlite3VdbeAddOp(v, OP_MemIncr, 1, iCntMem); + } + + if( triggers_exist ){ + /* Close all tables opened */ + if( !isView ){ + sqlite3VdbeAddOp(v, OP_Close, base, 0); + for(idx=1, pIdx=pTab->pIndex; pIdx; pIdx=pIdx->pNext, idx++){ + sqlite3VdbeAddOp(v, OP_Close, idx+base, 0); + } + } + + /* Code AFTER triggers */ + if( sqlite3CodeRowTrigger(pParse, TK_INSERT, 0, TRIGGER_AFTER, pTab, + newIdx, -1, onError, endOfLoop) ){ + goto insert_cleanup; + } + } + + /* The bottom of the loop, if the data source is a SELECT statement + */ + sqlite3VdbeResolveLabel(v, endOfLoop); + if( useTempTable ){ + sqlite3VdbeAddOp(v, OP_Next, srcTab, iCont); + sqlite3VdbeResolveLabel(v, iBreak); + sqlite3VdbeAddOp(v, OP_Close, srcTab, 0); + }else if( pSelect ){ + sqlite3VdbeAddOp(v, OP_Pop, nColumn, 0); + sqlite3VdbeAddOp(v, OP_Return, 0, 0); + sqlite3VdbeResolveLabel(v, iCleanup); + } + + if( !triggers_exist ){ + /* Close all tables opened */ + sqlite3VdbeAddOp(v, OP_Close, base, 0); + for(idx=1, pIdx=pTab->pIndex; pIdx; pIdx=pIdx->pNext, idx++){ + sqlite3VdbeAddOp(v, OP_Close, idx+base, 0); + } + } + +#ifndef SQLITE_OMIT_AUTOINCREMENT + /* Update the sqlite_sequence table by storing the content of the + ** counter value in memory counterMem back into the sqlite_sequence + ** table. + */ + if( pTab->autoInc ){ + int iCur = pParse->nTab; + int addr = sqlite3VdbeCurrentAddr(v); + sqlite3OpenTable(pParse, iCur, iDb, pDb->pSchema->pSeqTab, OP_OpenWrite); + sqlite3VdbeAddOp(v, OP_MemLoad, counterRowid, 0); + sqlite3VdbeAddOp(v, OP_NotNull, -1, addr+7); + sqlite3VdbeAddOp(v, OP_Pop, 1, 0); + sqlite3VdbeAddOp(v, OP_NewRowid, iCur, 0); + sqlite3VdbeOp3(v, OP_String8, 0, 0, pTab->zName, 0); + sqlite3VdbeAddOp(v, OP_MemLoad, counterMem, 0); + sqlite3VdbeAddOp(v, OP_MakeRecord, 2, 0); + sqlite3VdbeAddOp(v, OP_Insert, iCur, 0); + sqlite3VdbeAddOp(v, OP_Close, iCur, 0); + } +#endif + + /* + ** Return the number of rows inserted. If this routine is + ** generating code because of a call to sqlite3NestedParse(), do not + ** invoke the callback function. + */ + if( db->flags & SQLITE_CountRows && pParse->nested==0 && !pParse->trigStack ){ + sqlite3VdbeAddOp(v, OP_MemLoad, iCntMem, 0); + sqlite3VdbeAddOp(v, OP_Callback, 1, 0); + sqlite3VdbeSetNumCols(v, 1); + sqlite3VdbeSetColName(v, 0, COLNAME_NAME, "rows inserted", P3_STATIC); + } + +insert_cleanup: + sqlite3SrcListDelete(pTabList); + sqlite3ExprListDelete(pList); + sqlite3SelectDelete(pSelect); + sqlite3IdListDelete(pColumn); +} + +/* +** Generate code to do a constraint check prior to an INSERT or an UPDATE. +** +** When this routine is called, the stack contains (from bottom to top) +** the following values: +** +** 1. The rowid of the row to be updated before the update. This +** value is omitted unless we are doing an UPDATE that involves a +** change to the record number. +** +** 2. The rowid of the row after the update. +** +** 3. The data in the first column of the entry after the update. +** +** i. Data from middle columns... +** +** N. The data in the last column of the entry after the update. +** +** The old rowid shown as entry (1) above is omitted unless both isUpdate +** and rowidChng are 1. isUpdate is true for UPDATEs and false for +** INSERTs and rowidChng is true if the record number is being changed. +** +** The code generated by this routine pushes additional entries onto +** the stack which are the keys for new index entries for the new record. +** The order of index keys is the same as the order of the indices on +** the pTable->pIndex list. A key is only created for index i if +** aIdxUsed!=0 and aIdxUsed[i]!=0. +** +** This routine also generates code to check constraints. NOT NULL, +** CHECK, and UNIQUE constraints are all checked. If a constraint fails, +** then the appropriate action is performed. There are five possible +** actions: ROLLBACK, ABORT, FAIL, REPLACE, and IGNORE. +** +** Constraint type Action What Happens +** --------------- ---------- ---------------------------------------- +** any ROLLBACK The current transaction is rolled back and +** sqlite3_exec() returns immediately with a +** return code of SQLITE_CONSTRAINT. +** +** any ABORT Back out changes from the current command +** only (do not do a complete rollback) then +** cause sqlite3_exec() to return immediately +** with SQLITE_CONSTRAINT. +** +** any FAIL Sqlite_exec() returns immediately with a +** return code of SQLITE_CONSTRAINT. The +** transaction is not rolled back and any +** prior changes are retained. +** +** any IGNORE The record number and data is popped from +** the stack and there is an immediate jump +** to label ignoreDest. +** +** NOT NULL REPLACE The NULL value is replace by the default +** value for that column. If the default value +** is NULL, the action is the same as ABORT. +** +** UNIQUE REPLACE The other row that conflicts with the row +** being inserted is removed. +** +** CHECK REPLACE Illegal. The results in an exception. +** +** Which action to take is determined by the overrideError parameter. +** Or if overrideError==OE_Default, then the pParse->onError parameter +** is used. Or if pParse->onError==OE_Default then the onError value +** for the constraint is used. +** +** The calling routine must open a read/write cursor for pTab with +** cursor number "base". All indices of pTab must also have open +** read/write cursors with cursor number base+i for the i-th cursor. +** Except, if there is no possibility of a REPLACE action then +** cursors do not need to be open for indices where aIdxUsed[i]==0. +** +** If the isUpdate flag is true, it means that the "base" cursor is +** initially pointing to an entry that is being updated. The isUpdate +** flag causes extra code to be generated so that the "base" cursor +** is still pointing at the same entry after the routine returns. +** Without the isUpdate flag, the "base" cursor might be moved. +*/ +void sqlite3GenerateConstraintChecks( + Parse *pParse, /* The parser context */ + Table *pTab, /* the table into which we are inserting */ + int base, /* Index of a read/write cursor pointing at pTab */ + char *aIdxUsed, /* Which indices are used. NULL means all are used */ + int rowidChng, /* True if the record number will change */ + int isUpdate, /* True for UPDATE, False for INSERT */ + int overrideError, /* Override onError to this if not OE_Default */ + int ignoreDest /* Jump to this label on an OE_Ignore resolution */ +){ + int i; + Vdbe *v; + int nCol; + int onError; + int addr; + int extra; + int iCur; + Index *pIdx; + int seenReplace = 0; + int jumpInst1=0, jumpInst2; + int hasTwoRowids = (isUpdate && rowidChng); + + v = sqlite3GetVdbe(pParse); + assert( v!=0 ); + assert( pTab->pSelect==0 ); /* This table is not a VIEW */ + nCol = pTab->nCol; + + /* Test all NOT NULL constraints. + */ + for(i=0; iiPKey ){ + continue; + } + onError = pTab->aCol[i].notNull; + if( onError==OE_None ) continue; + if( overrideError!=OE_Default ){ + onError = overrideError; + }else if( onError==OE_Default ){ + onError = OE_Abort; + } + if( onError==OE_Replace && pTab->aCol[i].pDflt==0 ){ + onError = OE_Abort; + } + sqlite3VdbeAddOp(v, OP_Dup, nCol-1-i, 1); + addr = sqlite3VdbeAddOp(v, OP_NotNull, 1, 0); + assert( onError==OE_Rollback || onError==OE_Abort || onError==OE_Fail + || onError==OE_Ignore || onError==OE_Replace ); + switch( onError ){ + case OE_Rollback: + case OE_Abort: + case OE_Fail: { + char *zMsg = 0; + sqlite3VdbeAddOp(v, OP_Halt, SQLITE_CONSTRAINT, onError); + sqlite3SetString(&zMsg, pTab->zName, ".", pTab->aCol[i].zName, + " may not be NULL", (char*)0); + sqlite3VdbeChangeP3(v, -1, zMsg, P3_DYNAMIC); + break; + } + case OE_Ignore: { + sqlite3VdbeAddOp(v, OP_Pop, nCol+1+hasTwoRowids, 0); + sqlite3VdbeAddOp(v, OP_Goto, 0, ignoreDest); + break; + } + case OE_Replace: { + sqlite3ExprCode(pParse, pTab->aCol[i].pDflt); + sqlite3VdbeAddOp(v, OP_Push, nCol-i, 0); + break; + } + } + sqlite3VdbeJumpHere(v, addr); + } + + /* Test all CHECK constraints + */ +#ifndef SQLITE_OMIT_CHECK + if( pTab->pCheck && (pParse->db->flags & SQLITE_IgnoreChecks)==0 ){ + int allOk = sqlite3VdbeMakeLabel(v); + assert( pParse->ckOffset==0 ); + pParse->ckOffset = nCol; + sqlite3ExprIfTrue(pParse, pTab->pCheck, allOk, 1); + assert( pParse->ckOffset==nCol ); + pParse->ckOffset = 0; + sqlite3VdbeAddOp(v, OP_Halt, SQLITE_CONSTRAINT, OE_Abort); + sqlite3VdbeResolveLabel(v, allOk); + } +#endif /* !defined(SQLITE_OMIT_CHECK) */ + + /* If we have an INTEGER PRIMARY KEY, make sure the primary key + ** of the new record does not previously exist. Except, if this + ** is an UPDATE and the primary key is not changing, that is OK. + */ + if( rowidChng ){ + onError = pTab->keyConf; + if( overrideError!=OE_Default ){ + onError = overrideError; + }else if( onError==OE_Default ){ + onError = OE_Abort; + } + + if( isUpdate ){ + sqlite3VdbeAddOp(v, OP_Dup, nCol+1, 1); + sqlite3VdbeAddOp(v, OP_Dup, nCol+1, 1); + jumpInst1 = sqlite3VdbeAddOp(v, OP_Eq, 0, 0); + } + sqlite3VdbeAddOp(v, OP_Dup, nCol, 1); + jumpInst2 = sqlite3VdbeAddOp(v, OP_NotExists, base, 0); + switch( onError ){ + default: { + onError = OE_Abort; + /* Fall thru into the next case */ + } + case OE_Rollback: + case OE_Abort: + case OE_Fail: { + sqlite3VdbeOp3(v, OP_Halt, SQLITE_CONSTRAINT, onError, + "PRIMARY KEY must be unique", P3_STATIC); + break; + } + case OE_Replace: { + sqlite3GenerateRowIndexDelete(pParse->db, v, pTab, base, 0); + if( isUpdate ){ + sqlite3VdbeAddOp(v, OP_Dup, nCol+hasTwoRowids, 1); + sqlite3VdbeAddOp(v, OP_MoveGe, base, 0); + } + seenReplace = 1; + break; + } + case OE_Ignore: { + assert( seenReplace==0 ); + sqlite3VdbeAddOp(v, OP_Pop, nCol+1+hasTwoRowids, 0); + sqlite3VdbeAddOp(v, OP_Goto, 0, ignoreDest); + break; + } + } + sqlite3VdbeJumpHere(v, jumpInst2); + if( isUpdate ){ + sqlite3VdbeJumpHere(v, jumpInst1); + sqlite3VdbeAddOp(v, OP_Dup, nCol+1, 1); + sqlite3VdbeAddOp(v, OP_MoveGe, base, 0); + } + } + + /* Test all UNIQUE constraints by creating entries for each UNIQUE + ** index and making sure that duplicate entries do not already exist. + ** Add the new records to the indices as we go. + */ + extra = -1; + for(iCur=0, pIdx=pTab->pIndex; pIdx; pIdx=pIdx->pNext, iCur++){ + if( aIdxUsed && aIdxUsed[iCur]==0 ) continue; /* Skip unused indices */ + extra++; + + /* Create a key for accessing the index entry */ + sqlite3VdbeAddOp(v, OP_Dup, nCol+extra, 1); + for(i=0; inColumn; i++){ + int idx = pIdx->aiColumn[i]; + if( idx==pTab->iPKey ){ + sqlite3VdbeAddOp(v, OP_Dup, i+extra+nCol+1, 1); + }else{ + sqlite3VdbeAddOp(v, OP_Dup, i+extra+nCol-idx, 1); + } + } + jumpInst1 = sqlite3VdbeAddOp(v, OP_MakeIdxRec, pIdx->nColumn, 0); + sqlite3IndexAffinityStr(v, pIdx); + + /* Find out what action to take in case there is an indexing conflict */ + onError = pIdx->onError; + if( onError==OE_None ) continue; /* pIdx is not a UNIQUE index */ + if( overrideError!=OE_Default ){ + onError = overrideError; + }else if( onError==OE_Default ){ + onError = OE_Abort; + } + if( seenReplace ){ + if( onError==OE_Ignore ) onError = OE_Replace; + else if( onError==OE_Fail ) onError = OE_Abort; + } + + + /* Check to see if the new index entry will be unique */ + sqlite3VdbeAddOp(v, OP_Dup, extra+nCol+1+hasTwoRowids, 1); + jumpInst2 = sqlite3VdbeAddOp(v, OP_IsUnique, base+iCur+1, 0); + + /* Generate code that executes if the new index entry is not unique */ + assert( onError==OE_Rollback || onError==OE_Abort || onError==OE_Fail + || onError==OE_Ignore || onError==OE_Replace ); + switch( onError ){ + case OE_Rollback: + case OE_Abort: + case OE_Fail: { + int j, n1, n2; + char zErrMsg[200]; + strcpy(zErrMsg, pIdx->nColumn>1 ? "columns " : "column "); + n1 = strlen(zErrMsg); + for(j=0; jnColumn && n1aCol[pIdx->aiColumn[j]].zName; + n2 = strlen(zCol); + if( j>0 ){ + strcpy(&zErrMsg[n1], ", "); + n1 += 2; + } + if( n1+n2>sizeof(zErrMsg)-30 ){ + strcpy(&zErrMsg[n1], "..."); + n1 += 3; + break; + }else{ + strcpy(&zErrMsg[n1], zCol); + n1 += n2; + } + } + strcpy(&zErrMsg[n1], + pIdx->nColumn>1 ? " are not unique" : " is not unique"); + sqlite3VdbeOp3(v, OP_Halt, SQLITE_CONSTRAINT, onError, zErrMsg, 0); + break; + } + case OE_Ignore: { + assert( seenReplace==0 ); + sqlite3VdbeAddOp(v, OP_Pop, nCol+extra+3+hasTwoRowids, 0); + sqlite3VdbeAddOp(v, OP_Goto, 0, ignoreDest); + break; + } + case OE_Replace: { + sqlite3GenerateRowDelete(pParse->db, v, pTab, base, 0); + if( isUpdate ){ + sqlite3VdbeAddOp(v, OP_Dup, nCol+extra+1+hasTwoRowids, 1); + sqlite3VdbeAddOp(v, OP_MoveGe, base, 0); + } + seenReplace = 1; + break; + } + } +#if NULL_DISTINCT_FOR_UNIQUE + sqlite3VdbeJumpHere(v, jumpInst1); +#endif + sqlite3VdbeJumpHere(v, jumpInst2); + } +} + +/* +** This routine generates code to finish the INSERT or UPDATE operation +** that was started by a prior call to sqlite3GenerateConstraintChecks. +** The stack must contain keys for all active indices followed by data +** and the rowid for the new entry. This routine creates the new +** entries in all indices and in the main table. +** +** The arguments to this routine should be the same as the first six +** arguments to sqlite3GenerateConstraintChecks. +*/ +void sqlite3CompleteInsertion( + Parse *pParse, /* The parser context */ + Table *pTab, /* the table into which we are inserting */ + int base, /* Index of a read/write cursor pointing at pTab */ + char *aIdxUsed, /* Which indices are used. NULL means all are used */ + int rowidChng, /* True if the record number will change */ + int isUpdate, /* True for UPDATE, False for INSERT */ + int newIdx /* Index of NEW table for triggers. -1 if none */ +){ + int i; + Vdbe *v; + int nIdx; + Index *pIdx; + int pik_flags; + + v = sqlite3GetVdbe(pParse); + assert( v!=0 ); + assert( pTab->pSelect==0 ); /* This table is not a VIEW */ + for(nIdx=0, pIdx=pTab->pIndex; pIdx; pIdx=pIdx->pNext, nIdx++){} + for(i=nIdx-1; i>=0; i--){ + if( aIdxUsed && aIdxUsed[i]==0 ) continue; + sqlite3VdbeAddOp(v, OP_IdxInsert, base+i+1, 0); + } + sqlite3VdbeAddOp(v, OP_MakeRecord, pTab->nCol, 0); + sqlite3TableAffinityStr(v, pTab); +#ifndef SQLITE_OMIT_TRIGGER + if( newIdx>=0 ){ + sqlite3VdbeAddOp(v, OP_Dup, 1, 0); + sqlite3VdbeAddOp(v, OP_Dup, 1, 0); + sqlite3VdbeAddOp(v, OP_Insert, newIdx, 0); + } +#endif + if( pParse->nested ){ + pik_flags = 0; + }else{ + pik_flags = OPFLAG_NCHANGE; + pik_flags |= (isUpdate?OPFLAG_ISUPDATE:OPFLAG_LASTROWID); + } + sqlite3VdbeAddOp(v, OP_Insert, base, pik_flags); + if( !pParse->nested ){ + sqlite3VdbeChangeP3(v, -1, pTab->zName, P3_STATIC); + } + + if( isUpdate && rowidChng ){ + sqlite3VdbeAddOp(v, OP_Pop, 1, 0); + } +} + +/* +** Generate code that will open cursors for a table and for all +** indices of that table. The "base" parameter is the cursor number used +** for the table. Indices are opened on subsequent cursors. +*/ +void sqlite3OpenTableAndIndices( + Parse *pParse, /* Parsing context */ + Table *pTab, /* Table to be opened */ + int base, /* Cursor number assigned to the table */ + int op /* OP_OpenRead or OP_OpenWrite */ +){ + int i; + int iDb = sqlite3SchemaToIndex(pParse->db, pTab->pSchema); + Index *pIdx; + Vdbe *v = sqlite3GetVdbe(pParse); + assert( v!=0 ); + sqlite3OpenTable(pParse, base, iDb, pTab, op); + for(i=1, pIdx=pTab->pIndex; pIdx; pIdx=pIdx->pNext, i++){ + KeyInfo *pKey = sqlite3IndexKeyinfo(pParse, pIdx); + assert( pIdx->pSchema==pTab->pSchema ); + sqlite3VdbeAddOp(v, OP_Integer, iDb, 0); + VdbeComment((v, "# %s", pIdx->zName)); + sqlite3VdbeOp3(v, op, i+base, pIdx->tnum, (char*)pKey, P3_KEYINFO_HANDOFF); + } + if( pParse->nTab<=base+i ){ + pParse->nTab = base+i; + } +} Added: external/sqlite-source-3.3.4/keywordhash.h ============================================================================== --- (empty file) +++ external/sqlite-source-3.3.4/keywordhash.h Mon Apr 3 07:54:59 2006 @@ -0,0 +1,98 @@ +/* Hash score: 159 */ +static int keywordCode(const char *z, int n){ + static const char zText[537] = + "ABORTABLEFTEMPORARYADDATABASELECTHENDEFAULTRANSACTIONATURALTER" + "AISEACHECKEYAFTEREFERENCESCAPELSEXCEPTRIGGEREGEXPLAINITIALLYANALYZE" + "XCLUSIVEXISTSTATEMENTANDEFERRABLEATTACHAVINGLOBEFOREIGNOREINDEX" + "AUTOINCREMENTBEGINNERENAMEBETWEENOTNULLIKEBYCASCADEFERREDELETE" + "CASECASTCOLLATECOLUMNCOMMITCONFLICTCONSTRAINTERSECTCREATECROSS" + "CURRENT_DATECURRENT_TIMESTAMPLANDESCDETACHDISTINCTDROPRAGMATCH" + "FAILIMITFROMFULLGROUPDATEIFIMMEDIATEINSERTINSTEADINTOFFSETISNULL" + "JOINORDEREPLACEOUTERESTRICTPRIMARYQUERYRIGHTROLLBACKROWHENUNION" + "UNIQUEUSINGVACUUMVALUESVIEWHERE"; + static const unsigned char aHash[127] = { + 92, 80, 107, 91, 0, 4, 0, 0, 114, 0, 83, 0, 0, + 95, 44, 76, 93, 0, 106, 109, 97, 90, 0, 10, 0, 0, + 113, 0, 110, 103, 0, 28, 48, 0, 41, 0, 0, 65, 71, + 0, 63, 19, 0, 105, 36, 104, 0, 108, 74, 0, 0, 33, + 0, 61, 37, 0, 8, 0, 115, 38, 12, 0, 77, 40, 25, + 66, 0, 0, 31, 81, 53, 30, 50, 20, 88, 0, 34, 0, + 75, 26, 0, 72, 0, 0, 0, 64, 47, 67, 22, 87, 29, + 69, 86, 0, 1, 0, 9, 101, 58, 18, 0, 112, 82, 99, + 54, 6, 85, 0, 0, 49, 94, 0, 102, 0, 70, 0, 0, + 15, 0, 116, 51, 56, 0, 2, 55, 0, 111, + }; + static const unsigned char aNext[116] = { + 0, 0, 0, 0, 0, 3, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 17, 0, 0, 0, 0, + 0, 11, 0, 0, 0, 0, 5, 13, 0, 7, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 43, 0, 0, 0, 0, 0, + 0, 0, 16, 0, 23, 52, 0, 0, 0, 0, 45, 0, 59, + 0, 0, 0, 0, 0, 0, 0, 0, 73, 42, 0, 24, 60, + 21, 0, 79, 0, 0, 68, 0, 0, 84, 46, 0, 0, 0, + 0, 0, 0, 0, 0, 39, 96, 98, 0, 0, 100, 0, 32, + 0, 14, 27, 78, 0, 57, 89, 0, 35, 0, 62, 0, + }; + static const unsigned char aLen[116] = { + 5, 5, 4, 4, 9, 2, 3, 8, 2, 6, 4, 3, 7, + 11, 2, 7, 5, 5, 4, 5, 3, 5, 10, 6, 4, 6, + 7, 6, 7, 9, 3, 7, 9, 6, 9, 3, 10, 6, 6, + 4, 6, 3, 7, 6, 7, 5, 13, 2, 2, 5, 5, 6, + 7, 3, 7, 4, 4, 2, 7, 3, 8, 6, 4, 4, 7, + 6, 6, 8, 10, 9, 6, 5, 12, 12, 17, 4, 4, 6, + 8, 2, 4, 6, 5, 4, 5, 4, 4, 5, 6, 2, 9, + 6, 7, 4, 2, 6, 3, 6, 4, 5, 7, 5, 8, 7, + 5, 5, 8, 3, 4, 5, 6, 5, 6, 6, 4, 5, + }; + static const unsigned short int aOffset[116] = { + 0, 4, 7, 10, 10, 14, 19, 21, 26, 27, 32, 34, 36, + 42, 51, 52, 57, 61, 65, 67, 71, 74, 78, 86, 91, 94, + 99, 105, 108, 113, 118, 122, 128, 136, 141, 150, 152, 162, 167, + 172, 175, 177, 177, 181, 185, 187, 192, 194, 196, 205, 208, 212, + 218, 224, 224, 227, 230, 234, 236, 237, 241, 248, 254, 258, 262, + 269, 275, 281, 289, 296, 305, 311, 316, 328, 328, 344, 348, 352, + 358, 359, 366, 369, 373, 378, 381, 386, 390, 394, 397, 403, 405, + 414, 420, 427, 430, 430, 433, 436, 442, 446, 450, 457, 461, 469, + 476, 481, 486, 494, 496, 500, 505, 511, 516, 522, 528, 531, + }; + static const unsigned char aCode[116] = { + TK_ABORT, TK_TABLE, TK_JOIN_KW, TK_TEMP, TK_TEMP, + TK_OR, TK_ADD, TK_DATABASE, TK_AS, TK_SELECT, + TK_THEN, TK_END, TK_DEFAULT, TK_TRANSACTION,TK_ON, + TK_JOIN_KW, TK_ALTER, TK_RAISE, TK_EACH, TK_CHECK, + TK_KEY, TK_AFTER, TK_REFERENCES, TK_ESCAPE, TK_ELSE, + TK_EXCEPT, TK_TRIGGER, TK_LIKE_KW, TK_EXPLAIN, TK_INITIALLY, + TK_ALL, TK_ANALYZE, TK_EXCLUSIVE, TK_EXISTS, TK_STATEMENT, + TK_AND, TK_DEFERRABLE, TK_ATTACH, TK_HAVING, TK_LIKE_KW, + TK_BEFORE, TK_FOR, TK_FOREIGN, TK_IGNORE, TK_REINDEX, + TK_INDEX, TK_AUTOINCR, TK_TO, TK_IN, TK_BEGIN, + TK_JOIN_KW, TK_RENAME, TK_BETWEEN, TK_NOT, TK_NOTNULL, + TK_NULL, TK_LIKE_KW, TK_BY, TK_CASCADE, TK_ASC, + TK_DEFERRED, TK_DELETE, TK_CASE, TK_CAST, TK_COLLATE, + TK_COLUMNKW, TK_COMMIT, TK_CONFLICT, TK_CONSTRAINT, TK_INTERSECT, + TK_CREATE, TK_JOIN_KW, TK_CTIME_KW, TK_CTIME_KW, TK_CTIME_KW, + TK_PLAN, TK_DESC, TK_DETACH, TK_DISTINCT, TK_IS, + TK_DROP, TK_PRAGMA, TK_MATCH, TK_FAIL, TK_LIMIT, + TK_FROM, TK_JOIN_KW, TK_GROUP, TK_UPDATE, TK_IF, + TK_IMMEDIATE, TK_INSERT, TK_INSTEAD, TK_INTO, TK_OF, + TK_OFFSET, TK_SET, TK_ISNULL, TK_JOIN, TK_ORDER, + TK_REPLACE, TK_JOIN_KW, TK_RESTRICT, TK_PRIMARY, TK_QUERY, + TK_JOIN_KW, TK_ROLLBACK, TK_ROW, TK_WHEN, TK_UNION, + TK_UNIQUE, TK_USING, TK_VACUUM, TK_VALUES, TK_VIEW, + TK_WHERE, + }; + int h, i; + if( n<2 ) return TK_ID; + h = ((sqlite3UpperToLower[((unsigned char*)z)[0]]*4) ^ + (sqlite3UpperToLower[((unsigned char*)z)[n-1]]*3) ^ + n) % 127; + for(i=((int)aHash[h])-1; i>=0; i=((int)aNext[i])-1){ + if( aLen[i]==n && sqlite3StrNICmp(&zText[aOffset[i]],z,n)==0 ){ + return aCode[i]; + } + } + return TK_ID; +} +int sqlite3KeywordCode(const unsigned char *z, int n){ + return keywordCode((char*)z, n); +} Added: external/sqlite-source-3.3.4/legacy.c ============================================================================== --- (empty file) +++ external/sqlite-source-3.3.4/legacy.c Mon Apr 3 07:54:59 2006 @@ -0,0 +1,135 @@ +/* +** 2001 September 15 +** +** The author disclaims copyright to this source code. In place of +** a legal notice, here is a blessing: +** +** May you do good and not evil. +** May you find forgiveness for yourself and forgive others. +** May you share freely, never taking more than you give. +** +************************************************************************* +** Main file for the SQLite library. The routines in this file +** implement the programmer interface to the library. Routines in +** other files are for internal use by SQLite and should not be +** accessed by users of the library. +** +** $Id: legacy.c,v 1.13 2006/01/23 13:14:55 drh Exp $ +*/ + +#include "sqliteInt.h" +#include "os.h" +#include + +/* +** Execute SQL code. Return one of the SQLITE_ success/failure +** codes. Also write an error message into memory obtained from +** malloc() and make *pzErrMsg point to that message. +** +** If the SQL is a query, then for each row in the query result +** the xCallback() function is called. pArg becomes the first +** argument to xCallback(). If xCallback=NULL then no callback +** is invoked, even for queries. +*/ +int sqlite3_exec( + sqlite3 *db, /* The database on which the SQL executes */ + const char *zSql, /* The SQL to be executed */ + sqlite3_callback xCallback, /* Invoke this callback routine */ + void *pArg, /* First argument to xCallback() */ + char **pzErrMsg /* Write error messages here */ +){ + int rc = SQLITE_OK; + const char *zLeftover; + sqlite3_stmt *pStmt = 0; + char **azCols = 0; + + int nRetry = 0; + int nChange = 0; + int nCallback; + + if( zSql==0 ) return SQLITE_OK; + while( (rc==SQLITE_OK || (rc==SQLITE_SCHEMA && (++nRetry)<2)) && zSql[0] ){ + int nCol; + char **azVals = 0; + + pStmt = 0; + rc = sqlite3_prepare(db, zSql, -1, &pStmt, &zLeftover); + if( rc!=SQLITE_OK ){ + if( pStmt ) sqlite3_finalize(pStmt); + continue; + } + if( !pStmt ){ + /* this happens for a comment or white-space */ + zSql = zLeftover; + continue; + } + + db->nChange += nChange; + nCallback = 0; + + nCol = sqlite3_column_count(pStmt); + azCols = sqliteMalloc(2*nCol*sizeof(const char *) + 1); + if( azCols==0 ){ + goto exec_out; + } + + while( 1 ){ + int i; + rc = sqlite3_step(pStmt); + + /* Invoke the callback function if required */ + if( xCallback && (SQLITE_ROW==rc || + (SQLITE_DONE==rc && !nCallback && db->flags&SQLITE_NullCallback)) ){ + if( 0==nCallback ){ + for(i=0; ipVdbe==0 ){ + nChange = db->nChange; + } + if( rc!=SQLITE_SCHEMA ){ + nRetry = 0; + zSql = zLeftover; + while( isspace((unsigned char)zSql[0]) ) zSql++; + } + break; + } + } + + sqliteFree(azCols); + azCols = 0; + } + +exec_out: + if( pStmt ) sqlite3_finalize(pStmt); + if( azCols ) sqliteFree(azCols); + + rc = sqlite3ApiExit(0, rc); + if( rc!=SQLITE_OK && rc==sqlite3_errcode(db) && pzErrMsg ){ + *pzErrMsg = malloc(1+strlen(sqlite3_errmsg(db))); + if( *pzErrMsg ){ + strcpy(*pzErrMsg, sqlite3_errmsg(db)); + } + }else if( pzErrMsg ){ + *pzErrMsg = 0; + } + + return rc; +} Added: external/sqlite-source-3.3.4/main.c ============================================================================== --- (empty file) +++ external/sqlite-source-3.3.4/main.c Mon Apr 3 07:54:59 2006 @@ -0,0 +1,1233 @@ +/* +** 2001 September 15 +** +** The author disclaims copyright to this source code. In place of +** a legal notice, here is a blessing: +** +** May you do good and not evil. +** May you find forgiveness for yourself and forgive others. +** May you share freely, never taking more than you give. +** +************************************************************************* +** Main file for the SQLite library. The routines in this file +** implement the programmer interface to the library. Routines in +** other files are for internal use by SQLite and should not be +** accessed by users of the library. +** +** $Id: main.c,v 1.334 2006/02/09 13:43:29 danielk1977 Exp $ +*/ +#include "sqliteInt.h" +#include "os.h" +#include + +/* +** The following constant value is used by the SQLITE_BIGENDIAN and +** SQLITE_LITTLEENDIAN macros. +*/ +const int sqlite3one = 1; + +/* +** The version of the library +*/ +const char sqlite3_version[] = SQLITE_VERSION; +const char *sqlite3_libversion(void){ return sqlite3_version; } +int sqlite3_libversion_number(void){ return SQLITE_VERSION_NUMBER; } + +/* +** This is the default collating function named "BINARY" which is always +** available. +*/ +static int binCollFunc( + void *NotUsed, + int nKey1, const void *pKey1, + int nKey2, const void *pKey2 +){ + int rc, n; + n = nKey1lastRowid; +} + +/* +** Return the number of changes in the most recent call to sqlite3_exec(). +*/ +int sqlite3_changes(sqlite3 *db){ + return db->nChange; +} + +/* +** Return the number of changes since the database handle was opened. +*/ +int sqlite3_total_changes(sqlite3 *db){ + return db->nTotalChange; +} + +/* +** Close an existing SQLite database +*/ +int sqlite3_close(sqlite3 *db){ + HashElem *i; + int j; + + if( !db ){ + return SQLITE_OK; + } + if( sqlite3SafetyCheck(db) ){ + return SQLITE_MISUSE; + } + +#ifdef SQLITE_SSE + sqlite3_finalize(db->pFetch); +#endif + + /* If there are any outstanding VMs, return SQLITE_BUSY. */ + if( db->pVdbe ){ + sqlite3Error(db, SQLITE_BUSY, + "Unable to close due to unfinalised statements"); + return SQLITE_BUSY; + } + assert( !sqlite3SafetyCheck(db) ); + + /* FIX ME: db->magic may be set to SQLITE_MAGIC_CLOSED if the database + ** cannot be opened for some reason. So this routine needs to run in + ** that case. But maybe there should be an extra magic value for the + ** "failed to open" state. + */ + if( db->magic!=SQLITE_MAGIC_CLOSED && sqlite3SafetyOn(db) ){ + /* printf("DID NOT CLOSE\n"); fflush(stdout); */ + return SQLITE_ERROR; + } + + for(j=0; jnDb; j++){ + struct Db *pDb = &db->aDb[j]; + if( pDb->pBt ){ + sqlite3BtreeClose(pDb->pBt); + pDb->pBt = 0; + if( j!=1 ){ + pDb->pSchema = 0; + } + } + } + sqlite3ResetInternalSchema(db, 0); + assert( db->nDb<=2 ); + assert( db->aDb==db->aDbStatic ); + for(i=sqliteHashFirst(&db->aFunc); i; i=sqliteHashNext(i)){ + FuncDef *pFunc, *pNext; + for(pFunc = (FuncDef*)sqliteHashData(i); pFunc; pFunc=pNext){ + pNext = pFunc->pNext; + sqliteFree(pFunc); + } + } + + for(i=sqliteHashFirst(&db->aCollSeq); i; i=sqliteHashNext(i)){ + CollSeq *pColl = (CollSeq *)sqliteHashData(i); + sqliteFree(pColl); + } + sqlite3HashClear(&db->aCollSeq); + + sqlite3HashClear(&db->aFunc); + sqlite3Error(db, SQLITE_OK, 0); /* Deallocates any cached error strings. */ + if( db->pErr ){ + sqlite3ValueFree(db->pErr); + } + + db->magic = SQLITE_MAGIC_ERROR; + + /* The temp-database schema is allocated differently from the other schema + ** objects (using sqliteMalloc() directly, instead of sqlite3BtreeSchema()). + ** So it needs to be freed here. Todo: Why not roll the temp schema into + ** the same sqliteMalloc() as the one that allocates the database + ** structure? + */ + sqliteFree(db->aDb[1].pSchema); + sqliteFree(db); + sqlite3ReleaseThreadData(); + return SQLITE_OK; +} + +/* +** Rollback all database files. +*/ +void sqlite3RollbackAll(sqlite3 *db){ + int i; + int inTrans = 0; + for(i=0; inDb; i++){ + if( db->aDb[i].pBt ){ + if( sqlite3BtreeIsInTrans(db->aDb[i].pBt) ){ + inTrans = 1; + } + sqlite3BtreeRollback(db->aDb[i].pBt); + db->aDb[i].inTrans = 0; + } + } + if( db->flags&SQLITE_InternChanges ){ + sqlite3ResetInternalSchema(db, 0); + } + + /* If one has been configured, invoke the rollback-hook callback */ + if( db->xRollbackCallback && (inTrans || !db->autoCommit) ){ + db->xRollbackCallback(db->pRollbackArg); + } +} + +/* +** Return a static string that describes the kind of error specified in the +** argument. +*/ +const char *sqlite3ErrStr(int rc){ + const char *z; + switch( rc ){ + case SQLITE_ROW: + case SQLITE_DONE: + case SQLITE_OK: z = "not an error"; break; + case SQLITE_ERROR: z = "SQL logic error or missing database"; break; + case SQLITE_PERM: z = "access permission denied"; break; + case SQLITE_ABORT: z = "callback requested query abort"; break; + case SQLITE_BUSY: z = "database is locked"; break; + case SQLITE_LOCKED: z = "database table is locked"; break; + case SQLITE_NOMEM: z = "out of memory"; break; + case SQLITE_READONLY: z = "attempt to write a readonly database"; break; + case SQLITE_INTERRUPT: z = "interrupted"; break; + case SQLITE_IOERR: z = "disk I/O error"; break; + case SQLITE_CORRUPT: z = "database disk image is malformed"; break; + case SQLITE_FULL: z = "database or disk is full"; break; + case SQLITE_CANTOPEN: z = "unable to open database file"; break; + case SQLITE_PROTOCOL: z = "database locking protocol failure"; break; + case SQLITE_EMPTY: z = "table contains no data"; break; + case SQLITE_SCHEMA: z = "database schema has changed"; break; + case SQLITE_CONSTRAINT: z = "constraint failed"; break; + case SQLITE_MISMATCH: z = "datatype mismatch"; break; + case SQLITE_MISUSE: z = "library routine called out of sequence";break; + case SQLITE_NOLFS: z = "kernel lacks large file support"; break; + case SQLITE_AUTH: z = "authorization denied"; break; + case SQLITE_FORMAT: z = "auxiliary database format error"; break; + case SQLITE_RANGE: z = "bind or column index out of range"; break; + case SQLITE_NOTADB: z = "file is encrypted or is not a database";break; + default: z = "unknown error"; break; + } + return z; +} + +/* +** This routine implements a busy callback that sleeps and tries +** again until a timeout value is reached. The timeout value is +** an integer number of milliseconds passed in as the first +** argument. +*/ +static int sqliteDefaultBusyCallback( + void *ptr, /* Database connection */ + int count /* Number of times table has been busy */ +){ +#if SQLITE_MIN_SLEEP_MS==1 + static const u8 delays[] = + { 1, 2, 5, 10, 15, 20, 25, 25, 25, 50, 50, 100 }; + static const u8 totals[] = + { 0, 1, 3, 8, 18, 33, 53, 78, 103, 128, 178, 228 }; +# define NDELAY (sizeof(delays)/sizeof(delays[0])) + int timeout = ((sqlite3 *)ptr)->busyTimeout; + int delay, prior; + + assert( count>=0 ); + if( count < NDELAY ){ + delay = delays[count]; + prior = totals[count]; + }else{ + delay = delays[NDELAY-1]; + prior = totals[NDELAY-1] + delay*(count-(NDELAY-1)); + } + if( prior + delay > timeout ){ + delay = timeout - prior; + if( delay<=0 ) return 0; + } + sqlite3OsSleep(delay); + return 1; +#else + int timeout = ((sqlite3 *)ptr)->busyTimeout; + if( (count+1)*1000 > timeout ){ + return 0; + } + sqlite3OsSleep(1000); + return 1; +#endif +} + +/* +** Invoke the given busy handler. +** +** This routine is called when an operation failed with a lock. +** If this routine returns non-zero, the lock is retried. If it +** returns 0, the operation aborts with an SQLITE_BUSY error. +*/ +int sqlite3InvokeBusyHandler(BusyHandler *p){ + int rc; + if( p==0 || p->xFunc==0 || p->nBusy<0 ) return 0; + rc = p->xFunc(p->pArg, p->nBusy); + if( rc==0 ){ + p->nBusy = -1; + }else{ + p->nBusy++; + } + return rc; +} + +/* +** This routine sets the busy callback for an Sqlite database to the +** given callback function with the given argument. +*/ +int sqlite3_busy_handler( + sqlite3 *db, + int (*xBusy)(void*,int), + void *pArg +){ + if( sqlite3SafetyCheck(db) ){ + return SQLITE_MISUSE; + } + db->busyHandler.xFunc = xBusy; + db->busyHandler.pArg = pArg; + db->busyHandler.nBusy = 0; + return SQLITE_OK; +} + +#ifndef SQLITE_OMIT_PROGRESS_CALLBACK +/* +** This routine sets the progress callback for an Sqlite database to the +** given callback function with the given argument. The progress callback will +** be invoked every nOps opcodes. +*/ +void sqlite3_progress_handler( + sqlite3 *db, + int nOps, + int (*xProgress)(void*), + void *pArg +){ + if( !sqlite3SafetyCheck(db) ){ + if( nOps>0 ){ + db->xProgress = xProgress; + db->nProgressOps = nOps; + db->pProgressArg = pArg; + }else{ + db->xProgress = 0; + db->nProgressOps = 0; + db->pProgressArg = 0; + } + } +} +#endif + + +/* +** This routine installs a default busy handler that waits for the +** specified number of milliseconds before returning 0. +*/ +int sqlite3_busy_timeout(sqlite3 *db, int ms){ + if( ms>0 ){ + db->busyTimeout = ms; + sqlite3_busy_handler(db, sqliteDefaultBusyCallback, (void*)db); + }else{ + sqlite3_busy_handler(db, 0, 0); + } + return SQLITE_OK; +} + +/* +** Cause any pending operation to stop at its earliest opportunity. +*/ +void sqlite3_interrupt(sqlite3 *db){ + if( !sqlite3SafetyCheck(db) ){ + db->flags |= SQLITE_Interrupt; + } +} + +/* +** Windows systems should call this routine to free memory that +** is returned in the in the errmsg parameter of sqlite3_open() when +** SQLite is a DLL. For some reason, it does not work to call free() +** directly. +** +** Note that we need to call free() not sqliteFree() here. +*/ +void sqlite3_free(char *p){ free(p); } + +/* +** This function is exactly the same as sqlite3_create_function(), except +** that it is designed to be called by internal code. The difference is +** that if a malloc() fails in sqlite3_create_function(), an error code +** is returned and the mallocFailed flag cleared. +*/ +int sqlite3CreateFunc( + sqlite3 *db, + const char *zFunctionName, + int nArg, + int enc, + void *pUserData, + void (*xFunc)(sqlite3_context*,int,sqlite3_value **), + void (*xStep)(sqlite3_context*,int,sqlite3_value **), + void (*xFinal)(sqlite3_context*) +){ + FuncDef *p; + int nName; + + if( sqlite3SafetyCheck(db) ){ + return SQLITE_MISUSE; + } + if( zFunctionName==0 || + (xFunc && (xFinal || xStep)) || + (!xFunc && (xFinal && !xStep)) || + (!xFunc && (!xFinal && xStep)) || + (nArg<-1 || nArg>127) || + (255<(nName = strlen(zFunctionName))) ){ + return SQLITE_ERROR; + } + +#ifndef SQLITE_OMIT_UTF16 + /* If SQLITE_UTF16 is specified as the encoding type, transform this + ** to one of SQLITE_UTF16LE or SQLITE_UTF16BE using the + ** SQLITE_UTF16NATIVE macro. SQLITE_UTF16 is not used internally. + ** + ** If SQLITE_ANY is specified, add three versions of the function + ** to the hash table. + */ + if( enc==SQLITE_UTF16 ){ + enc = SQLITE_UTF16NATIVE; + }else if( enc==SQLITE_ANY ){ + int rc; + rc = sqlite3CreateFunc(db, zFunctionName, nArg, SQLITE_UTF8, + pUserData, xFunc, xStep, xFinal); + if( rc!=SQLITE_OK ) return rc; + rc = sqlite3CreateFunc(db, zFunctionName, nArg, SQLITE_UTF16LE, + pUserData, xFunc, xStep, xFinal); + if( rc!=SQLITE_OK ) return rc; + enc = SQLITE_UTF16BE; + } +#else + enc = SQLITE_UTF8; +#endif + + /* Check if an existing function is being overridden or deleted. If so, + ** and there are active VMs, then return SQLITE_BUSY. If a function + ** is being overridden/deleted but there are no active VMs, allow the + ** operation to continue but invalidate all precompiled statements. + */ + p = sqlite3FindFunction(db, zFunctionName, nName, nArg, enc, 0); + if( p && p->iPrefEnc==enc && p->nArg==nArg ){ + if( db->activeVdbeCnt ){ + sqlite3Error(db, SQLITE_BUSY, + "Unable to delete/modify user-function due to active statements"); + assert( !sqlite3MallocFailed() ); + return SQLITE_BUSY; + }else{ + sqlite3ExpirePreparedStatements(db); + } + } + + p = sqlite3FindFunction(db, zFunctionName, nName, nArg, enc, 1); + if( p ){ + p->flags = 0; + p->xFunc = xFunc; + p->xStep = xStep; + p->xFinalize = xFinal; + p->pUserData = pUserData; + } + return SQLITE_OK; +} + +/* +** Create new user functions. +*/ +int sqlite3_create_function( + sqlite3 *db, + const char *zFunctionName, + int nArg, + int enc, + void *p, + void (*xFunc)(sqlite3_context*,int,sqlite3_value **), + void (*xStep)(sqlite3_context*,int,sqlite3_value **), + void (*xFinal)(sqlite3_context*) +){ + int rc; + assert( !sqlite3MallocFailed() ); + rc = sqlite3CreateFunc(db, zFunctionName, nArg, enc, p, xFunc, xStep, xFinal); + + return sqlite3ApiExit(db, rc); +} + +#ifndef SQLITE_OMIT_UTF16 +int sqlite3_create_function16( + sqlite3 *db, + const void *zFunctionName, + int nArg, + int eTextRep, + void *p, + void (*xFunc)(sqlite3_context*,int,sqlite3_value**), + void (*xStep)(sqlite3_context*,int,sqlite3_value**), + void (*xFinal)(sqlite3_context*) +){ + int rc; + char *zFunc8; + assert( !sqlite3MallocFailed() ); + + zFunc8 = sqlite3utf16to8(zFunctionName, -1); + rc = sqlite3CreateFunc(db, zFunc8, nArg, eTextRep, p, xFunc, xStep, xFinal); + sqliteFree(zFunc8); + + return sqlite3ApiExit(db, rc); +} +#endif + +#ifndef SQLITE_OMIT_TRACE +/* +** Register a trace function. The pArg from the previously registered trace +** is returned. +** +** A NULL trace function means that no tracing is executes. A non-NULL +** trace is a pointer to a function that is invoked at the start of each +** SQL statement. +*/ +void *sqlite3_trace(sqlite3 *db, void (*xTrace)(void*,const char*), void *pArg){ + void *pOld = db->pTraceArg; + db->xTrace = xTrace; + db->pTraceArg = pArg; + return pOld; +} +/* +** Register a profile function. The pArg from the previously registered +** profile function is returned. +** +** A NULL profile function means that no profiling is executes. A non-NULL +** profile is a pointer to a function that is invoked at the conclusion of +** each SQL statement that is run. +*/ +void *sqlite3_profile( + sqlite3 *db, + void (*xProfile)(void*,const char*,sqlite_uint64), + void *pArg +){ + void *pOld = db->pProfileArg; + db->xProfile = xProfile; + db->pProfileArg = pArg; + return pOld; +} +#endif /* SQLITE_OMIT_TRACE */ + +/*** EXPERIMENTAL *** +** +** Register a function to be invoked when a transaction comments. +** If the invoked function returns non-zero, then the commit becomes a +** rollback. +*/ +void *sqlite3_commit_hook( + sqlite3 *db, /* Attach the hook to this database */ + int (*xCallback)(void*), /* Function to invoke on each commit */ + void *pArg /* Argument to the function */ +){ + void *pOld = db->pCommitArg; + db->xCommitCallback = xCallback; + db->pCommitArg = pArg; + return pOld; +} + +/* +** Register a callback to be invoked each time a row is updated, +** inserted or deleted using this database connection. +*/ +void *sqlite3_update_hook( + sqlite3 *db, /* Attach the hook to this database */ + void (*xCallback)(void*,int,char const *,char const *,sqlite_int64), + void *pArg /* Argument to the function */ +){ + void *pRet = db->pUpdateArg; + db->xUpdateCallback = xCallback; + db->pUpdateArg = pArg; + return pRet; +} + +/* +** Register a callback to be invoked each time a transaction is rolled +** back by this database connection. +*/ +void *sqlite3_rollback_hook( + sqlite3 *db, /* Attach the hook to this database */ + void (*xCallback)(void*), /* Callback function */ + void *pArg /* Argument to the function */ +){ + void *pRet = db->pRollbackArg; + db->xRollbackCallback = xCallback; + db->pRollbackArg = pArg; + return pRet; +} + +/* +** This routine is called to create a connection to a database BTree +** driver. If zFilename is the name of a file, then that file is +** opened and used. If zFilename is the magic name ":memory:" then +** the database is stored in memory (and is thus forgotten as soon as +** the connection is closed.) If zFilename is NULL then the database +** is a "virtual" database for transient use only and is deleted as +** soon as the connection is closed. +** +** A virtual database can be either a disk file (that is automatically +** deleted when the file is closed) or it an be held entirely in memory, +** depending on the values of the TEMP_STORE compile-time macro and the +** db->temp_store variable, according to the following chart: +** +** TEMP_STORE db->temp_store Location of temporary database +** ---------- -------------- ------------------------------ +** 0 any file +** 1 1 file +** 1 2 memory +** 1 0 file +** 2 1 file +** 2 2 memory +** 2 0 memory +** 3 any memory +*/ +int sqlite3BtreeFactory( + const sqlite3 *db, /* Main database when opening aux otherwise 0 */ + const char *zFilename, /* Name of the file containing the BTree database */ + int omitJournal, /* if TRUE then do not journal this file */ + int nCache, /* How many pages in the page cache */ + Btree **ppBtree /* Pointer to new Btree object written here */ +){ + int btree_flags = 0; + int rc; + + assert( ppBtree != 0); + if( omitJournal ){ + btree_flags |= BTREE_OMIT_JOURNAL; + } + if( db->flags & SQLITE_NoReadlock ){ + btree_flags |= BTREE_NO_READLOCK; + } + if( zFilename==0 ){ +#if TEMP_STORE==0 + /* Do nothing */ +#endif +#ifndef SQLITE_OMIT_MEMORYDB +#if TEMP_STORE==1 + if( db->temp_store==2 ) zFilename = ":memory:"; +#endif +#if TEMP_STORE==2 + if( db->temp_store!=1 ) zFilename = ":memory:"; +#endif +#if TEMP_STORE==3 + zFilename = ":memory:"; +#endif +#endif /* SQLITE_OMIT_MEMORYDB */ + } + + rc = sqlite3BtreeOpen(zFilename, (sqlite3 *)db, ppBtree, btree_flags); + if( rc==SQLITE_OK ){ + sqlite3BtreeSetBusyHandler(*ppBtree, (void*)&db->busyHandler); + sqlite3BtreeSetCacheSize(*ppBtree, nCache); + } + return rc; +} + +/* +** Return UTF-8 encoded English language explanation of the most recent +** error. +*/ +const char *sqlite3_errmsg(sqlite3 *db){ + const char *z; + if( !db || sqlite3MallocFailed() ){ + return sqlite3ErrStr(SQLITE_NOMEM); + } + if( sqlite3SafetyCheck(db) || db->errCode==SQLITE_MISUSE ){ + return sqlite3ErrStr(SQLITE_MISUSE); + } + z = (char*)sqlite3_value_text(db->pErr); + if( z==0 ){ + z = sqlite3ErrStr(db->errCode); + } + return z; +} + +#ifndef SQLITE_OMIT_UTF16 +/* +** Return UTF-16 encoded English language explanation of the most recent +** error. +*/ +const void *sqlite3_errmsg16(sqlite3 *db){ + /* Because all the characters in the string are in the unicode + ** range 0x00-0xFF, if we pad the big-endian string with a + ** zero byte, we can obtain the little-endian string with + ** &big_endian[1]. + */ + static const char outOfMemBe[] = { + 0, 'o', 0, 'u', 0, 't', 0, ' ', + 0, 'o', 0, 'f', 0, ' ', + 0, 'm', 0, 'e', 0, 'm', 0, 'o', 0, 'r', 0, 'y', 0, 0, 0 + }; + static const char misuseBe [] = { + 0, 'l', 0, 'i', 0, 'b', 0, 'r', 0, 'a', 0, 'r', 0, 'y', 0, ' ', + 0, 'r', 0, 'o', 0, 'u', 0, 't', 0, 'i', 0, 'n', 0, 'e', 0, ' ', + 0, 'c', 0, 'a', 0, 'l', 0, 'l', 0, 'e', 0, 'd', 0, ' ', + 0, 'o', 0, 'u', 0, 't', 0, ' ', + 0, 'o', 0, 'f', 0, ' ', + 0, 's', 0, 'e', 0, 'q', 0, 'u', 0, 'e', 0, 'n', 0, 'c', 0, 'e', 0, 0, 0 + }; + + const void *z; + if( sqlite3MallocFailed() ){ + return (void *)(&outOfMemBe[SQLITE_UTF16NATIVE==SQLITE_UTF16LE?1:0]); + } + if( sqlite3SafetyCheck(db) || db->errCode==SQLITE_MISUSE ){ + return (void *)(&misuseBe[SQLITE_UTF16NATIVE==SQLITE_UTF16LE?1:0]); + } + z = sqlite3_value_text16(db->pErr); + if( z==0 ){ + sqlite3ValueSetStr(db->pErr, -1, sqlite3ErrStr(db->errCode), + SQLITE_UTF8, SQLITE_STATIC); + z = sqlite3_value_text16(db->pErr); + } + sqlite3ApiExit(0, 0); + return z; +} +#endif /* SQLITE_OMIT_UTF16 */ + +/* +** Return the most recent error code generated by an SQLite routine. If NULL is +** passed to this function, we assume a malloc() failed during sqlite3_open(). +*/ +int sqlite3_errcode(sqlite3 *db){ + if( !db || sqlite3MallocFailed() ){ + return SQLITE_NOMEM; + } + if( sqlite3SafetyCheck(db) ){ + return SQLITE_MISUSE; + } + return db->errCode; +} + +/* +** Create a new collating function for database "db". The name is zName +** and the encoding is enc. +*/ +static int createCollation( + sqlite3* db, + const char *zName, + int enc, + void* pCtx, + int(*xCompare)(void*,int,const void*,int,const void*) +){ + CollSeq *pColl; + + if( sqlite3SafetyCheck(db) ){ + return SQLITE_MISUSE; + } + + /* If SQLITE_UTF16 is specified as the encoding type, transform this + ** to one of SQLITE_UTF16LE or SQLITE_UTF16BE using the + ** SQLITE_UTF16NATIVE macro. SQLITE_UTF16 is not used internally. + */ + if( enc==SQLITE_UTF16 ){ + enc = SQLITE_UTF16NATIVE; + } + + if( enc!=SQLITE_UTF8 && enc!=SQLITE_UTF16LE && enc!=SQLITE_UTF16BE ){ + sqlite3Error(db, SQLITE_ERROR, + "Param 3 to sqlite3_create_collation() must be one of " + "SQLITE_UTF8, SQLITE_UTF16, SQLITE_UTF16LE or SQLITE_UTF16BE" + ); + return SQLITE_ERROR; + } + + /* Check if this call is removing or replacing an existing collation + ** sequence. If so, and there are active VMs, return busy. If there + ** are no active VMs, invalidate any pre-compiled statements. + */ + pColl = sqlite3FindCollSeq(db, (u8)enc, zName, strlen(zName), 0); + if( pColl && pColl->xCmp ){ + if( db->activeVdbeCnt ){ + sqlite3Error(db, SQLITE_BUSY, + "Unable to delete/modify collation sequence due to active statements"); + return SQLITE_BUSY; + } + sqlite3ExpirePreparedStatements(db); + } + + pColl = sqlite3FindCollSeq(db, (u8)enc, zName, strlen(zName), 1); + if( pColl ){ + pColl->xCmp = xCompare; + pColl->pUser = pCtx; + pColl->enc = enc; + } + sqlite3Error(db, SQLITE_OK, 0); + return SQLITE_OK; +} + + +/* +** This routine does the work of opening a database on behalf of +** sqlite3_open() and sqlite3_open16(). The database filename "zFilename" +** is UTF-8 encoded. +*/ +static int openDatabase( + const char *zFilename, /* Database filename UTF-8 encoded */ + sqlite3 **ppDb /* OUT: Returned database handle */ +){ + sqlite3 *db; + int rc; + CollSeq *pColl; + + assert( !sqlite3MallocFailed() ); + + /* Allocate the sqlite data structure */ + db = sqliteMalloc( sizeof(sqlite3) ); + if( db==0 ) goto opendb_out; + db->priorNewRowid = 0; + db->magic = SQLITE_MAGIC_BUSY; + db->nDb = 2; + db->aDb = db->aDbStatic; + db->autoCommit = 1; + db->flags |= SQLITE_ShortColNames; + sqlite3HashInit(&db->aFunc, SQLITE_HASH_STRING, 0); + sqlite3HashInit(&db->aCollSeq, SQLITE_HASH_STRING, 0); + + /* Add the default collation sequence BINARY. BINARY works for both UTF-8 + ** and UTF-16, so add a version for each to avoid any unnecessary + ** conversions. The only error that can occur here is a malloc() failure. + */ + if( createCollation(db, "BINARY", SQLITE_UTF8, 0, binCollFunc) || + createCollation(db, "BINARY", SQLITE_UTF16BE, 0, binCollFunc) || + createCollation(db, "BINARY", SQLITE_UTF16LE, 0, binCollFunc) || + (db->pDfltColl = sqlite3FindCollSeq(db, SQLITE_UTF8, "BINARY", 6, 0))==0 + ){ + assert( sqlite3MallocFailed() ); + db->magic = SQLITE_MAGIC_CLOSED; + goto opendb_out; + } + + /* Also add a UTF-8 case-insensitive collation sequence. */ + createCollation(db, "NOCASE", SQLITE_UTF8, 0, nocaseCollatingFunc); + + /* Set flags on the built-in collating sequences */ + db->pDfltColl->type = SQLITE_COLL_BINARY; + pColl = sqlite3FindCollSeq(db, SQLITE_UTF8, "NOCASE", 6, 0); + if( pColl ){ + pColl->type = SQLITE_COLL_NOCASE; + } + + /* Open the backend database driver */ + rc = sqlite3BtreeFactory(db, zFilename, 0, MAX_PAGES, &db->aDb[0].pBt); + if( rc!=SQLITE_OK ){ + sqlite3Error(db, rc, 0); + db->magic = SQLITE_MAGIC_CLOSED; + goto opendb_out; + } +#ifndef SQLITE_OMIT_PARSER + db->aDb[0].pSchema = sqlite3SchemaGet(db->aDb[0].pBt); + db->aDb[1].pSchema = sqlite3SchemaGet(0); +#endif + + if( db->aDb[0].pSchema ){ + ENC(db) = SQLITE_UTF8; + } + + /* The default safety_level for the main database is 'full'; for the temp + ** database it is 'NONE'. This matches the pager layer defaults. + */ + db->aDb[0].zName = "main"; + db->aDb[0].safety_level = 3; +#ifndef SQLITE_OMIT_TEMPDB + db->aDb[1].zName = "temp"; + db->aDb[1].safety_level = 1; +#endif + + /* Register all built-in functions, but do not attempt to read the + ** database schema yet. This is delayed until the first time the database + ** is accessed. + */ + if( !sqlite3MallocFailed() ){ + sqlite3RegisterBuiltinFunctions(db); + sqlite3Error(db, SQLITE_OK, 0); + } + db->magic = SQLITE_MAGIC_OPEN; + +opendb_out: + if( SQLITE_NOMEM==(rc = sqlite3_errcode(db)) ){ + sqlite3_close(db); + db = 0; + } + *ppDb = db; + return sqlite3ApiExit(0, rc); +} + +/* +** Open a new database handle. +*/ +int sqlite3_open( + const char *zFilename, + sqlite3 **ppDb +){ + return openDatabase(zFilename, ppDb); +} + +#ifndef SQLITE_OMIT_UTF16 +/* +** Open a new database handle. +*/ +int sqlite3_open16( + const void *zFilename, + sqlite3 **ppDb +){ + char const *zFilename8; /* zFilename encoded in UTF-8 instead of UTF-16 */ + int rc = SQLITE_OK; + sqlite3_value *pVal; + + assert( zFilename ); + assert( ppDb ); + *ppDb = 0; + pVal = sqlite3ValueNew(); + sqlite3ValueSetStr(pVal, -1, zFilename, SQLITE_UTF16NATIVE, SQLITE_STATIC); + zFilename8 = sqlite3ValueText(pVal, SQLITE_UTF8); + if( zFilename8 ){ + rc = openDatabase(zFilename8, ppDb); + if( rc==SQLITE_OK && *ppDb ){ + rc = sqlite3_exec(*ppDb, "PRAGMA encoding = 'UTF-16'", 0, 0, 0); + if( rc!=SQLITE_OK ){ + sqlite3_close(*ppDb); + *ppDb = 0; + } + } + } + sqlite3ValueFree(pVal); + + return sqlite3ApiExit(0, rc); +} +#endif /* SQLITE_OMIT_UTF16 */ + +/* +** The following routine destroys a virtual machine that is created by +** the sqlite3_compile() routine. The integer returned is an SQLITE_ +** success/failure code that describes the result of executing the virtual +** machine. +** +** This routine sets the error code and string returned by +** sqlite3_errcode(), sqlite3_errmsg() and sqlite3_errmsg16(). +*/ +int sqlite3_finalize(sqlite3_stmt *pStmt){ + int rc; + if( pStmt==0 ){ + rc = SQLITE_OK; + }else{ + rc = sqlite3VdbeFinalize((Vdbe*)pStmt); + } + return rc; +} + +/* +** Terminate the current execution of an SQL statement and reset it +** back to its starting state so that it can be reused. A success code from +** the prior execution is returned. +** +** This routine sets the error code and string returned by +** sqlite3_errcode(), sqlite3_errmsg() and sqlite3_errmsg16(). +*/ +int sqlite3_reset(sqlite3_stmt *pStmt){ + int rc; + if( pStmt==0 ){ + rc = SQLITE_OK; + }else{ + rc = sqlite3VdbeReset((Vdbe*)pStmt); + sqlite3VdbeMakeReady((Vdbe*)pStmt, -1, 0, 0, 0); + } + return rc; +} + +/* +** Register a new collation sequence with the database handle db. +*/ +int sqlite3_create_collation( + sqlite3* db, + const char *zName, + int enc, + void* pCtx, + int(*xCompare)(void*,int,const void*,int,const void*) +){ + int rc; + assert( !sqlite3MallocFailed() ); + rc = createCollation(db, zName, enc, pCtx, xCompare); + return sqlite3ApiExit(db, rc); +} + +#ifndef SQLITE_OMIT_UTF16 +/* +** Register a new collation sequence with the database handle db. +*/ +int sqlite3_create_collation16( + sqlite3* db, + const char *zName, + int enc, + void* pCtx, + int(*xCompare)(void*,int,const void*,int,const void*) +){ + int rc = SQLITE_OK; + char *zName8; + assert( !sqlite3MallocFailed() ); + zName8 = sqlite3utf16to8(zName, -1); + if( zName8 ){ + rc = createCollation(db, zName8, enc, pCtx, xCompare); + sqliteFree(zName8); + } + return sqlite3ApiExit(db, rc); +} +#endif /* SQLITE_OMIT_UTF16 */ + +/* +** Register a collation sequence factory callback with the database handle +** db. Replace any previously installed collation sequence factory. +*/ +int sqlite3_collation_needed( + sqlite3 *db, + void *pCollNeededArg, + void(*xCollNeeded)(void*,sqlite3*,int eTextRep,const char*) +){ + if( sqlite3SafetyCheck(db) ){ + return SQLITE_MISUSE; + } + db->xCollNeeded = xCollNeeded; + db->xCollNeeded16 = 0; + db->pCollNeededArg = pCollNeededArg; + return SQLITE_OK; +} + +#ifndef SQLITE_OMIT_UTF16 +/* +** Register a collation sequence factory callback with the database handle +** db. Replace any previously installed collation sequence factory. +*/ +int sqlite3_collation_needed16( + sqlite3 *db, + void *pCollNeededArg, + void(*xCollNeeded16)(void*,sqlite3*,int eTextRep,const void*) +){ + if( sqlite3SafetyCheck(db) ){ + return SQLITE_MISUSE; + } + db->xCollNeeded = 0; + db->xCollNeeded16 = xCollNeeded16; + db->pCollNeededArg = pCollNeededArg; + return SQLITE_OK; +} +#endif /* SQLITE_OMIT_UTF16 */ + +#ifndef SQLITE_OMIT_GLOBALRECOVER +/* +** This function is now an anachronism. It used to be used to recover from a +** malloc() failure, but SQLite now does this automatically. +*/ +int sqlite3_global_recover(){ + return SQLITE_OK; +} +#endif + +/* +** Test to see whether or not the database connection is in autocommit +** mode. Return TRUE if it is and FALSE if not. Autocommit mode is on +** by default. Autocommit is disabled by a BEGIN statement and reenabled +** by the next COMMIT or ROLLBACK. +** +******* THIS IS AN EXPERIMENTAL API AND IS SUBJECT TO CHANGE ****** +*/ +int sqlite3_get_autocommit(sqlite3 *db){ + return db->autoCommit; +} + +#ifdef SQLITE_DEBUG +/* +** The following routine is subtituted for constant SQLITE_CORRUPT in +** debugging builds. This provides a way to set a breakpoint for when +** corruption is first detected. +*/ +int sqlite3Corrupt(void){ + return SQLITE_CORRUPT; +} +#endif + + +#ifndef SQLITE_OMIT_SHARED_CACHE +/* +** Enable or disable the shared pager and schema features for the +** current thread. +** +** This routine should only be called when there are no open +** database connections. +*/ +int sqlite3_enable_shared_cache(int enable){ + ThreadData *pTd = sqlite3ThreadData(); + if( pTd ){ + /* It is only legal to call sqlite3_enable_shared_cache() when there + ** are no currently open b-trees that were opened by the calling thread. + ** This condition is only easy to detect if the shared-cache were + ** previously enabled (and is being disabled). + */ + if( pTd->pBtree && !enable ){ + assert( pTd->useSharedData ); + return SQLITE_MISUSE; + } + + pTd->useSharedData = enable; + sqlite3ReleaseThreadData(); + } + return sqlite3ApiExit(0, SQLITE_OK); +} +#endif + +/* +** This is a convenience routine that makes sure that all thread-specific +** data for this thread has been deallocated. +*/ +void sqlite3_thread_cleanup(void){ + ThreadData *pTd = sqlite3OsThreadSpecificData(0); + if( pTd ){ + memset(pTd, 0, sizeof(*pTd)); + sqlite3OsThreadSpecificData(-1); + } +} + +/* +** Return meta information about a specific column of a database table. +** See comment in sqlite3.h (sqlite.h.in) for details. +*/ +#ifdef SQLITE_ENABLE_COLUMN_METADATA +int sqlite3_table_column_metadata( + sqlite3 *db, /* Connection handle */ + const char *zDbName, /* Database name or NULL */ + const char *zTableName, /* Table name */ + const char *zColumnName, /* Column name */ + char const **pzDataType, /* OUTPUT: Declared data type */ + char const **pzCollSeq, /* OUTPUT: Collation sequence name */ + int *pNotNull, /* OUTPUT: True if NOT NULL constraint exists */ + int *pPrimaryKey, /* OUTPUT: True if column part of PK */ + int *pAutoinc /* OUTPUT: True if colums is auto-increment */ +){ + int rc; + char *zErrMsg = 0; + Table *pTab = 0; + Column *pCol = 0; + int iCol; + + char const *zDataType = 0; + char const *zCollSeq = 0; + int notnull = 0; + int primarykey = 0; + int autoinc = 0; + + /* Ensure the database schema has been loaded */ + if( sqlite3SafetyOn(db) ){ + return SQLITE_MISUSE; + } + rc = sqlite3Init(db, &zErrMsg); + if( SQLITE_OK!=rc ){ + goto error_out; + } + + /* Locate the table in question */ + pTab = sqlite3FindTable(db, zTableName, zDbName); + if( !pTab || pTab->pSelect ){ + pTab = 0; + goto error_out; + } + + /* Find the column for which info is requested */ + if( sqlite3IsRowid(zColumnName) ){ + iCol = pTab->iPKey; + if( iCol>=0 ){ + pCol = &pTab->aCol[iCol]; + } + }else{ + for(iCol=0; iColnCol; iCol++){ + pCol = &pTab->aCol[iCol]; + if( 0==sqlite3StrICmp(pCol->zName, zColumnName) ){ + break; + } + } + if( iCol==pTab->nCol ){ + pTab = 0; + goto error_out; + } + } + + /* The following block stores the meta information that will be returned + ** to the caller in local variables zDataType, zCollSeq, notnull, primarykey + ** and autoinc. At this point there are two possibilities: + ** + ** 1. The specified column name was rowid", "oid" or "_rowid_" + ** and there is no explicitly declared IPK column. + ** + ** 2. The table is not a view and the column name identified an + ** explicitly declared column. Copy meta information from *pCol. + */ + if( pCol ){ + zDataType = pCol->zType; + zCollSeq = pCol->zColl; + notnull = (pCol->notNull?1:0); + primarykey = (pCol->isPrimKey?1:0); + autoinc = ((pTab->iPKey==iCol && pTab->autoInc)?1:0); + }else{ + zDataType = "INTEGER"; + primarykey = 1; + } + if( !zCollSeq ){ + zCollSeq = "BINARY"; + } + +error_out: + if( sqlite3SafetyOff(db) ){ + rc = SQLITE_MISUSE; + } + + /* Whether the function call succeeded or failed, set the output parameters + ** to whatever their local counterparts contain. If an error did occur, + ** this has the effect of zeroing all output parameters. + */ + if( pzDataType ) *pzDataType = zDataType; + if( pzCollSeq ) *pzCollSeq = zCollSeq; + if( pNotNull ) *pNotNull = notnull; + if( pPrimaryKey ) *pPrimaryKey = primarykey; + if( pAutoinc ) *pAutoinc = autoinc; + + if( SQLITE_OK==rc && !pTab ){ + sqlite3SetString(&zErrMsg, "no such table column: ", zTableName, ".", + zColumnName, 0); + rc = SQLITE_ERROR; + } + sqlite3Error(db, rc, (zErrMsg?"%s":0), zErrMsg); + sqliteFree(zErrMsg); + return sqlite3ApiExit(db, rc); +} +#endif + Added: external/sqlite-source-3.3.4/opcodes.c ============================================================================== --- (empty file) +++ external/sqlite-source-3.3.4/opcodes.c Mon Apr 3 07:54:59 2006 @@ -0,0 +1,147 @@ +/* Automatically generated. Do not edit */ +/* See the mkopcodec.awk script for details. */ +#if !defined(SQLITE_OMIT_EXPLAIN) || !defined(NDEBUG) || defined(VDBE_PROFILE) || defined(SQLITE_DEBUG) +const char *const sqlite3OpcodeNames[] = { "?", + /* 1 */ "MemLoad", + /* 2 */ "Column", + /* 3 */ "SetCookie", + /* 4 */ "IfMemPos", + /* 5 */ "Sequence", + /* 6 */ "MoveGt", + /* 7 */ "RowKey", + /* 8 */ "OpenWrite", + /* 9 */ "If", + /* 10 */ "Pop", + /* 11 */ "CollSeq", + /* 12 */ "OpenRead", + /* 13 */ "Expire", + /* 14 */ "AutoCommit", + /* 15 */ "IntegrityCk", + /* 16 */ "Not", + /* 17 */ "Sort", + /* 18 */ "Function", + /* 19 */ "Noop", + /* 20 */ "Return", + /* 21 */ "NewRowid", + /* 22 */ "IfMemNeg", + /* 23 */ "Variable", + /* 24 */ "String", + /* 25 */ "RealAffinity", + /* 26 */ "ParseSchema", + /* 27 */ "Close", + /* 28 */ "CreateIndex", + /* 29 */ "IsUnique", + /* 30 */ "IdxIsNull", + /* 31 */ "NotFound", + /* 32 */ "Int64", + /* 33 */ "MustBeInt", + /* 34 */ "Halt", + /* 35 */ "Rowid", + /* 36 */ "IdxLT", + /* 37 */ "AddImm", + /* 38 */ "Statement", + /* 39 */ "RowData", + /* 40 */ "MemMax", + /* 41 */ "Push", + /* 42 */ "NotExists", + /* 43 */ "MemIncr", + /* 44 */ "Gosub", + /* 45 */ "Integer", + /* 46 */ "MemInt", + /* 47 */ "Prev", + /* 48 */ "CreateTable", + /* 49 */ "Last", + /* 50 */ "IdxRowid", + /* 51 */ "MakeIdxRec", + /* 52 */ "ResetCount", + /* 53 */ "FifoWrite", + /* 54 */ "Callback", + /* 55 */ "ContextPush", + /* 56 */ "DropTrigger", + /* 57 */ "DropIndex", + /* 58 */ "IdxGE", + /* 59 */ "Or", + /* 60 */ "And", + /* 61 */ "IdxDelete", + /* 62 */ "Vacuum", + /* 63 */ "MoveLe", + /* 64 */ "IsNull", + /* 65 */ "NotNull", + /* 66 */ "Ne", + /* 67 */ "Eq", + /* 68 */ "Gt", + /* 69 */ "Le", + /* 70 */ "Lt", + /* 71 */ "Ge", + /* 72 */ "IfNot", + /* 73 */ "BitAnd", + /* 74 */ "BitOr", + /* 75 */ "ShiftLeft", + /* 76 */ "ShiftRight", + /* 77 */ "Add", + /* 78 */ "Subtract", + /* 79 */ "Multiply", + /* 80 */ "Divide", + /* 81 */ "Remainder", + /* 82 */ "Concat", + /* 83 */ "Negative", + /* 84 */ "DropTable", + /* 85 */ "BitNot", + /* 86 */ "String8", + /* 87 */ "MakeRecord", + /* 88 */ "Delete", + /* 89 */ "AggFinal", + /* 90 */ "Dup", + /* 91 */ "Goto", + /* 92 */ "TableLock", + /* 93 */ "FifoRead", + /* 94 */ "Clear", + /* 95 */ "IdxGT", + /* 96 */ "MoveLt", + /* 97 */ "VerifyCookie", + /* 98 */ "AggStep", + /* 99 */ "Pull", + /* 100 */ "SetNumColumns", + /* 101 */ "AbsValue", + /* 102 */ "Transaction", + /* 103 */ "ContextPop", + /* 104 */ "Next", + /* 105 */ "IdxInsert", + /* 106 */ "Distinct", + /* 107 */ "Insert", + /* 108 */ "Destroy", + /* 109 */ "ReadCookie", + /* 110 */ "ForceInt", + /* 111 */ "LoadAnalysis", + /* 112 */ "OpenVirtual", + /* 113 */ "Explain", + /* 114 */ "IfMemZero", + /* 115 */ "OpenPseudo", + /* 116 */ "Null", + /* 117 */ "Blob", + /* 118 */ "MemStore", + /* 119 */ "Rewind", + /* 120 */ "MoveGe", + /* 121 */ "MemMove", + /* 122 */ "MemNull", + /* 123 */ "Found", + /* 124 */ "Real", + /* 125 */ "HexBlob", + /* 126 */ "NullRow", + /* 127 */ "NotUsed_127", + /* 128 */ "NotUsed_128", + /* 129 */ "NotUsed_129", + /* 130 */ "NotUsed_130", + /* 131 */ "NotUsed_131", + /* 132 */ "NotUsed_132", + /* 133 */ "NotUsed_133", + /* 134 */ "NotUsed_134", + /* 135 */ "NotUsed_135", + /* 136 */ "NotUsed_136", + /* 137 */ "ToText", + /* 138 */ "ToBlob", + /* 139 */ "ToNumeric", + /* 140 */ "ToInt", + /* 141 */ "ToReal", +}; +#endif Added: external/sqlite-source-3.3.4/opcodes.h ============================================================================== --- (empty file) +++ external/sqlite-source-3.3.4/opcodes.h Mon Apr 3 07:54:59 2006 @@ -0,0 +1,159 @@ +/* Automatically generated. Do not edit */ +/* See the mkopcodeh.awk script for details */ +#define OP_MemLoad 1 +#define OP_HexBlob 125 /* same as TK_BLOB */ +#define OP_Column 2 +#define OP_SetCookie 3 +#define OP_IfMemPos 4 +#define OP_Real 124 /* same as TK_FLOAT */ +#define OP_Sequence 5 +#define OP_MoveGt 6 +#define OP_Ge 71 /* same as TK_GE */ +#define OP_RowKey 7 +#define OP_Eq 67 /* same as TK_EQ */ +#define OP_OpenWrite 8 +#define OP_NotNull 65 /* same as TK_NOTNULL */ +#define OP_If 9 +#define OP_ToInt 140 /* same as TK_TO_INT */ +#define OP_String8 86 /* same as TK_STRING */ +#define OP_Pop 10 +#define OP_CollSeq 11 +#define OP_OpenRead 12 +#define OP_Expire 13 +#define OP_AutoCommit 14 +#define OP_Gt 68 /* same as TK_GT */ +#define OP_IntegrityCk 15 +#define OP_Sort 17 +#define OP_Function 18 +#define OP_And 60 /* same as TK_AND */ +#define OP_Subtract 78 /* same as TK_MINUS */ +#define OP_Noop 19 +#define OP_Return 20 +#define OP_Remainder 81 /* same as TK_REM */ +#define OP_NewRowid 21 +#define OP_Multiply 79 /* same as TK_STAR */ +#define OP_IfMemNeg 22 +#define OP_Variable 23 +#define OP_String 24 +#define OP_RealAffinity 25 +#define OP_ParseSchema 26 +#define OP_Close 27 +#define OP_CreateIndex 28 +#define OP_IsUnique 29 +#define OP_IdxIsNull 30 +#define OP_NotFound 31 +#define OP_Int64 32 +#define OP_MustBeInt 33 +#define OP_Halt 34 +#define OP_Rowid 35 +#define OP_IdxLT 36 +#define OP_AddImm 37 +#define OP_Statement 38 +#define OP_RowData 39 +#define OP_MemMax 40 +#define OP_Push 41 +#define OP_Or 59 /* same as TK_OR */ +#define OP_NotExists 42 +#define OP_MemIncr 43 +#define OP_Gosub 44 +#define OP_Divide 80 /* same as TK_SLASH */ +#define OP_Integer 45 +#define OP_ToNumeric 139 /* same as TK_TO_NUMERIC*/ +#define OP_MemInt 46 +#define OP_Prev 47 +#define OP_Concat 82 /* same as TK_CONCAT */ +#define OP_BitAnd 73 /* same as TK_BITAND */ +#define OP_CreateTable 48 +#define OP_Last 49 +#define OP_IsNull 64 /* same as TK_ISNULL */ +#define OP_IdxRowid 50 +#define OP_MakeIdxRec 51 +#define OP_ShiftRight 76 /* same as TK_RSHIFT */ +#define OP_ResetCount 52 +#define OP_FifoWrite 53 +#define OP_Callback 54 +#define OP_ContextPush 55 +#define OP_DropTrigger 56 +#define OP_DropIndex 57 +#define OP_IdxGE 58 +#define OP_IdxDelete 61 +#define OP_Vacuum 62 +#define OP_MoveLe 63 +#define OP_IfNot 72 +#define OP_DropTable 84 +#define OP_MakeRecord 87 +#define OP_ToBlob 138 /* same as TK_TO_BLOB */ +#define OP_Delete 88 +#define OP_AggFinal 89 +#define OP_ShiftLeft 75 /* same as TK_LSHIFT */ +#define OP_Dup 90 +#define OP_Goto 91 +#define OP_TableLock 92 +#define OP_FifoRead 93 +#define OP_Clear 94 +#define OP_IdxGT 95 +#define OP_MoveLt 96 +#define OP_Le 69 /* same as TK_LE */ +#define OP_VerifyCookie 97 +#define OP_AggStep 98 +#define OP_Pull 99 +#define OP_ToText 137 /* same as TK_TO_TEXT */ +#define OP_Not 16 /* same as TK_NOT */ +#define OP_ToReal 141 /* same as TK_TO_REAL */ +#define OP_SetNumColumns 100 +#define OP_AbsValue 101 +#define OP_Transaction 102 +#define OP_Negative 83 /* same as TK_UMINUS */ +#define OP_Ne 66 /* same as TK_NE */ +#define OP_ContextPop 103 +#define OP_BitOr 74 /* same as TK_BITOR */ +#define OP_Next 104 +#define OP_IdxInsert 105 +#define OP_Distinct 106 +#define OP_Lt 70 /* same as TK_LT */ +#define OP_Insert 107 +#define OP_Destroy 108 +#define OP_ReadCookie 109 +#define OP_ForceInt 110 +#define OP_LoadAnalysis 111 +#define OP_OpenVirtual 112 +#define OP_Explain 113 +#define OP_IfMemZero 114 +#define OP_OpenPseudo 115 +#define OP_Null 116 +#define OP_Blob 117 +#define OP_Add 77 /* same as TK_PLUS */ +#define OP_MemStore 118 +#define OP_Rewind 119 +#define OP_MoveGe 120 +#define OP_BitNot 85 /* same as TK_BITNOT */ +#define OP_MemMove 121 +#define OP_MemNull 122 +#define OP_Found 123 +#define OP_NullRow 126 + +/* The following opcode values are never used */ +#define OP_NotUsed_127 127 +#define OP_NotUsed_128 128 +#define OP_NotUsed_129 129 +#define OP_NotUsed_130 130 +#define OP_NotUsed_131 131 +#define OP_NotUsed_132 132 +#define OP_NotUsed_133 133 +#define OP_NotUsed_134 134 +#define OP_NotUsed_135 135 +#define OP_NotUsed_136 136 + +/* Opcodes that are guaranteed to never push a value onto the stack +** contain a 1 their corresponding position of the following mask +** set. See the opcodeNoPush() function in vdbeaux.c */ +#define NOPUSH_MASK_0 0x7f58 +#define NOPUSH_MASK_1 0xee5b +#define NOPUSH_MASK_2 0x9f76 +#define NOPUSH_MASK_3 0xfff2 +#define NOPUSH_MASK_4 0xffff +#define NOPUSH_MASK_5 0xdb3b +#define NOPUSH_MASK_6 0xcfdf +#define NOPUSH_MASK_7 0x49cd +#define NOPUSH_MASK_8 0x3e00 +#define NOPUSH_MASK_9 0x0000 Added: external/sqlite-source-3.3.4/os.c ============================================================================== --- (empty file) +++ external/sqlite-source-3.3.4/os.c Mon Apr 3 07:54:59 2006 @@ -0,0 +1,92 @@ +/* +** 2005 November 29 +** +** The author disclaims copyright to this source code. In place of +** a legal notice, here is a blessing: +** +** May you do good and not evil. +** May you find forgiveness for yourself and forgive others. +** May you share freely, never taking more than you give. +** +****************************************************************************** +** +** This file contains OS interface code that is common to all +** architectures. +*/ +#define _SQLITE_OS_C_ 1 +#include "sqliteInt.h" +#include "os.h" + +/* +** The following routines are convenience wrappers around methods +** of the OsFile object. This is mostly just syntactic sugar. All +** of this would be completely automatic if SQLite were coded using +** C++ instead of plain old C. +*/ +int sqlite3OsClose(OsFile **pId){ + OsFile *id; + if( pId!=0 && (id = *pId)!=0 ){ + return id->pMethod->xClose(pId); + }else{ + return SQLITE_OK; + } +} +int sqlite3OsOpenDirectory(OsFile *id, const char *zName){ + return id->pMethod->xOpenDirectory(id, zName); +} +int sqlite3OsRead(OsFile *id, void *pBuf, int amt){ + return id->pMethod->xRead(id, pBuf, amt); +} +int sqlite3OsWrite(OsFile *id, const void *pBuf, int amt){ + return id->pMethod->xWrite(id, pBuf, amt); +} +int sqlite3OsSeek(OsFile *id, i64 offset){ + return id->pMethod->xSeek(id, offset); +} +int sqlite3OsTruncate(OsFile *id, i64 size){ + return id->pMethod->xTruncate(id, size); +} +int sqlite3OsSync(OsFile *id, int fullsync){ + return id->pMethod->xSync(id, fullsync); +} +void sqlite3OsSetFullSync(OsFile *id, int value){ + id->pMethod->xSetFullSync(id, value); +} +#if defined(SQLITE_TEST) || defined(SQLITE_DEBUG) +/* This method is currently only used while interactively debugging the +** pager. More specificly, it can only be used when sqlite3DebugPrintf() is +** included in the build. */ +int sqlite3OsFileHandle(OsFile *id){ + return id->pMethod->xFileHandle(id); +} +#endif +int sqlite3OsFileSize(OsFile *id, i64 *pSize){ + return id->pMethod->xFileSize(id, pSize); +} +int sqlite3OsLock(OsFile *id, int lockType){ + return id->pMethod->xLock(id, lockType); +} +int sqlite3OsUnlock(OsFile *id, int lockType){ + return id->pMethod->xUnlock(id, lockType); +} +int sqlite3OsLockState(OsFile *id){ + return id->pMethod->xLockState(id); +} +int sqlite3OsCheckReservedLock(OsFile *id){ + return id->pMethod->xCheckReservedLock(id); +} + +#ifdef SQLITE_ENABLE_REDEF_IO +/* +** A function to return a pointer to the virtual function table. +** This routine really does not accomplish very much since the +** virtual function table is a global variable and anybody who +** can call this function can just as easily access the variable +** for themselves. Nevertheless, we include this routine for +** backwards compatibility with an earlier redefinable I/O +** interface design. +*/ +struct sqlite3OsVtbl *sqlite3_os_switch(void){ + return &sqlite3Os; +} +#endif Added: external/sqlite-source-3.3.4/os.h ============================================================================== --- (empty file) +++ external/sqlite-source-3.3.4/os.h Mon Apr 3 07:54:59 2006 @@ -0,0 +1,440 @@ +/* +** 2001 September 16 +** +** The author disclaims copyright to this source code. In place of +** a legal notice, here is a blessing: +** +** May you do good and not evil. +** May you find forgiveness for yourself and forgive others. +** May you share freely, never taking more than you give. +** +****************************************************************************** +** +** This header file (together with is companion C source-code file +** "os.c") attempt to abstract the underlying operating system so that +** the SQLite library will work on both POSIX and windows systems. +*/ +#ifndef _SQLITE_OS_H_ +#define _SQLITE_OS_H_ + +/* +** Figure out if we are dealing with Unix, Windows, or some other +** operating system. +*/ +#if !defined(OS_UNIX) && !defined(OS_OTHER) +# define OS_OTHER 0 +# ifndef OS_WIN +# if defined(_WIN32) || defined(WIN32) || defined(__CYGWIN__) || defined(__MINGW32__) || defined(__BORLANDC__) +# define OS_WIN 1 +# define OS_UNIX 0 +# else +# define OS_WIN 0 +# define OS_UNIX 1 +# endif +# else +# define OS_UNIX 0 +# endif +#else +# ifndef OS_WIN +# define OS_WIN 0 +# endif +#endif + + +/* +** Define the maximum size of a temporary filename +*/ +#if OS_WIN +# include +# define SQLITE_TEMPNAME_SIZE (MAX_PATH+50) +#else +# define SQLITE_TEMPNAME_SIZE 200 +#endif + +/* If the SET_FULLSYNC macro is not defined above, then make it +** a no-op +*/ +#ifndef SET_FULLSYNC +# define SET_FULLSYNC(x,y) +#endif + +/* +** Temporary files are named starting with this prefix followed by 16 random +** alphanumeric characters, and no file extension. They are stored in the +** OS's standard temporary file directory, and are deleted prior to exit. +** If sqlite is being embedded in another program, you may wish to change the +** prefix to reflect your program's name, so that if your program exits +** prematurely, old temporary files can be easily identified. This can be done +** using -DTEMP_FILE_PREFIX=myprefix_ on the compiler command line. +*/ +#ifndef TEMP_FILE_PREFIX +# define TEMP_FILE_PREFIX "sqlite_" +#endif + +/* +** Define the interfaces for Unix and for Windows. +*/ +#if OS_UNIX +#define sqlite3OsOpenReadWrite sqlite3UnixOpenReadWrite +#define sqlite3OsOpenExclusive sqlite3UnixOpenExclusive +#define sqlite3OsOpenReadOnly sqlite3UnixOpenReadOnly +#define sqlite3OsDelete sqlite3UnixDelete +#define sqlite3OsFileExists sqlite3UnixFileExists +#define sqlite3OsFullPathname sqlite3UnixFullPathname +#define sqlite3OsIsDirWritable sqlite3UnixIsDirWritable +#define sqlite3OsSyncDirectory sqlite3UnixSyncDirectory +#define sqlite3OsTempFileName sqlite3UnixTempFileName +#define sqlite3OsRandomSeed sqlite3UnixRandomSeed +#define sqlite3OsSleep sqlite3UnixSleep +#define sqlite3OsCurrentTime sqlite3UnixCurrentTime +#define sqlite3OsEnterMutex sqlite3UnixEnterMutex +#define sqlite3OsLeaveMutex sqlite3UnixLeaveMutex +#define sqlite3OsInMutex sqlite3UnixInMutex +#define sqlite3OsThreadSpecificData sqlite3UnixThreadSpecificData +#define sqlite3OsMalloc sqlite3GenericMalloc +#define sqlite3OsRealloc sqlite3GenericRealloc +#define sqlite3OsFree sqlite3GenericFree +#define sqlite3OsAllocationSize sqlite3GenericAllocationSize +#endif +#if OS_WIN +#define sqlite3OsOpenReadWrite sqlite3WinOpenReadWrite +#define sqlite3OsOpenExclusive sqlite3WinOpenExclusive +#define sqlite3OsOpenReadOnly sqlite3WinOpenReadOnly +#define sqlite3OsDelete sqlite3WinDelete +#define sqlite3OsFileExists sqlite3WinFileExists +#define sqlite3OsFullPathname sqlite3WinFullPathname +#define sqlite3OsIsDirWritable sqlite3WinIsDirWritable +#define sqlite3OsSyncDirectory sqlite3WinSyncDirectory +#define sqlite3OsTempFileName sqlite3WinTempFileName +#define sqlite3OsRandomSeed sqlite3WinRandomSeed +#define sqlite3OsSleep sqlite3WinSleep +#define sqlite3OsCurrentTime sqlite3WinCurrentTime +#define sqlite3OsEnterMutex sqlite3WinEnterMutex +#define sqlite3OsLeaveMutex sqlite3WinLeaveMutex +#define sqlite3OsInMutex sqlite3WinInMutex +#define sqlite3OsThreadSpecificData sqlite3WinThreadSpecificData +#define sqlite3OsMalloc sqlite3GenericMalloc +#define sqlite3OsRealloc sqlite3GenericRealloc +#define sqlite3OsFree sqlite3GenericFree +#define sqlite3OsAllocationSize sqlite3GenericAllocationSize +#endif + +/* +** If using an alternative OS interface, then we must have an "os_other.h" +** header file available for that interface. Presumably the "os_other.h" +** header file contains #defines similar to those above. +*/ +#if OS_OTHER +# include "os_other.h" +#endif + + + +/* +** Forward declarations +*/ +typedef struct OsFile OsFile; +typedef struct IoMethod IoMethod; + +/* +** An instance of the following structure contains pointers to all +** methods on an OsFile object. +*/ +struct IoMethod { + int (*xClose)(OsFile**); + int (*xOpenDirectory)(OsFile*, const char*); + int (*xRead)(OsFile*, void*, int amt); + int (*xWrite)(OsFile*, const void*, int amt); + int (*xSeek)(OsFile*, i64 offset); + int (*xTruncate)(OsFile*, i64 size); + int (*xSync)(OsFile*, int); + void (*xSetFullSync)(OsFile *id, int setting); + int (*xFileHandle)(OsFile *id); + int (*xFileSize)(OsFile*, i64 *pSize); + int (*xLock)(OsFile*, int); + int (*xUnlock)(OsFile*, int); + int (*xLockState)(OsFile *id); + int (*xCheckReservedLock)(OsFile *id); +}; + +/* +** The OsFile object describes an open disk file in an OS-dependent way. +** The version of OsFile defined here is a generic version. Each OS +** implementation defines its own subclass of this structure that contains +** additional information needed to handle file I/O. But the pMethod +** entry (pointing to the virtual function table) always occurs first +** so that we can always find the appropriate methods. +*/ +struct OsFile { + IoMethod const *pMethod; +}; + +/* +** The following values may be passed as the second argument to +** sqlite3OsLock(). The various locks exhibit the following semantics: +** +** SHARED: Any number of processes may hold a SHARED lock simultaneously. +** RESERVED: A single process may hold a RESERVED lock on a file at +** any time. Other processes may hold and obtain new SHARED locks. +** PENDING: A single process may hold a PENDING lock on a file at +** any one time. Existing SHARED locks may persist, but no new +** SHARED locks may be obtained by other processes. +** EXCLUSIVE: An EXCLUSIVE lock precludes all other locks. +** +** PENDING_LOCK may not be passed directly to sqlite3OsLock(). Instead, a +** process that requests an EXCLUSIVE lock may actually obtain a PENDING +** lock. This can be upgraded to an EXCLUSIVE lock by a subsequent call to +** sqlite3OsLock(). +*/ +#define NO_LOCK 0 +#define SHARED_LOCK 1 +#define RESERVED_LOCK 2 +#define PENDING_LOCK 3 +#define EXCLUSIVE_LOCK 4 + +/* +** File Locking Notes: (Mostly about windows but also some info for Unix) +** +** We cannot use LockFileEx() or UnlockFileEx() on Win95/98/ME because +** those functions are not available. So we use only LockFile() and +** UnlockFile(). +** +** LockFile() prevents not just writing but also reading by other processes. +** A SHARED_LOCK is obtained by locking a single randomly-chosen +** byte out of a specific range of bytes. The lock byte is obtained at +** random so two separate readers can probably access the file at the +** same time, unless they are unlucky and choose the same lock byte. +** An EXCLUSIVE_LOCK is obtained by locking all bytes in the range. +** There can only be one writer. A RESERVED_LOCK is obtained by locking +** a single byte of the file that is designated as the reserved lock byte. +** A PENDING_LOCK is obtained by locking a designated byte different from +** the RESERVED_LOCK byte. +** +** On WinNT/2K/XP systems, LockFileEx() and UnlockFileEx() are available, +** which means we can use reader/writer locks. When reader/writer locks +** are used, the lock is placed on the same range of bytes that is used +** for probabilistic locking in Win95/98/ME. Hence, the locking scheme +** will support two or more Win95 readers or two or more WinNT readers. +** But a single Win95 reader will lock out all WinNT readers and a single +** WinNT reader will lock out all other Win95 readers. +** +** The following #defines specify the range of bytes used for locking. +** SHARED_SIZE is the number of bytes available in the pool from which +** a random byte is selected for a shared lock. The pool of bytes for +** shared locks begins at SHARED_FIRST. +** +** These #defines are available in sqlite_aux.h so that adaptors for +** connecting SQLite to other operating systems can use the same byte +** ranges for locking. In particular, the same locking strategy and +** byte ranges are used for Unix. This leaves open the possiblity of having +** clients on win95, winNT, and unix all talking to the same shared file +** and all locking correctly. To do so would require that samba (or whatever +** tool is being used for file sharing) implements locks correctly between +** windows and unix. I'm guessing that isn't likely to happen, but by +** using the same locking range we are at least open to the possibility. +** +** Locking in windows is manditory. For this reason, we cannot store +** actual data in the bytes used for locking. The pager never allocates +** the pages involved in locking therefore. SHARED_SIZE is selected so +** that all locks will fit on a single page even at the minimum page size. +** PENDING_BYTE defines the beginning of the locks. By default PENDING_BYTE +** is set high so that we don't have to allocate an unused page except +** for very large databases. But one should test the page skipping logic +** by setting PENDING_BYTE low and running the entire regression suite. +** +** Changing the value of PENDING_BYTE results in a subtly incompatible +** file format. Depending on how it is changed, you might not notice +** the incompatibility right away, even running a full regression test. +** The default location of PENDING_BYTE is the first byte past the +** 1GB boundary. +** +*/ +#ifndef SQLITE_TEST +#define PENDING_BYTE 0x40000000 /* First byte past the 1GB boundary */ +#else +extern unsigned int sqlite3_pending_byte; +#define PENDING_BYTE sqlite3_pending_byte +#endif + +#define RESERVED_BYTE (PENDING_BYTE+1) +#define SHARED_FIRST (PENDING_BYTE+2) +#define SHARED_SIZE 510 + +/* +** Prototypes for operating system interface routines. +*/ +int sqlite3OsClose(OsFile**); +int sqlite3OsOpenDirectory(OsFile*, const char*); +int sqlite3OsRead(OsFile*, void*, int amt); +int sqlite3OsWrite(OsFile*, const void*, int amt); +int sqlite3OsSeek(OsFile*, i64 offset); +int sqlite3OsTruncate(OsFile*, i64 size); +int sqlite3OsSync(OsFile*, int); +void sqlite3OsSetFullSync(OsFile *id, int setting); +int sqlite3OsFileHandle(OsFile *id); +int sqlite3OsFileSize(OsFile*, i64 *pSize); +int sqlite3OsLock(OsFile*, int); +int sqlite3OsUnlock(OsFile*, int); +int sqlite3OsLockState(OsFile *id); +int sqlite3OsCheckReservedLock(OsFile *id); +int sqlite3OsOpenReadWrite(const char*, OsFile**, int*); +int sqlite3OsOpenExclusive(const char*, OsFile**, int); +int sqlite3OsOpenReadOnly(const char*, OsFile**); +int sqlite3OsDelete(const char*); +int sqlite3OsFileExists(const char*); +char *sqlite3OsFullPathname(const char*); +int sqlite3OsIsDirWritable(char*); +int sqlite3OsSyncDirectory(const char*); +int sqlite3OsTempFileName(char*); +int sqlite3OsRandomSeed(char*); +int sqlite3OsSleep(int ms); +int sqlite3OsCurrentTime(double*); +void sqlite3OsEnterMutex(void); +void sqlite3OsLeaveMutex(void); +int sqlite3OsInMutex(int); +ThreadData *sqlite3OsThreadSpecificData(int); +void *sqlite3OsMalloc(int); +void *sqlite3OsRealloc(void *, int); +void sqlite3OsFree(void *); +int sqlite3OsAllocationSize(void *); + +/* +** If the SQLITE_ENABLE_REDEF_IO macro is defined, then the OS-layer +** interface routines are not called directly but are invoked using +** pointers to functions. This allows the implementation of various +** OS-layer interface routines to be modified at run-time. There are +** obscure but legitimate reasons for wanting to do this. But for +** most users, a direct call to the underlying interface is preferable +** so the the redefinable I/O interface is turned off by default. +*/ +#ifdef SQLITE_ENABLE_REDEF_IO + +/* +** When redefinable I/O is enabled, a single global instance of the +** following structure holds pointers to the routines that SQLite +** uses to talk with the underlying operating system. Modify this +** structure (before using any SQLite API!) to accomodate perculiar +** operating system interfaces or behaviors. +*/ +struct sqlite3OsVtbl { + int (*xOpenReadWrite)(const char*, OsFile**, int*); + int (*xOpenExclusive)(const char*, OsFile**, int); + int (*xOpenReadOnly)(const char*, OsFile**); + + int (*xDelete)(const char*); + int (*xFileExists)(const char*); + char *(*xFullPathname)(const char*); + int (*xIsDirWritable)(char*); + int (*xSyncDirectory)(const char*); + int (*xTempFileName)(char*); + + int (*xRandomSeed)(char*); + int (*xSleep)(int ms); + int (*xCurrentTime)(double*); + + void (*xEnterMutex)(void); + void (*xLeaveMutex)(void); + int (*xInMutex)(int); + ThreadData *(*xThreadSpecificData)(int); + + void *(*xMalloc)(int); + void *(*xRealloc)(void *, int); + void (*xFree)(void *); + int (*xAllocationSize)(void *); +}; + +/* Macro used to comment out routines that do not exists when there is +** no disk I/O +*/ +#ifdef SQLITE_OMIT_DISKIO +# define IF_DISKIO(X) 0 +#else +# define IF_DISKIO(X) X +#endif + +#ifdef _SQLITE_OS_C_ + /* + ** The os.c file implements the global virtual function table. + */ + struct sqlite3OsVtbl sqlite3Os = { + IF_DISKIO( sqlite3OsOpenReadWrite ), + IF_DISKIO( sqlite3OsOpenExclusive ), + IF_DISKIO( sqlite3OsOpenReadOnly ), + IF_DISKIO( sqlite3OsDelete ), + IF_DISKIO( sqlite3OsFileExists ), + IF_DISKIO( sqlite3OsFullPathname ), + IF_DISKIO( sqlite3OsIsDirWritable ), + IF_DISKIO( sqlite3OsSyncDirectory ), + IF_DISKIO( sqlite3OsTempFileName ), + sqlite3OsRandomSeed, + sqlite3OsSleep, + sqlite3OsCurrentTime, + sqlite3OsEnterMutex, + sqlite3OsLeaveMutex, + sqlite3OsInMutex, + sqlite3OsThreadSpecificData, + sqlite3OsMalloc, + sqlite3OsRealloc, + sqlite3OsFree, + sqlite3OsAllocationSize + }; +#else + /* + ** Files other than os.c just reference the global virtual function table. + */ + extern struct sqlite3OsVtbl sqlite3Os; +#endif /* _SQLITE_OS_C_ */ + + +/* This additional API routine is available with redefinable I/O */ +struct sqlite3OsVtbl *sqlite3_os_switch(void); + + +/* +** Redefine the OS interface to go through the virtual function table +** rather than calling routines directly. +*/ +#undef sqlite3OsOpenReadWrite +#undef sqlite3OsOpenExclusive +#undef sqlite3OsOpenReadOnly +#undef sqlite3OsDelete +#undef sqlite3OsFileExists +#undef sqlite3OsFullPathname +#undef sqlite3OsIsDirWritable +#undef sqlite3OsSyncDirectory +#undef sqlite3OsTempFileName +#undef sqlite3OsRandomSeed +#undef sqlite3OsSleep +#undef sqlite3OsCurrentTime +#undef sqlite3OsEnterMutex +#undef sqlite3OsLeaveMutex +#undef sqlite3OsInMutex +#undef sqlite3OsThreadSpecificData +#undef sqlite3OsMalloc +#undef sqlite3OsRealloc +#undef sqlite3OsFree +#undef sqlite3OsAllocationSize +#define sqlite3OsOpenReadWrite sqlite3Os.xOpenReadWrite +#define sqlite3OsOpenExclusive sqlite3Os.xOpenExclusive +#define sqlite3OsOpenReadOnly sqlite3Os.xOpenReadOnly +#define sqlite3OsDelete sqlite3Os.xDelete +#define sqlite3OsFileExists sqlite3Os.xFileExists +#define sqlite3OsFullPathname sqlite3Os.xFullPathname +#define sqlite3OsIsDirWritable sqlite3Os.xIsDirWritable +#define sqlite3OsSyncDirectory sqlite3Os.xSyncDirectory +#define sqlite3OsTempFileName sqlite3Os.xTempFileName +#define sqlite3OsRandomSeed sqlite3Os.xRandomSeed +#define sqlite3OsSleep sqlite3Os.xSleep +#define sqlite3OsCurrentTime sqlite3Os.xCurrentTime +#define sqlite3OsEnterMutex sqlite3Os.xEnterMutex +#define sqlite3OsLeaveMutex sqlite3Os.xLeaveMutex +#define sqlite3OsInMutex sqlite3Os.xInMutex +#define sqlite3OsThreadSpecificData sqlite3Os.xThreadSpecificData +#define sqlite3OsMalloc sqlite3Os.xMalloc +#define sqlite3OsRealloc sqlite3Os.xRealloc +#define sqlite3OsFree sqlite3Os.xFree +#define sqlite3OsAllocationSize sqlite3Os.xAllocationSize + +#endif /* SQLITE_ENABLE_REDEF_IO */ + +#endif /* _SQLITE_OS_H_ */ Added: external/sqlite-source-3.3.4/os_common.h ============================================================================== --- (empty file) +++ external/sqlite-source-3.3.4/os_common.h Mon Apr 3 07:54:59 2006 @@ -0,0 +1,188 @@ +/* +** 2004 May 22 +** +** The author disclaims copyright to this source code. In place of +** a legal notice, here is a blessing: +** +** May you do good and not evil. +** May you find forgiveness for yourself and forgive others. +** May you share freely, never taking more than you give. +** +****************************************************************************** +** +** This file contains macros and a little bit of code that is common to +** all of the platform-specific files (os_*.c) and is #included into those +** files. +** +** This file should be #included by the os_*.c files only. It is not a +** general purpose header file. +*/ + +/* +** At least two bugs have slipped in because we changed the MEMORY_DEBUG +** macro to SQLITE_DEBUG and some older makefiles have not yet made the +** switch. The following code should catch this problem at compile-time. +*/ +#ifdef MEMORY_DEBUG +# error "The MEMORY_DEBUG macro is obsolete. Use SQLITE_DEBUG instead." +#endif + + +/* + * When testing, this global variable stores the location of the + * pending-byte in the database file. + */ +#ifdef SQLITE_TEST +unsigned int sqlite3_pending_byte = 0x40000000; +#endif + +int sqlite3_os_trace = 0; +#ifdef SQLITE_DEBUG +static int last_page = 0; +#define SEEK(X) last_page=(X) +#define TRACE1(X) if( sqlite3_os_trace ) sqlite3DebugPrintf(X) +#define TRACE2(X,Y) if( sqlite3_os_trace ) sqlite3DebugPrintf(X,Y) +#define TRACE3(X,Y,Z) if( sqlite3_os_trace ) sqlite3DebugPrintf(X,Y,Z) +#define TRACE4(X,Y,Z,A) if( sqlite3_os_trace ) sqlite3DebugPrintf(X,Y,Z,A) +#define TRACE5(X,Y,Z,A,B) if( sqlite3_os_trace ) sqlite3DebugPrintf(X,Y,Z,A,B) +#define TRACE6(X,Y,Z,A,B,C) if(sqlite3_os_trace) sqlite3DebugPrintf(X,Y,Z,A,B,C) +#define TRACE7(X,Y,Z,A,B,C,D) \ + if(sqlite3_os_trace) sqlite3DebugPrintf(X,Y,Z,A,B,C,D) +#else +#define SEEK(X) +#define TRACE1(X) +#define TRACE2(X,Y) +#define TRACE3(X,Y,Z) +#define TRACE4(X,Y,Z,A) +#define TRACE5(X,Y,Z,A,B) +#define TRACE6(X,Y,Z,A,B,C) +#define TRACE7(X,Y,Z,A,B,C,D) +#endif + +/* +** Macros for performance tracing. Normally turned off. Only works +** on i486 hardware. +*/ +#ifdef SQLITE_PERFORMANCE_TRACE +__inline__ unsigned long long int hwtime(void){ + unsigned long long int x; + __asm__("rdtsc\n\t" + "mov %%edx, %%ecx\n\t" + :"=A" (x)); + return x; +} +static unsigned long long int g_start; +static unsigned int elapse; +#define TIMER_START g_start=hwtime() +#define TIMER_END elapse=hwtime()-g_start +#define TIMER_ELAPSED elapse +#else +#define TIMER_START +#define TIMER_END +#define TIMER_ELAPSED 0 +#endif + +/* +** If we compile with the SQLITE_TEST macro set, then the following block +** of code will give us the ability to simulate a disk I/O error. This +** is used for testing the I/O recovery logic. +*/ +#ifdef SQLITE_TEST +int sqlite3_io_error_hit = 0; +int sqlite3_io_error_pending = 0; +int sqlite3_diskfull_pending = 0; +int sqlite3_diskfull = 0; +#define SimulateIOError(A) \ + if( sqlite3_io_error_pending ) \ + if( sqlite3_io_error_pending-- == 1 ){ local_ioerr(); return A; } +static void local_ioerr(){ + sqlite3_io_error_hit = 1; /* Really just a place to set a breakpoint */ +} +#define SimulateDiskfullError \ + if( sqlite3_diskfull_pending ){ \ + if( sqlite3_diskfull_pending == 1 ){ \ + local_ioerr(); \ + sqlite3_diskfull = 1; \ + return SQLITE_FULL; \ + }else{ \ + sqlite3_diskfull_pending--; \ + } \ + } +#else +#define SimulateIOError(A) +#define SimulateDiskfullError +#endif + +/* +** When testing, keep a count of the number of open files. +*/ +#ifdef SQLITE_TEST +int sqlite3_open_file_count = 0; +#define OpenCounter(X) sqlite3_open_file_count+=(X) +#else +#define OpenCounter(X) +#endif + +/* +** sqlite3GenericMalloc +** sqlite3GenericRealloc +** sqlite3GenericOsFree +** sqlite3GenericAllocationSize +** +** Implementation of the os level dynamic memory allocation interface in terms +** of the standard malloc(), realloc() and free() found in many operating +** systems. No rocket science here. +** +** There are two versions of these four functions here. The version +** implemented here is only used if memory-management or memory-debugging is +** enabled. This version allocates an extra 8-bytes at the beginning of each +** block and stores the size of the allocation there. +** +** If neither memory-management or debugging is enabled, the second +** set of implementations is used instead. +*/ +#if defined(SQLITE_ENABLE_MEMORY_MANAGEMENT) || defined (SQLITE_MEMDEBUG) +void *sqlite3GenericMalloc(int n){ + char *p = (char *)malloc(n+8); + assert(n>0); + assert(sizeof(int)<=8); + if( p ){ + *(int *)p = n; + p += 8; + } + return (void *)p; +} +void *sqlite3GenericRealloc(void *p, int n){ + char *p2 = ((char *)p - 8); + assert(n>0); + p2 = (char*)realloc(p2, n+8); + if( p2 ){ + *(int *)p2 = n; + p2 += 8; + } + return (void *)p2; +} +void sqlite3GenericFree(void *p){ + assert(p); + free((void *)((char *)p - 8)); +} +int sqlite3GenericAllocationSize(void *p){ + return p ? *(int *)((char *)p - 8) : 0; +} +#else +void *sqlite3GenericMalloc(int n){ + char *p = (char *)malloc(n); + return (void *)p; +} +void *sqlite3GenericRealloc(void *p, int n){ + assert(n>0); + p = realloc(p, n); + return p; +} +void sqlite3GenericFree(void *p){ + assert(p); + free(p); +} +/* Never actually used, but needed for the linker */ +int sqlite3GenericAllocationSize(void *p){ return 0; } +#endif Added: external/sqlite-source-3.3.4/os_unix.c ============================================================================== --- (empty file) +++ external/sqlite-source-3.3.4/os_unix.c Mon Apr 3 07:54:59 2006 @@ -0,0 +1,1928 @@ +/* +** 2004 May 22 +** +** The author disclaims copyright to this source code. In place of +** a legal notice, here is a blessing: +** +** May you do good and not evil. +** May you find forgiveness for yourself and forgive others. +** May you share freely, never taking more than you give. +** +****************************************************************************** +** +** This file contains code that is specific to Unix systems. +*/ +#include "sqliteInt.h" +#include "os.h" +#if OS_UNIX /* This file is used on unix only */ + +/* +** These #defines should enable >2GB file support on Posix if the +** underlying operating system supports it. If the OS lacks +** large file support, these should be no-ops. +** +** Large file support can be disabled using the -DSQLITE_DISABLE_LFS switch +** on the compiler command line. This is necessary if you are compiling +** on a recent machine (ex: RedHat 7.2) but you want your code to work +** on an older machine (ex: RedHat 6.0). If you compile on RedHat 7.2 +** without this option, LFS is enable. But LFS does not exist in the kernel +** in RedHat 6.0, so the code won't work. Hence, for maximum binary +** portability you should omit LFS. +*/ +#ifndef SQLITE_DISABLE_LFS +# define _LARGE_FILE 1 +# ifndef _FILE_OFFSET_BITS +# define _FILE_OFFSET_BITS 64 +# endif +# define _LARGEFILE_SOURCE 1 +#endif + +/* +** standard include files. +*/ +#include +#include +#include +#include +#include +#include +#include + +/* +** If we are to be thread-safe, include the pthreads header and define +** the SQLITE_UNIX_THREADS macro. +*/ +#if defined(THREADSAFE) && THREADSAFE +# include +# define SQLITE_UNIX_THREADS 1 +#endif + +/* +** Default permissions when creating a new file +*/ +#ifndef SQLITE_DEFAULT_FILE_PERMISSIONS +# define SQLITE_DEFAULT_FILE_PERMISSIONS 0644 +#endif + + + +/* +** The unixFile structure is subclass of OsFile specific for the unix +** protability layer. +*/ +typedef struct unixFile unixFile; +struct unixFile { + IoMethod const *pMethod; /* Always the first entry */ + struct openCnt *pOpen; /* Info about all open fd's on this inode */ + struct lockInfo *pLock; /* Info about locks on this inode */ + int h; /* The file descriptor */ + unsigned char locktype; /* The type of lock held on this fd */ + unsigned char isOpen; /* True if needs to be closed */ + unsigned char fullSync; /* Use F_FULLSYNC if available */ + int dirfd; /* File descriptor for the directory */ +#ifdef SQLITE_UNIX_THREADS + pthread_t tid; /* The thread that "owns" this OsFile */ +#endif +}; + +/* +** Provide the ability to override some OS-layer functions during +** testing. This is used to simulate OS crashes to verify that +** commits are atomic even in the event of an OS crash. +*/ +#ifdef SQLITE_CRASH_TEST + extern int sqlite3CrashTestEnable; + extern int sqlite3CrashOpenReadWrite(const char*, OsFile**, int*); + extern int sqlite3CrashOpenExclusive(const char*, OsFile**, int); + extern int sqlite3CrashOpenReadOnly(const char*, OsFile**, int); +# define CRASH_TEST_OVERRIDE(X,A,B,C) \ + if(sqlite3CrashTestEnable){ return X(A,B,C); } +#else +# define CRASH_TEST_OVERRIDE(X,A,B,C) /* no-op */ +#endif + + +/* +** Include code that is common to all os_*.c files +*/ +#include "os_common.h" + +/* +** Do not include any of the File I/O interface procedures if the +** SQLITE_OMIT_DISKIO macro is defined (indicating that the database +** will be in-memory only) +*/ +#ifndef SQLITE_OMIT_DISKIO + + +/* +** Define various macros that are missing from some systems. +*/ +#ifndef O_LARGEFILE +# define O_LARGEFILE 0 +#endif +#ifdef SQLITE_DISABLE_LFS +# undef O_LARGEFILE +# define O_LARGEFILE 0 +#endif +#ifndef O_NOFOLLOW +# define O_NOFOLLOW 0 +#endif +#ifndef O_BINARY +# define O_BINARY 0 +#endif + +/* +** The DJGPP compiler environment looks mostly like Unix, but it +** lacks the fcntl() system call. So redefine fcntl() to be something +** that always succeeds. This means that locking does not occur under +** DJGPP. But it's DOS - what did you expect? +*/ +#ifdef __DJGPP__ +# define fcntl(A,B,C) 0 +#endif + +/* +** The threadid macro resolves to the thread-id or to 0. Used for +** testing and debugging only. +*/ +#ifdef SQLITE_UNIX_THREADS +#define threadid pthread_self() +#else +#define threadid 0 +#endif + +/* +** Set or check the OsFile.tid field. This field is set when an OsFile +** is first opened. All subsequent uses of the OsFile verify that the +** same thread is operating on the OsFile. Some operating systems do +** not allow locks to be overridden by other threads and that restriction +** means that sqlite3* database handles cannot be moved from one thread +** to another. This logic makes sure a user does not try to do that +** by mistake. +** +** Version 3.3.1 (2006-01-15): OsFiles can be moved from one thread to +** another as long as we are running on a system that supports threads +** overriding each others locks (which now the most common behavior) +** or if no locks are held. But the OsFile.pLock field needs to be +** recomputed because its key includes the thread-id. See the +** transferOwnership() function below for additional information +*/ +#if defined(SQLITE_UNIX_THREADS) +# define SET_THREADID(X) (X)->tid = pthread_self() +# define CHECK_THREADID(X) (threadsOverrideEachOthersLocks==0 && \ + !pthread_equal((X)->tid, pthread_self())) +#else +# define SET_THREADID(X) +# define CHECK_THREADID(X) 0 +#endif + +/* +** Here is the dirt on POSIX advisory locks: ANSI STD 1003.1 (1996) +** section 6.5.2.2 lines 483 through 490 specify that when a process +** sets or clears a lock, that operation overrides any prior locks set +** by the same process. It does not explicitly say so, but this implies +** that it overrides locks set by the same process using a different +** file descriptor. Consider this test case: +** +** int fd1 = open("./file1", O_RDWR|O_CREAT, 0644); +** int fd2 = open("./file2", O_RDWR|O_CREAT, 0644); +** +** Suppose ./file1 and ./file2 are really the same file (because +** one is a hard or symbolic link to the other) then if you set +** an exclusive lock on fd1, then try to get an exclusive lock +** on fd2, it works. I would have expected the second lock to +** fail since there was already a lock on the file due to fd1. +** But not so. Since both locks came from the same process, the +** second overrides the first, even though they were on different +** file descriptors opened on different file names. +** +** Bummer. If you ask me, this is broken. Badly broken. It means +** that we cannot use POSIX locks to synchronize file access among +** competing threads of the same process. POSIX locks will work fine +** to synchronize access for threads in separate processes, but not +** threads within the same process. +** +** To work around the problem, SQLite has to manage file locks internally +** on its own. Whenever a new database is opened, we have to find the +** specific inode of the database file (the inode is determined by the +** st_dev and st_ino fields of the stat structure that fstat() fills in) +** and check for locks already existing on that inode. When locks are +** created or removed, we have to look at our own internal record of the +** locks to see if another thread has previously set a lock on that same +** inode. +** +** The OsFile structure for POSIX is no longer just an integer file +** descriptor. It is now a structure that holds the integer file +** descriptor and a pointer to a structure that describes the internal +** locks on the corresponding inode. There is one locking structure +** per inode, so if the same inode is opened twice, both OsFile structures +** point to the same locking structure. The locking structure keeps +** a reference count (so we will know when to delete it) and a "cnt" +** field that tells us its internal lock status. cnt==0 means the +** file is unlocked. cnt==-1 means the file has an exclusive lock. +** cnt>0 means there are cnt shared locks on the file. +** +** Any attempt to lock or unlock a file first checks the locking +** structure. The fcntl() system call is only invoked to set a +** POSIX lock if the internal lock structure transitions between +** a locked and an unlocked state. +** +** 2004-Jan-11: +** More recent discoveries about POSIX advisory locks. (The more +** I discover, the more I realize the a POSIX advisory locks are +** an abomination.) +** +** If you close a file descriptor that points to a file that has locks, +** all locks on that file that are owned by the current process are +** released. To work around this problem, each OsFile structure contains +** a pointer to an openCnt structure. There is one openCnt structure +** per open inode, which means that multiple OsFiles can point to a single +** openCnt. When an attempt is made to close an OsFile, if there are +** other OsFiles open on the same inode that are holding locks, the call +** to close() the file descriptor is deferred until all of the locks clear. +** The openCnt structure keeps a list of file descriptors that need to +** be closed and that list is walked (and cleared) when the last lock +** clears. +** +** First, under Linux threads, because each thread has a separate +** process ID, lock operations in one thread do not override locks +** to the same file in other threads. Linux threads behave like +** separate processes in this respect. But, if you close a file +** descriptor in linux threads, all locks are cleared, even locks +** on other threads and even though the other threads have different +** process IDs. Linux threads is inconsistent in this respect. +** (I'm beginning to think that linux threads is an abomination too.) +** The consequence of this all is that the hash table for the lockInfo +** structure has to include the process id as part of its key because +** locks in different threads are treated as distinct. But the +** openCnt structure should not include the process id in its +** key because close() clears lock on all threads, not just the current +** thread. Were it not for this goofiness in linux threads, we could +** combine the lockInfo and openCnt structures into a single structure. +** +** 2004-Jun-28: +** On some versions of linux, threads can override each others locks. +** On others not. Sometimes you can change the behavior on the same +** system by setting the LD_ASSUME_KERNEL environment variable. The +** POSIX standard is silent as to which behavior is correct, as far +** as I can tell, so other versions of unix might show the same +** inconsistency. There is no little doubt in my mind that posix +** advisory locks and linux threads are profoundly broken. +** +** To work around the inconsistencies, we have to test at runtime +** whether or not threads can override each others locks. This test +** is run once, the first time any lock is attempted. A static +** variable is set to record the results of this test for future +** use. +*/ + +/* +** An instance of the following structure serves as the key used +** to locate a particular lockInfo structure given its inode. +** +** If threads cannot override each others locks, then we set the +** lockKey.tid field to the thread ID. If threads can override +** each others locks then tid is always set to zero. tid is omitted +** if we compile without threading support. +*/ +struct lockKey { + dev_t dev; /* Device number */ + ino_t ino; /* Inode number */ +#ifdef SQLITE_UNIX_THREADS + pthread_t tid; /* Thread ID or zero if threads can override each other */ +#endif +}; + +/* +** An instance of the following structure is allocated for each open +** inode on each thread with a different process ID. (Threads have +** different process IDs on linux, but not on most other unixes.) +** +** A single inode can have multiple file descriptors, so each OsFile +** structure contains a pointer to an instance of this object and this +** object keeps a count of the number of OsFiles pointing to it. +*/ +struct lockInfo { + struct lockKey key; /* The lookup key */ + int cnt; /* Number of SHARED locks held */ + int locktype; /* One of SHARED_LOCK, RESERVED_LOCK etc. */ + int nRef; /* Number of pointers to this structure */ +}; + +/* +** An instance of the following structure serves as the key used +** to locate a particular openCnt structure given its inode. This +** is the same as the lockKey except that the thread ID is omitted. +*/ +struct openKey { + dev_t dev; /* Device number */ + ino_t ino; /* Inode number */ +}; + +/* +** An instance of the following structure is allocated for each open +** inode. This structure keeps track of the number of locks on that +** inode. If a close is attempted against an inode that is holding +** locks, the close is deferred until all locks clear by adding the +** file descriptor to be closed to the pending list. +*/ +struct openCnt { + struct openKey key; /* The lookup key */ + int nRef; /* Number of pointers to this structure */ + int nLock; /* Number of outstanding locks */ + int nPending; /* Number of pending close() operations */ + int *aPending; /* Malloced space holding fd's awaiting a close() */ +}; + +/* +** These hash tables map inodes and file descriptors (really, lockKey and +** openKey structures) into lockInfo and openCnt structures. Access to +** these hash tables must be protected by a mutex. +*/ +static Hash lockHash = { SQLITE_HASH_BINARY, 0, 0, 0, 0, 0 }; +static Hash openHash = { SQLITE_HASH_BINARY, 0, 0, 0, 0, 0 }; + + +#ifdef SQLITE_UNIX_THREADS +/* +** This variable records whether or not threads can override each others +** locks. +** +** 0: No. Threads cannot override each others locks. +** 1: Yes. Threads can override each others locks. +** -1: We don't know yet. +** +** On some systems, we know at compile-time if threads can override each +** others locks. On those systems, the SQLITE_THREAD_OVERRIDE_LOCK macro +** will be set appropriately. On other systems, we have to check at +** runtime. On these latter systems, SQLTIE_THREAD_OVERRIDE_LOCK is +** undefined. +** +** This variable normally has file scope only. But during testing, we make +** it a global so that the test code can change its value in order to verify +** that the right stuff happens in either case. +*/ +#ifndef SQLITE_THREAD_OVERRIDE_LOCK +# define SQLITE_THREAD_OVERRIDE_LOCK -1 +#endif +#ifdef SQLITE_TEST +int threadsOverrideEachOthersLocks = SQLITE_THREAD_OVERRIDE_LOCK; +#else +static int threadsOverrideEachOthersLocks = SQLITE_THREAD_OVERRIDE_LOCK; +#endif + +/* +** This structure holds information passed into individual test +** threads by the testThreadLockingBehavior() routine. +*/ +struct threadTestData { + int fd; /* File to be locked */ + struct flock lock; /* The locking operation */ + int result; /* Result of the locking operation */ +}; + +#ifdef SQLITE_LOCK_TRACE +/* +** Print out information about all locking operations. +** +** This routine is used for troubleshooting locks on multithreaded +** platforms. Enable by compiling with the -DSQLITE_LOCK_TRACE +** command-line option on the compiler. This code is normally +** turned off. +*/ +static int lockTrace(int fd, int op, struct flock *p){ + char *zOpName, *zType; + int s; + int savedErrno; + if( op==F_GETLK ){ + zOpName = "GETLK"; + }else if( op==F_SETLK ){ + zOpName = "SETLK"; + }else{ + s = fcntl(fd, op, p); + sqlite3DebugPrintf("fcntl unknown %d %d %d\n", fd, op, s); + return s; + } + if( p->l_type==F_RDLCK ){ + zType = "RDLCK"; + }else if( p->l_type==F_WRLCK ){ + zType = "WRLCK"; + }else if( p->l_type==F_UNLCK ){ + zType = "UNLCK"; + }else{ + assert( 0 ); + } + assert( p->l_whence==SEEK_SET ); + s = fcntl(fd, op, p); + savedErrno = errno; + sqlite3DebugPrintf("fcntl %d %d %s %s %d %d %d %d\n", + threadid, fd, zOpName, zType, (int)p->l_start, (int)p->l_len, + (int)p->l_pid, s); + if( s && op==F_SETLK && (p->l_type==F_RDLCK || p->l_type==F_WRLCK) ){ + struct flock l2; + l2 = *p; + fcntl(fd, F_GETLK, &l2); + if( l2.l_type==F_RDLCK ){ + zType = "RDLCK"; + }else if( l2.l_type==F_WRLCK ){ + zType = "WRLCK"; + }else if( l2.l_type==F_UNLCK ){ + zType = "UNLCK"; + }else{ + assert( 0 ); + } + sqlite3DebugPrintf("fcntl-failure-reason: %s %d %d %d\n", + zType, (int)l2.l_start, (int)l2.l_len, (int)l2.l_pid); + } + errno = savedErrno; + return s; +} +#define fcntl lockTrace +#endif /* SQLITE_LOCK_TRACE */ + +/* +** The testThreadLockingBehavior() routine launches two separate +** threads on this routine. This routine attempts to lock a file +** descriptor then returns. The success or failure of that attempt +** allows the testThreadLockingBehavior() procedure to determine +** whether or not threads can override each others locks. +*/ +static void *threadLockingTest(void *pArg){ + struct threadTestData *pData = (struct threadTestData*)pArg; + pData->result = fcntl(pData->fd, F_SETLK, &pData->lock); + return pArg; +} + +/* +** This procedure attempts to determine whether or not threads +** can override each others locks then sets the +** threadsOverrideEachOthersLocks variable appropriately. +*/ +static void testThreadLockingBehavior(int fd_orig){ + int fd; + struct threadTestData d[2]; + pthread_t t[2]; + + fd = dup(fd_orig); + if( fd<0 ) return; + memset(d, 0, sizeof(d)); + d[0].fd = fd; + d[0].lock.l_type = F_RDLCK; + d[0].lock.l_len = 1; + d[0].lock.l_start = 0; + d[0].lock.l_whence = SEEK_SET; + d[1] = d[0]; + d[1].lock.l_type = F_WRLCK; + pthread_create(&t[0], 0, threadLockingTest, &d[0]); + pthread_create(&t[1], 0, threadLockingTest, &d[1]); + pthread_join(t[0], 0); + pthread_join(t[1], 0); + close(fd); + threadsOverrideEachOthersLocks = d[0].result==0 && d[1].result==0; +} +#endif /* SQLITE_UNIX_THREADS */ + +/* +** Release a lockInfo structure previously allocated by findLockInfo(). +*/ +static void releaseLockInfo(struct lockInfo *pLock){ + assert( sqlite3OsInMutex(1) ); + pLock->nRef--; + if( pLock->nRef==0 ){ + sqlite3HashInsert(&lockHash, &pLock->key, sizeof(pLock->key), 0); + sqliteFree(pLock); + } +} + +/* +** Release a openCnt structure previously allocated by findLockInfo(). +*/ +static void releaseOpenCnt(struct openCnt *pOpen){ + assert( sqlite3OsInMutex(1) ); + pOpen->nRef--; + if( pOpen->nRef==0 ){ + sqlite3HashInsert(&openHash, &pOpen->key, sizeof(pOpen->key), 0); + free(pOpen->aPending); + sqliteFree(pOpen); + } +} + +/* +** Given a file descriptor, locate lockInfo and openCnt structures that +** describes that file descriptor. Create new ones if necessary. The +** return values might be uninitialized if an error occurs. +** +** Return the number of errors. +*/ +static int findLockInfo( + int fd, /* The file descriptor used in the key */ + struct lockInfo **ppLock, /* Return the lockInfo structure here */ + struct openCnt **ppOpen /* Return the openCnt structure here */ +){ + int rc; + struct lockKey key1; + struct openKey key2; + struct stat statbuf; + struct lockInfo *pLock; + struct openCnt *pOpen; + rc = fstat(fd, &statbuf); + if( rc!=0 ) return 1; + + assert( sqlite3OsInMutex(1) ); + memset(&key1, 0, sizeof(key1)); + key1.dev = statbuf.st_dev; + key1.ino = statbuf.st_ino; +#ifdef SQLITE_UNIX_THREADS + if( threadsOverrideEachOthersLocks<0 ){ + testThreadLockingBehavior(fd); + } + key1.tid = threadsOverrideEachOthersLocks ? 0 : pthread_self(); +#endif + memset(&key2, 0, sizeof(key2)); + key2.dev = statbuf.st_dev; + key2.ino = statbuf.st_ino; + pLock = (struct lockInfo*)sqlite3HashFind(&lockHash, &key1, sizeof(key1)); + if( pLock==0 ){ + struct lockInfo *pOld; + pLock = sqliteMallocRaw( sizeof(*pLock) ); + if( pLock==0 ){ + rc = 1; + goto exit_findlockinfo; + } + pLock->key = key1; + pLock->nRef = 1; + pLock->cnt = 0; + pLock->locktype = 0; + pOld = sqlite3HashInsert(&lockHash, &pLock->key, sizeof(key1), pLock); + if( pOld!=0 ){ + assert( pOld==pLock ); + sqliteFree(pLock); + rc = 1; + goto exit_findlockinfo; + } + }else{ + pLock->nRef++; + } + *ppLock = pLock; + if( ppOpen!=0 ){ + pOpen = (struct openCnt*)sqlite3HashFind(&openHash, &key2, sizeof(key2)); + if( pOpen==0 ){ + struct openCnt *pOld; + pOpen = sqliteMallocRaw( sizeof(*pOpen) ); + if( pOpen==0 ){ + releaseLockInfo(pLock); + rc = 1; + goto exit_findlockinfo; + } + pOpen->key = key2; + pOpen->nRef = 1; + pOpen->nLock = 0; + pOpen->nPending = 0; + pOpen->aPending = 0; + pOld = sqlite3HashInsert(&openHash, &pOpen->key, sizeof(key2), pOpen); + if( pOld!=0 ){ + assert( pOld==pOpen ); + sqliteFree(pOpen); + releaseLockInfo(pLock); + rc = 1; + goto exit_findlockinfo; + } + }else{ + pOpen->nRef++; + } + *ppOpen = pOpen; + } + +exit_findlockinfo: + return rc; +} + +#ifdef SQLITE_DEBUG +/* +** Helper function for printing out trace information from debugging +** binaries. This returns the string represetation of the supplied +** integer lock-type. +*/ +static const char *locktypeName(int locktype){ + switch( locktype ){ + case NO_LOCK: return "NONE"; + case SHARED_LOCK: return "SHARED"; + case RESERVED_LOCK: return "RESERVED"; + case PENDING_LOCK: return "PENDING"; + case EXCLUSIVE_LOCK: return "EXCLUSIVE"; + } + return "ERROR"; +} +#endif + +/* +** If we are currently in a different thread than the thread that the +** unixFile argument belongs to, then transfer ownership of the unixFile +** over to the current thread. +** +** A unixFile is only owned by a thread on systems where one thread is +** unable to override locks created by a different thread. RedHat9 is +** an example of such a system. +** +** Ownership transfer is only allowed if the unixFile is currently unlocked. +** If the unixFile is locked and an ownership is wrong, then return +** SQLITE_MISUSE. SQLITE_OK is returned if everything works. +*/ +#ifdef SQLITE_UNIX_THREADS +static int transferOwnership(unixFile *pFile){ + int rc; + pthread_t hSelf; + if( threadsOverrideEachOthersLocks ){ + /* Ownership transfers not needed on this system */ + return SQLITE_OK; + } + hSelf = pthread_self(); + if( pthread_equal(pFile->tid, hSelf) ){ + /* We are still in the same thread */ + TRACE1("No-transfer, same thread\n"); + return SQLITE_OK; + } + if( pFile->locktype!=NO_LOCK ){ + /* We cannot change ownership while we are holding a lock! */ + return SQLITE_MISUSE; + } + TRACE4("Transfer ownership of %d from %d to %d\n", pFile->h,pFile->tid,hSelf); + pFile->tid = hSelf; + releaseLockInfo(pFile->pLock); + rc = findLockInfo(pFile->h, &pFile->pLock, 0); + TRACE5("LOCK %d is now %s(%s,%d)\n", pFile->h, + locktypeName(pFile->locktype), + locktypeName(pFile->pLock->locktype), pFile->pLock->cnt); + return rc; +} +#else + /* On single-threaded builds, ownership transfer is a no-op */ +# define transferOwnership(X) SQLITE_OK +#endif + +/* +** Delete the named file +*/ +int sqlite3UnixDelete(const char *zFilename){ + unlink(zFilename); + return SQLITE_OK; +} + +/* +** Return TRUE if the named file exists. +*/ +int sqlite3UnixFileExists(const char *zFilename){ + return access(zFilename, 0)==0; +} + +/* Forward declaration */ +static int allocateUnixFile(unixFile *pInit, OsFile **pId); + +/* +** Attempt to open a file for both reading and writing. If that +** fails, try opening it read-only. If the file does not exist, +** try to create it. +** +** On success, a handle for the open file is written to *id +** and *pReadonly is set to 0 if the file was opened for reading and +** writing or 1 if the file was opened read-only. The function returns +** SQLITE_OK. +** +** On failure, the function returns SQLITE_CANTOPEN and leaves +** *id and *pReadonly unchanged. +*/ +int sqlite3UnixOpenReadWrite( + const char *zFilename, + OsFile **pId, + int *pReadonly +){ + int rc; + unixFile f; + + CRASH_TEST_OVERRIDE(sqlite3CrashOpenReadWrite, zFilename, pId, pReadonly); + assert( 0==*pId ); + f.h = open(zFilename, O_RDWR|O_CREAT|O_LARGEFILE|O_BINARY, + SQLITE_DEFAULT_FILE_PERMISSIONS); + if( f.h<0 ){ +#ifdef EISDIR + if( errno==EISDIR ){ + return SQLITE_CANTOPEN; + } +#endif + f.h = open(zFilename, O_RDONLY|O_LARGEFILE|O_BINARY); + if( f.h<0 ){ + return SQLITE_CANTOPEN; + } + *pReadonly = 1; + }else{ + *pReadonly = 0; + } + sqlite3OsEnterMutex(); + rc = findLockInfo(f.h, &f.pLock, &f.pOpen); + sqlite3OsLeaveMutex(); + if( rc ){ + close(f.h); + return SQLITE_NOMEM; + } + TRACE3("OPEN %-3d %s\n", f.h, zFilename); + return allocateUnixFile(&f, pId); +} + + +/* +** Attempt to open a new file for exclusive access by this process. +** The file will be opened for both reading and writing. To avoid +** a potential security problem, we do not allow the file to have +** previously existed. Nor do we allow the file to be a symbolic +** link. +** +** If delFlag is true, then make arrangements to automatically delete +** the file when it is closed. +** +** On success, write the file handle into *id and return SQLITE_OK. +** +** On failure, return SQLITE_CANTOPEN. +*/ +int sqlite3UnixOpenExclusive(const char *zFilename, OsFile **pId, int delFlag){ + int rc; + unixFile f; + + CRASH_TEST_OVERRIDE(sqlite3CrashOpenExclusive, zFilename, pId, delFlag); + assert( 0==*pId ); + if( access(zFilename, 0)==0 ){ + return SQLITE_CANTOPEN; + } + f.h = open(zFilename, + O_RDWR|O_CREAT|O_EXCL|O_NOFOLLOW|O_LARGEFILE|O_BINARY, + SQLITE_DEFAULT_FILE_PERMISSIONS); + if( f.h<0 ){ + return SQLITE_CANTOPEN; + } + sqlite3OsEnterMutex(); + rc = findLockInfo(f.h, &f.pLock, &f.pOpen); + sqlite3OsLeaveMutex(); + if( rc ){ + close(f.h); + unlink(zFilename); + return SQLITE_NOMEM; + } + if( delFlag ){ + unlink(zFilename); + } + TRACE3("OPEN-EX %-3d %s\n", f.h, zFilename); + return allocateUnixFile(&f, pId); +} + +/* +** Attempt to open a new file for read-only access. +** +** On success, write the file handle into *id and return SQLITE_OK. +** +** On failure, return SQLITE_CANTOPEN. +*/ +int sqlite3UnixOpenReadOnly(const char *zFilename, OsFile **pId){ + int rc; + unixFile f; + + CRASH_TEST_OVERRIDE(sqlite3CrashOpenReadOnly, zFilename, pId, 0); + assert( 0==*pId ); + f.h = open(zFilename, O_RDONLY|O_LARGEFILE|O_BINARY); + if( f.h<0 ){ + return SQLITE_CANTOPEN; + } + sqlite3OsEnterMutex(); + rc = findLockInfo(f.h, &f.pLock, &f.pOpen); + sqlite3OsLeaveMutex(); + if( rc ){ + close(f.h); + return SQLITE_NOMEM; + } + TRACE3("OPEN-RO %-3d %s\n", f.h, zFilename); + return allocateUnixFile(&f, pId); +} + +/* +** Attempt to open a file descriptor for the directory that contains a +** file. This file descriptor can be used to fsync() the directory +** in order to make sure the creation of a new file is actually written +** to disk. +** +** This routine is only meaningful for Unix. It is a no-op under +** windows since windows does not support hard links. +** +** On success, a handle for a previously open file at *id is +** updated with the new directory file descriptor and SQLITE_OK is +** returned. +** +** On failure, the function returns SQLITE_CANTOPEN and leaves +** *id unchanged. +*/ +static int unixOpenDirectory( + OsFile *id, + const char *zDirname +){ + unixFile *pFile = (unixFile*)id; + if( pFile==0 ){ + /* Do not open the directory if the corresponding file is not already + ** open. */ + return SQLITE_CANTOPEN; + } + SET_THREADID(pFile); + assert( pFile->dirfd<0 ); + pFile->dirfd = open(zDirname, O_RDONLY|O_BINARY, 0); + if( pFile->dirfd<0 ){ + return SQLITE_CANTOPEN; + } + TRACE3("OPENDIR %-3d %s\n", pFile->dirfd, zDirname); + return SQLITE_OK; +} + +/* +** If the following global variable points to a string which is the +** name of a directory, then that directory will be used to store +** temporary files. +** +** See also the "PRAGMA temp_store_directory" SQL command. +*/ +char *sqlite3_temp_directory = 0; + +/* +** Create a temporary file name in zBuf. zBuf must be big enough to +** hold at least SQLITE_TEMPNAME_SIZE characters. +*/ +int sqlite3UnixTempFileName(char *zBuf){ + static const char *azDirs[] = { + 0, + "/var/tmp", + "/usr/tmp", + "/tmp", + ".", + }; + static const unsigned char zChars[] = + "abcdefghijklmnopqrstuvwxyz" + "ABCDEFGHIJKLMNOPQRSTUVWXYZ" + "0123456789"; + int i, j; + struct stat buf; + const char *zDir = "."; + azDirs[0] = sqlite3_temp_directory; + for(i=0; ih, pBuf, amt); + TIMER_END; + TRACE5("READ %-3d %5d %7d %d\n", ((unixFile*)id)->h, got, + last_page, TIMER_ELAPSED); + SEEK(0); + /* if( got<0 ) got = 0; */ + if( got==amt ){ + return SQLITE_OK; + }else{ + return SQLITE_IOERR; + } +} + +/* +** Write data from a buffer into a file. Return SQLITE_OK on success +** or some other error code on failure. +*/ +static int unixWrite(OsFile *id, const void *pBuf, int amt){ + int wrote = 0; + assert( id ); + assert( amt>0 ); + SimulateIOError(SQLITE_IOERR); + SimulateDiskfullError; + TIMER_START; + while( amt>0 && (wrote = write(((unixFile*)id)->h, pBuf, amt))>0 ){ + amt -= wrote; + pBuf = &((char*)pBuf)[wrote]; + } + TIMER_END; + TRACE5("WRITE %-3d %5d %7d %d\n", ((unixFile*)id)->h, wrote, + last_page, TIMER_ELAPSED); + SEEK(0); + if( amt>0 ){ + return SQLITE_FULL; + } + return SQLITE_OK; +} + +/* +** Move the read/write pointer in a file. +*/ +static int unixSeek(OsFile *id, i64 offset){ + assert( id ); + SEEK(offset/1024 + 1); +#ifdef SQLITE_TEST + if( offset ) SimulateDiskfullError +#endif + lseek(((unixFile*)id)->h, offset, SEEK_SET); + return SQLITE_OK; +} + +#ifdef SQLITE_TEST +/* +** Count the number of fullsyncs and normal syncs. This is used to test +** that syncs and fullsyncs are occuring at the right times. +*/ +int sqlite3_sync_count = 0; +int sqlite3_fullsync_count = 0; +#endif + +/* +** Use the fdatasync() API only if the HAVE_FDATASYNC macro is defined. +** Otherwise use fsync() in its place. +*/ +#ifndef HAVE_FDATASYNC +# define fdatasync fsync +#endif + +/* +** Define HAVE_FULLFSYNC to 0 or 1 depending on whether or not +** the F_FULLFSYNC macro is defined. F_FULLFSYNC is currently +** only available on Mac OS X. But that could change. +*/ +#ifdef F_FULLFSYNC +# define HAVE_FULLFSYNC 1 +#else +# define HAVE_FULLFSYNC 0 +#endif + + +/* +** The fsync() system call does not work as advertised on many +** unix systems. The following procedure is an attempt to make +** it work better. +** +** The SQLITE_NO_SYNC macro disables all fsync()s. This is useful +** for testing when we want to run through the test suite quickly. +** You are strongly advised *not* to deploy with SQLITE_NO_SYNC +** enabled, however, since with SQLITE_NO_SYNC enabled, an OS crash +** or power failure will likely corrupt the database file. +*/ +static int full_fsync(int fd, int fullSync, int dataOnly){ + int rc; + + /* Record the number of times that we do a normal fsync() and + ** FULLSYNC. This is used during testing to verify that this procedure + ** gets called with the correct arguments. + */ +#ifdef SQLITE_TEST + if( fullSync ) sqlite3_fullsync_count++; + sqlite3_sync_count++; +#endif + + /* If we compiled with the SQLITE_NO_SYNC flag, then syncing is a + ** no-op + */ +#ifdef SQLITE_NO_SYNC + rc = SQLITE_OK; +#else + +#if HAVE_FULLFSYNC + if( fullSync ){ + rc = fcntl(fd, F_FULLFSYNC, 0); + }else{ + rc = 1; + } + /* If the FULLSYNC failed, try to do a normal fsync() */ + if( rc ) rc = fsync(fd); + +#else /* if !defined(F_FULLSYNC) */ + if( dataOnly ){ + rc = fdatasync(fd); + }else{ + rc = fsync(fd); + } +#endif /* defined(F_FULLFSYNC) */ +#endif /* defined(SQLITE_NO_SYNC) */ + + return rc; +} + +/* +** Make sure all writes to a particular file are committed to disk. +** +** If dataOnly==0 then both the file itself and its metadata (file +** size, access time, etc) are synced. If dataOnly!=0 then only the +** file data is synced. +** +** Under Unix, also make sure that the directory entry for the file +** has been created by fsync-ing the directory that contains the file. +** If we do not do this and we encounter a power failure, the directory +** entry for the journal might not exist after we reboot. The next +** SQLite to access the file will not know that the journal exists (because +** the directory entry for the journal was never created) and the transaction +** will not roll back - possibly leading to database corruption. +*/ +static int unixSync(OsFile *id, int dataOnly){ + unixFile *pFile = (unixFile*)id; + assert( pFile ); + SimulateIOError(SQLITE_IOERR); + TRACE2("SYNC %-3d\n", pFile->h); + if( full_fsync(pFile->h, pFile->fullSync, dataOnly) ){ + return SQLITE_IOERR; + } + if( pFile->dirfd>=0 ){ + TRACE4("DIRSYNC %-3d (have_fullfsync=%d fullsync=%d)\n", pFile->dirfd, + HAVE_FULLFSYNC, pFile->fullSync); +#ifndef SQLITE_DISABLE_DIRSYNC + /* The directory sync is only attempted if full_fsync is + ** turned off or unavailable. If a full_fsync occurred above, + ** then the directory sync is superfluous. + */ + if( (!HAVE_FULLFSYNC || !pFile->fullSync) && full_fsync(pFile->dirfd,0,0) ){ + /* + ** We have received multiple reports of fsync() returning + ** errors when applied to directories on certain file systems. + ** A failed directory sync is not a big deal. So it seems + ** better to ignore the error. Ticket #1657 + */ + /* return SQLITE_IOERR; */ + } +#endif + close(pFile->dirfd); /* Only need to sync once, so close the directory */ + pFile->dirfd = -1; /* when we are done. */ + } + return SQLITE_OK; +} + +/* +** Sync the directory zDirname. This is a no-op on operating systems other +** than UNIX. +** +** This is used to make sure the master journal file has truely been deleted +** before making changes to individual journals on a multi-database commit. +** The F_FULLFSYNC option is not needed here. +*/ +int sqlite3UnixSyncDirectory(const char *zDirname){ +#ifdef SQLITE_DISABLE_DIRSYNC + return SQLITE_OK; +#else + int fd; + int r; + SimulateIOError(SQLITE_IOERR); + fd = open(zDirname, O_RDONLY|O_BINARY, 0); + TRACE3("DIRSYNC %-3d (%s)\n", fd, zDirname); + if( fd<0 ){ + return SQLITE_CANTOPEN; + } + r = fsync(fd); + close(fd); + return ((r==0)?SQLITE_OK:SQLITE_IOERR); +#endif +} + +/* +** Truncate an open file to a specified size +*/ +static int unixTruncate(OsFile *id, i64 nByte){ + assert( id ); + SimulateIOError(SQLITE_IOERR); + return ftruncate(((unixFile*)id)->h, nByte)==0 ? SQLITE_OK : SQLITE_IOERR; +} + +/* +** Determine the current size of a file in bytes +*/ +static int unixFileSize(OsFile *id, i64 *pSize){ + struct stat buf; + assert( id ); + SimulateIOError(SQLITE_IOERR); + if( fstat(((unixFile*)id)->h, &buf)!=0 ){ + return SQLITE_IOERR; + } + *pSize = buf.st_size; + return SQLITE_OK; +} + +/* +** This routine checks if there is a RESERVED lock held on the specified +** file by this or any other process. If such a lock is held, return +** non-zero. If the file is unlocked or holds only SHARED locks, then +** return zero. +*/ +static int unixCheckReservedLock(OsFile *id){ + int r = 0; + unixFile *pFile = (unixFile*)id; + + assert( pFile ); + sqlite3OsEnterMutex(); /* Because pFile->pLock is shared across threads */ + + /* Check if a thread in this process holds such a lock */ + if( pFile->pLock->locktype>SHARED_LOCK ){ + r = 1; + } + + /* Otherwise see if some other process holds it. + */ + if( !r ){ + struct flock lock; + lock.l_whence = SEEK_SET; + lock.l_start = RESERVED_BYTE; + lock.l_len = 1; + lock.l_type = F_WRLCK; + fcntl(pFile->h, F_GETLK, &lock); + if( lock.l_type!=F_UNLCK ){ + r = 1; + } + } + + sqlite3OsLeaveMutex(); + TRACE3("TEST WR-LOCK %d %d\n", pFile->h, r); + + return r; +} + +/* +** Lock the file with the lock specified by parameter locktype - one +** of the following: +** +** (1) SHARED_LOCK +** (2) RESERVED_LOCK +** (3) PENDING_LOCK +** (4) EXCLUSIVE_LOCK +** +** Sometimes when requesting one lock state, additional lock states +** are inserted in between. The locking might fail on one of the later +** transitions leaving the lock state different from what it started but +** still short of its goal. The following chart shows the allowed +** transitions and the inserted intermediate states: +** +** UNLOCKED -> SHARED +** SHARED -> RESERVED +** SHARED -> (PENDING) -> EXCLUSIVE +** RESERVED -> (PENDING) -> EXCLUSIVE +** PENDING -> EXCLUSIVE +** +** This routine will only increase a lock. Use the sqlite3OsUnlock() +** routine to lower a locking level. +*/ +static int unixLock(OsFile *id, int locktype){ + /* The following describes the implementation of the various locks and + ** lock transitions in terms of the POSIX advisory shared and exclusive + ** lock primitives (called read-locks and write-locks below, to avoid + ** confusion with SQLite lock names). The algorithms are complicated + ** slightly in order to be compatible with windows systems simultaneously + ** accessing the same database file, in case that is ever required. + ** + ** Symbols defined in os.h indentify the 'pending byte' and the 'reserved + ** byte', each single bytes at well known offsets, and the 'shared byte + ** range', a range of 510 bytes at a well known offset. + ** + ** To obtain a SHARED lock, a read-lock is obtained on the 'pending + ** byte'. If this is successful, a random byte from the 'shared byte + ** range' is read-locked and the lock on the 'pending byte' released. + ** + ** A process may only obtain a RESERVED lock after it has a SHARED lock. + ** A RESERVED lock is implemented by grabbing a write-lock on the + ** 'reserved byte'. + ** + ** A process may only obtain a PENDING lock after it has obtained a + ** SHARED lock. A PENDING lock is implemented by obtaining a write-lock + ** on the 'pending byte'. This ensures that no new SHARED locks can be + ** obtained, but existing SHARED locks are allowed to persist. A process + ** does not have to obtain a RESERVED lock on the way to a PENDING lock. + ** This property is used by the algorithm for rolling back a journal file + ** after a crash. + ** + ** An EXCLUSIVE lock, obtained after a PENDING lock is held, is + ** implemented by obtaining a write-lock on the entire 'shared byte + ** range'. Since all other locks require a read-lock on one of the bytes + ** within this range, this ensures that no other locks are held on the + ** database. + ** + ** The reason a single byte cannot be used instead of the 'shared byte + ** range' is that some versions of windows do not support read-locks. By + ** locking a random byte from a range, concurrent SHARED locks may exist + ** even if the locking primitive used is always a write-lock. + */ + int rc = SQLITE_OK; + unixFile *pFile = (unixFile*)id; + struct lockInfo *pLock = pFile->pLock; + struct flock lock; + int s; + + assert( pFile ); + TRACE7("LOCK %d %s was %s(%s,%d) pid=%d\n", pFile->h, + locktypeName(locktype), locktypeName(pFile->locktype), + locktypeName(pLock->locktype), pLock->cnt , getpid()); + + /* If there is already a lock of this type or more restrictive on the + ** OsFile, do nothing. Don't use the end_lock: exit path, as + ** sqlite3OsEnterMutex() hasn't been called yet. + */ + if( pFile->locktype>=locktype ){ + TRACE3("LOCK %d %s ok (already held)\n", pFile->h, + locktypeName(locktype)); + return SQLITE_OK; + } + + /* Make sure the locking sequence is correct + */ + assert( pFile->locktype!=NO_LOCK || locktype==SHARED_LOCK ); + assert( locktype!=PENDING_LOCK ); + assert( locktype!=RESERVED_LOCK || pFile->locktype==SHARED_LOCK ); + + /* This mutex is needed because pFile->pLock is shared across threads + */ + sqlite3OsEnterMutex(); + + /* Make sure the current thread owns the pFile. + */ + rc = transferOwnership(pFile); + if( rc!=SQLITE_OK ){ + sqlite3OsLeaveMutex(); + return rc; + } + pLock = pFile->pLock; + + /* If some thread using this PID has a lock via a different OsFile* + ** handle that precludes the requested lock, return BUSY. + */ + if( (pFile->locktype!=pLock->locktype && + (pLock->locktype>=PENDING_LOCK || locktype>SHARED_LOCK)) + ){ + rc = SQLITE_BUSY; + goto end_lock; + } + + /* If a SHARED lock is requested, and some thread using this PID already + ** has a SHARED or RESERVED lock, then increment reference counts and + ** return SQLITE_OK. + */ + if( locktype==SHARED_LOCK && + (pLock->locktype==SHARED_LOCK || pLock->locktype==RESERVED_LOCK) ){ + assert( locktype==SHARED_LOCK ); + assert( pFile->locktype==0 ); + assert( pLock->cnt>0 ); + pFile->locktype = SHARED_LOCK; + pLock->cnt++; + pFile->pOpen->nLock++; + goto end_lock; + } + + lock.l_len = 1L; + + lock.l_whence = SEEK_SET; + + /* A PENDING lock is needed before acquiring a SHARED lock and before + ** acquiring an EXCLUSIVE lock. For the SHARED lock, the PENDING will + ** be released. + */ + if( locktype==SHARED_LOCK + || (locktype==EXCLUSIVE_LOCK && pFile->locktypeh, F_SETLK, &lock); + if( s ){ + rc = (errno==EINVAL) ? SQLITE_NOLFS : SQLITE_BUSY; + goto end_lock; + } + } + + + /* If control gets to this point, then actually go ahead and make + ** operating system calls for the specified lock. + */ + if( locktype==SHARED_LOCK ){ + assert( pLock->cnt==0 ); + assert( pLock->locktype==0 ); + + /* Now get the read-lock */ + lock.l_start = SHARED_FIRST; + lock.l_len = SHARED_SIZE; + s = fcntl(pFile->h, F_SETLK, &lock); + + /* Drop the temporary PENDING lock */ + lock.l_start = PENDING_BYTE; + lock.l_len = 1L; + lock.l_type = F_UNLCK; + if( fcntl(pFile->h, F_SETLK, &lock)!=0 ){ + rc = SQLITE_IOERR; /* This should never happen */ + goto end_lock; + } + if( s ){ + rc = (errno==EINVAL) ? SQLITE_NOLFS : SQLITE_BUSY; + }else{ + pFile->locktype = SHARED_LOCK; + pFile->pOpen->nLock++; + pLock->cnt = 1; + } + }else if( locktype==EXCLUSIVE_LOCK && pLock->cnt>1 ){ + /* We are trying for an exclusive lock but another thread in this + ** same process is still holding a shared lock. */ + rc = SQLITE_BUSY; + }else{ + /* The request was for a RESERVED or EXCLUSIVE lock. It is + ** assumed that there is a SHARED or greater lock on the file + ** already. + */ + assert( 0!=pFile->locktype ); + lock.l_type = F_WRLCK; + switch( locktype ){ + case RESERVED_LOCK: + lock.l_start = RESERVED_BYTE; + break; + case EXCLUSIVE_LOCK: + lock.l_start = SHARED_FIRST; + lock.l_len = SHARED_SIZE; + break; + default: + assert(0); + } + s = fcntl(pFile->h, F_SETLK, &lock); + if( s ){ + rc = (errno==EINVAL) ? SQLITE_NOLFS : SQLITE_BUSY; + } + } + + if( rc==SQLITE_OK ){ + pFile->locktype = locktype; + pLock->locktype = locktype; + }else if( locktype==EXCLUSIVE_LOCK ){ + pFile->locktype = PENDING_LOCK; + pLock->locktype = PENDING_LOCK; + } + +end_lock: + sqlite3OsLeaveMutex(); + TRACE4("LOCK %d %s %s\n", pFile->h, locktypeName(locktype), + rc==SQLITE_OK ? "ok" : "failed"); + return rc; +} + +/* +** Lower the locking level on file descriptor pFile to locktype. locktype +** must be either NO_LOCK or SHARED_LOCK. +** +** If the locking level of the file descriptor is already at or below +** the requested locking level, this routine is a no-op. +*/ +static int unixUnlock(OsFile *id, int locktype){ + struct lockInfo *pLock; + struct flock lock; + int rc = SQLITE_OK; + unixFile *pFile = (unixFile*)id; + + assert( pFile ); + TRACE7("UNLOCK %d %d was %d(%d,%d) pid=%d\n", pFile->h, locktype, + pFile->locktype, pFile->pLock->locktype, pFile->pLock->cnt, getpid()); + + assert( locktype<=SHARED_LOCK ); + if( pFile->locktype<=locktype ){ + return SQLITE_OK; + } + if( CHECK_THREADID(pFile) ){ + return SQLITE_MISUSE; + } + sqlite3OsEnterMutex(); + pLock = pFile->pLock; + assert( pLock->cnt!=0 ); + if( pFile->locktype>SHARED_LOCK ){ + assert( pLock->locktype==pFile->locktype ); + if( locktype==SHARED_LOCK ){ + lock.l_type = F_RDLCK; + lock.l_whence = SEEK_SET; + lock.l_start = SHARED_FIRST; + lock.l_len = SHARED_SIZE; + if( fcntl(pFile->h, F_SETLK, &lock)!=0 ){ + /* This should never happen */ + rc = SQLITE_IOERR; + } + } + lock.l_type = F_UNLCK; + lock.l_whence = SEEK_SET; + lock.l_start = PENDING_BYTE; + lock.l_len = 2L; assert( PENDING_BYTE+1==RESERVED_BYTE ); + if( fcntl(pFile->h, F_SETLK, &lock)==0 ){ + pLock->locktype = SHARED_LOCK; + }else{ + rc = SQLITE_IOERR; /* This should never happen */ + } + } + if( locktype==NO_LOCK ){ + struct openCnt *pOpen; + + /* Decrement the shared lock counter. Release the lock using an + ** OS call only when all threads in this same process have released + ** the lock. + */ + pLock->cnt--; + if( pLock->cnt==0 ){ + lock.l_type = F_UNLCK; + lock.l_whence = SEEK_SET; + lock.l_start = lock.l_len = 0L; + if( fcntl(pFile->h, F_SETLK, &lock)==0 ){ + pLock->locktype = NO_LOCK; + }else{ + rc = SQLITE_IOERR; /* This should never happen */ + } + } + + /* Decrement the count of locks against this same file. When the + ** count reaches zero, close any other file descriptors whose close + ** was deferred because of outstanding locks. + */ + pOpen = pFile->pOpen; + pOpen->nLock--; + assert( pOpen->nLock>=0 ); + if( pOpen->nLock==0 && pOpen->nPending>0 ){ + int i; + for(i=0; inPending; i++){ + close(pOpen->aPending[i]); + } + free(pOpen->aPending); + pOpen->nPending = 0; + pOpen->aPending = 0; + } + } + sqlite3OsLeaveMutex(); + pFile->locktype = locktype; + return rc; +} + +/* +** Close a file. +*/ +static int unixClose(OsFile **pId){ + unixFile *id = (unixFile*)*pId; + + if( !id ) return SQLITE_OK; + unixUnlock(*pId, NO_LOCK); + if( id->dirfd>=0 ) close(id->dirfd); + id->dirfd = -1; + sqlite3OsEnterMutex(); + + if( id->pOpen->nLock ){ + /* If there are outstanding locks, do not actually close the file just + ** yet because that would clear those locks. Instead, add the file + ** descriptor to pOpen->aPending. It will be automatically closed when + ** the last lock is cleared. + */ + int *aNew; + struct openCnt *pOpen = id->pOpen; + aNew = realloc( pOpen->aPending, (pOpen->nPending+1)*sizeof(int) ); + if( aNew==0 ){ + /* If a malloc fails, just leak the file descriptor */ + }else{ + pOpen->aPending = aNew; + pOpen->aPending[pOpen->nPending] = id->h; + pOpen->nPending++; + } + }else{ + /* There are no outstanding locks so we can close the file immediately */ + close(id->h); + } + releaseLockInfo(id->pLock); + releaseOpenCnt(id->pOpen); + + sqlite3OsLeaveMutex(); + id->isOpen = 0; + TRACE2("CLOSE %-3d\n", id->h); + OpenCounter(-1); + sqliteFree(id); + *pId = 0; + return SQLITE_OK; +} + +/* +** Turn a relative pathname into a full pathname. Return a pointer +** to the full pathname stored in space obtained from sqliteMalloc(). +** The calling function is responsible for freeing this space once it +** is no longer needed. +*/ +char *sqlite3UnixFullPathname(const char *zRelative){ + char *zFull = 0; + if( zRelative[0]=='/' ){ + sqlite3SetString(&zFull, zRelative, (char*)0); + }else{ + char *zBuf = sqliteMalloc(5000); + if( zBuf==0 ){ + return 0; + } + zBuf[0] = 0; + sqlite3SetString(&zFull, getcwd(zBuf, 5000), "/", zRelative, + (char*)0); + sqliteFree(zBuf); + } + return zFull; +} + +/* +** Change the value of the fullsync flag in the given file descriptor. +*/ +static void unixSetFullSync(OsFile *id, int v){ + ((unixFile*)id)->fullSync = v; +} + +/* +** Return the underlying file handle for an OsFile +*/ +static int unixFileHandle(OsFile *id){ + return ((unixFile*)id)->h; +} + +/* +** Return an integer that indices the type of lock currently held +** by this handle. (Used for testing and analysis only.) +*/ +static int unixLockState(OsFile *id){ + return ((unixFile*)id)->locktype; +} + +/* +** This vector defines all the methods that can operate on an OsFile +** for unix. +*/ +static const IoMethod sqlite3UnixIoMethod = { + unixClose, + unixOpenDirectory, + unixRead, + unixWrite, + unixSeek, + unixTruncate, + unixSync, + unixSetFullSync, + unixFileHandle, + unixFileSize, + unixLock, + unixUnlock, + unixLockState, + unixCheckReservedLock, +}; + +/* +** Allocate memory for a unixFile. Initialize the new unixFile +** to the value given in pInit and return a pointer to the new +** OsFile. If we run out of memory, close the file and return NULL. +*/ +static int allocateUnixFile(unixFile *pInit, OsFile **pId){ + unixFile *pNew; + pInit->dirfd = -1; + pInit->fullSync = 0; + pInit->locktype = 0; + SET_THREADID(pInit); + pNew = sqliteMalloc( sizeof(unixFile) ); + if( pNew==0 ){ + close(pInit->h); + sqlite3OsEnterMutex(); + releaseLockInfo(pInit->pLock); + releaseOpenCnt(pInit->pOpen); + sqlite3OsLeaveMutex(); + *pId = 0; + return SQLITE_NOMEM; + }else{ + *pNew = *pInit; + pNew->pMethod = &sqlite3UnixIoMethod; + *pId = (OsFile*)pNew; + OpenCounter(+1); + return SQLITE_OK; + } +} + + +#endif /* SQLITE_OMIT_DISKIO */ +/*************************************************************************** +** Everything above deals with file I/O. Everything that follows deals +** with other miscellanous aspects of the operating system interface +****************************************************************************/ + + +/* +** Get information to seed the random number generator. The seed +** is written into the buffer zBuf[256]. The calling function must +** supply a sufficiently large buffer. +*/ +int sqlite3UnixRandomSeed(char *zBuf){ + /* We have to initialize zBuf to prevent valgrind from reporting + ** errors. The reports issued by valgrind are incorrect - we would + ** prefer that the randomness be increased by making use of the + ** uninitialized space in zBuf - but valgrind errors tend to worry + ** some users. Rather than argue, it seems easier just to initialize + ** the whole array and silence valgrind, even if that means less randomness + ** in the random seed. + ** + ** When testing, initializing zBuf[] to zero is all we do. That means + ** that we always use the same random number sequence. This makes the + ** tests repeatable. + */ + memset(zBuf, 0, 256); +#if !defined(SQLITE_TEST) + { + int pid, fd; + fd = open("/dev/urandom", O_RDONLY); + if( fd<0 ){ + time_t t; + time(&t); + memcpy(zBuf, &t, sizeof(t)); + pid = getpid(); + memcpy(&zBuf[sizeof(time_t)], &pid, sizeof(pid)); + }else{ + read(fd, zBuf, 256); + close(fd); + } + } +#endif + return SQLITE_OK; +} + +/* +** Sleep for a little while. Return the amount of time slept. +** The argument is the number of milliseconds we want to sleep. +*/ +int sqlite3UnixSleep(int ms){ +#if defined(HAVE_USLEEP) && HAVE_USLEEP + usleep(ms*1000); + return ms; +#else + sleep((ms+999)/1000); + return 1000*((ms+999)/1000); +#endif +} + +/* +** Static variables used for thread synchronization. +** +** inMutex the nesting depth of the recursive mutex. The thread +** holding mutexMain can read this variable at any time. +** But is must hold mutexAux to change this variable. Other +** threads must hold mutexAux to read the variable and can +** never write. +** +** mutexOwner The thread id of the thread holding mutexMain. Same +** access rules as for inMutex. +** +** mutexOwnerValid True if the value in mutexOwner is valid. The same +** access rules apply as for inMutex. +** +** mutexMain The main mutex. Hold this mutex in order to get exclusive +** access to SQLite data structures. +** +** mutexAux An auxiliary mutex needed to access variables defined above. +** +** Mutexes are always acquired in this order: mutexMain mutexAux. It +** is not necessary to acquire mutexMain in order to get mutexAux - just +** do not attempt to acquire them in the reverse order: mutexAux mutexMain. +** Either get the mutexes with mutexMain first or get mutexAux only. +** +** When running on a platform where the three variables inMutex, mutexOwner, +** and mutexOwnerValid can be set atomically, the mutexAux is not required. +** On many systems, all three are 32-bit integers and writing to a 32-bit +** integer is atomic. I think. But there are no guarantees. So it seems +** safer to protect them using mutexAux. +*/ +static int inMutex = 0; +#ifdef SQLITE_UNIX_THREADS +static pthread_t mutexOwner; /* Thread holding mutexMain */ +static int mutexOwnerValid = 0; /* True if mutexOwner is valid */ +static pthread_mutex_t mutexMain = PTHREAD_MUTEX_INITIALIZER; /* The mutex */ +static pthread_mutex_t mutexAux = PTHREAD_MUTEX_INITIALIZER; /* Aux mutex */ +#endif + +/* +** The following pair of routine implement mutual exclusion for +** multi-threaded processes. Only a single thread is allowed to +** executed code that is surrounded by EnterMutex() and LeaveMutex(). +** +** SQLite uses only a single Mutex. There is not much critical +** code and what little there is executes quickly and without blocking. +** +** As of version 3.3.2, this mutex must be recursive. +*/ +void sqlite3UnixEnterMutex(){ +#ifdef SQLITE_UNIX_THREADS + pthread_mutex_lock(&mutexAux); + if( !mutexOwnerValid || !pthread_equal(mutexOwner, pthread_self()) ){ + pthread_mutex_unlock(&mutexAux); + pthread_mutex_lock(&mutexMain); + assert( inMutex==0 ); + assert( !mutexOwnerValid ); + pthread_mutex_lock(&mutexAux); + mutexOwner = pthread_self(); + mutexOwnerValid = 1; + } + inMutex++; + pthread_mutex_unlock(&mutexAux); +#else + inMutex++; +#endif +} +void sqlite3UnixLeaveMutex(){ + assert( inMutex>0 ); +#ifdef SQLITE_UNIX_THREADS + pthread_mutex_lock(&mutexAux); + inMutex--; + assert( pthread_equal(mutexOwner, pthread_self()) ); + if( inMutex==0 ){ + assert( mutexOwnerValid ); + mutexOwnerValid = 0; + pthread_mutex_unlock(&mutexMain); + } + pthread_mutex_unlock(&mutexAux); +#else + inMutex--; +#endif +} + +/* +** Return TRUE if the mutex is currently held. +** +** If the thisThrd parameter is true, return true only if the +** calling thread holds the mutex. If the parameter is false, return +** true if any thread holds the mutex. +*/ +int sqlite3UnixInMutex(int thisThrd){ +#ifdef SQLITE_UNIX_THREADS + int rc; + pthread_mutex_lock(&mutexAux); + rc = inMutex>0 && (thisThrd==0 || pthread_equal(mutexOwner,pthread_self())); + pthread_mutex_unlock(&mutexAux); + return rc; +#else + return inMutex>0; +#endif +} + +/* +** Remember the number of thread-specific-data blocks allocated. +** Use this to verify that we are not leaking thread-specific-data. +** Ticket #1601 +*/ +#ifdef SQLITE_TEST +int sqlite3_tsd_count = 0; +# ifdef SQLITE_UNIX_THREADS + static pthread_mutex_t tsd_counter_mutex = PTHREAD_MUTEX_INITIALIZER; +# define TSD_COUNTER(N) \ + pthread_mutex_lock(&tsd_counter_mutex); \ + sqlite3_tsd_count += N; \ + pthread_mutex_unlock(&tsd_counter_mutex); +# else +# define TSD_COUNTER(N) sqlite3_tsd_count += N +# endif +#else +# define TSD_COUNTER(N) /* no-op */ +#endif + +/* +** If called with allocateFlag>0, then return a pointer to thread +** specific data for the current thread. Allocate and zero the +** thread-specific data if it does not already exist. +** +** If called with allocateFlag==0, then check the current thread +** specific data. Return it if it exists. If it does not exist, +** then return NULL. +** +** If called with allocateFlag<0, check to see if the thread specific +** data is allocated and is all zero. If it is then deallocate it. +** Return a pointer to the thread specific data or NULL if it is +** unallocated or gets deallocated. +*/ +ThreadData *sqlite3UnixThreadSpecificData(int allocateFlag){ + static const ThreadData zeroData = {0}; /* Initializer to silence warnings + ** from broken compilers */ +#ifdef SQLITE_UNIX_THREADS + static pthread_key_t key; + static int keyInit = 0; + ThreadData *pTsd; + + if( !keyInit ){ + sqlite3OsEnterMutex(); + if( !keyInit ){ + int rc; + rc = pthread_key_create(&key, 0); + if( rc ){ + sqlite3OsLeaveMutex(); + return 0; + } + keyInit = 1; + } + sqlite3OsLeaveMutex(); + } + + pTsd = pthread_getspecific(key); + if( allocateFlag>0 ){ + if( pTsd==0 ){ + if( !sqlite3TestMallocFail() ){ + pTsd = sqlite3OsMalloc(sizeof(zeroData)); + } +#ifdef SQLITE_MEMDEBUG + sqlite3_isFail = 0; +#endif + if( pTsd ){ + *pTsd = zeroData; + pthread_setspecific(key, pTsd); + TSD_COUNTER(+1); + } + } + }else if( pTsd!=0 && allocateFlag<0 + && memcmp(pTsd, &zeroData, sizeof(ThreadData))==0 ){ + sqlite3OsFree(pTsd); + pthread_setspecific(key, 0); + TSD_COUNTER(-1); + pTsd = 0; + } + return pTsd; +#else + static ThreadData *pTsd = 0; + if( allocateFlag>0 ){ + if( pTsd==0 ){ + if( !sqlite3TestMallocFail() ){ + pTsd = sqlite3OsMalloc( sizeof(zeroData) ); + } +#ifdef SQLITE_MEMDEBUG + sqlite3_isFail = 0; +#endif + if( pTsd ){ + *pTsd = zeroData; + TSD_COUNTER(+1); + } + } + }else if( pTsd!=0 && allocateFlag<0 + && memcmp(pTsd, &zeroData, sizeof(ThreadData))==0 ){ + sqlite3OsFree(pTsd); + TSD_COUNTER(-1); + pTsd = 0; + } + return pTsd; +#endif +} + +/* +** The following variable, if set to a non-zero value, becomes the result +** returned from sqlite3OsCurrentTime(). This is used for testing. +*/ +#ifdef SQLITE_TEST +int sqlite3_current_time = 0; +#endif + +/* +** Find the current time (in Universal Coordinated Time). Write the +** current time and date as a Julian Day number into *prNow and +** return 0. Return 1 if the time and date cannot be found. +*/ +int sqlite3UnixCurrentTime(double *prNow){ +#ifdef NO_GETTOD + time_t t; + time(&t); + *prNow = t/86400.0 + 2440587.5; +#else + struct timeval sNow; + struct timezone sTz; /* Not used */ + gettimeofday(&sNow, &sTz); + *prNow = 2440587.5 + sNow.tv_sec/86400.0 + sNow.tv_usec/86400000000.0; +#endif +#ifdef SQLITE_TEST + if( sqlite3_current_time ){ + *prNow = sqlite3_current_time/86400.0 + 2440587.5; + } +#endif + return 0; +} + +#endif /* OS_UNIX */ Added: external/sqlite-source-3.3.4/os_win.c ============================================================================== --- (empty file) +++ external/sqlite-source-3.3.4/os_win.c Mon Apr 3 07:54:59 2006 @@ -0,0 +1,1516 @@ +/* +** 2004 May 22 +** +** The author disclaims copyright to this source code. In place of +** a legal notice, here is a blessing: +** +** May you do good and not evil. +** May you find forgiveness for yourself and forgive others. +** May you share freely, never taking more than you give. +** +****************************************************************************** +** +** This file contains code that is specific to windows. +*/ +#include "sqliteInt.h" +#include "os.h" +#if OS_WIN /* This file is used for windows only */ + +#include + +#ifdef __CYGWIN__ +# include +#endif + +/* +** Macros used to determine whether or not to use threads. +*/ +#if defined(THREADSAFE) && THREADSAFE +# define SQLITE_W32_THREADS 1 +#endif + +/* +** Include code that is common to all os_*.c files +*/ +#include "os_common.h" + +/* +** Determine if we are dealing with WindowsCE - which has a much +** reduced API. +*/ +#if defined(_WIN32_WCE) +# define OS_WINCE 1 +#else +# define OS_WINCE 0 +#endif + +/* +** WinCE lacks native support for file locking so we have to fake it +** with some code of our own. +*/ +#if OS_WINCE +typedef struct winceLock { + int nReaders; /* Number of reader locks obtained */ + BOOL bPending; /* Indicates a pending lock has been obtained */ + BOOL bReserved; /* Indicates a reserved lock has been obtained */ + BOOL bExclusive; /* Indicates an exclusive lock has been obtained */ +} winceLock; +#endif + +/* +** The winFile structure is a subclass of OsFile specific to the win32 +** portability layer. +*/ +typedef struct winFile winFile; +struct winFile { + IoMethod const *pMethod;/* Must be first */ + HANDLE h; /* Handle for accessing the file */ + unsigned char locktype; /* Type of lock currently held on this file */ + short sharedLockByte; /* Randomly chosen byte used as a shared lock */ +#if OS_WINCE + WCHAR *zDeleteOnClose; /* Name of file to delete when closing */ + HANDLE hMutex; /* Mutex used to control access to shared lock */ + HANDLE hShared; /* Shared memory segment used for locking */ + winceLock local; /* Locks obtained by this instance of winFile */ + winceLock *shared; /* Global shared lock memory for the file */ +#endif +}; + + +/* +** Do not include any of the File I/O interface procedures if the +** SQLITE_OMIT_DISKIO macro is defined (indicating that there database +** will be in-memory only) +*/ +#ifndef SQLITE_OMIT_DISKIO + +/* +** The following variable is (normally) set once and never changes +** thereafter. It records whether the operating system is Win95 +** or WinNT. +** +** 0: Operating system unknown. +** 1: Operating system is Win95. +** 2: Operating system is WinNT. +** +** In order to facilitate testing on a WinNT system, the test fixture +** can manually set this value to 1 to emulate Win98 behavior. +*/ +int sqlite3_os_type = 0; + +/* +** Return true (non-zero) if we are running under WinNT, Win2K, WinXP, +** or WinCE. Return false (zero) for Win95, Win98, or WinME. +** +** Here is an interesting observation: Win95, Win98, and WinME lack +** the LockFileEx() API. But we can still statically link against that +** API as long as we don't call it win running Win95/98/ME. A call to +** this routine is used to determine if the host is Win95/98/ME or +** WinNT/2K/XP so that we will know whether or not we can safely call +** the LockFileEx() API. +*/ +#if OS_WINCE +# define isNT() (1) +#else + static int isNT(void){ + if( sqlite3_os_type==0 ){ + OSVERSIONINFO sInfo; + sInfo.dwOSVersionInfoSize = sizeof(sInfo); + GetVersionEx(&sInfo); + sqlite3_os_type = sInfo.dwPlatformId==VER_PLATFORM_WIN32_NT ? 2 : 1; + } + return sqlite3_os_type==2; + } +#endif /* OS_WINCE */ + +/* +** Convert a UTF-8 string to UTF-32. Space to hold the returned string +** is obtained from sqliteMalloc. +*/ +static WCHAR *utf8ToUnicode(const char *zFilename){ + int nByte; + WCHAR *zWideFilename; + + if( !isNT() ){ + return 0; + } + nByte = MultiByteToWideChar(CP_UTF8, 0, zFilename, -1, NULL, 0)*sizeof(WCHAR); + zWideFilename = sqliteMalloc( nByte*sizeof(zWideFilename[0]) ); + if( zWideFilename==0 ){ + return 0; + } + nByte = MultiByteToWideChar(CP_UTF8, 0, zFilename, -1, zWideFilename, nByte); + if( nByte==0 ){ + sqliteFree(zWideFilename); + zWideFilename = 0; + } + return zWideFilename; +} + +/* +** Convert UTF-32 to UTF-8. Space to hold the returned string is +** obtained from sqliteMalloc(). +*/ +static char *unicodeToUtf8(const WCHAR *zWideFilename){ + int nByte; + char *zFilename; + + nByte = WideCharToMultiByte(CP_UTF8, 0, zWideFilename, -1, 0, 0, 0, 0); + zFilename = sqliteMalloc( nByte ); + if( zFilename==0 ){ + return 0; + } + nByte = WideCharToMultiByte(CP_UTF8, 0, zWideFilename, -1, zFilename, nByte, + 0, 0); + if( nByte == 0 ){ + sqliteFree(zFilename); + zFilename = 0; + } + return zFilename; +} + +#if OS_WINCE +/************************************************************************* +** This section contains code for WinCE only. +*/ +/* +** WindowsCE does not have a localtime() function. So create a +** substitute. +*/ +#include +struct tm *__cdecl localtime(const time_t *t) +{ + static struct tm y; + FILETIME uTm, lTm; + SYSTEMTIME pTm; + i64 t64; + t64 = *t; + t64 = (t64 + 11644473600)*10000000; + uTm.dwLowDateTime = t64 & 0xFFFFFFFF; + uTm.dwHighDateTime= t64 >> 32; + FileTimeToLocalFileTime(&uTm,&lTm); + FileTimeToSystemTime(&lTm,&pTm); + y.tm_year = pTm.wYear - 1900; + y.tm_mon = pTm.wMonth - 1; + y.tm_wday = pTm.wDayOfWeek; + y.tm_mday = pTm.wDay; + y.tm_hour = pTm.wHour; + y.tm_min = pTm.wMinute; + y.tm_sec = pTm.wSecond; + return &y; +} + +/* This will never be called, but defined to make the code compile */ +#define GetTempPathA(a,b) + +#define LockFile(a,b,c,d,e) winceLockFile(&a, b, c, d, e) +#define UnlockFile(a,b,c,d,e) winceUnlockFile(&a, b, c, d, e) +#define LockFileEx(a,b,c,d,e,f) winceLockFileEx(&a, b, c, d, e, f) + +#define HANDLE_TO_WINFILE(a) (winFile*)&((char*)a)[-offsetof(winFile,h)] + +/* +** Acquire a lock on the handle h +*/ +static void winceMutexAcquire(HANDLE h){ + DWORD dwErr; + do { + dwErr = WaitForSingleObject(h, INFINITE); + } while (dwErr != WAIT_OBJECT_0 && dwErr != WAIT_ABANDONED); +} +/* +** Release a lock acquired by winceMutexAcquire() +*/ +#define winceMutexRelease(h) ReleaseMutex(h) + +/* +** Create the mutex and shared memory used for locking in the file +** descriptor pFile +*/ +static BOOL winceCreateLock(const char *zFilename, winFile *pFile){ + WCHAR *zTok; + WCHAR *zName = utf8ToUnicode(zFilename); + BOOL bInit = TRUE; + + /* Initialize the local lockdata */ + ZeroMemory(&pFile->local, sizeof(pFile->local)); + + /* Replace the backslashes from the filename and lowercase it + ** to derive a mutex name. */ + zTok = CharLowerW(zName); + for (;*zTok;zTok++){ + if (*zTok == '\\') *zTok = '_'; + } + + /* Create/open the named mutex */ + pFile->hMutex = CreateMutexW(NULL, FALSE, zName); + if (!pFile->hMutex){ + sqliteFree(zName); + return FALSE; + } + + /* Acquire the mutex before continuing */ + winceMutexAcquire(pFile->hMutex); + + /* Since the names of named mutexes, semaphores, file mappings etc are + ** case-sensitive, take advantage of that by uppercasing the mutex name + ** and using that as the shared filemapping name. + */ + CharUpperW(zName); + pFile->hShared = CreateFileMappingW(INVALID_HANDLE_VALUE, NULL, + PAGE_READWRITE, 0, sizeof(winceLock), + zName); + + /* Set a flag that indicates we're the first to create the memory so it + ** must be zero-initialized */ + if (GetLastError() == ERROR_ALREADY_EXISTS){ + bInit = FALSE; + } + + sqliteFree(zName); + + /* If we succeeded in making the shared memory handle, map it. */ + if (pFile->hShared){ + pFile->shared = (winceLock*)MapViewOfFile(pFile->hShared, + FILE_MAP_READ|FILE_MAP_WRITE, 0, 0, sizeof(winceLock)); + /* If mapping failed, close the shared memory handle and erase it */ + if (!pFile->shared){ + CloseHandle(pFile->hShared); + pFile->hShared = NULL; + } + } + + /* If shared memory could not be created, then close the mutex and fail */ + if (pFile->hShared == NULL){ + winceMutexRelease(pFile->hMutex); + CloseHandle(pFile->hMutex); + pFile->hMutex = NULL; + return FALSE; + } + + /* Initialize the shared memory if we're supposed to */ + if (bInit) { + ZeroMemory(pFile->shared, sizeof(winceLock)); + } + + winceMutexRelease(pFile->hMutex); + return TRUE; +} + +/* +** Destroy the part of winFile that deals with wince locks +*/ +static void winceDestroyLock(winFile *pFile){ + if (pFile->hMutex){ + /* Acquire the mutex */ + winceMutexAcquire(pFile->hMutex); + + /* The following blocks should probably assert in debug mode, but they + are to cleanup in case any locks remained open */ + if (pFile->local.nReaders){ + pFile->shared->nReaders --; + } + if (pFile->local.bReserved){ + pFile->shared->bReserved = FALSE; + } + if (pFile->local.bPending){ + pFile->shared->bPending = FALSE; + } + if (pFile->local.bExclusive){ + pFile->shared->bExclusive = FALSE; + } + + /* De-reference and close our copy of the shared memory handle */ + UnmapViewOfFile(pFile->shared); + CloseHandle(pFile->hShared); + + /* Done with the mutex */ + winceMutexRelease(pFile->hMutex); + CloseHandle(pFile->hMutex); + pFile->hMutex = NULL; + } +} + +/* +** An implementation of the LockFile() API of windows for wince +*/ +static BOOL winceLockFile( + HANDLE *phFile, + DWORD dwFileOffsetLow, + DWORD dwFileOffsetHigh, + DWORD nNumberOfBytesToLockLow, + DWORD nNumberOfBytesToLockHigh +){ + winFile *pFile = HANDLE_TO_WINFILE(phFile); + BOOL bReturn = FALSE; + + if (!pFile->hMutex) return TRUE; + winceMutexAcquire(pFile->hMutex); + + /* Wanting an exclusive lock? */ + if (dwFileOffsetLow == SHARED_FIRST + && nNumberOfBytesToLockLow == SHARED_SIZE){ + if (pFile->shared->nReaders == 0 && pFile->shared->bExclusive == 0){ + pFile->shared->bExclusive = TRUE; + pFile->local.bExclusive = TRUE; + bReturn = TRUE; + } + } + + /* Want a read-only lock? */ + else if ((dwFileOffsetLow >= SHARED_FIRST && + dwFileOffsetLow < SHARED_FIRST + SHARED_SIZE) && + nNumberOfBytesToLockLow == 1){ + if (pFile->shared->bExclusive == 0){ + pFile->local.nReaders ++; + if (pFile->local.nReaders == 1){ + pFile->shared->nReaders ++; + } + bReturn = TRUE; + } + } + + /* Want a pending lock? */ + else if (dwFileOffsetLow == PENDING_BYTE && nNumberOfBytesToLockLow == 1){ + /* If no pending lock has been acquired, then acquire it */ + if (pFile->shared->bPending == 0) { + pFile->shared->bPending = TRUE; + pFile->local.bPending = TRUE; + bReturn = TRUE; + } + } + /* Want a reserved lock? */ + else if (dwFileOffsetLow == RESERVED_BYTE && nNumberOfBytesToLockLow == 1){ + if (pFile->shared->bReserved == 0) { + pFile->shared->bReserved = TRUE; + pFile->local.bReserved = TRUE; + bReturn = TRUE; + } + } + + winceMutexRelease(pFile->hMutex); + return bReturn; +} + +/* +** An implementation of the UnlockFile API of windows for wince +*/ +static BOOL winceUnlockFile( + HANDLE *phFile, + DWORD dwFileOffsetLow, + DWORD dwFileOffsetHigh, + DWORD nNumberOfBytesToUnlockLow, + DWORD nNumberOfBytesToUnlockHigh +){ + winFile *pFile = HANDLE_TO_WINFILE(phFile); + BOOL bReturn = FALSE; + + if (!pFile->hMutex) return TRUE; + winceMutexAcquire(pFile->hMutex); + + /* Releasing a reader lock or an exclusive lock */ + if (dwFileOffsetLow >= SHARED_FIRST && + dwFileOffsetLow < SHARED_FIRST + SHARED_SIZE){ + /* Did we have an exclusive lock? */ + if (pFile->local.bExclusive){ + pFile->local.bExclusive = FALSE; + pFile->shared->bExclusive = FALSE; + bReturn = TRUE; + } + + /* Did we just have a reader lock? */ + else if (pFile->local.nReaders){ + pFile->local.nReaders --; + if (pFile->local.nReaders == 0) + { + pFile->shared->nReaders --; + } + bReturn = TRUE; + } + } + + /* Releasing a pending lock */ + else if (dwFileOffsetLow == PENDING_BYTE && nNumberOfBytesToUnlockLow == 1){ + if (pFile->local.bPending){ + pFile->local.bPending = FALSE; + pFile->shared->bPending = FALSE; + bReturn = TRUE; + } + } + /* Releasing a reserved lock */ + else if (dwFileOffsetLow == RESERVED_BYTE && nNumberOfBytesToUnlockLow == 1){ + if (pFile->local.bReserved) { + pFile->local.bReserved = FALSE; + pFile->shared->bReserved = FALSE; + bReturn = TRUE; + } + } + + winceMutexRelease(pFile->hMutex); + return bReturn; +} + +/* +** An implementation of the LockFileEx() API of windows for wince +*/ +static BOOL winceLockFileEx( + HANDLE *phFile, + DWORD dwFlags, + DWORD dwReserved, + DWORD nNumberOfBytesToLockLow, + DWORD nNumberOfBytesToLockHigh, + LPOVERLAPPED lpOverlapped +){ + /* If the caller wants a shared read lock, forward this call + ** to winceLockFile */ + if (lpOverlapped->Offset == SHARED_FIRST && + dwFlags == 1 && + nNumberOfBytesToLockLow == SHARED_SIZE){ + return winceLockFile(phFile, SHARED_FIRST, 0, 1, 0); + } + return FALSE; +} +/* +** End of the special code for wince +*****************************************************************************/ +#endif /* OS_WINCE */ + +/* +** Delete the named file +*/ +int sqlite3WinDelete(const char *zFilename){ + WCHAR *zWide = utf8ToUnicode(zFilename); + if( zWide ){ + DeleteFileW(zWide); + sqliteFree(zWide); + }else{ +#if OS_WINCE + return SQLITE_NOMEM; +#else + DeleteFileA(zFilename); +#endif + } + TRACE2("DELETE \"%s\"\n", zFilename); + return SQLITE_OK; +} + +/* +** Return TRUE if the named file exists. +*/ +int sqlite3WinFileExists(const char *zFilename){ + int exists = 0; + WCHAR *zWide = utf8ToUnicode(zFilename); + if( zWide ){ + exists = GetFileAttributesW(zWide) != 0xffffffff; + sqliteFree(zWide); + }else{ +#if OS_WINCE + return SQLITE_NOMEM; +#else + exists = GetFileAttributesA(zFilename) != 0xffffffff; +#endif + } + return exists; +} + +/* Forward declaration */ +static int allocateWinFile(winFile *pInit, OsFile **pId); + +/* +** Attempt to open a file for both reading and writing. If that +** fails, try opening it read-only. If the file does not exist, +** try to create it. +** +** On success, a handle for the open file is written to *id +** and *pReadonly is set to 0 if the file was opened for reading and +** writing or 1 if the file was opened read-only. The function returns +** SQLITE_OK. +** +** On failure, the function returns SQLITE_CANTOPEN and leaves +** *id and *pReadonly unchanged. +*/ +int sqlite3WinOpenReadWrite( + const char *zFilename, + OsFile **pId, + int *pReadonly +){ + winFile f; + HANDLE h; + WCHAR *zWide = utf8ToUnicode(zFilename); + assert( *pId==0 ); + if( zWide ){ + h = CreateFileW(zWide, + GENERIC_READ | GENERIC_WRITE, + FILE_SHARE_READ | FILE_SHARE_WRITE, + NULL, + OPEN_ALWAYS, + FILE_ATTRIBUTE_NORMAL | FILE_FLAG_RANDOM_ACCESS, + NULL + ); + if( h==INVALID_HANDLE_VALUE ){ + h = CreateFileW(zWide, + GENERIC_READ, + FILE_SHARE_READ, + NULL, + OPEN_ALWAYS, + FILE_ATTRIBUTE_NORMAL | FILE_FLAG_RANDOM_ACCESS, + NULL + ); + if( h==INVALID_HANDLE_VALUE ){ + sqliteFree(zWide); + return SQLITE_CANTOPEN; + } + *pReadonly = 1; + }else{ + *pReadonly = 0; + } +#if OS_WINCE + if (!winceCreateLock(zFilename, &f)){ + CloseHandle(h); + sqliteFree(zWide); + return SQLITE_CANTOPEN; + } +#endif + sqliteFree(zWide); + }else{ +#if OS_WINCE + return SQLITE_NOMEM; +#else + h = CreateFileA(zFilename, + GENERIC_READ | GENERIC_WRITE, + FILE_SHARE_READ | FILE_SHARE_WRITE, + NULL, + OPEN_ALWAYS, + FILE_ATTRIBUTE_NORMAL | FILE_FLAG_RANDOM_ACCESS, + NULL + ); + if( h==INVALID_HANDLE_VALUE ){ + h = CreateFileA(zFilename, + GENERIC_READ, + FILE_SHARE_READ, + NULL, + OPEN_ALWAYS, + FILE_ATTRIBUTE_NORMAL | FILE_FLAG_RANDOM_ACCESS, + NULL + ); + if( h==INVALID_HANDLE_VALUE ){ + return SQLITE_CANTOPEN; + } + *pReadonly = 1; + }else{ + *pReadonly = 0; + } +#endif /* OS_WINCE */ + } + f.h = h; +#if OS_WINCE + f.zDeleteOnClose = 0; +#endif + TRACE3("OPEN R/W %d \"%s\"\n", h, zFilename); + return allocateWinFile(&f, pId); +} + + +/* +** Attempt to open a new file for exclusive access by this process. +** The file will be opened for both reading and writing. To avoid +** a potential security problem, we do not allow the file to have +** previously existed. Nor do we allow the file to be a symbolic +** link. +** +** If delFlag is true, then make arrangements to automatically delete +** the file when it is closed. +** +** On success, write the file handle into *id and return SQLITE_OK. +** +** On failure, return SQLITE_CANTOPEN. +*/ +int sqlite3WinOpenExclusive(const char *zFilename, OsFile **pId, int delFlag){ + winFile f; + HANDLE h; + int fileflags; + WCHAR *zWide = utf8ToUnicode(zFilename); + assert( *pId == 0 ); + fileflags = FILE_FLAG_RANDOM_ACCESS; +#if !OS_WINCE + if( delFlag ){ + fileflags |= FILE_ATTRIBUTE_TEMPORARY | FILE_FLAG_DELETE_ON_CLOSE; + } +#endif + if( zWide ){ + h = CreateFileW(zWide, + GENERIC_READ | GENERIC_WRITE, + 0, + NULL, + CREATE_ALWAYS, + fileflags, + NULL + ); + sqliteFree(zWide); + }else{ +#if OS_WINCE + return SQLITE_NOMEM; +#else + h = CreateFileA(zFilename, + GENERIC_READ | GENERIC_WRITE, + 0, + NULL, + CREATE_ALWAYS, + fileflags, + NULL + ); +#endif /* OS_WINCE */ + } + if( h==INVALID_HANDLE_VALUE ){ + return SQLITE_CANTOPEN; + } + f.h = h; +#if OS_WINCE + f.zDeleteOnClose = delFlag ? utf8ToUnicode(zFilename) : 0; + f.hMutex = NULL; +#endif + TRACE3("OPEN EX %d \"%s\"\n", h, zFilename); + return allocateWinFile(&f, pId); +} + +/* +** Attempt to open a new file for read-only access. +** +** On success, write the file handle into *id and return SQLITE_OK. +** +** On failure, return SQLITE_CANTOPEN. +*/ +int sqlite3WinOpenReadOnly(const char *zFilename, OsFile **pId){ + winFile f; + HANDLE h; + WCHAR *zWide = utf8ToUnicode(zFilename); + assert( *pId==0 ); + if( zWide ){ + h = CreateFileW(zWide, + GENERIC_READ, + 0, + NULL, + OPEN_EXISTING, + FILE_ATTRIBUTE_NORMAL | FILE_FLAG_RANDOM_ACCESS, + NULL + ); + sqliteFree(zWide); + }else{ +#if OS_WINCE + return SQLITE_NOMEM; +#else + h = CreateFileA(zFilename, + GENERIC_READ, + 0, + NULL, + OPEN_EXISTING, + FILE_ATTRIBUTE_NORMAL | FILE_FLAG_RANDOM_ACCESS, + NULL + ); +#endif + } + if( h==INVALID_HANDLE_VALUE ){ + return SQLITE_CANTOPEN; + } + f.h = h; +#if OS_WINCE + f.zDeleteOnClose = 0; + f.hMutex = NULL; +#endif + TRACE3("OPEN RO %d \"%s\"\n", h, zFilename); + return allocateWinFile(&f, pId); +} + +/* +** Attempt to open a file descriptor for the directory that contains a +** file. This file descriptor can be used to fsync() the directory +** in order to make sure the creation of a new file is actually written +** to disk. +** +** This routine is only meaningful for Unix. It is a no-op under +** windows since windows does not support hard links. +** +** On success, a handle for a previously open file is at *id is +** updated with the new directory file descriptor and SQLITE_OK is +** returned. +** +** On failure, the function returns SQLITE_CANTOPEN and leaves +** *id unchanged. +*/ +static int winOpenDirectory( + OsFile *id, + const char *zDirname +){ + return SQLITE_OK; +} + +/* +** If the following global variable points to a string which is the +** name of a directory, then that directory will be used to store +** temporary files. +*/ +char *sqlite3_temp_directory = 0; + +/* +** Create a temporary file name in zBuf. zBuf must be big enough to +** hold at least SQLITE_TEMPNAME_SIZE characters. +*/ +int sqlite3WinTempFileName(char *zBuf){ + static char zChars[] = + "abcdefghijklmnopqrstuvwxyz" + "ABCDEFGHIJKLMNOPQRSTUVWXYZ" + "0123456789"; + int i, j; + char zTempPath[SQLITE_TEMPNAME_SIZE]; + if( sqlite3_temp_directory ){ + strncpy(zTempPath, sqlite3_temp_directory, SQLITE_TEMPNAME_SIZE-30); + zTempPath[SQLITE_TEMPNAME_SIZE-30] = 0; + }else if( isNT() ){ + char *zMulti; + WCHAR zWidePath[SQLITE_TEMPNAME_SIZE]; + GetTempPathW(SQLITE_TEMPNAME_SIZE-30, zWidePath); + zMulti = unicodeToUtf8(zWidePath); + if( zMulti ){ + strncpy(zTempPath, zMulti, SQLITE_TEMPNAME_SIZE-30); + zTempPath[SQLITE_TEMPNAME_SIZE-30] = 0; + sqliteFree(zMulti); + } + }else{ + GetTempPathA(SQLITE_TEMPNAME_SIZE-30, zTempPath); + } + for(i=strlen(zTempPath); i>0 && zTempPath[i-1]=='\\'; i--){} + zTempPath[i] = 0; + for(;;){ + sprintf(zBuf, "%s\\"TEMP_FILE_PREFIX, zTempPath); + j = strlen(zBuf); + sqlite3Randomness(15, &zBuf[j]); + for(i=0; i<15; i++, j++){ + zBuf[j] = (char)zChars[ ((unsigned char)zBuf[j])%(sizeof(zChars)-1) ]; + } + zBuf[j] = 0; + if( !sqlite3OsFileExists(zBuf) ) break; + } + TRACE2("TEMP FILENAME: %s\n", zBuf); + return SQLITE_OK; +} + +/* +** Close a file. +*/ +static int winClose(OsFile **pId){ + winFile *pFile; + if( pId && (pFile = (winFile*)*pId)!=0 ){ + TRACE2("CLOSE %d\n", pFile->h); + CloseHandle(pFile->h); +#if OS_WINCE + winceDestroyLock(pFile); + if( pFile->zDeleteOnClose ){ + DeleteFileW(pFile->zDeleteOnClose); + sqliteFree(pFile->zDeleteOnClose); + } +#endif + OpenCounter(-1); + sqliteFree(pFile); + *pId = 0; + } + return SQLITE_OK; +} + +/* +** Read data from a file into a buffer. Return SQLITE_OK if all +** bytes were read successfully and SQLITE_IOERR if anything goes +** wrong. +*/ +static int winRead(OsFile *id, void *pBuf, int amt){ + DWORD got; + assert( id!=0 ); + SimulateIOError(SQLITE_IOERR); + TRACE3("READ %d lock=%d\n", ((winFile*)id)->h, ((winFile*)id)->locktype); + if( !ReadFile(((winFile*)id)->h, pBuf, amt, &got, 0) ){ + got = 0; + } + if( got==(DWORD)amt ){ + return SQLITE_OK; + }else{ + return SQLITE_IOERR; + } +} + +/* +** Write data from a buffer into a file. Return SQLITE_OK on success +** or some other error code on failure. +*/ +static int winWrite(OsFile *id, const void *pBuf, int amt){ + int rc = 0; + DWORD wrote; + assert( id!=0 ); + SimulateIOError(SQLITE_IOERR); + SimulateDiskfullError; + TRACE3("WRITE %d lock=%d\n", ((winFile*)id)->h, ((winFile*)id)->locktype); + assert( amt>0 ); + while( amt>0 && (rc = WriteFile(((winFile*)id)->h, pBuf, amt, &wrote, 0))!=0 + && wrote>0 ){ + amt -= wrote; + pBuf = &((char*)pBuf)[wrote]; + } + if( !rc || amt>(int)wrote ){ + return SQLITE_FULL; + } + return SQLITE_OK; +} + +/* +** Some microsoft compilers lack this definition. +*/ +#ifndef INVALID_SET_FILE_POINTER +# define INVALID_SET_FILE_POINTER ((DWORD)-1) +#endif + +/* +** Move the read/write pointer in a file. +*/ +static int winSeek(OsFile *id, i64 offset){ + LONG upperBits = offset>>32; + LONG lowerBits = offset & 0xffffffff; + DWORD rc; + assert( id!=0 ); +#ifdef SQLITE_TEST + if( offset ) SimulateDiskfullError +#endif + SEEK(offset/1024 + 1); + rc = SetFilePointer(((winFile*)id)->h, lowerBits, &upperBits, FILE_BEGIN); + TRACE3("SEEK %d %lld\n", ((winFile*)id)->h, offset); + if( rc==INVALID_SET_FILE_POINTER && GetLastError()!=NO_ERROR ){ + return SQLITE_FULL; + } + return SQLITE_OK; +} + +/* +** Make sure all writes to a particular file are committed to disk. +*/ +static int winSync(OsFile *id, int dataOnly){ + assert( id!=0 ); + TRACE3("SYNC %d lock=%d\n", ((winFile*)id)->h, ((winFile*)id)->locktype); + if( FlushFileBuffers(((winFile*)id)->h) ){ + return SQLITE_OK; + }else{ + return SQLITE_IOERR; + } +} + +/* +** Sync the directory zDirname. This is a no-op on operating systems other +** than UNIX. +*/ +int sqlite3WinSyncDirectory(const char *zDirname){ + SimulateIOError(SQLITE_IOERR); + return SQLITE_OK; +} + +/* +** Truncate an open file to a specified size +*/ +static int winTruncate(OsFile *id, i64 nByte){ + LONG upperBits = nByte>>32; + assert( id!=0 ); + TRACE3("TRUNCATE %d %lld\n", ((winFile*)id)->h, nByte); + SimulateIOError(SQLITE_IOERR); + SetFilePointer(((winFile*)id)->h, nByte, &upperBits, FILE_BEGIN); + SetEndOfFile(((winFile*)id)->h); + return SQLITE_OK; +} + +/* +** Determine the current size of a file in bytes +*/ +static int winFileSize(OsFile *id, i64 *pSize){ + DWORD upperBits, lowerBits; + assert( id!=0 ); + SimulateIOError(SQLITE_IOERR); + lowerBits = GetFileSize(((winFile*)id)->h, &upperBits); + *pSize = (((i64)upperBits)<<32) + lowerBits; + return SQLITE_OK; +} + +/* +** LOCKFILE_FAIL_IMMEDIATELY is undefined on some Windows systems. +*/ +#ifndef LOCKFILE_FAIL_IMMEDIATELY +# define LOCKFILE_FAIL_IMMEDIATELY 1 +#endif + +/* +** Acquire a reader lock. +** Different API routines are called depending on whether or not this +** is Win95 or WinNT. +*/ +static int getReadLock(winFile *id){ + int res; + if( isNT() ){ + OVERLAPPED ovlp; + ovlp.Offset = SHARED_FIRST; + ovlp.OffsetHigh = 0; + ovlp.hEvent = 0; + res = LockFileEx(id->h, LOCKFILE_FAIL_IMMEDIATELY, 0, SHARED_SIZE,0,&ovlp); + }else{ + int lk; + sqlite3Randomness(sizeof(lk), &lk); + id->sharedLockByte = (lk & 0x7fffffff)%(SHARED_SIZE - 1); + res = LockFile(id->h, SHARED_FIRST+id->sharedLockByte, 0, 1, 0); + } + return res; +} + +/* +** Undo a readlock +*/ +static int unlockReadLock(winFile *pFile){ + int res; + if( isNT() ){ + res = UnlockFile(pFile->h, SHARED_FIRST, 0, SHARED_SIZE, 0); + }else{ + res = UnlockFile(pFile->h, SHARED_FIRST + pFile->sharedLockByte, 0, 1, 0); + } + return res; +} + +#ifndef SQLITE_OMIT_PAGER_PRAGMAS +/* +** Check that a given pathname is a directory and is writable +** +*/ +int sqlite3WinIsDirWritable(char *zDirname){ + int fileAttr; + WCHAR *zWide; + if( zDirname==0 ) return 0; + if( !isNT() && strlen(zDirname)>MAX_PATH ) return 0; + zWide = utf8ToUnicode(zDirname); + if( zWide ){ + fileAttr = GetFileAttributesW(zWide); + sqliteFree(zWide); + }else{ +#if OS_WINCE + return 0; +#else + fileAttr = GetFileAttributesA(zDirname); +#endif + } + if( fileAttr == 0xffffffff ) return 0; + if( (fileAttr & FILE_ATTRIBUTE_DIRECTORY) != FILE_ATTRIBUTE_DIRECTORY ){ + return 0; + } + return 1; +} +#endif /* SQLITE_OMIT_PAGER_PRAGMAS */ + +/* +** Lock the file with the lock specified by parameter locktype - one +** of the following: +** +** (1) SHARED_LOCK +** (2) RESERVED_LOCK +** (3) PENDING_LOCK +** (4) EXCLUSIVE_LOCK +** +** Sometimes when requesting one lock state, additional lock states +** are inserted in between. The locking might fail on one of the later +** transitions leaving the lock state different from what it started but +** still short of its goal. The following chart shows the allowed +** transitions and the inserted intermediate states: +** +** UNLOCKED -> SHARED +** SHARED -> RESERVED +** SHARED -> (PENDING) -> EXCLUSIVE +** RESERVED -> (PENDING) -> EXCLUSIVE +** PENDING -> EXCLUSIVE +** +** This routine will only increase a lock. The winUnlock() routine +** erases all locks at once and returns us immediately to locking level 0. +** It is not possible to lower the locking level one step at a time. You +** must go straight to locking level 0. +*/ +static int winLock(OsFile *id, int locktype){ + int rc = SQLITE_OK; /* Return code from subroutines */ + int res = 1; /* Result of a windows lock call */ + int newLocktype; /* Set id->locktype to this value before exiting */ + int gotPendingLock = 0;/* True if we acquired a PENDING lock this time */ + winFile *pFile = (winFile*)id; + + assert( pFile!=0 ); + TRACE5("LOCK %d %d was %d(%d)\n", + pFile->h, locktype, pFile->locktype, pFile->sharedLockByte); + + /* If there is already a lock of this type or more restrictive on the + ** OsFile, do nothing. Don't use the end_lock: exit path, as + ** sqlite3OsEnterMutex() hasn't been called yet. + */ + if( pFile->locktype>=locktype ){ + return SQLITE_OK; + } + + /* Make sure the locking sequence is correct + */ + assert( pFile->locktype!=NO_LOCK || locktype==SHARED_LOCK ); + assert( locktype!=PENDING_LOCK ); + assert( locktype!=RESERVED_LOCK || pFile->locktype==SHARED_LOCK ); + + /* Lock the PENDING_LOCK byte if we need to acquire a PENDING lock or + ** a SHARED lock. If we are acquiring a SHARED lock, the acquisition of + ** the PENDING_LOCK byte is temporary. + */ + newLocktype = pFile->locktype; + if( pFile->locktype==NO_LOCK + || (locktype==EXCLUSIVE_LOCK && pFile->locktype==RESERVED_LOCK) + ){ + int cnt = 3; + while( cnt-->0 && (res = LockFile(pFile->h, PENDING_BYTE, 0, 1, 0))==0 ){ + /* Try 3 times to get the pending lock. The pending lock might be + ** held by another reader process who will release it momentarily. + */ + TRACE2("could not get a PENDING lock. cnt=%d\n", cnt); + Sleep(1); + } + gotPendingLock = res; + } + + /* Acquire a shared lock + */ + if( locktype==SHARED_LOCK && res ){ + assert( pFile->locktype==NO_LOCK ); + res = getReadLock(pFile); + if( res ){ + newLocktype = SHARED_LOCK; + } + } + + /* Acquire a RESERVED lock + */ + if( locktype==RESERVED_LOCK && res ){ + assert( pFile->locktype==SHARED_LOCK ); + res = LockFile(pFile->h, RESERVED_BYTE, 0, 1, 0); + if( res ){ + newLocktype = RESERVED_LOCK; + } + } + + /* Acquire a PENDING lock + */ + if( locktype==EXCLUSIVE_LOCK && res ){ + newLocktype = PENDING_LOCK; + gotPendingLock = 0; + } + + /* Acquire an EXCLUSIVE lock + */ + if( locktype==EXCLUSIVE_LOCK && res ){ + assert( pFile->locktype>=SHARED_LOCK ); + res = unlockReadLock(pFile); + TRACE2("unreadlock = %d\n", res); + res = LockFile(pFile->h, SHARED_FIRST, 0, SHARED_SIZE, 0); + if( res ){ + newLocktype = EXCLUSIVE_LOCK; + }else{ + TRACE2("error-code = %d\n", GetLastError()); + } + } + + /* If we are holding a PENDING lock that ought to be released, then + ** release it now. + */ + if( gotPendingLock && locktype==SHARED_LOCK ){ + UnlockFile(pFile->h, PENDING_BYTE, 0, 1, 0); + } + + /* Update the state of the lock has held in the file descriptor then + ** return the appropriate result code. + */ + if( res ){ + rc = SQLITE_OK; + }else{ + TRACE4("LOCK FAILED %d trying for %d but got %d\n", pFile->h, + locktype, newLocktype); + rc = SQLITE_BUSY; + } + pFile->locktype = newLocktype; + return rc; +} + +/* +** This routine checks if there is a RESERVED lock held on the specified +** file by this or any other process. If such a lock is held, return +** non-zero, otherwise zero. +*/ +static int winCheckReservedLock(OsFile *id){ + int rc; + winFile *pFile = (winFile*)id; + assert( pFile!=0 ); + if( pFile->locktype>=RESERVED_LOCK ){ + rc = 1; + TRACE3("TEST WR-LOCK %d %d (local)\n", pFile->h, rc); + }else{ + rc = LockFile(pFile->h, RESERVED_BYTE, 0, 1, 0); + if( rc ){ + UnlockFile(pFile->h, RESERVED_BYTE, 0, 1, 0); + } + rc = !rc; + TRACE3("TEST WR-LOCK %d %d (remote)\n", pFile->h, rc); + } + return rc; +} + +/* +** Lower the locking level on file descriptor id to locktype. locktype +** must be either NO_LOCK or SHARED_LOCK. +** +** If the locking level of the file descriptor is already at or below +** the requested locking level, this routine is a no-op. +** +** It is not possible for this routine to fail if the second argument +** is NO_LOCK. If the second argument is SHARED_LOCK then this routine +** might return SQLITE_IOERR; +*/ +static int winUnlock(OsFile *id, int locktype){ + int type; + int rc = SQLITE_OK; + winFile *pFile = (winFile*)id; + assert( pFile!=0 ); + assert( locktype<=SHARED_LOCK ); + TRACE5("UNLOCK %d to %d was %d(%d)\n", pFile->h, locktype, + pFile->locktype, pFile->sharedLockByte); + type = pFile->locktype; + if( type>=EXCLUSIVE_LOCK ){ + UnlockFile(pFile->h, SHARED_FIRST, 0, SHARED_SIZE, 0); + if( locktype==SHARED_LOCK && !getReadLock(pFile) ){ + /* This should never happen. We should always be able to + ** reacquire the read lock */ + rc = SQLITE_IOERR; + } + } + if( type>=RESERVED_LOCK ){ + UnlockFile(pFile->h, RESERVED_BYTE, 0, 1, 0); + } + if( locktype==NO_LOCK && type>=SHARED_LOCK ){ + unlockReadLock(pFile); + } + if( type>=PENDING_LOCK ){ + UnlockFile(pFile->h, PENDING_BYTE, 0, 1, 0); + } + pFile->locktype = locktype; + return rc; +} + +/* +** Turn a relative pathname into a full pathname. Return a pointer +** to the full pathname stored in space obtained from sqliteMalloc(). +** The calling function is responsible for freeing this space once it +** is no longer needed. +*/ +char *sqlite3WinFullPathname(const char *zRelative){ + char *zFull; +#if defined(__CYGWIN__) + int nByte; + nByte = strlen(zRelative) + MAX_PATH + 1001; + zFull = sqliteMalloc( nByte ); + if( zFull==0 ) return 0; + if( cygwin_conv_to_full_win32_path(zRelative, zFull) ) return 0; +#elif OS_WINCE + /* WinCE has no concept of a relative pathname, or so I am told. */ + zFull = sqliteStrDup(zRelative); +#else + char *zNotUsed; + WCHAR *zWide; + int nByte; + zWide = utf8ToUnicode(zRelative); + if( zWide ){ + WCHAR *zTemp, *zNotUsedW; + nByte = GetFullPathNameW(zWide, 0, 0, &zNotUsedW) + 1; + zTemp = sqliteMalloc( nByte*sizeof(zTemp[0]) ); + if( zTemp==0 ) return 0; + GetFullPathNameW(zWide, nByte, zTemp, &zNotUsedW); + sqliteFree(zWide); + zFull = unicodeToUtf8(zTemp); + sqliteFree(zTemp); + }else{ + nByte = GetFullPathNameA(zRelative, 0, 0, &zNotUsed) + 1; + zFull = sqliteMalloc( nByte*sizeof(zFull[0]) ); + if( zFull==0 ) return 0; + GetFullPathNameA(zRelative, nByte, zFull, &zNotUsed); + } +#endif + return zFull; +} + +/* +** The fullSync option is meaningless on windows. This is a no-op. +*/ +static void winSetFullSync(OsFile *id, int v){ + return; +} + +/* +** Return the underlying file handle for an OsFile +*/ +static int winFileHandle(OsFile *id){ + return (int)((winFile*)id)->h; +} + +/* +** Return an integer that indices the type of lock currently held +** by this handle. (Used for testing and analysis only.) +*/ +static int winLockState(OsFile *id){ + return ((winFile*)id)->locktype; +} + +/* +** This vector defines all the methods that can operate on an OsFile +** for win32. +*/ +static const IoMethod sqlite3WinIoMethod = { + winClose, + winOpenDirectory, + winRead, + winWrite, + winSeek, + winTruncate, + winSync, + winSetFullSync, + winFileHandle, + winFileSize, + winLock, + winUnlock, + winLockState, + winCheckReservedLock, +}; + +/* +** Allocate memory for an OsFile. Initialize the new OsFile +** to the value given in pInit and return a pointer to the new +** OsFile. If we run out of memory, close the file and return NULL. +*/ +static int allocateWinFile(winFile *pInit, OsFile **pId){ + winFile *pNew; + pNew = sqliteMalloc( sizeof(*pNew) ); + if( pNew==0 ){ + CloseHandle(pInit->h); +#if OS_WINCE + sqliteFree(pInit->zDeleteOnClose); +#endif + *pId = 0; + return SQLITE_NOMEM; + }else{ + *pNew = *pInit; + pNew->pMethod = &sqlite3WinIoMethod; + pNew->locktype = NO_LOCK; + pNew->sharedLockByte = 0; + *pId = (OsFile*)pNew; + OpenCounter(+1); + return SQLITE_OK; + } +} + + +#endif /* SQLITE_OMIT_DISKIO */ +/*************************************************************************** +** Everything above deals with file I/O. Everything that follows deals +** with other miscellanous aspects of the operating system interface +****************************************************************************/ + +/* +** Get information to seed the random number generator. The seed +** is written into the buffer zBuf[256]. The calling function must +** supply a sufficiently large buffer. +*/ +int sqlite3WinRandomSeed(char *zBuf){ + /* We have to initialize zBuf to prevent valgrind from reporting + ** errors. The reports issued by valgrind are incorrect - we would + ** prefer that the randomness be increased by making use of the + ** uninitialized space in zBuf - but valgrind errors tend to worry + ** some users. Rather than argue, it seems easier just to initialize + ** the whole array and silence valgrind, even if that means less randomness + ** in the random seed. + ** + ** When testing, initializing zBuf[] to zero is all we do. That means + ** that we always use the same random number sequence.* This makes the + ** tests repeatable. + */ + memset(zBuf, 0, 256); + GetSystemTime((LPSYSTEMTIME)zBuf); + return SQLITE_OK; +} + +/* +** Sleep for a little while. Return the amount of time slept. +*/ +int sqlite3WinSleep(int ms){ + Sleep(ms); + return ms; +} + +/* +** Static variables used for thread synchronization +*/ +static int inMutex = 0; +#ifdef SQLITE_W32_THREADS + static DWORD mutexOwner; + static CRITICAL_SECTION cs; +#endif + +/* +** The following pair of routines implement mutual exclusion for +** multi-threaded processes. Only a single thread is allowed to +** executed code that is surrounded by EnterMutex() and LeaveMutex(). +** +** SQLite uses only a single Mutex. There is not much critical +** code and what little there is executes quickly and without blocking. +** +** Version 3.3.1 and earlier used a simple mutex. Beginning with +** version 3.3.2, a recursive mutex is required. +*/ +void sqlite3WinEnterMutex(){ +#ifdef SQLITE_W32_THREADS + static int isInit = 0; + while( !isInit ){ + static long lock = 0; + if( InterlockedIncrement(&lock)==1 ){ + InitializeCriticalSection(&cs); + isInit = 1; + }else{ + Sleep(1); + } + } + EnterCriticalSection(&cs); + mutexOwner = GetCurrentThreadId(); +#endif + inMutex++; +} +void sqlite3WinLeaveMutex(){ + assert( inMutex ); + inMutex--; +#ifdef SQLITE_W32_THREADS + assert( mutexOwner==GetCurrentThreadId() ); + LeaveCriticalSection(&cs); +#endif +} + +/* +** Return TRUE if the mutex is currently held. +** +** If the thisThreadOnly parameter is true, return true if and only if the +** calling thread holds the mutex. If the parameter is false, return +** true if any thread holds the mutex. +*/ +int sqlite3WinInMutex(int thisThreadOnly){ +#ifdef SQLITE_W32_THREADS + return inMutex>0 && (thisThreadOnly==0 || mutexOwner==GetCurrentThreadId()); +#else + return inMutex>0; +#endif +} + + +/* +** The following variable, if set to a non-zero value, becomes the result +** returned from sqlite3OsCurrentTime(). This is used for testing. +*/ +#ifdef SQLITE_TEST +int sqlite3_current_time = 0; +#endif + +/* +** Find the current time (in Universal Coordinated Time). Write the +** current time and date as a Julian Day number into *prNow and +** return 0. Return 1 if the time and date cannot be found. +*/ +int sqlite3WinCurrentTime(double *prNow){ + FILETIME ft; + /* FILETIME structure is a 64-bit value representing the number of + 100-nanosecond intervals since January 1, 1601 (= JD 2305813.5). + */ + double now; +#if OS_WINCE + SYSTEMTIME time; + GetSystemTime(&time); + SystemTimeToFileTime(&time,&ft); +#else + GetSystemTimeAsFileTime( &ft ); +#endif + now = ((double)ft.dwHighDateTime) * 4294967296.0; + *prNow = (now + ft.dwLowDateTime)/864000000000.0 + 2305813.5; +#ifdef SQLITE_TEST + if( sqlite3_current_time ){ + *prNow = sqlite3_current_time/86400.0 + 2440587.5; + } +#endif + return 0; +} + +/* +** Remember the number of thread-specific-data blocks allocated. +** Use this to verify that we are not leaking thread-specific-data. +** Ticket #1601 +*/ +#ifdef SQLITE_TEST +int sqlite3_tsd_count = 0; +# define TSD_COUNTER_INCR InterlockedIncrement(&sqlite3_tsd_count) +# define TSD_COUNTER_DECR InterlockedDecrement(&sqlite3_tsd_count) +#else +# define TSD_COUNTER_INCR /* no-op */ +# define TSD_COUNTER_DECR /* no-op */ +#endif + + + +/* +** If called with allocateFlag>1, then return a pointer to thread +** specific data for the current thread. Allocate and zero the +** thread-specific data if it does not already exist necessary. +** +** If called with allocateFlag==0, then check the current thread +** specific data. Return it if it exists. If it does not exist, +** then return NULL. +** +** If called with allocateFlag<0, check to see if the thread specific +** data is allocated and is all zero. If it is then deallocate it. +** Return a pointer to the thread specific data or NULL if it is +** unallocated or gets deallocated. +*/ +ThreadData *sqlite3WinThreadSpecificData(int allocateFlag){ + static int key; + static int keyInit = 0; + static const ThreadData zeroData = {0}; + ThreadData *pTsd; + + if( !keyInit ){ + sqlite3OsEnterMutex(); + if( !keyInit ){ + key = TlsAlloc(); + if( key==0xffffffff ){ + sqlite3OsLeaveMutex(); + return 0; + } + keyInit = 1; + } + sqlite3OsLeaveMutex(); + } + pTsd = TlsGetValue(key); + if( allocateFlag>0 ){ + if( !pTsd ){ + pTsd = sqlite3OsMalloc( sizeof(zeroData) ); + if( pTsd ){ + *pTsd = zeroData; + TlsSetValue(key, pTsd); + TSD_COUNTER_INCR; + } + } + }else if( pTsd!=0 && allocateFlag<0 + && memcmp(pTsd, &zeroData, sizeof(ThreadData))==0 ){ + sqlite3OsFree(pTsd); + TlsSetValue(key, 0); + TSD_COUNTER_DECR; + pTsd = 0; + } + return pTsd; +} +#endif /* OS_WIN */ Added: external/sqlite-source-3.3.4/pager.c ============================================================================== --- (empty file) +++ external/sqlite-source-3.3.4/pager.c Mon Apr 3 07:54:59 2006 @@ -0,0 +1,3806 @@ +/* +** 2001 September 15 +** +** The author disclaims copyright to this source code. In place of +** a legal notice, here is a blessing: +** +** May you do good and not evil. +** May you find forgiveness for yourself and forgive others. +** May you share freely, never taking more than you give. +** +************************************************************************* +** This is the implementation of the page cache subsystem or "pager". +** +** The pager is used to access a database disk file. It implements +** atomic commit and rollback through the use of a journal file that +** is separate from the database file. The pager also implements file +** locking to prevent two processes from writing the same database +** file simultaneously, or one process from reading the database while +** another is writing. +** +** @(#) $Id: pager.c,v 1.258 2006/02/11 01:25:51 drh Exp $ +*/ +#ifndef SQLITE_OMIT_DISKIO +#include "sqliteInt.h" +#include "os.h" +#include "pager.h" +#include +#include + +/* +** Macros for troubleshooting. Normally turned off +*/ +#if 0 +#define TRACE1(X) sqlite3DebugPrintf(X) +#define TRACE2(X,Y) sqlite3DebugPrintf(X,Y) +#define TRACE3(X,Y,Z) sqlite3DebugPrintf(X,Y,Z) +#define TRACE4(X,Y,Z,W) sqlite3DebugPrintf(X,Y,Z,W) +#define TRACE5(X,Y,Z,W,V) sqlite3DebugPrintf(X,Y,Z,W,V) +#else +#define TRACE1(X) +#define TRACE2(X,Y) +#define TRACE3(X,Y,Z) +#define TRACE4(X,Y,Z,W) +#define TRACE5(X,Y,Z,W,V) +#endif + +/* +** The following two macros are used within the TRACEX() macros above +** to print out file-descriptors. +** +** PAGERID() takes a pointer to a Pager struct as it's argument. The +** associated file-descriptor is returned. FILEHANDLEID() takes an OsFile +** struct as it's argument. +*/ +#define PAGERID(p) FILEHANDLEID(&(p)->fd) +#define FILEHANDLEID(fd) (sqlite3OsFileHandle(&fd)) + +/* +** The page cache as a whole is always in one of the following +** states: +** +** PAGER_UNLOCK The page cache is not currently reading or +** writing the database file. There is no +** data held in memory. This is the initial +** state. +** +** PAGER_SHARED The page cache is reading the database. +** Writing is not permitted. There can be +** multiple readers accessing the same database +** file at the same time. +** +** PAGER_RESERVED This process has reserved the database for writing +** but has not yet made any changes. Only one process +** at a time can reserve the database. The original +** database file has not been modified so other +** processes may still be reading the on-disk +** database file. +** +** PAGER_EXCLUSIVE The page cache is writing the database. +** Access is exclusive. No other processes or +** threads can be reading or writing while one +** process is writing. +** +** PAGER_SYNCED The pager moves to this state from PAGER_EXCLUSIVE +** after all dirty pages have been written to the +** database file and the file has been synced to +** disk. All that remains to do is to remove the +** journal file and the transaction will be +** committed. +** +** The page cache comes up in PAGER_UNLOCK. The first time a +** sqlite3pager_get() occurs, the state transitions to PAGER_SHARED. +** After all pages have been released using sqlite_page_unref(), +** the state transitions back to PAGER_UNLOCK. The first time +** that sqlite3pager_write() is called, the state transitions to +** PAGER_RESERVED. (Note that sqlite_page_write() can only be +** called on an outstanding page which means that the pager must +** be in PAGER_SHARED before it transitions to PAGER_RESERVED.) +** The transition to PAGER_EXCLUSIVE occurs when before any changes +** are made to the database file. After an sqlite3pager_rollback() +** or sqlite_pager_commit(), the state goes back to PAGER_SHARED. +*/ +#define PAGER_UNLOCK 0 +#define PAGER_SHARED 1 /* same as SHARED_LOCK */ +#define PAGER_RESERVED 2 /* same as RESERVED_LOCK */ +#define PAGER_EXCLUSIVE 4 /* same as EXCLUSIVE_LOCK */ +#define PAGER_SYNCED 5 + +/* +** If the SQLITE_BUSY_RESERVED_LOCK macro is set to true at compile-time, +** then failed attempts to get a reserved lock will invoke the busy callback. +** This is off by default. To see why, consider the following scenario: +** +** Suppose thread A already has a shared lock and wants a reserved lock. +** Thread B already has a reserved lock and wants an exclusive lock. If +** both threads are using their busy callbacks, it might be a long time +** be for one of the threads give up and allows the other to proceed. +** But if the thread trying to get the reserved lock gives up quickly +** (if it never invokes its busy callback) then the contention will be +** resolved quickly. +*/ +#ifndef SQLITE_BUSY_RESERVED_LOCK +# define SQLITE_BUSY_RESERVED_LOCK 0 +#endif + +/* +** This macro rounds values up so that if the value is an address it +** is guaranteed to be an address that is aligned to an 8-byte boundary. +*/ +#define FORCE_ALIGNMENT(X) (((X)+7)&~7) + +/* +** Each in-memory image of a page begins with the following header. +** This header is only visible to this pager module. The client +** code that calls pager sees only the data that follows the header. +** +** Client code should call sqlite3pager_write() on a page prior to making +** any modifications to that page. The first time sqlite3pager_write() +** is called, the original page contents are written into the rollback +** journal and PgHdr.inJournal and PgHdr.needSync are set. Later, once +** the journal page has made it onto the disk surface, PgHdr.needSync +** is cleared. The modified page cannot be written back into the original +** database file until the journal pages has been synced to disk and the +** PgHdr.needSync has been cleared. +** +** The PgHdr.dirty flag is set when sqlite3pager_write() is called and +** is cleared again when the page content is written back to the original +** database file. +*/ +typedef struct PgHdr PgHdr; +struct PgHdr { + Pager *pPager; /* The pager to which this page belongs */ + Pgno pgno; /* The page number for this page */ + PgHdr *pNextHash, *pPrevHash; /* Hash collision chain for PgHdr.pgno */ + PgHdr *pNextFree, *pPrevFree; /* Freelist of pages where nRef==0 */ + PgHdr *pNextAll; /* A list of all pages */ + PgHdr *pNextStmt, *pPrevStmt; /* List of pages in the statement journal */ + u8 inJournal; /* TRUE if has been written to journal */ + u8 inStmt; /* TRUE if in the statement subjournal */ + u8 dirty; /* TRUE if we need to write back changes */ + u8 needSync; /* Sync journal before writing this page */ + u8 alwaysRollback; /* Disable dont_rollback() for this page */ + short int nRef; /* Number of users of this page */ + PgHdr *pDirty; /* Dirty pages sorted by PgHdr.pgno */ +#ifdef SQLITE_CHECK_PAGES + u32 pageHash; +#endif + /* pPager->pageSize bytes of page data follow this header */ + /* Pager.nExtra bytes of local data follow the page data */ +}; + +/* +** For an in-memory only database, some extra information is recorded about +** each page so that changes can be rolled back. (Journal files are not +** used for in-memory databases.) The following information is added to +** the end of every EXTRA block for in-memory databases. +** +** This information could have been added directly to the PgHdr structure. +** But then it would take up an extra 8 bytes of storage on every PgHdr +** even for disk-based databases. Splitting it out saves 8 bytes. This +** is only a savings of 0.8% but those percentages add up. +*/ +typedef struct PgHistory PgHistory; +struct PgHistory { + u8 *pOrig; /* Original page text. Restore to this on a full rollback */ + u8 *pStmt; /* Text as it was at the beginning of the current statement */ +}; + +/* +** A macro used for invoking the codec if there is one +*/ +#ifdef SQLITE_HAS_CODEC +# define CODEC(P,D,N,X) if( P->xCodec ){ P->xCodec(P->pCodecArg,D,N,X); } +#else +# define CODEC(P,D,N,X) +#endif + +/* +** Convert a pointer to a PgHdr into a pointer to its data +** and back again. +*/ +#define PGHDR_TO_DATA(P) ((void*)(&(P)[1])) +#define DATA_TO_PGHDR(D) (&((PgHdr*)(D))[-1]) +#define PGHDR_TO_EXTRA(G,P) ((void*)&((char*)(&(G)[1]))[(P)->pageSize]) +#define PGHDR_TO_HIST(P,PGR) \ + ((PgHistory*)&((char*)(&(P)[1]))[(PGR)->pageSize+(PGR)->nExtra]) + +/* +** How big to make the hash table used for locating in-memory pages +** by page number. This macro looks a little silly, but is evaluated +** at compile-time, not run-time (at least for gcc this is true). +*/ +#define N_PG_HASH (\ + (MAX_PAGES>1024)?2048: \ + (MAX_PAGES>512)?1024: \ + (MAX_PAGES>256)?512: \ + (MAX_PAGES>128)?256: \ + (MAX_PAGES>64)?128:64 \ +) + +/* +** Hash a page number +*/ +#define pager_hash(PN) ((PN)&(N_PG_HASH-1)) + +/* +** A open page cache is an instance of the following structure. +** +** Pager.errCode may be set to SQLITE_IOERR, SQLITE_CORRUPT, SQLITE_PROTOCOL +** or SQLITE_FULL. Once one of the first three errors occurs, it persists +** and is returned as the result of every major pager API call. The +** SQLITE_FULL return code is slightly different. It persists only until the +** next successful rollback is performed on the pager cache. Also, +** SQLITE_FULL does not affect the sqlite3pager_get() and sqlite3pager_lookup() +** APIs, they may still be used successfully. +*/ +struct Pager { + u8 journalOpen; /* True if journal file descriptors is valid */ + u8 journalStarted; /* True if header of journal is synced */ + u8 useJournal; /* Use a rollback journal on this file */ + u8 noReadlock; /* Do not bother to obtain readlocks */ + u8 stmtOpen; /* True if the statement subjournal is open */ + u8 stmtInUse; /* True we are in a statement subtransaction */ + u8 stmtAutoopen; /* Open stmt journal when main journal is opened*/ + u8 noSync; /* Do not sync the journal if true */ + u8 fullSync; /* Do extra syncs of the journal for robustness */ + u8 full_fsync; /* Use F_FULLFSYNC when available */ + u8 state; /* PAGER_UNLOCK, _SHARED, _RESERVED, etc. */ + u8 errCode; /* One of several kinds of errors */ + u8 tempFile; /* zFilename is a temporary file */ + u8 readOnly; /* True for a read-only database */ + u8 needSync; /* True if an fsync() is needed on the journal */ + u8 dirtyCache; /* True if cached pages have changed */ + u8 alwaysRollback; /* Disable dont_rollback() for all pages */ + u8 memDb; /* True to inhibit all file I/O */ + u8 setMaster; /* True if a m-j name has been written to jrnl */ + int dbSize; /* Number of pages in the file */ + int origDbSize; /* dbSize before the current change */ + int stmtSize; /* Size of database (in pages) at stmt_begin() */ + int nRec; /* Number of pages written to the journal */ + u32 cksumInit; /* Quasi-random value added to every checksum */ + int stmtNRec; /* Number of records in stmt subjournal */ + int nExtra; /* Add this many bytes to each in-memory page */ + int pageSize; /* Number of bytes in a page */ + int nPage; /* Total number of in-memory pages */ + int nMaxPage; /* High water mark of nPage */ + int nRef; /* Number of in-memory pages with PgHdr.nRef>0 */ + int mxPage; /* Maximum number of pages to hold in cache */ + u8 *aInJournal; /* One bit for each page in the database file */ + u8 *aInStmt; /* One bit for each page in the database */ + char *zFilename; /* Name of the database file */ + char *zJournal; /* Name of the journal file */ + char *zDirectory; /* Directory hold database and journal files */ + OsFile *fd, *jfd; /* File descriptors for database and journal */ + OsFile *stfd; /* File descriptor for the statement subjournal*/ + BusyHandler *pBusyHandler; /* Pointer to sqlite.busyHandler */ + PgHdr *pFirst, *pLast; /* List of free pages */ + PgHdr *pFirstSynced; /* First free page with PgHdr.needSync==0 */ + PgHdr *pAll; /* List of all pages */ + PgHdr *pStmt; /* List of pages in the statement subjournal */ + i64 journalOff; /* Current byte offset in the journal file */ + i64 journalHdr; /* Byte offset to previous journal header */ + i64 stmtHdrOff; /* First journal header written this statement */ + i64 stmtCksum; /* cksumInit when statement was started */ + i64 stmtJSize; /* Size of journal at stmt_begin() */ + int sectorSize; /* Assumed sector size during rollback */ +#ifdef SQLITE_TEST + int nHit, nMiss, nOvfl; /* Cache hits, missing, and LRU overflows */ + int nRead,nWrite; /* Database pages read/written */ +#endif + void (*xDestructor)(void*,int); /* Call this routine when freeing pages */ + void (*xReiniter)(void*,int); /* Call this routine when reloading pages */ + void (*xCodec)(void*,void*,Pgno,int); /* Routine for en/decoding data */ + void *pCodecArg; /* First argument to xCodec() */ + PgHdr *aHash[N_PG_HASH]; /* Hash table to map page number to PgHdr */ +#ifdef SQLITE_ENABLE_MEMORY_MANAGEMENT + Pager *pNext; /* Linked list of pagers in this thread */ +#endif +}; + +/* +** If SQLITE_TEST is defined then increment the variable given in +** the argument +*/ +#ifdef SQLITE_TEST +# define TEST_INCR(x) x++ +#else +# define TEST_INCR(x) +#endif + +/* +** Journal files begin with the following magic string. The data +** was obtained from /dev/random. It is used only as a sanity check. +** +** Since version 2.8.0, the journal format contains additional sanity +** checking information. If the power fails while the journal is begin +** written, semi-random garbage data might appear in the journal +** file after power is restored. If an attempt is then made +** to roll the journal back, the database could be corrupted. The additional +** sanity checking data is an attempt to discover the garbage in the +** journal and ignore it. +** +** The sanity checking information for the new journal format consists +** of a 32-bit checksum on each page of data. The checksum covers both +** the page number and the pPager->pageSize bytes of data for the page. +** This cksum is initialized to a 32-bit random value that appears in the +** journal file right after the header. The random initializer is important, +** because garbage data that appears at the end of a journal is likely +** data that was once in other files that have now been deleted. If the +** garbage data came from an obsolete journal file, the checksums might +** be correct. But by initializing the checksum to random value which +** is different for every journal, we minimize that risk. +*/ +static const unsigned char aJournalMagic[] = { + 0xd9, 0xd5, 0x05, 0xf9, 0x20, 0xa1, 0x63, 0xd7, +}; + +/* +** The size of the header and of each page in the journal is determined +** by the following macros. +*/ +#define JOURNAL_PG_SZ(pPager) ((pPager->pageSize) + 8) + +/* +** The journal header size for this pager. In the future, this could be +** set to some value read from the disk controller. The important +** characteristic is that it is the same size as a disk sector. +*/ +#define JOURNAL_HDR_SZ(pPager) (pPager->sectorSize) + +/* +** The macro MEMDB is true if we are dealing with an in-memory database. +** We do this as a macro so that if the SQLITE_OMIT_MEMORYDB macro is set, +** the value of MEMDB will be a constant and the compiler will optimize +** out code that would never execute. +*/ +#ifdef SQLITE_OMIT_MEMORYDB +# define MEMDB 0 +#else +# define MEMDB pPager->memDb +#endif + +/* +** The default size of a disk sector +*/ +#define PAGER_SECTOR_SIZE 512 + +/* +** Page number PAGER_MJ_PGNO is never used in an SQLite database (it is +** reserved for working around a windows/posix incompatibility). It is +** used in the journal to signify that the remainder of the journal file +** is devoted to storing a master journal name - there are no more pages to +** roll back. See comments for function writeMasterJournal() for details. +*/ +/* #define PAGER_MJ_PGNO(x) (PENDING_BYTE/((x)->pageSize)) */ +#define PAGER_MJ_PGNO(x) ((PENDING_BYTE/((x)->pageSize))+1) + +/* +** The maximum legal page number is (2^31 - 1). +*/ +#define PAGER_MAX_PGNO 2147483647 + +/* +** Enable reference count tracking (for debugging) here: +*/ +#ifdef SQLITE_DEBUG + int pager3_refinfo_enable = 0; + static void pager_refinfo(PgHdr *p){ + static int cnt = 0; + if( !pager3_refinfo_enable ) return; + sqlite3DebugPrintf( + "REFCNT: %4d addr=%p nRef=%d\n", + p->pgno, PGHDR_TO_DATA(p), p->nRef + ); + cnt++; /* Something to set a breakpoint on */ + } +# define REFINFO(X) pager_refinfo(X) +#else +# define REFINFO(X) +#endif + +/* +** Read a 32-bit integer from the given file descriptor. Store the integer +** that is read in *pRes. Return SQLITE_OK if everything worked, or an +** error code is something goes wrong. +** +** All values are stored on disk as big-endian. +*/ +static int read32bits(OsFile *fd, u32 *pRes){ + unsigned char ac[4]; + int rc = sqlite3OsRead(fd, ac, sizeof(ac)); + if( rc==SQLITE_OK ){ + *pRes = (ac[0]<<24) | (ac[1]<<16) | (ac[2]<<8) | ac[3]; + } + return rc; +} + +/* +** Write a 32-bit integer into a string buffer in big-endian byte order. +*/ +static void put32bits(char *ac, u32 val){ + ac[0] = (val>>24) & 0xff; + ac[1] = (val>>16) & 0xff; + ac[2] = (val>>8) & 0xff; + ac[3] = val & 0xff; +} + +/* +** Write a 32-bit integer into the given file descriptor. Return SQLITE_OK +** on success or an error code is something goes wrong. +*/ +static int write32bits(OsFile *fd, u32 val){ + char ac[4]; + put32bits(ac, val); + return sqlite3OsWrite(fd, ac, 4); +} + +/* +** Write the 32-bit integer 'val' into the page identified by page header +** 'p' at offset 'offset'. +*/ +static void store32bits(u32 val, PgHdr *p, int offset){ + char *ac; + ac = &((char*)PGHDR_TO_DATA(p))[offset]; + put32bits(ac, val); +} + +/* +** Read a 32-bit integer at offset 'offset' from the page identified by +** page header 'p'. +*/ +static u32 retrieve32bits(PgHdr *p, int offset){ + unsigned char *ac; + ac = &((unsigned char*)PGHDR_TO_DATA(p))[offset]; + return (ac[0]<<24) | (ac[1]<<16) | (ac[2]<<8) | ac[3]; +} + + +/* +** This function should be called when an error occurs within the pager +** code. The first argument is a pointer to the pager structure, the +** second the error-code about to be returned by a pager API function. +** The value returned is a copy of the second argument to this function. +** +** If the second argument is SQLITE_IOERR, SQLITE_CORRUPT or SQLITE_PROTOCOL, +** the error becomes persistent. All subsequent API calls on this Pager +** will immediately return the same error code. +*/ +static int pager_error(Pager *pPager, int rc){ + assert( pPager->errCode==SQLITE_FULL || pPager->errCode==SQLITE_OK ); + if( + rc==SQLITE_FULL || + rc==SQLITE_IOERR || + rc==SQLITE_CORRUPT || + rc==SQLITE_PROTOCOL + ){ + pPager->errCode = rc; + } + return rc; +} + +#ifdef SQLITE_CHECK_PAGES +/* +** Return a 32-bit hash of the page data for pPage. +*/ +static u32 pager_pagehash(PgHdr *pPage){ + u32 hash = 0; + int i; + unsigned char *pData = (unsigned char *)PGHDR_TO_DATA(pPage); + for(i=0; ipPager->pageSize; i++){ + hash = (hash+i)^pData[i]; + } + return hash; +} + +/* +** The CHECK_PAGE macro takes a PgHdr* as an argument. If SQLITE_CHECK_PAGES +** is defined, and NDEBUG is not defined, an assert() statement checks +** that the page is either dirty or still matches the calculated page-hash. +*/ +#define CHECK_PAGE(x) checkPage(x) +static void checkPage(PgHdr *pPg){ + Pager *pPager = pPg->pPager; + assert( !pPg->pageHash || pPager->errCode || MEMDB || pPg->dirty || + pPg->pageHash==pager_pagehash(pPg) ); +} + +#else +#define CHECK_PAGE(x) +#endif + +/* +** When this is called the journal file for pager pPager must be open. +** The master journal file name is read from the end of the file and +** written into memory obtained from sqliteMalloc(). *pzMaster is +** set to point at the memory and SQLITE_OK returned. The caller must +** sqliteFree() *pzMaster. +** +** If no master journal file name is present *pzMaster is set to 0 and +** SQLITE_OK returned. +*/ +static int readMasterJournal(OsFile *pJrnl, char **pzMaster){ + int rc; + u32 len; + i64 szJ; + u32 cksum; + int i; + unsigned char aMagic[8]; /* A buffer to hold the magic header */ + + *pzMaster = 0; + + rc = sqlite3OsFileSize(pJrnl, &szJ); + if( rc!=SQLITE_OK || szJ<16 ) return rc; + + rc = sqlite3OsSeek(pJrnl, szJ-16); + if( rc!=SQLITE_OK ) return rc; + + rc = read32bits(pJrnl, &len); + if( rc!=SQLITE_OK ) return rc; + + rc = read32bits(pJrnl, &cksum); + if( rc!=SQLITE_OK ) return rc; + + rc = sqlite3OsRead(pJrnl, aMagic, 8); + if( rc!=SQLITE_OK || memcmp(aMagic, aJournalMagic, 8) ) return rc; + + rc = sqlite3OsSeek(pJrnl, szJ-16-len); + if( rc!=SQLITE_OK ) return rc; + + *pzMaster = (char *)sqliteMalloc(len+1); + if( !*pzMaster ){ + return SQLITE_NOMEM; + } + rc = sqlite3OsRead(pJrnl, *pzMaster, len); + if( rc!=SQLITE_OK ){ + sqliteFree(*pzMaster); + *pzMaster = 0; + return rc; + } + + /* See if the checksum matches the master journal name */ + for(i=0; ijournalOff; + if( c ){ + offset = ((c-1)/JOURNAL_HDR_SZ(pPager) + 1) * JOURNAL_HDR_SZ(pPager); + } + assert( offset%JOURNAL_HDR_SZ(pPager)==0 ); + assert( offset>=c ); + assert( (offset-c)journalOff = offset; + return sqlite3OsSeek(pPager->jfd, pPager->journalOff); +} + +/* +** The journal file must be open when this routine is called. A journal +** header (JOURNAL_HDR_SZ bytes) is written into the journal file at the +** current location. +** +** The format for the journal header is as follows: +** - 8 bytes: Magic identifying journal format. +** - 4 bytes: Number of records in journal, or -1 no-sync mode is on. +** - 4 bytes: Random number used for page hash. +** - 4 bytes: Initial database page count. +** - 4 bytes: Sector size used by the process that wrote this journal. +** +** Followed by (JOURNAL_HDR_SZ - 24) bytes of unused space. +*/ +static int writeJournalHdr(Pager *pPager){ + char zHeader[sizeof(aJournalMagic)+16]; + + int rc = seekJournalHdr(pPager); + if( rc ) return rc; + + pPager->journalHdr = pPager->journalOff; + if( pPager->stmtHdrOff==0 ){ + pPager->stmtHdrOff = pPager->journalHdr; + } + pPager->journalOff += JOURNAL_HDR_SZ(pPager); + + /* FIX ME: + ** + ** Possibly for a pager not in no-sync mode, the journal magic should not + ** be written until nRec is filled in as part of next syncJournal(). + ** + ** Actually maybe the whole journal header should be delayed until that + ** point. Think about this. + */ + memcpy(zHeader, aJournalMagic, sizeof(aJournalMagic)); + /* The nRec Field. 0xFFFFFFFF for no-sync journals. */ + put32bits(&zHeader[sizeof(aJournalMagic)], pPager->noSync ? 0xffffffff : 0); + /* The random check-hash initialiser */ + sqlite3Randomness(sizeof(pPager->cksumInit), &pPager->cksumInit); + put32bits(&zHeader[sizeof(aJournalMagic)+4], pPager->cksumInit); + /* The initial database size */ + put32bits(&zHeader[sizeof(aJournalMagic)+8], pPager->dbSize); + /* The assumed sector size for this process */ + put32bits(&zHeader[sizeof(aJournalMagic)+12], pPager->sectorSize); + rc = sqlite3OsWrite(pPager->jfd, zHeader, sizeof(zHeader)); + + /* The journal header has been written successfully. Seek the journal + ** file descriptor to the end of the journal header sector. + */ + if( rc==SQLITE_OK ){ + rc = sqlite3OsSeek(pPager->jfd, pPager->journalOff-1); + if( rc==SQLITE_OK ){ + rc = sqlite3OsWrite(pPager->jfd, "\000", 1); + } + } + return rc; +} + +/* +** The journal file must be open when this is called. A journal header file +** (JOURNAL_HDR_SZ bytes) is read from the current location in the journal +** file. See comments above function writeJournalHdr() for a description of +** the journal header format. +** +** If the header is read successfully, *nRec is set to the number of +** page records following this header and *dbSize is set to the size of the +** database before the transaction began, in pages. Also, pPager->cksumInit +** is set to the value read from the journal header. SQLITE_OK is returned +** in this case. +** +** If the journal header file appears to be corrupted, SQLITE_DONE is +** returned and *nRec and *dbSize are not set. If JOURNAL_HDR_SZ bytes +** cannot be read from the journal file an error code is returned. +*/ +static int readJournalHdr( + Pager *pPager, + i64 journalSize, + u32 *pNRec, + u32 *pDbSize +){ + int rc; + unsigned char aMagic[8]; /* A buffer to hold the magic header */ + + rc = seekJournalHdr(pPager); + if( rc ) return rc; + + if( pPager->journalOff+JOURNAL_HDR_SZ(pPager) > journalSize ){ + return SQLITE_DONE; + } + + rc = sqlite3OsRead(pPager->jfd, aMagic, sizeof(aMagic)); + if( rc ) return rc; + + if( memcmp(aMagic, aJournalMagic, sizeof(aMagic))!=0 ){ + return SQLITE_DONE; + } + + rc = read32bits(pPager->jfd, pNRec); + if( rc ) return rc; + + rc = read32bits(pPager->jfd, &pPager->cksumInit); + if( rc ) return rc; + + rc = read32bits(pPager->jfd, pDbSize); + if( rc ) return rc; + + /* Update the assumed sector-size to match the value used by + ** the process that created this journal. If this journal was + ** created by a process other than this one, then this routine + ** is being called from within pager_playback(). The local value + ** of Pager.sectorSize is restored at the end of that routine. + */ + rc = read32bits(pPager->jfd, (u32 *)&pPager->sectorSize); + if( rc ) return rc; + + pPager->journalOff += JOURNAL_HDR_SZ(pPager); + rc = sqlite3OsSeek(pPager->jfd, pPager->journalOff); + return rc; +} + + +/* +** Write the supplied master journal name into the journal file for pager +** pPager at the current location. The master journal name must be the last +** thing written to a journal file. If the pager is in full-sync mode, the +** journal file descriptor is advanced to the next sector boundary before +** anything is written. The format is: +** +** + 4 bytes: PAGER_MJ_PGNO. +** + N bytes: length of master journal name. +** + 4 bytes: N +** + 4 bytes: Master journal name checksum. +** + 8 bytes: aJournalMagic[]. +** +** The master journal page checksum is the sum of the bytes in the master +** journal name. +** +** If zMaster is a NULL pointer (occurs for a single database transaction), +** this call is a no-op. +*/ +static int writeMasterJournal(Pager *pPager, const char *zMaster){ + int rc; + int len; + int i; + u32 cksum = 0; + char zBuf[sizeof(aJournalMagic)+2*4]; + + if( !zMaster || pPager->setMaster) return SQLITE_OK; + pPager->setMaster = 1; + + len = strlen(zMaster); + for(i=0; ifullSync ){ + rc = seekJournalHdr(pPager); + if( rc!=SQLITE_OK ) return rc; + } + pPager->journalOff += (len+20); + + rc = write32bits(pPager->jfd, PAGER_MJ_PGNO(pPager)); + if( rc!=SQLITE_OK ) return rc; + + rc = sqlite3OsWrite(pPager->jfd, zMaster, len); + if( rc!=SQLITE_OK ) return rc; + + put32bits(zBuf, len); + put32bits(&zBuf[4], cksum); + memcpy(&zBuf[8], aJournalMagic, sizeof(aJournalMagic)); + rc = sqlite3OsWrite(pPager->jfd, zBuf, 8+sizeof(aJournalMagic)); + pPager->needSync = !pPager->noSync; + return rc; +} + +/* +** Add or remove a page from the list of all pages that are in the +** statement journal. +** +** The Pager keeps a separate list of pages that are currently in +** the statement journal. This helps the sqlite3pager_stmt_commit() +** routine run MUCH faster for the common case where there are many +** pages in memory but only a few are in the statement journal. +*/ +static void page_add_to_stmt_list(PgHdr *pPg){ + Pager *pPager = pPg->pPager; + if( pPg->inStmt ) return; + assert( pPg->pPrevStmt==0 && pPg->pNextStmt==0 ); + pPg->pPrevStmt = 0; + if( pPager->pStmt ){ + pPager->pStmt->pPrevStmt = pPg; + } + pPg->pNextStmt = pPager->pStmt; + pPager->pStmt = pPg; + pPg->inStmt = 1; +} +static void page_remove_from_stmt_list(PgHdr *pPg){ + if( !pPg->inStmt ) return; + if( pPg->pPrevStmt ){ + assert( pPg->pPrevStmt->pNextStmt==pPg ); + pPg->pPrevStmt->pNextStmt = pPg->pNextStmt; + }else{ + assert( pPg->pPager->pStmt==pPg ); + pPg->pPager->pStmt = pPg->pNextStmt; + } + if( pPg->pNextStmt ){ + assert( pPg->pNextStmt->pPrevStmt==pPg ); + pPg->pNextStmt->pPrevStmt = pPg->pPrevStmt; + } + pPg->pNextStmt = 0; + pPg->pPrevStmt = 0; + pPg->inStmt = 0; +} + +/* +** Find a page in the hash table given its page number. Return +** a pointer to the page or NULL if not found. +*/ +static PgHdr *pager_lookup(Pager *pPager, Pgno pgno){ + PgHdr *p = pPager->aHash[pager_hash(pgno)]; + while( p && p->pgno!=pgno ){ + p = p->pNextHash; + } + return p; +} + +/* +** Unlock the database and clear the in-memory cache. This routine +** sets the state of the pager back to what it was when it was first +** opened. Any outstanding pages are invalidated and subsequent attempts +** to access those pages will likely result in a coredump. +*/ +static void pager_reset(Pager *pPager){ + PgHdr *pPg, *pNext; + if( pPager->errCode ) return; + for(pPg=pPager->pAll; pPg; pPg=pNext){ + pNext = pPg->pNextAll; + sqliteFree(pPg); + } + pPager->pFirst = 0; + pPager->pFirstSynced = 0; + pPager->pLast = 0; + pPager->pAll = 0; + memset(pPager->aHash, 0, sizeof(pPager->aHash)); + pPager->nPage = 0; + if( pPager->state>=PAGER_RESERVED ){ + sqlite3pager_rollback(pPager); + } + sqlite3OsUnlock(pPager->fd, NO_LOCK); + pPager->state = PAGER_UNLOCK; + pPager->dbSize = -1; + pPager->nRef = 0; + assert( pPager->journalOpen==0 ); +} + +/* +** When this routine is called, the pager has the journal file open and +** a RESERVED or EXCLUSIVE lock on the database. This routine releases +** the database lock and acquires a SHARED lock in its place. The journal +** file is deleted and closed. +** +** TODO: Consider keeping the journal file open for temporary databases. +** This might give a performance improvement on windows where opening +** a file is an expensive operation. +*/ +static int pager_unwritelock(Pager *pPager){ + PgHdr *pPg; + int rc; + assert( !MEMDB ); + if( pPager->statestmtOpen ){ + sqlite3OsClose(&pPager->stfd); + pPager->stmtOpen = 0; + } + if( pPager->journalOpen ){ + sqlite3OsClose(&pPager->jfd); + pPager->journalOpen = 0; + sqlite3OsDelete(pPager->zJournal); + sqliteFree( pPager->aInJournal ); + pPager->aInJournal = 0; + for(pPg=pPager->pAll; pPg; pPg=pPg->pNextAll){ + pPg->inJournal = 0; + pPg->dirty = 0; + pPg->needSync = 0; +#ifdef SQLITE_CHECK_PAGES + pPg->pageHash = pager_pagehash(pPg); +#endif + } + pPager->dirtyCache = 0; + pPager->nRec = 0; + }else{ + assert( pPager->aInJournal==0 ); + assert( pPager->dirtyCache==0 || pPager->useJournal==0 ); + } + rc = sqlite3OsUnlock(pPager->fd, SHARED_LOCK); + pPager->state = PAGER_SHARED; + pPager->origDbSize = 0; + pPager->setMaster = 0; + pPager->needSync = 0; + pPager->pFirstSynced = pPager->pFirst; + return rc; +} + +/* +** Compute and return a checksum for the page of data. +** +** This is not a real checksum. It is really just the sum of the +** random initial value and the page number. We experimented with +** a checksum of the entire data, but that was found to be too slow. +** +** Note that the page number is stored at the beginning of data and +** the checksum is stored at the end. This is important. If journal +** corruption occurs due to a power failure, the most likely scenario +** is that one end or the other of the record will be changed. It is +** much less likely that the two ends of the journal record will be +** correct and the middle be corrupt. Thus, this "checksum" scheme, +** though fast and simple, catches the mostly likely kind of corruption. +** +** FIX ME: Consider adding every 200th (or so) byte of the data to the +** checksum. That way if a single page spans 3 or more disk sectors and +** only the middle sector is corrupt, we will still have a reasonable +** chance of failing the checksum and thus detecting the problem. +*/ +static u32 pager_cksum(Pager *pPager, Pgno pgno, const u8 *aData){ + u32 cksum = pPager->cksumInit; + int i = pPager->pageSize-200; + while( i>0 ){ + cksum += aData[i]; + i -= 200; + } + return cksum; +} + +/* +** Read a single page from the journal file opened on file descriptor +** jfd. Playback this one page. +** +** If useCksum==0 it means this journal does not use checksums. Checksums +** are not used in statement journals because statement journals do not +** need to survive power failures. +*/ +static int pager_playback_one_page(Pager *pPager, OsFile *jfd, int useCksum){ + int rc; + PgHdr *pPg; /* An existing page in the cache */ + Pgno pgno; /* The page number of a page in journal */ + u32 cksum; /* Checksum used for sanity checking */ + u8 aData[SQLITE_MAX_PAGE_SIZE]; /* Temp storage for a page */ + + /* useCksum should be true for the main journal and false for + ** statement journals. Verify that this is always the case + */ + assert( jfd == (useCksum ? pPager->jfd : pPager->stfd) ); + + + rc = read32bits(jfd, &pgno); + if( rc!=SQLITE_OK ) return rc; + rc = sqlite3OsRead(jfd, &aData, pPager->pageSize); + if( rc!=SQLITE_OK ) return rc; + pPager->journalOff += pPager->pageSize + 4; + + /* Sanity checking on the page. This is more important that I originally + ** thought. If a power failure occurs while the journal is being written, + ** it could cause invalid data to be written into the journal. We need to + ** detect this invalid data (with high probability) and ignore it. + */ + if( pgno==0 || pgno==PAGER_MJ_PGNO(pPager) ){ + return SQLITE_DONE; + } + if( pgno>(unsigned)pPager->dbSize ){ + return SQLITE_OK; + } + if( useCksum ){ + rc = read32bits(jfd, &cksum); + if( rc ) return rc; + pPager->journalOff += 4; + if( pager_cksum(pPager, pgno, aData)!=cksum ){ + return SQLITE_DONE; + } + } + + assert( pPager->state==PAGER_RESERVED || pPager->state>=PAGER_EXCLUSIVE ); + + /* If the pager is in RESERVED state, then there must be a copy of this + ** page in the pager cache. In this case just update the pager cache, + ** not the database file. The page is left marked dirty in this case. + ** + ** If in EXCLUSIVE state, then we update the pager cache if it exists + ** and the main file. The page is then marked not dirty. + ** + ** Ticket #1171: The statement journal might contain page content that is + ** different from the page content at the start of the transaction. + ** This occurs when a page is changed prior to the start of a statement + ** then changed again within the statement. When rolling back such a + ** statement we must not write to the original database unless we know + ** for certain that original page contents are in the main rollback + ** journal. Otherwise, if a full ROLLBACK occurs after the statement + ** rollback the full ROLLBACK will not restore the page to its original + ** content. Two conditions must be met before writing to the database + ** files. (1) the database must be locked. (2) we know that the original + ** page content is in the main journal either because the page is not in + ** cache or else it is marked as needSync==0. + */ + pPg = pager_lookup(pPager, pgno); + assert( pPager->state>=PAGER_EXCLUSIVE || pPg!=0 ); + TRACE3("PLAYBACK %d page %d\n", PAGERID(pPager), pgno); + if( pPager->state>=PAGER_EXCLUSIVE && (pPg==0 || pPg->needSync==0) ){ + rc = sqlite3OsSeek(pPager->fd, (pgno-1)*(i64)pPager->pageSize); + if( rc==SQLITE_OK ){ + rc = sqlite3OsWrite(pPager->fd, aData, pPager->pageSize); + } + if( pPg ) pPg->dirty = 0; + } + if( pPg ){ + /* No page should ever be explicitly rolled back that is in use, except + ** for page 1 which is held in use in order to keep the lock on the + ** database active. However such a page may be rolled back as a result + ** of an internal error resulting in an automatic call to + ** sqlite3pager_rollback(). + */ + void *pData; + /* assert( pPg->nRef==0 || pPg->pgno==1 ); */ + pData = PGHDR_TO_DATA(pPg); + memcpy(pData, aData, pPager->pageSize); + if( pPager->xDestructor ){ /*** FIX ME: Should this be xReinit? ***/ + pPager->xDestructor(pData, pPager->pageSize); + } +#ifdef SQLITE_CHECK_PAGES + pPg->pageHash = pager_pagehash(pPg); +#endif + CODEC(pPager, pData, pPg->pgno, 3); + } + return rc; +} + +/* +** Parameter zMaster is the name of a master journal file. A single journal +** file that referred to the master journal file has just been rolled back. +** This routine checks if it is possible to delete the master journal file, +** and does so if it is. +** +** The master journal file contains the names of all child journals. +** To tell if a master journal can be deleted, check to each of the +** children. If all children are either missing or do not refer to +** a different master journal, then this master journal can be deleted. +*/ +static int pager_delmaster(const char *zMaster){ + int rc; + int master_open = 0; + OsFile *master = 0; + char *zMasterJournal = 0; /* Contents of master journal file */ + i64 nMasterJournal; /* Size of master journal file */ + + /* Open the master journal file exclusively in case some other process + ** is running this routine also. Not that it makes too much difference. + */ + rc = sqlite3OsOpenReadOnly(zMaster, &master); + if( rc!=SQLITE_OK ) goto delmaster_out; + master_open = 1; + rc = sqlite3OsFileSize(master, &nMasterJournal); + if( rc!=SQLITE_OK ) goto delmaster_out; + + if( nMasterJournal>0 ){ + char *zJournal; + char *zMasterPtr = 0; + + /* Load the entire master journal file into space obtained from + ** sqliteMalloc() and pointed to by zMasterJournal. + */ + zMasterJournal = (char *)sqliteMalloc(nMasterJournal); + if( !zMasterJournal ){ + rc = SQLITE_NOMEM; + goto delmaster_out; + } + rc = sqlite3OsRead(master, zMasterJournal, nMasterJournal); + if( rc!=SQLITE_OK ) goto delmaster_out; + + zJournal = zMasterJournal; + while( (zJournal-zMasterJournal)pAll; pPg; pPg=pPg->pNextAll){ + char zBuf[SQLITE_MAX_PAGE_SIZE]; + if( !pPg->dirty ) continue; + if( (int)pPg->pgno <= pPager->origDbSize ){ + rc = sqlite3OsSeek(pPager->fd, pPager->pageSize*(i64)(pPg->pgno-1)); + if( rc==SQLITE_OK ){ + rc = sqlite3OsRead(pPager->fd, zBuf, pPager->pageSize); + } + TRACE3("REFETCH %d page %d\n", PAGERID(pPager), pPg->pgno); + if( rc ) break; + CODEC(pPager, zBuf, pPg->pgno, 2); + }else{ + memset(zBuf, 0, pPager->pageSize); + } + if( pPg->nRef==0 || memcmp(zBuf, PGHDR_TO_DATA(pPg), pPager->pageSize) ){ + memcpy(PGHDR_TO_DATA(pPg), zBuf, pPager->pageSize); + if( pPager->xReiniter ){ + pPager->xReiniter(PGHDR_TO_DATA(pPg), pPager->pageSize); + }else{ + memset(PGHDR_TO_EXTRA(pPg, pPager), 0, pPager->nExtra); + } + } + pPg->needSync = 0; + pPg->dirty = 0; +#ifdef SQLITE_CHECK_PAGES + pPg->pageHash = pager_pagehash(pPg); +#endif + } + return rc; +} + +/* +** Truncate the main file of the given pager to the number of pages +** indicated. +*/ +static int pager_truncate(Pager *pPager, int nPage){ + assert( pPager->state>=PAGER_EXCLUSIVE ); + return sqlite3OsTruncate(pPager->fd, pPager->pageSize*(i64)nPage); +} + +/* +** Playback the journal and thus restore the database file to +** the state it was in before we started making changes. +** +** The journal file format is as follows: +** +** (1) 8 byte prefix. A copy of aJournalMagic[]. +** (2) 4 byte big-endian integer which is the number of valid page records +** in the journal. If this value is 0xffffffff, then compute the +** number of page records from the journal size. +** (3) 4 byte big-endian integer which is the initial value for the +** sanity checksum. +** (4) 4 byte integer which is the number of pages to truncate the +** database to during a rollback. +** (5) 4 byte integer which is the number of bytes in the master journal +** name. The value may be zero (indicate that there is no master +** journal.) +** (6) N bytes of the master journal name. The name will be nul-terminated +** and might be shorter than the value read from (5). If the first byte +** of the name is \000 then there is no master journal. The master +** journal name is stored in UTF-8. +** (7) Zero or more pages instances, each as follows: +** + 4 byte page number. +** + pPager->pageSize bytes of data. +** + 4 byte checksum +** +** When we speak of the journal header, we mean the first 6 items above. +** Each entry in the journal is an instance of the 7th item. +** +** Call the value from the second bullet "nRec". nRec is the number of +** valid page entries in the journal. In most cases, you can compute the +** value of nRec from the size of the journal file. But if a power +** failure occurred while the journal was being written, it could be the +** case that the size of the journal file had already been increased but +** the extra entries had not yet made it safely to disk. In such a case, +** the value of nRec computed from the file size would be too large. For +** that reason, we always use the nRec value in the header. +** +** If the nRec value is 0xffffffff it means that nRec should be computed +** from the file size. This value is used when the user selects the +** no-sync option for the journal. A power failure could lead to corruption +** in this case. But for things like temporary table (which will be +** deleted when the power is restored) we don't care. +** +** If the file opened as the journal file is not a well-formed +** journal file then all pages up to the first corrupted page are rolled +** back (or no pages if the journal header is corrupted). The journal file +** is then deleted and SQLITE_OK returned, just as if no corruption had +** been encountered. +** +** If an I/O or malloc() error occurs, the journal-file is not deleted +** and an error code is returned. +*/ +static int pager_playback(Pager *pPager){ + i64 szJ; /* Size of the journal file in bytes */ + u32 nRec; /* Number of Records in the journal */ + int i; /* Loop counter */ + Pgno mxPg = 0; /* Size of the original file in pages */ + int rc; /* Result code of a subroutine */ + char *zMaster = 0; /* Name of master journal file if any */ + + /* Figure out how many records are in the journal. Abort early if + ** the journal is empty. + */ + assert( pPager->journalOpen ); + rc = sqlite3OsFileSize(pPager->jfd, &szJ); + if( rc!=SQLITE_OK ){ + goto end_playback; + } + + /* Read the master journal name from the journal, if it is present. + ** If a master journal file name is specified, but the file is not + ** present on disk, then the journal is not hot and does not need to be + ** played back. + */ + rc = readMasterJournal(pPager->jfd, &zMaster); + assert( rc!=SQLITE_DONE ); + if( rc!=SQLITE_OK || (zMaster && !sqlite3OsFileExists(zMaster)) ){ + sqliteFree(zMaster); + zMaster = 0; + if( rc==SQLITE_DONE ) rc = SQLITE_OK; + goto end_playback; + } + sqlite3OsSeek(pPager->jfd, 0); + pPager->journalOff = 0; + + /* This loop terminates either when the readJournalHdr() call returns + ** SQLITE_DONE or an IO error occurs. */ + while( 1 ){ + + /* Read the next journal header from the journal file. If there are + ** not enough bytes left in the journal file for a complete header, or + ** it is corrupted, then a process must of failed while writing it. + ** This indicates nothing more needs to be rolled back. + */ + rc = readJournalHdr(pPager, szJ, &nRec, &mxPg); + if( rc!=SQLITE_OK ){ + if( rc==SQLITE_DONE ){ + rc = SQLITE_OK; + } + goto end_playback; + } + + /* If nRec is 0xffffffff, then this journal was created by a process + ** working in no-sync mode. This means that the rest of the journal + ** file consists of pages, there are no more journal headers. Compute + ** the value of nRec based on this assumption. + */ + if( nRec==0xffffffff ){ + assert( pPager->journalOff==JOURNAL_HDR_SZ(pPager) ); + nRec = (szJ - JOURNAL_HDR_SZ(pPager))/JOURNAL_PG_SZ(pPager); + } + + /* If this is the first header read from the journal, truncate the + ** database file back to it's original size. + */ + if( pPager->state>=PAGER_EXCLUSIVE && + pPager->journalOff==JOURNAL_HDR_SZ(pPager) ){ + assert( pPager->origDbSize==0 || pPager->origDbSize==mxPg ); + rc = pager_truncate(pPager, mxPg); + if( rc!=SQLITE_OK ){ + goto end_playback; + } + pPager->dbSize = mxPg; + } + + /* rc = sqlite3OsSeek(pPager->jfd, JOURNAL_HDR_SZ(pPager)); */ + if( rc!=SQLITE_OK ) goto end_playback; + + /* Copy original pages out of the journal and back into the database file. + */ + for(i=0; ijfd, 1); + if( rc!=SQLITE_OK ){ + if( rc==SQLITE_DONE ){ + rc = SQLITE_OK; + pPager->journalOff = szJ; + break; + }else{ + goto end_playback; + } + } + } + } + + /* Pages that have been written to the journal but never synced + ** where not restored by the loop above. We have to restore those + ** pages by reading them back from the original database. + */ + assert( rc==SQLITE_OK ); + pager_reload_cache(pPager); + +end_playback: + if( rc==SQLITE_OK ){ + rc = pager_unwritelock(pPager); + } + if( zMaster ){ + /* If there was a master journal and this routine will return true, + ** see if it is possible to delete the master journal. + */ + if( rc==SQLITE_OK ){ + rc = pager_delmaster(zMaster); + } + sqliteFree(zMaster); + } + + /* The Pager.sectorSize variable may have been updated while rolling + ** back a journal created by a process with a different PAGER_SECTOR_SIZE + ** value. Reset it to the correct value for this process. + */ + pPager->sectorSize = PAGER_SECTOR_SIZE; + return rc; +} + +/* +** Playback the statement journal. +** +** This is similar to playing back the transaction journal but with +** a few extra twists. +** +** (1) The number of pages in the database file at the start of +** the statement is stored in pPager->stmtSize, not in the +** journal file itself. +** +** (2) In addition to playing back the statement journal, also +** playback all pages of the transaction journal beginning +** at offset pPager->stmtJSize. +*/ +static int pager_stmt_playback(Pager *pPager){ + i64 szJ; /* Size of the full journal */ + i64 hdrOff; + int nRec; /* Number of Records */ + int i; /* Loop counter */ + int rc; + + szJ = pPager->journalOff; +#ifndef NDEBUG + { + i64 os_szJ; + rc = sqlite3OsFileSize(pPager->jfd, &os_szJ); + if( rc!=SQLITE_OK ) return rc; + assert( szJ==os_szJ ); + } +#endif + + /* Set hdrOff to be the offset to the first journal header written + ** this statement transaction, or the end of the file if no journal + ** header was written. + */ + hdrOff = pPager->stmtHdrOff; + assert( pPager->fullSync || !hdrOff ); + if( !hdrOff ){ + hdrOff = szJ; + } + + /* Truncate the database back to its original size. + */ + if( pPager->state>=PAGER_EXCLUSIVE ){ + rc = pager_truncate(pPager, pPager->stmtSize); + } + pPager->dbSize = pPager->stmtSize; + + /* Figure out how many records are in the statement journal. + */ + assert( pPager->stmtInUse && pPager->journalOpen ); + sqlite3OsSeek(pPager->stfd, 0); + nRec = pPager->stmtNRec; + + /* Copy original pages out of the statement journal and back into the + ** database file. Note that the statement journal omits checksums from + ** each record since power-failure recovery is not important to statement + ** journals. + */ + for(i=nRec-1; i>=0; i--){ + rc = pager_playback_one_page(pPager, pPager->stfd, 0); + assert( rc!=SQLITE_DONE ); + if( rc!=SQLITE_OK ) goto end_stmt_playback; + } + + /* Now roll some pages back from the transaction journal. Pager.stmtJSize + ** was the size of the journal file when this statement was started, so + ** everything after that needs to be rolled back, either into the + ** database, the memory cache, or both. + ** + ** If it is not zero, then Pager.stmtHdrOff is the offset to the start + ** of the first journal header written during this statement transaction. + */ + rc = sqlite3OsSeek(pPager->jfd, pPager->stmtJSize); + if( rc!=SQLITE_OK ){ + goto end_stmt_playback; + } + pPager->journalOff = pPager->stmtJSize; + pPager->cksumInit = pPager->stmtCksum; + assert( JOURNAL_HDR_SZ(pPager)<(pPager->pageSize+8) ); + while( pPager->journalOff <= (hdrOff-(pPager->pageSize+8)) ){ + rc = pager_playback_one_page(pPager, pPager->jfd, 1); + assert( rc!=SQLITE_DONE ); + if( rc!=SQLITE_OK ) goto end_stmt_playback; + } + + while( pPager->journalOff < szJ ){ + u32 nJRec; /* Number of Journal Records */ + u32 dummy; + rc = readJournalHdr(pPager, szJ, &nJRec, &dummy); + if( rc!=SQLITE_OK ){ + assert( rc!=SQLITE_DONE ); + goto end_stmt_playback; + } + if( nJRec==0 ){ + nJRec = (szJ - pPager->journalOff) / (pPager->pageSize+8); + } + for(i=nJRec-1; i>=0 && pPager->journalOff < szJ; i--){ + rc = pager_playback_one_page(pPager, pPager->jfd, 1); + assert( rc!=SQLITE_DONE ); + if( rc!=SQLITE_OK ) goto end_stmt_playback; + } + } + + pPager->journalOff = szJ; + +end_stmt_playback: + if( rc==SQLITE_OK) { + pPager->journalOff = szJ; + /* pager_reload_cache(pPager); */ + } + return rc; +} + +/* +** Change the maximum number of in-memory pages that are allowed. +*/ +void sqlite3pager_set_cachesize(Pager *pPager, int mxPage){ + if( mxPage>10 ){ + pPager->mxPage = mxPage; + }else{ + pPager->mxPage = 10; + } +} + +/* +** Adjust the robustness of the database to damage due to OS crashes +** or power failures by changing the number of syncs()s when writing +** the rollback journal. There are three levels: +** +** OFF sqlite3OsSync() is never called. This is the default +** for temporary and transient files. +** +** NORMAL The journal is synced once before writes begin on the +** database. This is normally adequate protection, but +** it is theoretically possible, though very unlikely, +** that an inopertune power failure could leave the journal +** in a state which would cause damage to the database +** when it is rolled back. +** +** FULL The journal is synced twice before writes begin on the +** database (with some additional information - the nRec field +** of the journal header - being written in between the two +** syncs). If we assume that writing a +** single disk sector is atomic, then this mode provides +** assurance that the journal will not be corrupted to the +** point of causing damage to the database during rollback. +** +** Numeric values associated with these states are OFF==1, NORMAL=2, +** and FULL=3. +*/ +#ifndef SQLITE_OMIT_PAGER_PRAGMAS +void sqlite3pager_set_safety_level(Pager *pPager, int level, int full_fsync){ + pPager->noSync = level==1 || pPager->tempFile; + pPager->fullSync = level==3 && !pPager->tempFile; + pPager->full_fsync = full_fsync; + if( pPager->noSync ) pPager->needSync = 0; +} +#endif + +/* +** The following global variable is incremented whenever the library +** attempts to open a temporary file. This information is used for +** testing and analysis only. +*/ +int sqlite3_opentemp_count = 0; + +/* +** Open a temporary file. Write the name of the file into zFile +** (zFile must be at least SQLITE_TEMPNAME_SIZE bytes long.) Write +** the file descriptor into *fd. Return SQLITE_OK on success or some +** other error code if we fail. +** +** The OS will automatically delete the temporary file when it is +** closed. +*/ +static int sqlite3pager_opentemp(char *zFile, OsFile **pFd){ + int cnt = 8; + int rc; + sqlite3_opentemp_count++; /* Used for testing and analysis only */ + do{ + cnt--; + sqlite3OsTempFileName(zFile); + rc = sqlite3OsOpenExclusive(zFile, pFd, 1); + }while( cnt>0 && rc!=SQLITE_OK && rc!=SQLITE_NOMEM ); + return rc; +} + +/* +** Create a new page cache and put a pointer to the page cache in *ppPager. +** The file to be cached need not exist. The file is not locked until +** the first call to sqlite3pager_get() and is only held open until the +** last page is released using sqlite3pager_unref(). +** +** If zFilename is NULL then a randomly-named temporary file is created +** and used as the file to be cached. The file will be deleted +** automatically when it is closed. +** +** If zFilename is ":memory:" then all information is held in cache. +** It is never written to disk. This can be used to implement an +** in-memory database. +*/ +int sqlite3pager_open( + Pager **ppPager, /* Return the Pager structure here */ + const char *zFilename, /* Name of the database file to open */ + int nExtra, /* Extra bytes append to each in-memory page */ + int flags /* flags controlling this file */ +){ + Pager *pPager = 0; + char *zFullPathname = 0; + int nameLen; /* Compiler is wrong. This is always initialized before use */ + OsFile *fd; + int rc = SQLITE_OK; + int i; + int tempFile = 0; + int memDb = 0; + int readOnly = 0; + int useJournal = (flags & PAGER_OMIT_JOURNAL)==0; + int noReadlock = (flags & PAGER_NO_READLOCK)!=0; + char zTemp[SQLITE_TEMPNAME_SIZE]; +#ifdef SQLITE_ENABLE_MEMORY_MANAGEMENT + /* A malloc() cannot fail in sqlite3ThreadData() as one or more calls to + ** malloc() must have already been made by this thread before it gets + ** to this point. This means the ThreadData must have been allocated already + ** so that ThreadData.nAlloc can be set. It would be nice to assert + ** that ThreadData.nAlloc is non-zero, but alas this breaks test cases + ** written to invoke the pager directly. + */ + ThreadData *pTsd = sqlite3ThreadData(); + assert( pTsd ); +#endif + + /* If malloc() has already failed return SQLITE_NOMEM. Before even + ** testing for this, set *ppPager to NULL so the caller knows the pager + ** structure was never allocated. + */ + *ppPager = 0; + if( sqlite3MallocFailed() ){ + return SQLITE_NOMEM; + } + memset(&fd, 0, sizeof(fd)); + + /* Open the pager file and set zFullPathname to point at malloc()ed + ** memory containing the complete filename (i.e. including the directory). + */ + if( zFilename && zFilename[0] ){ +#ifndef SQLITE_OMIT_MEMORYDB + if( strcmp(zFilename,":memory:")==0 ){ + memDb = 1; + zFullPathname = sqliteStrDup(""); + }else +#endif + { + zFullPathname = sqlite3OsFullPathname(zFilename); + if( zFullPathname ){ + rc = sqlite3OsOpenReadWrite(zFullPathname, &fd, &readOnly); + } + } + }else{ + rc = sqlite3pager_opentemp(zTemp, &fd); + zFilename = zTemp; + zFullPathname = sqlite3OsFullPathname(zFilename); + if( rc==SQLITE_OK ){ + tempFile = 1; + } + } + + /* Allocate the Pager structure. As part of the same allocation, allocate + ** space for the full paths of the file, directory and journal + ** (Pager.zFilename, Pager.zDirectory and Pager.zJournal). + */ + if( zFullPathname ){ + nameLen = strlen(zFullPathname); + pPager = sqliteMalloc( sizeof(*pPager) + nameLen*3 + 30 ); + } + + /* If an error occured in either of the blocks above, free the memory + ** pointed to by zFullPathname, free the Pager structure and close the + ** file. Since the pager is not allocated there is no need to set + ** any Pager.errMask variables. + */ + if( !pPager || !zFullPathname || rc!=SQLITE_OK ){ + sqlite3OsClose(&fd); + sqliteFree(zFullPathname); + sqliteFree(pPager); + return ((rc==SQLITE_OK)?SQLITE_NOMEM:rc); + } + + TRACE3("OPEN %d %s\n", FILEHANDLEID(fd), zFullPathname); + pPager->zFilename = (char*)&pPager[1]; + pPager->zDirectory = &pPager->zFilename[nameLen+1]; + pPager->zJournal = &pPager->zDirectory[nameLen+1]; + strcpy(pPager->zFilename, zFullPathname); + strcpy(pPager->zDirectory, zFullPathname); + + for(i=nameLen; i>0 && pPager->zDirectory[i-1]!='/'; i--){} + if( i>0 ) pPager->zDirectory[i-1] = 0; + strcpy(pPager->zJournal, zFullPathname); + sqliteFree(zFullPathname); + strcpy(&pPager->zJournal[nameLen], "-journal"); + pPager->fd = fd; + /* pPager->journalOpen = 0; */ + pPager->useJournal = useJournal && !memDb; + pPager->noReadlock = noReadlock && readOnly; + /* pPager->stmtOpen = 0; */ + /* pPager->stmtInUse = 0; */ + /* pPager->nRef = 0; */ + pPager->dbSize = memDb-1; + pPager->pageSize = SQLITE_DEFAULT_PAGE_SIZE; + /* pPager->stmtSize = 0; */ + /* pPager->stmtJSize = 0; */ + /* pPager->nPage = 0; */ + /* pPager->nMaxPage = 0; */ + pPager->mxPage = 100; + assert( PAGER_UNLOCK==0 ); + /* pPager->state = PAGER_UNLOCK; */ + /* pPager->errMask = 0; */ + pPager->tempFile = tempFile; + pPager->memDb = memDb; + pPager->readOnly = readOnly; + /* pPager->needSync = 0; */ + pPager->noSync = pPager->tempFile || !useJournal; + pPager->fullSync = (pPager->noSync?0:1); + /* pPager->pFirst = 0; */ + /* pPager->pFirstSynced = 0; */ + /* pPager->pLast = 0; */ + pPager->nExtra = FORCE_ALIGNMENT(nExtra); + pPager->sectorSize = PAGER_SECTOR_SIZE; + /* pPager->pBusyHandler = 0; */ + /* memset(pPager->aHash, 0, sizeof(pPager->aHash)); */ + *ppPager = pPager; +#ifdef SQLITE_ENABLE_MEMORY_MANAGEMENT + pPager->pNext = pTsd->pPager; + pTsd->pPager = pPager; +#endif + return SQLITE_OK; +} + +/* +** Set the busy handler function. +*/ +void sqlite3pager_set_busyhandler(Pager *pPager, BusyHandler *pBusyHandler){ + pPager->pBusyHandler = pBusyHandler; +} + +/* +** Set the destructor for this pager. If not NULL, the destructor is called +** when the reference count on each page reaches zero. The destructor can +** be used to clean up information in the extra segment appended to each page. +** +** The destructor is not called as a result sqlite3pager_close(). +** Destructors are only called by sqlite3pager_unref(). +*/ +void sqlite3pager_set_destructor(Pager *pPager, void (*xDesc)(void*,int)){ + pPager->xDestructor = xDesc; +} + +/* +** Set the reinitializer for this pager. If not NULL, the reinitializer +** is called when the content of a page in cache is restored to its original +** value as a result of a rollback. The callback gives higher-level code +** an opportunity to restore the EXTRA section to agree with the restored +** page data. +*/ +void sqlite3pager_set_reiniter(Pager *pPager, void (*xReinit)(void*,int)){ + pPager->xReiniter = xReinit; +} + +/* +** Set the page size. Return the new size. If the suggest new page +** size is inappropriate, then an alternative page size is selected +** and returned. +*/ +int sqlite3pager_set_pagesize(Pager *pPager, int pageSize){ + assert( pageSize>=512 && pageSize<=SQLITE_MAX_PAGE_SIZE ); + if( !pPager->memDb ){ + pPager->pageSize = pageSize; + } + return pPager->pageSize; +} + +/* +** The following set of routines are used to disable the simulated +** I/O error mechanism. These routines are used to avoid simulated +** errors in places where we do not care about errors. +** +** Unless -DSQLITE_TEST=1 is used, these routines are all no-ops +** and generate no code. +*/ +#ifdef SQLITE_TEST +extern int sqlite3_io_error_pending; +extern int sqlite3_io_error_hit; +static int saved_cnt; +void clear_simulated_io_error(){ + sqlite3_io_error_hit = 0; +} +void disable_simulated_io_errors(void){ + saved_cnt = sqlite3_io_error_pending; + sqlite3_io_error_pending = -1; +} +void enable_simulated_io_errors(void){ + sqlite3_io_error_pending = saved_cnt; +} +#else +# define clear_simulated_io_error() +# define disable_simulated_io_errors() +# define enable_simulated_io_errors() +#endif + +/* +** Read the first N bytes from the beginning of the file into memory +** that pDest points to. +** +** No error checking is done. The rational for this is that this function +** may be called even if the file does not exist or contain a header. In +** these cases sqlite3OsRead() will return an error, to which the correct +** response is to zero the memory at pDest and continue. A real IO error +** will presumably recur and be picked up later (Todo: Think about this). +*/ +void sqlite3pager_read_fileheader(Pager *pPager, int N, unsigned char *pDest){ + memset(pDest, 0, N); + if( MEMDB==0 ){ + disable_simulated_io_errors(); + sqlite3OsSeek(pPager->fd, 0); + sqlite3OsRead(pPager->fd, pDest, N); + enable_simulated_io_errors(); + } +} + +/* +** Return the total number of pages in the disk file associated with +** pPager. +** +** If the PENDING_BYTE lies on the page directly after the end of the +** file, then consider this page part of the file too. For example, if +** PENDING_BYTE is byte 4096 (the first byte of page 5) and the size of the +** file is 4096 bytes, 5 is returned instead of 4. +*/ +int sqlite3pager_pagecount(Pager *pPager){ + i64 n; + assert( pPager!=0 ); + if( pPager->dbSize>=0 ){ + n = pPager->dbSize; + } else { + if( sqlite3OsFileSize(pPager->fd, &n)!=SQLITE_OK ){ + pager_error(pPager, SQLITE_IOERR); + return 0; + } + if( n>0 && npageSize ){ + n = 1; + }else{ + n /= pPager->pageSize; + } + if( pPager->state!=PAGER_UNLOCK ){ + pPager->dbSize = n; + } + } + if( n==(PENDING_BYTE/pPager->pageSize) ){ + n++; + } + return n; +} + +/* +** Forward declaration +*/ +static int syncJournal(Pager*); + + +/* +** Unlink pPg from it's hash chain. Also set the page number to 0 to indicate +** that the page is not part of any hash chain. This is required because the +** sqlite3pager_movepage() routine can leave a page in the +** pNextFree/pPrevFree list that is not a part of any hash-chain. +*/ +static void unlinkHashChain(Pager *pPager, PgHdr *pPg){ + if( pPg->pgno==0 ){ + /* If the page number is zero, then this page is not in any hash chain. */ + return; + } + if( pPg->pNextHash ){ + pPg->pNextHash->pPrevHash = pPg->pPrevHash; + } + if( pPg->pPrevHash ){ + assert( pPager->aHash[pager_hash(pPg->pgno)]!=pPg ); + pPg->pPrevHash->pNextHash = pPg->pNextHash; + }else{ + int h = pager_hash(pPg->pgno); + assert( pPager->aHash[h]==pPg ); + pPager->aHash[h] = pPg->pNextHash; + } + + pPg->pgno = 0; + pPg->pNextHash = pPg->pPrevHash = 0; +} + +/* +** Unlink a page from the free list (the list of all pages where nRef==0) +** and from its hash collision chain. +*/ +static void unlinkPage(PgHdr *pPg){ + Pager *pPager = pPg->pPager; + + /* Keep the pFirstSynced pointer pointing at the first synchronized page */ + if( pPg==pPager->pFirstSynced ){ + PgHdr *p = pPg->pNextFree; + while( p && p->needSync ){ p = p->pNextFree; } + pPager->pFirstSynced = p; + } + + /* Unlink from the freelist */ + if( pPg->pPrevFree ){ + pPg->pPrevFree->pNextFree = pPg->pNextFree; + }else{ + assert( pPager->pFirst==pPg ); + pPager->pFirst = pPg->pNextFree; + } + if( pPg->pNextFree ){ + pPg->pNextFree->pPrevFree = pPg->pPrevFree; + }else{ + assert( pPager->pLast==pPg ); + pPager->pLast = pPg->pPrevFree; + } + pPg->pNextFree = pPg->pPrevFree = 0; + + /* Unlink from the pgno hash table */ + unlinkHashChain(pPager, pPg); +} + +#ifndef SQLITE_OMIT_MEMORYDB +/* +** This routine is used to truncate an in-memory database. Delete +** all pages whose pgno is larger than pPager->dbSize and is unreferenced. +** Referenced pages larger than pPager->dbSize are zeroed. +*/ +static void memoryTruncate(Pager *pPager){ + PgHdr *pPg; + PgHdr **ppPg; + int dbSize = pPager->dbSize; + + ppPg = &pPager->pAll; + while( (pPg = *ppPg)!=0 ){ + if( pPg->pgno<=dbSize ){ + ppPg = &pPg->pNextAll; + }else if( pPg->nRef>0 ){ + memset(PGHDR_TO_DATA(pPg), 0, pPager->pageSize); + ppPg = &pPg->pNextAll; + }else{ + *ppPg = pPg->pNextAll; + unlinkPage(pPg); + sqliteFree(pPg); + pPager->nPage--; + } + } +} +#else +#define memoryTruncate(p) +#endif + +/* +** Try to obtain a lock on a file. Invoke the busy callback if the lock +** is currently not available. Repeat until the busy callback returns +** false or until the lock succeeds. +** +** Return SQLITE_OK on success and an error code if we cannot obtain +** the lock. +*/ +static int pager_wait_on_lock(Pager *pPager, int locktype){ + int rc; + assert( PAGER_SHARED==SHARED_LOCK ); + assert( PAGER_RESERVED==RESERVED_LOCK ); + assert( PAGER_EXCLUSIVE==EXCLUSIVE_LOCK ); + if( pPager->state>=locktype ){ + rc = SQLITE_OK; + }else{ + do { + rc = sqlite3OsLock(pPager->fd, locktype); + }while( rc==SQLITE_BUSY && sqlite3InvokeBusyHandler(pPager->pBusyHandler) ); + if( rc==SQLITE_OK ){ + pPager->state = locktype; + } + } + return rc; +} + +/* +** Truncate the file to the number of pages specified. +*/ +int sqlite3pager_truncate(Pager *pPager, Pgno nPage){ + int rc; + sqlite3pager_pagecount(pPager); + if( pPager->errCode ){ + rc = pPager->errCode; + return rc; + } + if( nPage>=(unsigned)pPager->dbSize ){ + return SQLITE_OK; + } + if( MEMDB ){ + pPager->dbSize = nPage; + memoryTruncate(pPager); + return SQLITE_OK; + } + rc = syncJournal(pPager); + if( rc!=SQLITE_OK ){ + return rc; + } + + /* Get an exclusive lock on the database before truncating. */ + rc = pager_wait_on_lock(pPager, EXCLUSIVE_LOCK); + if( rc!=SQLITE_OK ){ + return rc; + } + + rc = pager_truncate(pPager, nPage); + if( rc==SQLITE_OK ){ + pPager->dbSize = nPage; + } + return rc; +} + +/* +** Shutdown the page cache. Free all memory and close all files. +** +** If a transaction was in progress when this routine is called, that +** transaction is rolled back. All outstanding pages are invalidated +** and their memory is freed. Any attempt to use a page associated +** with this page cache after this function returns will likely +** result in a coredump. +** +** This function always succeeds. If a transaction is active an attempt +** is made to roll it back. If an error occurs during the rollback +** a hot journal may be left in the filesystem but no error is returned +** to the caller. +*/ +int sqlite3pager_close(Pager *pPager){ + PgHdr *pPg, *pNext; +#ifdef SQLITE_ENABLE_MEMORY_MANAGEMENT + /* A malloc() cannot fail in sqlite3ThreadData() as one or more calls to + ** malloc() must have already been made by this thread before it gets + ** to this point. This means the ThreadData must have been allocated already + ** so that ThreadData.nAlloc can be set. + */ + ThreadData *pTsd = sqlite3ThreadData(); + assert( pPager ); + assert( pTsd && pTsd->nAlloc ); +#endif + + switch( pPager->state ){ + case PAGER_RESERVED: + case PAGER_SYNCED: + case PAGER_EXCLUSIVE: { + /* We ignore any IO errors that occur during the rollback + ** operation. So disable IO error simulation so that testing + ** works more easily. + */ + disable_simulated_io_errors(); + sqlite3pager_rollback(pPager); + enable_simulated_io_errors(); + if( !MEMDB ){ + sqlite3OsUnlock(pPager->fd, NO_LOCK); + } + assert( pPager->errCode || pPager->journalOpen==0 ); + break; + } + case PAGER_SHARED: { + if( !MEMDB ){ + sqlite3OsUnlock(pPager->fd, NO_LOCK); + } + break; + } + default: { + /* Do nothing */ + break; + } + } + for(pPg=pPager->pAll; pPg; pPg=pNext){ +#ifndef NDEBUG + if( MEMDB ){ + PgHistory *pHist = PGHDR_TO_HIST(pPg, pPager); + assert( !pPg->alwaysRollback ); + assert( !pHist->pOrig ); + assert( !pHist->pStmt ); + } +#endif + pNext = pPg->pNextAll; + sqliteFree(pPg); + } + TRACE2("CLOSE %d\n", PAGERID(pPager)); + assert( pPager->errCode || (pPager->journalOpen==0 && pPager->stmtOpen==0) ); + if( pPager->journalOpen ){ + sqlite3OsClose(&pPager->jfd); + } + sqliteFree(pPager->aInJournal); + if( pPager->stmtOpen ){ + sqlite3OsClose(&pPager->stfd); + } + sqlite3OsClose(&pPager->fd); + /* Temp files are automatically deleted by the OS + ** if( pPager->tempFile ){ + ** sqlite3OsDelete(pPager->zFilename); + ** } + */ + +#ifdef SQLITE_ENABLE_MEMORY_MANAGEMENT + /* Remove the pager from the linked list of pagers starting at + ** ThreadData.pPager if memory-management is enabled. + */ + if( pPager==pTsd->pPager ){ + pTsd->pPager = pPager->pNext; + }else{ + Pager *pTmp; + for(pTmp = pTsd->pPager; pTmp->pNext!=pPager; pTmp=pTmp->pNext); + pTmp->pNext = pPager->pNext; + } +#endif + + sqliteFree(pPager); + return SQLITE_OK; +} + +/* +** Return the page number for the given page data. +*/ +Pgno sqlite3pager_pagenumber(void *pData){ + PgHdr *p = DATA_TO_PGHDR(pData); + return p->pgno; +} + +/* +** The page_ref() function increments the reference count for a page. +** If the page is currently on the freelist (the reference count is zero) then +** remove it from the freelist. +** +** For non-test systems, page_ref() is a macro that calls _page_ref() +** online of the reference count is zero. For test systems, page_ref() +** is a real function so that we can set breakpoints and trace it. +*/ +static void _page_ref(PgHdr *pPg){ + if( pPg->nRef==0 ){ + /* The page is currently on the freelist. Remove it. */ + if( pPg==pPg->pPager->pFirstSynced ){ + PgHdr *p = pPg->pNextFree; + while( p && p->needSync ){ p = p->pNextFree; } + pPg->pPager->pFirstSynced = p; + } + if( pPg->pPrevFree ){ + pPg->pPrevFree->pNextFree = pPg->pNextFree; + }else{ + pPg->pPager->pFirst = pPg->pNextFree; + } + if( pPg->pNextFree ){ + pPg->pNextFree->pPrevFree = pPg->pPrevFree; + }else{ + pPg->pPager->pLast = pPg->pPrevFree; + } + pPg->pPager->nRef++; + } + pPg->nRef++; + REFINFO(pPg); +} +#ifdef SQLITE_DEBUG + static void page_ref(PgHdr *pPg){ + if( pPg->nRef==0 ){ + _page_ref(pPg); + }else{ + pPg->nRef++; + REFINFO(pPg); + } + } +#else +# define page_ref(P) ((P)->nRef==0?_page_ref(P):(void)(P)->nRef++) +#endif + +/* +** Increment the reference count for a page. The input pointer is +** a reference to the page data. +*/ +int sqlite3pager_ref(void *pData){ + PgHdr *pPg = DATA_TO_PGHDR(pData); + page_ref(pPg); + return SQLITE_OK; +} + +/* +** Sync the journal. In other words, make sure all the pages that have +** been written to the journal have actually reached the surface of the +** disk. It is not safe to modify the original database file until after +** the journal has been synced. If the original database is modified before +** the journal is synced and a power failure occurs, the unsynced journal +** data would be lost and we would be unable to completely rollback the +** database changes. Database corruption would occur. +** +** This routine also updates the nRec field in the header of the journal. +** (See comments on the pager_playback() routine for additional information.) +** If the sync mode is FULL, two syncs will occur. First the whole journal +** is synced, then the nRec field is updated, then a second sync occurs. +** +** For temporary databases, we do not care if we are able to rollback +** after a power failure, so sync occurs. +** +** This routine clears the needSync field of every page current held in +** memory. +*/ +static int syncJournal(Pager *pPager){ + PgHdr *pPg; + int rc = SQLITE_OK; + + /* Sync the journal before modifying the main database + ** (assuming there is a journal and it needs to be synced.) + */ + if( pPager->needSync ){ + if( !pPager->tempFile ){ + assert( pPager->journalOpen ); + /* assert( !pPager->noSync ); // noSync might be set if synchronous + ** was turned off after the transaction was started. Ticket #615 */ +#ifndef NDEBUG + { + /* Make sure the pPager->nRec counter we are keeping agrees + ** with the nRec computed from the size of the journal file. + */ + i64 jSz; + rc = sqlite3OsFileSize(pPager->jfd, &jSz); + if( rc!=0 ) return rc; + assert( pPager->journalOff==jSz ); + } +#endif + { + /* Write the nRec value into the journal file header. If in + ** full-synchronous mode, sync the journal first. This ensures that + ** all data has really hit the disk before nRec is updated to mark + ** it as a candidate for rollback. + */ + if( pPager->fullSync ){ + TRACE2("SYNC journal of %d\n", PAGERID(pPager)); + rc = sqlite3OsSync(pPager->jfd, 0); + if( rc!=0 ) return rc; + } + rc = sqlite3OsSeek(pPager->jfd, + pPager->journalHdr + sizeof(aJournalMagic)); + if( rc ) return rc; + rc = write32bits(pPager->jfd, pPager->nRec); + if( rc ) return rc; + + rc = sqlite3OsSeek(pPager->jfd, pPager->journalOff); + if( rc ) return rc; + } + TRACE2("SYNC journal of %d\n", PAGERID(pPager)); + rc = sqlite3OsSync(pPager->jfd, pPager->full_fsync); + if( rc!=0 ) return rc; + pPager->journalStarted = 1; + } + pPager->needSync = 0; + + /* Erase the needSync flag from every page. + */ + for(pPg=pPager->pAll; pPg; pPg=pPg->pNextAll){ + pPg->needSync = 0; + } + pPager->pFirstSynced = pPager->pFirst; + } + +#ifndef NDEBUG + /* If the Pager.needSync flag is clear then the PgHdr.needSync + ** flag must also be clear for all pages. Verify that this + ** invariant is true. + */ + else{ + for(pPg=pPager->pAll; pPg; pPg=pPg->pNextAll){ + assert( pPg->needSync==0 ); + } + assert( pPager->pFirstSynced==pPager->pFirst ); + } +#endif + + return rc; +} + +/* +** Given a list of pages (connected by the PgHdr.pDirty pointer) write +** every one of those pages out to the database file and mark them all +** as clean. +*/ +static int pager_write_pagelist(PgHdr *pList){ + Pager *pPager; + int rc; + + if( pList==0 ) return SQLITE_OK; + pPager = pList->pPager; + + /* At this point there may be either a RESERVED or EXCLUSIVE lock on the + ** database file. If there is already an EXCLUSIVE lock, the following + ** calls to sqlite3OsLock() are no-ops. + ** + ** Moving the lock from RESERVED to EXCLUSIVE actually involves going + ** through an intermediate state PENDING. A PENDING lock prevents new + ** readers from attaching to the database but is unsufficient for us to + ** write. The idea of a PENDING lock is to prevent new readers from + ** coming in while we wait for existing readers to clear. + ** + ** While the pager is in the RESERVED state, the original database file + ** is unchanged and we can rollback without having to playback the + ** journal into the original database file. Once we transition to + ** EXCLUSIVE, it means the database file has been changed and any rollback + ** will require a journal playback. + */ + rc = pager_wait_on_lock(pPager, EXCLUSIVE_LOCK); + if( rc!=SQLITE_OK ){ + return rc; + } + + while( pList ){ + assert( pList->dirty ); + rc = sqlite3OsSeek(pPager->fd, (pList->pgno-1)*(i64)pPager->pageSize); + if( rc ) return rc; + /* If there are dirty pages in the page cache with page numbers greater + ** than Pager.dbSize, this means sqlite3pager_truncate() was called to + ** make the file smaller (presumably by auto-vacuum code). Do not write + ** any such pages to the file. + */ + if( pList->pgno<=pPager->dbSize ){ + CODEC(pPager, PGHDR_TO_DATA(pList), pList->pgno, 6); + TRACE3("STORE %d page %d\n", PAGERID(pPager), pList->pgno); + rc = sqlite3OsWrite(pPager->fd, PGHDR_TO_DATA(pList), + pPager->pageSize); + CODEC(pPager, PGHDR_TO_DATA(pList), pList->pgno, 0); + TEST_INCR(pPager->nWrite); + } +#ifndef NDEBUG + else{ + TRACE3("NOSTORE %d page %d\n", PAGERID(pPager), pList->pgno); + } +#endif + if( rc ) return rc; + pList->dirty = 0; +#ifdef SQLITE_CHECK_PAGES + pList->pageHash = pager_pagehash(pList); +#endif + pList = pList->pDirty; + } + return SQLITE_OK; +} + +/* +** Collect every dirty page into a dirty list and +** return a pointer to the head of that list. All pages are +** collected even if they are still in use. +*/ +static PgHdr *pager_get_all_dirty_pages(Pager *pPager){ + PgHdr *p, *pList; + pList = 0; + for(p=pPager->pAll; p; p=p->pNextAll){ + if( p->dirty ){ + p->pDirty = pList; + pList = p; + } + } + return pList; +} + +/* +** Return TRUE if there is a hot journal on the given pager. +** A hot journal is one that needs to be played back. +** +** If the current size of the database file is 0 but a journal file +** exists, that is probably an old journal left over from a prior +** database with the same name. Just delete the journal. +*/ +static int hasHotJournal(Pager *pPager){ + if( !pPager->useJournal ) return 0; + if( !sqlite3OsFileExists(pPager->zJournal) ) return 0; + if( sqlite3OsCheckReservedLock(pPager->fd) ) return 0; + if( sqlite3pager_pagecount(pPager)==0 ){ + sqlite3OsDelete(pPager->zJournal); + return 0; + }else{ + return 1; + } +} + +/* +** Try to find a page in the cache that can be recycled. +** +** This routine may return SQLITE_IOERR, SQLITE_FULL or SQLITE_OK. It +** does not set the pPager->errCode variable. +*/ +static int pager_recycle(Pager *pPager, int syncOk, PgHdr **ppPg){ + PgHdr *pPg; + *ppPg = 0; + + /* Find a page to recycle. Try to locate a page that does not + ** require us to do an fsync() on the journal. + */ + pPg = pPager->pFirstSynced; + + /* If we could not find a page that does not require an fsync() + ** on the journal file then fsync the journal file. This is a + ** very slow operation, so we work hard to avoid it. But sometimes + ** it can't be helped. + */ + if( pPg==0 && pPager->pFirst && syncOk && !MEMDB){ + int rc = syncJournal(pPager); + if( rc!=0 ){ + return rc; + } + if( pPager->fullSync ){ + /* If in full-sync mode, write a new journal header into the + ** journal file. This is done to avoid ever modifying a journal + ** header that is involved in the rollback of pages that have + ** already been written to the database (in case the header is + ** trashed when the nRec field is updated). + */ + pPager->nRec = 0; + assert( pPager->journalOff > 0 ); + rc = writeJournalHdr(pPager); + if( rc!=0 ){ + return rc; + } + } + pPg = pPager->pFirst; + } + if( pPg==0 ){ + return SQLITE_OK; + } + + assert( pPg->nRef==0 ); + + /* Write the page to the database file if it is dirty. + */ + if( pPg->dirty ){ + int rc; + assert( pPg->needSync==0 ); + pPg->pDirty = 0; + rc = pager_write_pagelist( pPg ); + if( rc!=SQLITE_OK ){ + return rc; + } + } + assert( pPg->dirty==0 ); + + /* If the page we are recycling is marked as alwaysRollback, then + ** set the global alwaysRollback flag, thus disabling the + ** sqlite_dont_rollback() optimization for the rest of this transaction. + ** It is necessary to do this because the page marked alwaysRollback + ** might be reloaded at a later time but at that point we won't remember + ** that is was marked alwaysRollback. This means that all pages must + ** be marked as alwaysRollback from here on out. + */ + if( pPg->alwaysRollback ){ + pPager->alwaysRollback = 1; + } + + /* Unlink the old page from the free list and the hash table + */ + unlinkPage(pPg); + TEST_INCR(pPager->nOvfl); + + *ppPg = pPg; + return SQLITE_OK; +} + +/* +** This function is called to free superfluous dynamically allocated memory +** held by the pager system. Memory in use by any SQLite pager allocated +** by the current thread may be sqliteFree()ed. +** +** nReq is the number of bytes of memory required. Once this much has +** been released, the function returns. A negative value for nReq means +** free as much memory as possible. The return value is the total number +** of bytes of memory released. +*/ +#ifdef SQLITE_ENABLE_MEMORY_MANAGEMENT +int sqlite3pager_release_memory(int nReq){ + const ThreadData *pTsdro = sqlite3ThreadDataReadOnly(); + Pager *p; + int nReleased = 0; + int i; + + /* If the the global mutex is held, this subroutine becomes a + ** o-op; zero bytes of memory are freed. This is because + ** some of the code invoked by this function may also + ** try to obtain the mutex, resulting in a deadlock. + */ + if( sqlite3OsInMutex(0) ){ + return 0; + } + + /* Outermost loop runs for at most two iterations. First iteration we + ** try to find memory that can be released without calling fsync(). Second + ** iteration (which only runs if the first failed to free nReq bytes of + ** memory) is permitted to call fsync(). This is of course much more + ** expensive. + */ + for(i=0; i<=1; i++){ + + /* Loop through all the SQLite pagers opened by the current thread. */ + for(p=pTsdro->pPager; p && (nReq<0 || nReleasedpNext){ + PgHdr *pPg; + int rc; + + /* For each pager, try to free as many pages as possible (without + ** calling fsync() if this is the first iteration of the outermost + ** loop). + */ + while( SQLITE_OK==(rc = pager_recycle(p, i, &pPg)) && pPg) { + /* We've found a page to free. At this point the page has been + ** removed from the page hash-table, free-list and synced-list + ** (pFirstSynced). It is still in the all pages (pAll) list. + ** Remove it from this list before freeing. + ** + ** Todo: Check the Pager.pStmt list to make sure this is Ok. It + ** probably is though. + */ + PgHdr *pTmp; + assert( pPg ); + page_remove_from_stmt_list(pPg); + if( pPg==p->pAll ){ + p->pAll = pPg->pNextAll; + }else{ + for( pTmp=p->pAll; pTmp->pNextAll!=pPg; pTmp=pTmp->pNextAll ); + pTmp->pNextAll = pPg->pNextAll; + } + nReleased += sqliteAllocSize(pPg); + sqliteFree(pPg); + } + + if( rc!=SQLITE_OK ){ + /* An error occured whilst writing to the database file or + ** journal in pager_recycle(). The error is not returned to the + ** caller of this function. Instead, set the Pager.errCode variable. + ** The error will be returned to the user (or users, in the case + ** of a shared pager cache) of the pager for which the error occured. + */ + assert( rc==SQLITE_IOERR || rc==SQLITE_FULL ); + assert( p->state>=PAGER_RESERVED ); + pager_error(p, rc); + } + } + } + + return nReleased; +} +#endif /* SQLITE_ENABLE_MEMORY_MANAGEMENT */ + +/* +** Acquire a page. +** +** A read lock on the disk file is obtained when the first page is acquired. +** This read lock is dropped when the last page is released. +** +** A _get works for any page number greater than 0. If the database +** file is smaller than the requested page, then no actual disk +** read occurs and the memory image of the page is initialized to +** all zeros. The extra data appended to a page is always initialized +** to zeros the first time a page is loaded into memory. +** +** The acquisition might fail for several reasons. In all cases, +** an appropriate error code is returned and *ppPage is set to NULL. +** +** See also sqlite3pager_lookup(). Both this routine and _lookup() attempt +** to find a page in the in-memory cache first. If the page is not already +** in memory, this routine goes to disk to read it in whereas _lookup() +** just returns 0. This routine acquires a read-lock the first time it +** has to go to disk, and could also playback an old journal if necessary. +** Since _lookup() never goes to disk, it never has to deal with locks +** or journal files. +*/ +int sqlite3pager_get(Pager *pPager, Pgno pgno, void **ppPage){ + PgHdr *pPg; + int rc; + + /* The maximum page number is 2^31. Return SQLITE_CORRUPT if a page + ** number greater than this, or zero, is requested. + */ + if( pgno>PAGER_MAX_PGNO || pgno==0 || pgno==PAGER_MJ_PGNO(pPager) ){ + return SQLITE_CORRUPT_BKPT; + } + + /* Make sure we have not hit any critical errors. + */ + assert( pPager!=0 ); + *ppPage = 0; + if( pPager->errCode && pPager->errCode!=SQLITE_FULL ){ + return pPager->errCode; + } + + /* If this is the first page accessed, then get a SHARED lock + ** on the database file. + */ + if( pPager->nRef==0 && !MEMDB ){ + if( !pPager->noReadlock ){ + rc = pager_wait_on_lock(pPager, SHARED_LOCK); + if( rc!=SQLITE_OK ){ + return pager_error(pPager, rc); + } + } + + /* If a journal file exists, and there is no RESERVED lock on the + ** database file, then it either needs to be played back or deleted. + */ + if( hasHotJournal(pPager) ){ + /* Get an EXCLUSIVE lock on the database file. At this point it is + ** important that a RESERVED lock is not obtained on the way to the + ** EXCLUSIVE lock. If it were, another process might open the + ** database file, detect the RESERVED lock, and conclude that the + ** database is safe to read while this process is still rolling it + ** back. + ** + ** Because the intermediate RESERVED lock is not requested, the + ** second process will get to this point in the code and fail to + ** obtain it's own EXCLUSIVE lock on the database file. + */ + rc = sqlite3OsLock(pPager->fd, EXCLUSIVE_LOCK); + if( rc!=SQLITE_OK ){ + sqlite3OsUnlock(pPager->fd, NO_LOCK); + pPager->state = PAGER_UNLOCK; + return pager_error(pPager, rc); + } + pPager->state = PAGER_EXCLUSIVE; + + /* Open the journal for reading only. Return SQLITE_BUSY if + ** we are unable to open the journal file. + ** + ** The journal file does not need to be locked itself. The + ** journal file is never open unless the main database file holds + ** a write lock, so there is never any chance of two or more + ** processes opening the journal at the same time. + */ + rc = sqlite3OsOpenReadOnly(pPager->zJournal, &pPager->jfd); + if( rc!=SQLITE_OK ){ + sqlite3OsUnlock(pPager->fd, NO_LOCK); + pPager->state = PAGER_UNLOCK; + return SQLITE_BUSY; + } + pPager->journalOpen = 1; + pPager->journalStarted = 0; + pPager->journalOff = 0; + pPager->setMaster = 0; + pPager->journalHdr = 0; + + /* Playback and delete the journal. Drop the database write + ** lock and reacquire the read lock. + */ + rc = pager_playback(pPager); + if( rc!=SQLITE_OK ){ + return pager_error(pPager, rc); + } + } + pPg = 0; + }else{ + /* Search for page in cache */ + pPg = pager_lookup(pPager, pgno); + if( MEMDB && pPager->state==PAGER_UNLOCK ){ + pPager->state = PAGER_SHARED; + } + } + if( pPg==0 ){ + /* The requested page is not in the page cache. */ + int h; + TEST_INCR(pPager->nMiss); + if( pPager->nPagemxPage || pPager->pFirst==0 || MEMDB ){ + /* Create a new page */ + pPg = sqliteMallocRaw( sizeof(*pPg) + pPager->pageSize + + sizeof(u32) + pPager->nExtra + + MEMDB*sizeof(PgHistory) ); + if( pPg==0 ){ + return SQLITE_NOMEM; + } + memset(pPg, 0, sizeof(*pPg)); + if( MEMDB ){ + memset(PGHDR_TO_HIST(pPg, pPager), 0, sizeof(PgHistory)); + } + pPg->pPager = pPager; + pPg->pNextAll = pPager->pAll; + pPager->pAll = pPg; + pPager->nPage++; + if( pPager->nPage>pPager->nMaxPage ){ + assert( pPager->nMaxPage==(pPager->nPage-1) ); + pPager->nMaxPage++; + } + }else{ + rc = pager_recycle(pPager, 1, &pPg); + if( rc!=SQLITE_OK ){ + return rc; + } + assert(pPg) ; + } + pPg->pgno = pgno; + if( pPager->aInJournal && (int)pgno<=pPager->origDbSize ){ + sqlite3CheckMemory(pPager->aInJournal, pgno/8); + assert( pPager->journalOpen ); + pPg->inJournal = (pPager->aInJournal[pgno/8] & (1<<(pgno&7)))!=0; + pPg->needSync = 0; + }else{ + pPg->inJournal = 0; + pPg->needSync = 0; + } + if( pPager->aInStmt && (int)pgno<=pPager->stmtSize + && (pPager->aInStmt[pgno/8] & (1<<(pgno&7)))!=0 ){ + page_add_to_stmt_list(pPg); + }else{ + page_remove_from_stmt_list(pPg); + } + pPg->dirty = 0; + pPg->nRef = 1; + REFINFO(pPg); + + pPager->nRef++; + if( pPager->nExtra>0 ){ + memset(PGHDR_TO_EXTRA(pPg, pPager), 0, pPager->nExtra); + } + if( pPager->errCode ){ + sqlite3pager_unref(PGHDR_TO_DATA(pPg)); + rc = pPager->errCode; + return rc; + } + + /* Populate the page with data, either by reading from the database + ** file, or by setting the entire page to zero. + */ + if( sqlite3pager_pagecount(pPager)<(int)pgno || MEMDB ){ + memset(PGHDR_TO_DATA(pPg), 0, pPager->pageSize); + }else{ + assert( MEMDB==0 ); + rc = sqlite3OsSeek(pPager->fd, (pgno-1)*(i64)pPager->pageSize); + if( rc==SQLITE_OK ){ + rc = sqlite3OsRead(pPager->fd, PGHDR_TO_DATA(pPg), + pPager->pageSize); + } + TRACE3("FETCH %d page %d\n", PAGERID(pPager), pPg->pgno); + CODEC(pPager, PGHDR_TO_DATA(pPg), pPg->pgno, 3); + if( rc!=SQLITE_OK ){ + i64 fileSize; + int rc2 = sqlite3OsFileSize(pPager->fd, &fileSize); + if( rc2!=SQLITE_OK || fileSize>=pgno*pPager->pageSize ){ + /* An IO error occured in one of the the sqlite3OsSeek() or + ** sqlite3OsRead() calls above. */ + pPg->pgno = 0; + sqlite3pager_unref(PGHDR_TO_DATA(pPg)); + return rc; + }else{ + clear_simulated_io_error(); + memset(PGHDR_TO_DATA(pPg), 0, pPager->pageSize); + } + }else{ + TEST_INCR(pPager->nRead); + } + } + + /* Link the page into the page hash table */ + h = pager_hash(pgno); + pPg->pNextHash = pPager->aHash[h]; + pPager->aHash[h] = pPg; + if( pPg->pNextHash ){ + assert( pPg->pNextHash->pPrevHash==0 ); + pPg->pNextHash->pPrevHash = pPg; + } + +#ifdef SQLITE_CHECK_PAGES + pPg->pageHash = pager_pagehash(pPg); +#endif + }else{ + /* The requested page is in the page cache. */ + TEST_INCR(pPager->nHit); + page_ref(pPg); + } + *ppPage = PGHDR_TO_DATA(pPg); + return SQLITE_OK; +} + +/* +** Acquire a page if it is already in the in-memory cache. Do +** not read the page from disk. Return a pointer to the page, +** or 0 if the page is not in cache. +** +** See also sqlite3pager_get(). The difference between this routine +** and sqlite3pager_get() is that _get() will go to the disk and read +** in the page if the page is not already in cache. This routine +** returns NULL if the page is not in cache or if a disk I/O error +** has ever happened. +*/ +void *sqlite3pager_lookup(Pager *pPager, Pgno pgno){ + PgHdr *pPg; + + assert( pPager!=0 ); + assert( pgno!=0 ); + if( pPager->errCode && pPager->errCode!=SQLITE_FULL ){ + return 0; + } + pPg = pager_lookup(pPager, pgno); + if( pPg==0 ) return 0; + page_ref(pPg); + return PGHDR_TO_DATA(pPg); +} + +/* +** Release a page. +** +** If the number of references to the page drop to zero, then the +** page is added to the LRU list. When all references to all pages +** are released, a rollback occurs and the lock on the database is +** removed. +*/ +int sqlite3pager_unref(void *pData){ + PgHdr *pPg; + + /* Decrement the reference count for this page + */ + pPg = DATA_TO_PGHDR(pData); + assert( pPg->nRef>0 ); + pPg->nRef--; + REFINFO(pPg); + + CHECK_PAGE(pPg); + + /* When the number of references to a page reach 0, call the + ** destructor and add the page to the freelist. + */ + if( pPg->nRef==0 ){ + Pager *pPager; + pPager = pPg->pPager; + pPg->pNextFree = 0; + pPg->pPrevFree = pPager->pLast; + pPager->pLast = pPg; + if( pPg->pPrevFree ){ + pPg->pPrevFree->pNextFree = pPg; + }else{ + pPager->pFirst = pPg; + } + if( pPg->needSync==0 && pPager->pFirstSynced==0 ){ + pPager->pFirstSynced = pPg; + } + if( pPager->xDestructor ){ + pPager->xDestructor(pData, pPager->pageSize); + } + + /* When all pages reach the freelist, drop the read lock from + ** the database file. + */ + pPager->nRef--; + assert( pPager->nRef>=0 ); + if( pPager->nRef==0 && !MEMDB ){ + pager_reset(pPager); + } + } + return SQLITE_OK; +} + +/* +** Create a journal file for pPager. There should already be a RESERVED +** or EXCLUSIVE lock on the database file when this routine is called. +** +** Return SQLITE_OK if everything. Return an error code and release the +** write lock if anything goes wrong. +*/ +static int pager_open_journal(Pager *pPager){ + int rc; + assert( !MEMDB ); + assert( pPager->state>=PAGER_RESERVED ); + assert( pPager->journalOpen==0 ); + assert( pPager->useJournal ); + assert( pPager->aInJournal==0 ); + sqlite3pager_pagecount(pPager); + pPager->aInJournal = sqliteMalloc( pPager->dbSize/8 + 1 ); + if( pPager->aInJournal==0 ){ + rc = SQLITE_NOMEM; + goto failed_to_open_journal; + } + rc = sqlite3OsOpenExclusive(pPager->zJournal, &pPager->jfd, + pPager->tempFile); + pPager->journalOff = 0; + pPager->setMaster = 0; + pPager->journalHdr = 0; + if( rc!=SQLITE_OK ){ + goto failed_to_open_journal; + } + sqlite3OsSetFullSync(pPager->jfd, pPager->full_fsync); + sqlite3OsSetFullSync(pPager->fd, pPager->full_fsync); + sqlite3OsOpenDirectory(pPager->jfd, pPager->zDirectory); + pPager->journalOpen = 1; + pPager->journalStarted = 0; + pPager->needSync = 0; + pPager->alwaysRollback = 0; + pPager->nRec = 0; + if( pPager->errCode ){ + rc = pPager->errCode; + goto failed_to_open_journal; + } + pPager->origDbSize = pPager->dbSize; + + rc = writeJournalHdr(pPager); + + if( pPager->stmtAutoopen && rc==SQLITE_OK ){ + rc = sqlite3pager_stmt_begin(pPager); + } + if( rc!=SQLITE_OK && rc!=SQLITE_NOMEM ){ + rc = pager_unwritelock(pPager); + if( rc==SQLITE_OK ){ + rc = SQLITE_FULL; + } + } + return rc; + +failed_to_open_journal: + sqliteFree(pPager->aInJournal); + pPager->aInJournal = 0; + if( rc==SQLITE_NOMEM ){ + /* If this was a malloc() failure, then we will not be closing the pager + ** file. So delete any journal file we may have just created. Otherwise, + ** the system will get confused, we have a read-lock on the file and a + ** mysterious journal has appeared in the filesystem. + */ + sqlite3OsDelete(pPager->zJournal); + }else{ + sqlite3OsUnlock(pPager->fd, NO_LOCK); + pPager->state = PAGER_UNLOCK; + } + return rc; +} + +/* +** Acquire a write-lock on the database. The lock is removed when +** the any of the following happen: +** +** * sqlite3pager_commit() is called. +** * sqlite3pager_rollback() is called. +** * sqlite3pager_close() is called. +** * sqlite3pager_unref() is called to on every outstanding page. +** +** The first parameter to this routine is a pointer to any open page of the +** database file. Nothing changes about the page - it is used merely to +** acquire a pointer to the Pager structure and as proof that there is +** already a read-lock on the database. +** +** The second parameter indicates how much space in bytes to reserve for a +** master journal file-name at the start of the journal when it is created. +** +** A journal file is opened if this is not a temporary file. For temporary +** files, the opening of the journal file is deferred until there is an +** actual need to write to the journal. +** +** If the database is already reserved for writing, this routine is a no-op. +** +** If exFlag is true, go ahead and get an EXCLUSIVE lock on the file +** immediately instead of waiting until we try to flush the cache. The +** exFlag is ignored if a transaction is already active. +*/ +int sqlite3pager_begin(void *pData, int exFlag){ + PgHdr *pPg = DATA_TO_PGHDR(pData); + Pager *pPager = pPg->pPager; + int rc = SQLITE_OK; + assert( pPg->nRef>0 ); + assert( pPager->state!=PAGER_UNLOCK ); + if( pPager->state==PAGER_SHARED ){ + assert( pPager->aInJournal==0 ); + if( MEMDB ){ + pPager->state = PAGER_EXCLUSIVE; + pPager->origDbSize = pPager->dbSize; + }else{ + rc = sqlite3OsLock(pPager->fd, RESERVED_LOCK); + if( rc==SQLITE_OK ){ + pPager->state = PAGER_RESERVED; + if( exFlag ){ + rc = pager_wait_on_lock(pPager, EXCLUSIVE_LOCK); + } + } + if( rc!=SQLITE_OK ){ + return rc; + } + pPager->dirtyCache = 0; + TRACE2("TRANSACTION %d\n", PAGERID(pPager)); + if( pPager->useJournal && !pPager->tempFile ){ + rc = pager_open_journal(pPager); + } + } + } + return rc; +} + +/* +** Mark a data page as writeable. The page is written into the journal +** if it is not there already. This routine must be called before making +** changes to a page. +** +** The first time this routine is called, the pager creates a new +** journal and acquires a RESERVED lock on the database. If the RESERVED +** lock could not be acquired, this routine returns SQLITE_BUSY. The +** calling routine must check for that return value and be careful not to +** change any page data until this routine returns SQLITE_OK. +** +** If the journal file could not be written because the disk is full, +** then this routine returns SQLITE_FULL and does an immediate rollback. +** All subsequent write attempts also return SQLITE_FULL until there +** is a call to sqlite3pager_commit() or sqlite3pager_rollback() to +** reset. +*/ +int sqlite3pager_write(void *pData){ + PgHdr *pPg = DATA_TO_PGHDR(pData); + Pager *pPager = pPg->pPager; + int rc = SQLITE_OK; + + /* Check for errors + */ + if( pPager->errCode ){ + return pPager->errCode; + } + if( pPager->readOnly ){ + return SQLITE_PERM; + } + + assert( !pPager->setMaster ); + + CHECK_PAGE(pPg); + + /* Mark the page as dirty. If the page has already been written + ** to the journal then we can return right away. + */ + pPg->dirty = 1; + if( pPg->inJournal && (pPg->inStmt || pPager->stmtInUse==0) ){ + pPager->dirtyCache = 1; + }else{ + + /* If we get this far, it means that the page needs to be + ** written to the transaction journal or the ckeckpoint journal + ** or both. + ** + ** First check to see that the transaction journal exists and + ** create it if it does not. + */ + assert( pPager->state!=PAGER_UNLOCK ); + rc = sqlite3pager_begin(pData, 0); + if( rc!=SQLITE_OK ){ + return rc; + } + assert( pPager->state>=PAGER_RESERVED ); + if( !pPager->journalOpen && pPager->useJournal ){ + rc = pager_open_journal(pPager); + if( rc!=SQLITE_OK ) return rc; + } + assert( pPager->journalOpen || !pPager->useJournal ); + pPager->dirtyCache = 1; + + /* The transaction journal now exists and we have a RESERVED or an + ** EXCLUSIVE lock on the main database file. Write the current page to + ** the transaction journal if it is not there already. + */ + if( !pPg->inJournal && (pPager->useJournal || MEMDB) ){ + if( (int)pPg->pgno <= pPager->origDbSize ){ + int szPg; + u32 saved; + if( MEMDB ){ + PgHistory *pHist = PGHDR_TO_HIST(pPg, pPager); + TRACE3("JOURNAL %d page %d\n", PAGERID(pPager), pPg->pgno); + assert( pHist->pOrig==0 ); + pHist->pOrig = sqliteMallocRaw( pPager->pageSize ); + if( pHist->pOrig ){ + memcpy(pHist->pOrig, PGHDR_TO_DATA(pPg), pPager->pageSize); + } + }else{ + u32 cksum; + /* We should never write to the journal file the page that + ** contains the database locks. The following assert verifies + ** that we do not. */ + assert( pPg->pgno!=PAGER_MJ_PGNO(pPager) ); + CODEC(pPager, pData, pPg->pgno, 7); + cksum = pager_cksum(pPager, pPg->pgno, pData); + saved = *(u32*)PGHDR_TO_EXTRA(pPg, pPager); + store32bits(cksum, pPg, pPager->pageSize); + szPg = pPager->pageSize+8; + store32bits(pPg->pgno, pPg, -4); + + rc = sqlite3OsWrite(pPager->jfd, &((char*)pData)[-4], szPg); + pPager->journalOff += szPg; + TRACE4("JOURNAL %d page %d needSync=%d\n", + PAGERID(pPager), pPg->pgno, pPg->needSync); + CODEC(pPager, pData, pPg->pgno, 0); + *(u32*)PGHDR_TO_EXTRA(pPg, pPager) = saved; + + /* An error has occured writing to the journal file. The + ** transaction will be rolled back by the layer above. + */ + if( rc!=SQLITE_OK ){ + return rc; + } + + pPager->nRec++; + assert( pPager->aInJournal!=0 ); + pPager->aInJournal[pPg->pgno/8] |= 1<<(pPg->pgno&7); + pPg->needSync = !pPager->noSync; + if( pPager->stmtInUse ){ + pPager->aInStmt[pPg->pgno/8] |= 1<<(pPg->pgno&7); + page_add_to_stmt_list(pPg); + } + } + }else{ + pPg->needSync = !pPager->journalStarted && !pPager->noSync; + TRACE4("APPEND %d page %d needSync=%d\n", + PAGERID(pPager), pPg->pgno, pPg->needSync); + } + if( pPg->needSync ){ + pPager->needSync = 1; + } + pPg->inJournal = 1; + } + + /* If the statement journal is open and the page is not in it, + ** then write the current page to the statement journal. Note that + ** the statement journal format differs from the standard journal format + ** in that it omits the checksums and the header. + */ + if( pPager->stmtInUse && !pPg->inStmt && (int)pPg->pgno<=pPager->stmtSize ){ + assert( pPg->inJournal || (int)pPg->pgno>pPager->origDbSize ); + if( MEMDB ){ + PgHistory *pHist = PGHDR_TO_HIST(pPg, pPager); + assert( pHist->pStmt==0 ); + pHist->pStmt = sqliteMallocRaw( pPager->pageSize ); + if( pHist->pStmt ){ + memcpy(pHist->pStmt, PGHDR_TO_DATA(pPg), pPager->pageSize); + } + TRACE3("STMT-JOURNAL %d page %d\n", PAGERID(pPager), pPg->pgno); + }else{ + store32bits(pPg->pgno, pPg, -4); + CODEC(pPager, pData, pPg->pgno, 7); + rc = sqlite3OsWrite(pPager->stfd,((char*)pData)-4, + pPager->pageSize+4); + TRACE3("STMT-JOURNAL %d page %d\n", PAGERID(pPager), pPg->pgno); + CODEC(pPager, pData, pPg->pgno, 0); + if( rc!=SQLITE_OK ){ + return rc; + } + pPager->stmtNRec++; + assert( pPager->aInStmt!=0 ); + pPager->aInStmt[pPg->pgno/8] |= 1<<(pPg->pgno&7); + } + page_add_to_stmt_list(pPg); + } + } + + /* Update the database size and return. + */ + if( pPager->dbSize<(int)pPg->pgno ){ + pPager->dbSize = pPg->pgno; + if( !MEMDB && pPager->dbSize==PENDING_BYTE/pPager->pageSize ){ + pPager->dbSize++; + } + } + return rc; +} + +/* +** Return TRUE if the page given in the argument was previously passed +** to sqlite3pager_write(). In other words, return TRUE if it is ok +** to change the content of the page. +*/ +#ifndef NDEBUG +int sqlite3pager_iswriteable(void *pData){ + PgHdr *pPg = DATA_TO_PGHDR(pData); + return pPg->dirty; +} +#endif + +#ifndef SQLITE_OMIT_VACUUM +/* +** Replace the content of a single page with the information in the third +** argument. +*/ +int sqlite3pager_overwrite(Pager *pPager, Pgno pgno, void *pData){ + void *pPage; + int rc; + + rc = sqlite3pager_get(pPager, pgno, &pPage); + if( rc==SQLITE_OK ){ + rc = sqlite3pager_write(pPage); + if( rc==SQLITE_OK ){ + memcpy(pPage, pData, pPager->pageSize); + } + sqlite3pager_unref(pPage); + } + return rc; +} +#endif + +/* +** A call to this routine tells the pager that it is not necessary to +** write the information on page "pgno" back to the disk, even though +** that page might be marked as dirty. +** +** The overlying software layer calls this routine when all of the data +** on the given page is unused. The pager marks the page as clean so +** that it does not get written to disk. +** +** Tests show that this optimization, together with the +** sqlite3pager_dont_rollback() below, more than double the speed +** of large INSERT operations and quadruple the speed of large DELETEs. +** +** When this routine is called, set the alwaysRollback flag to true. +** Subsequent calls to sqlite3pager_dont_rollback() for the same page +** will thereafter be ignored. This is necessary to avoid a problem +** where a page with data is added to the freelist during one part of +** a transaction then removed from the freelist during a later part +** of the same transaction and reused for some other purpose. When it +** is first added to the freelist, this routine is called. When reused, +** the dont_rollback() routine is called. But because the page contains +** critical data, we still need to be sure it gets rolled back in spite +** of the dont_rollback() call. +*/ +void sqlite3pager_dont_write(Pager *pPager, Pgno pgno){ + PgHdr *pPg; + + if( MEMDB ) return; + + pPg = pager_lookup(pPager, pgno); + pPg->alwaysRollback = 1; + if( pPg && pPg->dirty && !pPager->stmtInUse ){ + if( pPager->dbSize==(int)pPg->pgno && pPager->origDbSizedbSize ){ + /* If this pages is the last page in the file and the file has grown + ** during the current transaction, then do NOT mark the page as clean. + ** When the database file grows, we must make sure that the last page + ** gets written at least once so that the disk file will be the correct + ** size. If you do not write this page and the size of the file + ** on the disk ends up being too small, that can lead to database + ** corruption during the next transaction. + */ + }else{ + TRACE3("DONT_WRITE page %d of %d\n", pgno, PAGERID(pPager)); + pPg->dirty = 0; +#ifdef SQLITE_CHECK_PAGES + pPg->pageHash = pager_pagehash(pPg); +#endif + } + } +} + +/* +** A call to this routine tells the pager that if a rollback occurs, +** it is not necessary to restore the data on the given page. This +** means that the pager does not have to record the given page in the +** rollback journal. +*/ +void sqlite3pager_dont_rollback(void *pData){ + PgHdr *pPg = DATA_TO_PGHDR(pData); + Pager *pPager = pPg->pPager; + + if( pPager->state!=PAGER_EXCLUSIVE || pPager->journalOpen==0 ) return; + if( pPg->alwaysRollback || pPager->alwaysRollback || MEMDB ) return; + if( !pPg->inJournal && (int)pPg->pgno <= pPager->origDbSize ){ + assert( pPager->aInJournal!=0 ); + pPager->aInJournal[pPg->pgno/8] |= 1<<(pPg->pgno&7); + pPg->inJournal = 1; + if( pPager->stmtInUse ){ + pPager->aInStmt[pPg->pgno/8] |= 1<<(pPg->pgno&7); + page_add_to_stmt_list(pPg); + } + TRACE3("DONT_ROLLBACK page %d of %d\n", pPg->pgno, PAGERID(pPager)); + } + if( pPager->stmtInUse && !pPg->inStmt && (int)pPg->pgno<=pPager->stmtSize ){ + assert( pPg->inJournal || (int)pPg->pgno>pPager->origDbSize ); + assert( pPager->aInStmt!=0 ); + pPager->aInStmt[pPg->pgno/8] |= 1<<(pPg->pgno&7); + page_add_to_stmt_list(pPg); + } +} + + +#ifndef SQLITE_OMIT_MEMORYDB +/* +** Clear a PgHistory block +*/ +static void clearHistory(PgHistory *pHist){ + sqliteFree(pHist->pOrig); + sqliteFree(pHist->pStmt); + pHist->pOrig = 0; + pHist->pStmt = 0; +} +#else +#define clearHistory(x) +#endif + +/* +** Commit all changes to the database and release the write lock. +** +** If the commit fails for any reason, a rollback attempt is made +** and an error code is returned. If the commit worked, SQLITE_OK +** is returned. +*/ +int sqlite3pager_commit(Pager *pPager){ + int rc; + PgHdr *pPg; + + if( pPager->errCode ){ + return pPager->errCode; + } + if( pPager->statedirty = 0; + pPg->inJournal = 0; + pPg->inStmt = 0; + pPg->pPrevStmt = pPg->pNextStmt = 0; + pPg = pPg->pDirty; + } +#ifndef NDEBUG + for(pPg=pPager->pAll; pPg; pPg=pPg->pNextAll){ + PgHistory *pHist = PGHDR_TO_HIST(pPg, pPager); + assert( !pPg->alwaysRollback ); + assert( !pHist->pOrig ); + assert( !pHist->pStmt ); + } +#endif + pPager->pStmt = 0; + pPager->state = PAGER_SHARED; + return SQLITE_OK; + } + if( pPager->dirtyCache==0 ){ + /* Exit early (without doing the time-consuming sqlite3OsSync() calls) + ** if there have been no changes to the database file. */ + assert( pPager->needSync==0 ); + rc = pager_unwritelock(pPager); + pPager->dbSize = -1; + return rc; + } + assert( pPager->journalOpen ); + rc = sqlite3pager_sync(pPager, 0, 0); + if( rc==SQLITE_OK ){ + rc = pager_unwritelock(pPager); + pPager->dbSize = -1; + } + return rc; +} + +/* +** Rollback all changes. The database falls back to PAGER_SHARED mode. +** All in-memory cache pages revert to their original data contents. +** The journal is deleted. +** +** This routine cannot fail unless some other process is not following +** the correct locking protocol (SQLITE_PROTOCOL) or unless some other +** process is writing trash into the journal file (SQLITE_CORRUPT) or +** unless a prior malloc() failed (SQLITE_NOMEM). Appropriate error +** codes are returned for all these occasions. Otherwise, +** SQLITE_OK is returned. +*/ +int sqlite3pager_rollback(Pager *pPager){ + int rc; + TRACE2("ROLLBACK %d\n", PAGERID(pPager)); + if( MEMDB ){ + PgHdr *p; + for(p=pPager->pAll; p; p=p->pNextAll){ + PgHistory *pHist; + assert( !p->alwaysRollback ); + if( !p->dirty ){ + assert( !((PgHistory *)PGHDR_TO_HIST(p, pPager))->pOrig ); + assert( !((PgHistory *)PGHDR_TO_HIST(p, pPager))->pStmt ); + continue; + } + + pHist = PGHDR_TO_HIST(p, pPager); + if( pHist->pOrig ){ + memcpy(PGHDR_TO_DATA(p), pHist->pOrig, pPager->pageSize); + TRACE3("ROLLBACK-PAGE %d of %d\n", p->pgno, PAGERID(pPager)); + }else{ + TRACE3("PAGE %d is clean on %d\n", p->pgno, PAGERID(pPager)); + } + clearHistory(pHist); + p->dirty = 0; + p->inJournal = 0; + p->inStmt = 0; + p->pPrevStmt = p->pNextStmt = 0; + + if( pPager->xReiniter ){ + pPager->xReiniter(PGHDR_TO_DATA(p), pPager->pageSize); + } + + } + pPager->pStmt = 0; + pPager->dbSize = pPager->origDbSize; + memoryTruncate(pPager); + pPager->stmtInUse = 0; + pPager->state = PAGER_SHARED; + return SQLITE_OK; + } + + if( !pPager->dirtyCache || !pPager->journalOpen ){ + rc = pager_unwritelock(pPager); + pPager->dbSize = -1; + return rc; + } + + if( pPager->errCode && pPager->errCode!=SQLITE_FULL ){ + if( pPager->state>=PAGER_EXCLUSIVE ){ + pager_playback(pPager); + } + return pPager->errCode; + } + if( pPager->state==PAGER_RESERVED ){ + int rc2; + rc = pager_reload_cache(pPager); + rc2 = pager_unwritelock(pPager); + if( rc==SQLITE_OK ){ + rc = rc2; + } + }else{ + rc = pager_playback(pPager); + } + pPager->dbSize = -1; + + /* If an error occurs during a ROLLBACK, we can no longer trust the pager + ** cache. So call pager_error() on the way out to make any error + ** persistent. + */ + return pager_error(pPager, rc); +} + +/* +** Return TRUE if the database file is opened read-only. Return FALSE +** if the database is (in theory) writable. +*/ +int sqlite3pager_isreadonly(Pager *pPager){ + return pPager->readOnly; +} + +/* +** This routine is used for testing and analysis only. +*/ +int *sqlite3pager_stats(Pager *pPager){ + static int a[11]; + a[0] = pPager->nRef; + a[1] = pPager->nPage; + a[2] = pPager->mxPage; + a[3] = pPager->dbSize; + a[4] = pPager->state; + a[5] = pPager->errCode; +#ifdef SQLITE_TEST + a[6] = pPager->nHit; + a[7] = pPager->nMiss; + a[8] = pPager->nOvfl; + a[9] = pPager->nRead; + a[10] = pPager->nWrite; +#endif + return a; +} + +/* +** Set the statement rollback point. +** +** This routine should be called with the transaction journal already +** open. A new statement journal is created that can be used to rollback +** changes of a single SQL command within a larger transaction. +*/ +int sqlite3pager_stmt_begin(Pager *pPager){ + int rc; + char zTemp[SQLITE_TEMPNAME_SIZE]; + assert( !pPager->stmtInUse ); + assert( pPager->dbSize>=0 ); + TRACE2("STMT-BEGIN %d\n", PAGERID(pPager)); + if( MEMDB ){ + pPager->stmtInUse = 1; + pPager->stmtSize = pPager->dbSize; + return SQLITE_OK; + } + if( !pPager->journalOpen ){ + pPager->stmtAutoopen = 1; + return SQLITE_OK; + } + assert( pPager->journalOpen ); + pPager->aInStmt = sqliteMalloc( pPager->dbSize/8 + 1 ); + if( pPager->aInStmt==0 ){ + /* sqlite3OsLock(pPager->fd, SHARED_LOCK); */ + return SQLITE_NOMEM; + } +#ifndef NDEBUG + rc = sqlite3OsFileSize(pPager->jfd, &pPager->stmtJSize); + if( rc ) goto stmt_begin_failed; + assert( pPager->stmtJSize == pPager->journalOff ); +#endif + pPager->stmtJSize = pPager->journalOff; + pPager->stmtSize = pPager->dbSize; + pPager->stmtHdrOff = 0; + pPager->stmtCksum = pPager->cksumInit; + if( !pPager->stmtOpen ){ + rc = sqlite3pager_opentemp(zTemp, &pPager->stfd); + if( rc ) goto stmt_begin_failed; + pPager->stmtOpen = 1; + pPager->stmtNRec = 0; + } + pPager->stmtInUse = 1; + return SQLITE_OK; + +stmt_begin_failed: + if( pPager->aInStmt ){ + sqliteFree(pPager->aInStmt); + pPager->aInStmt = 0; + } + return rc; +} + +/* +** Commit a statement. +*/ +int sqlite3pager_stmt_commit(Pager *pPager){ + if( pPager->stmtInUse ){ + PgHdr *pPg, *pNext; + TRACE2("STMT-COMMIT %d\n", PAGERID(pPager)); + if( !MEMDB ){ + sqlite3OsSeek(pPager->stfd, 0); + /* sqlite3OsTruncate(pPager->stfd, 0); */ + sqliteFree( pPager->aInStmt ); + pPager->aInStmt = 0; + } + for(pPg=pPager->pStmt; pPg; pPg=pNext){ + pNext = pPg->pNextStmt; + assert( pPg->inStmt ); + pPg->inStmt = 0; + pPg->pPrevStmt = pPg->pNextStmt = 0; + if( MEMDB ){ + PgHistory *pHist = PGHDR_TO_HIST(pPg, pPager); + sqliteFree(pHist->pStmt); + pHist->pStmt = 0; + } + } + pPager->stmtNRec = 0; + pPager->stmtInUse = 0; + pPager->pStmt = 0; + } + pPager->stmtAutoopen = 0; + return SQLITE_OK; +} + +/* +** Rollback a statement. +*/ +int sqlite3pager_stmt_rollback(Pager *pPager){ + int rc; + if( pPager->stmtInUse ){ + TRACE2("STMT-ROLLBACK %d\n", PAGERID(pPager)); + if( MEMDB ){ + PgHdr *pPg; + for(pPg=pPager->pStmt; pPg; pPg=pPg->pNextStmt){ + PgHistory *pHist = PGHDR_TO_HIST(pPg, pPager); + if( pHist->pStmt ){ + memcpy(PGHDR_TO_DATA(pPg), pHist->pStmt, pPager->pageSize); + sqliteFree(pHist->pStmt); + pHist->pStmt = 0; + } + } + pPager->dbSize = pPager->stmtSize; + memoryTruncate(pPager); + rc = SQLITE_OK; + }else{ + rc = pager_stmt_playback(pPager); + } + sqlite3pager_stmt_commit(pPager); + }else{ + rc = SQLITE_OK; + } + pPager->stmtAutoopen = 0; + return rc; +} + +/* +** Return the full pathname of the database file. +*/ +const char *sqlite3pager_filename(Pager *pPager){ + return pPager->zFilename; +} + +/* +** Return the directory of the database file. +*/ +const char *sqlite3pager_dirname(Pager *pPager){ + return pPager->zDirectory; +} + +/* +** Return the full pathname of the journal file. +*/ +const char *sqlite3pager_journalname(Pager *pPager){ + return pPager->zJournal; +} + +/* +** Return true if fsync() calls are disabled for this pager. Return FALSE +** if fsync()s are executed normally. +*/ +int sqlite3pager_nosync(Pager *pPager){ + return pPager->noSync; +} + +/* +** Set the codec for this pager +*/ +void sqlite3pager_set_codec( + Pager *pPager, + void (*xCodec)(void*,void*,Pgno,int), + void *pCodecArg +){ + pPager->xCodec = xCodec; + pPager->pCodecArg = pCodecArg; +} + +/* +** This routine is called to increment the database file change-counter, +** stored at byte 24 of the pager file. +*/ +static int pager_incr_changecounter(Pager *pPager){ + void *pPage; + PgHdr *pPgHdr; + u32 change_counter; + int rc; + + /* Open page 1 of the file for writing. */ + rc = sqlite3pager_get(pPager, 1, &pPage); + if( rc!=SQLITE_OK ) return rc; + rc = sqlite3pager_write(pPage); + if( rc!=SQLITE_OK ) return rc; + + /* Read the current value at byte 24. */ + pPgHdr = DATA_TO_PGHDR(pPage); + change_counter = retrieve32bits(pPgHdr, 24); + + /* Increment the value just read and write it back to byte 24. */ + change_counter++; + store32bits(change_counter, pPgHdr, 24); + + /* Release the page reference. */ + sqlite3pager_unref(pPage); + return SQLITE_OK; +} + +/* +** Sync the database file for the pager pPager. zMaster points to the name +** of a master journal file that should be written into the individual +** journal file. zMaster may be NULL, which is interpreted as no master +** journal (a single database transaction). +** +** This routine ensures that the journal is synced, all dirty pages written +** to the database file and the database file synced. The only thing that +** remains to commit the transaction is to delete the journal file (or +** master journal file if specified). +** +** Note that if zMaster==NULL, this does not overwrite a previous value +** passed to an sqlite3pager_sync() call. +** +** If parameter nTrunc is non-zero, then the pager file is truncated to +** nTrunc pages (this is used by auto-vacuum databases). +*/ +int sqlite3pager_sync(Pager *pPager, const char *zMaster, Pgno nTrunc){ + int rc = SQLITE_OK; + + TRACE4("DATABASE SYNC: File=%s zMaster=%s nTrunc=%d\n", + pPager->zFilename, zMaster, nTrunc); + + /* If this is an in-memory db, or no pages have been written to, or this + ** function has already been called, it is a no-op. + */ + if( pPager->state!=PAGER_SYNCED && !MEMDB && pPager->dirtyCache ){ + PgHdr *pPg; + assert( pPager->journalOpen ); + + /* If a master journal file name has already been written to the + ** journal file, then no sync is required. This happens when it is + ** written, then the process fails to upgrade from a RESERVED to an + ** EXCLUSIVE lock. The next time the process tries to commit the + ** transaction the m-j name will have already been written. + */ + if( !pPager->setMaster ){ + rc = pager_incr_changecounter(pPager); + if( rc!=SQLITE_OK ) goto sync_exit; +#ifndef SQLITE_OMIT_AUTOVACUUM + if( nTrunc!=0 ){ + /* If this transaction has made the database smaller, then all pages + ** being discarded by the truncation must be written to the journal + ** file. + */ + Pgno i; + void *pPage; + int iSkip = PAGER_MJ_PGNO(pPager); + for( i=nTrunc+1; i<=pPager->origDbSize; i++ ){ + if( !(pPager->aInJournal[i/8] & (1<<(i&7))) && i!=iSkip ){ + rc = sqlite3pager_get(pPager, i, &pPage); + if( rc!=SQLITE_OK ) goto sync_exit; + rc = sqlite3pager_write(pPage); + sqlite3pager_unref(pPage); + if( rc!=SQLITE_OK ) goto sync_exit; + } + } + } +#endif + rc = writeMasterJournal(pPager, zMaster); + if( rc!=SQLITE_OK ) goto sync_exit; + rc = syncJournal(pPager); + if( rc!=SQLITE_OK ) goto sync_exit; + } + +#ifndef SQLITE_OMIT_AUTOVACUUM + if( nTrunc!=0 ){ + rc = sqlite3pager_truncate(pPager, nTrunc); + if( rc!=SQLITE_OK ) goto sync_exit; + } +#endif + + /* Write all dirty pages to the database file */ + pPg = pager_get_all_dirty_pages(pPager); + rc = pager_write_pagelist(pPg); + if( rc!=SQLITE_OK ) goto sync_exit; + + /* Sync the database file. */ + if( !pPager->noSync ){ + rc = sqlite3OsSync(pPager->fd, 0); + } + + pPager->state = PAGER_SYNCED; + } + +sync_exit: + return rc; +} + +#ifndef SQLITE_OMIT_AUTOVACUUM +/* +** Move the page identified by pData to location pgno in the file. +** +** There must be no references to the current page pgno. If current page +** pgno is not already in the rollback journal, it is not written there by +** by this routine. The same applies to the page pData refers to on entry to +** this routine. +** +** References to the page refered to by pData remain valid. Updating any +** meta-data associated with page pData (i.e. data stored in the nExtra bytes +** allocated along with the page) is the responsibility of the caller. +** +** A transaction must be active when this routine is called. It used to be +** required that a statement transaction was not active, but this restriction +** has been removed (CREATE INDEX needs to move a page when a statement +** transaction is active). +*/ +int sqlite3pager_movepage(Pager *pPager, void *pData, Pgno pgno){ + PgHdr *pPg = DATA_TO_PGHDR(pData); + PgHdr *pPgOld; + int h; + Pgno needSyncPgno = 0; + + assert( pPg->nRef>0 ); + + TRACE5("MOVE %d page %d (needSync=%d) moves to %d\n", + PAGERID(pPager), pPg->pgno, pPg->needSync, pgno); + + if( pPg->needSync ){ + needSyncPgno = pPg->pgno; + assert( pPg->inJournal ); + assert( pPg->dirty ); + assert( pPager->needSync ); + } + + /* Unlink pPg from it's hash-chain */ + unlinkHashChain(pPager, pPg); + + /* If the cache contains a page with page-number pgno, remove it + ** from it's hash chain. Also, if the PgHdr.needSync was set for + ** page pgno before the 'move' operation, it needs to be retained + ** for the page moved there. + */ + pPgOld = pager_lookup(pPager, pgno); + if( pPgOld ){ + assert( pPgOld->nRef==0 ); + unlinkHashChain(pPager, pPgOld); + pPgOld->dirty = 0; + if( pPgOld->needSync ){ + assert( pPgOld->inJournal ); + pPg->inJournal = 1; + pPg->needSync = 1; + assert( pPager->needSync ); + } + } + + /* Change the page number for pPg and insert it into the new hash-chain. */ + pPg->pgno = pgno; + h = pager_hash(pgno); + if( pPager->aHash[h] ){ + assert( pPager->aHash[h]->pPrevHash==0 ); + pPager->aHash[h]->pPrevHash = pPg; + } + pPg->pNextHash = pPager->aHash[h]; + pPager->aHash[h] = pPg; + pPg->pPrevHash = 0; + + pPg->dirty = 1; + pPager->dirtyCache = 1; + + if( needSyncPgno ){ + /* If needSyncPgno is non-zero, then the journal file needs to be + ** sync()ed before any data is written to database file page needSyncPgno. + ** Currently, no such page exists in the page-cache and the + ** Pager.aInJournal bit has been set. This needs to be remedied by loading + ** the page into the pager-cache and setting the PgHdr.needSync flag. + ** + ** The sqlite3pager_get() call may cause the journal to sync. So make + ** sure the Pager.needSync flag is set too. + */ + int rc; + void *pNeedSync; + assert( pPager->needSync ); + rc = sqlite3pager_get(pPager, needSyncPgno, &pNeedSync); + if( rc!=SQLITE_OK ) return rc; + pPager->needSync = 1; + DATA_TO_PGHDR(pNeedSync)->needSync = 1; + DATA_TO_PGHDR(pNeedSync)->inJournal = 1; + DATA_TO_PGHDR(pNeedSync)->dirty = 1; + sqlite3pager_unref(pNeedSync); + } + + return SQLITE_OK; +} +#endif + +#if defined(SQLITE_DEBUG) || defined(SQLITE_TEST) +/* +** Return the current state of the file lock for the given pager. +** The return value is one of NO_LOCK, SHARED_LOCK, RESERVED_LOCK, +** PENDING_LOCK, or EXCLUSIVE_LOCK. +*/ +int sqlite3pager_lockstate(Pager *pPager){ + return sqlite3OsLockState(pPager->fd); +} +#endif + +#ifdef SQLITE_DEBUG +/* +** Print a listing of all referenced pages and their ref count. +*/ +void sqlite3pager_refdump(Pager *pPager){ + PgHdr *pPg; + for(pPg=pPager->pAll; pPg; pPg=pPg->pNextAll){ + if( pPg->nRef<=0 ) continue; + sqlite3DebugPrintf("PAGE %3d addr=%p nRef=%d\n", + pPg->pgno, PGHDR_TO_DATA(pPg), pPg->nRef); + } +} +#endif + +#endif /* SQLITE_OMIT_DISKIO */ Added: external/sqlite-source-3.3.4/pager.h ============================================================================== --- (empty file) +++ external/sqlite-source-3.3.4/pager.h Mon Apr 3 07:54:59 2006 @@ -0,0 +1,122 @@ +/* +** 2001 September 15 +** +** The author disclaims copyright to this source code. In place of +** a legal notice, here is a blessing: +** +** May you do good and not evil. +** May you find forgiveness for yourself and forgive others. +** May you share freely, never taking more than you give. +** +************************************************************************* +** This header file defines the interface that the sqlite page cache +** subsystem. The page cache subsystem reads and writes a file a page +** at a time and provides a journal for rollback. +** +** @(#) $Id: pager.h,v 1.49 2006/02/11 01:25:51 drh Exp $ +*/ + +#ifndef _PAGER_H_ +#define _PAGER_H_ + +/* +** The default size of a database page. +*/ +#ifndef SQLITE_DEFAULT_PAGE_SIZE +# define SQLITE_DEFAULT_PAGE_SIZE 1024 +#endif + +/* Maximum page size. The upper bound on this value is 32768. This a limit +** imposed by necessity of storing the value in a 2-byte unsigned integer +** and the fact that the page size must be a power of 2. +** +** This value is used to initialize certain arrays on the stack at +** various places in the code. On embedded machines where stack space +** is limited and the flexibility of having large pages is not needed, +** it makes good sense to reduce the maximum page size to something more +** reasonable, like 1024. +*/ +#ifndef SQLITE_MAX_PAGE_SIZE +# define SQLITE_MAX_PAGE_SIZE 32768 +#endif + +/* +** Maximum number of pages in one database. +*/ +#define SQLITE_MAX_PAGE 1073741823 + +/* +** The type used to represent a page number. The first page in a file +** is called page 1. 0 is used to represent "not a page". +*/ +typedef unsigned int Pgno; + +/* +** Each open file is managed by a separate instance of the "Pager" structure. +*/ +typedef struct Pager Pager; + +/* +** Allowed values for the flags parameter to sqlite3pager_open(). +** +** NOTE: This values must match the corresponding BTREE_ values in btree.h. +*/ +#define PAGER_OMIT_JOURNAL 0x0001 /* Do not use a rollback journal */ +#define PAGER_NO_READLOCK 0x0002 /* Omit readlocks on readonly files */ + + +/* +** See source code comments for a detailed description of the following +** routines: +*/ +int sqlite3pager_open(Pager **ppPager, const char *zFilename, + int nExtra, int flags); +void sqlite3pager_set_busyhandler(Pager*, BusyHandler *pBusyHandler); +void sqlite3pager_set_destructor(Pager*, void(*)(void*,int)); +void sqlite3pager_set_reiniter(Pager*, void(*)(void*,int)); +int sqlite3pager_set_pagesize(Pager*, int); +void sqlite3pager_read_fileheader(Pager*, int, unsigned char*); +void sqlite3pager_set_cachesize(Pager*, int); +int sqlite3pager_close(Pager *pPager); +int sqlite3pager_get(Pager *pPager, Pgno pgno, void **ppPage); +void *sqlite3pager_lookup(Pager *pPager, Pgno pgno); +int sqlite3pager_ref(void*); +int sqlite3pager_unref(void*); +Pgno sqlite3pager_pagenumber(void*); +int sqlite3pager_write(void*); +int sqlite3pager_iswriteable(void*); +int sqlite3pager_overwrite(Pager *pPager, Pgno pgno, void*); +int sqlite3pager_pagecount(Pager*); +int sqlite3pager_truncate(Pager*,Pgno); +int sqlite3pager_begin(void*, int exFlag); +int sqlite3pager_commit(Pager*); +int sqlite3pager_sync(Pager*,const char *zMaster, Pgno); +int sqlite3pager_rollback(Pager*); +int sqlite3pager_isreadonly(Pager*); +int sqlite3pager_stmt_begin(Pager*); +int sqlite3pager_stmt_commit(Pager*); +int sqlite3pager_stmt_rollback(Pager*); +void sqlite3pager_dont_rollback(void*); +void sqlite3pager_dont_write(Pager*, Pgno); +int *sqlite3pager_stats(Pager*); +void sqlite3pager_set_safety_level(Pager*,int,int); +const char *sqlite3pager_filename(Pager*); +const char *sqlite3pager_dirname(Pager*); +const char *sqlite3pager_journalname(Pager*); +int sqlite3pager_nosync(Pager*); +int sqlite3pager_rename(Pager*, const char *zNewName); +void sqlite3pager_set_codec(Pager*,void(*)(void*,void*,Pgno,int),void*); +int sqlite3pager_movepage(Pager*,void*,Pgno); +int sqlite3pager_reset(Pager*); +int sqlite3pager_release_memory(int); + +#if defined(SQLITE_DEBUG) || defined(SQLITE_TEST) +int sqlite3pager_lockstate(Pager*); +#endif + +#ifdef SQLITE_TEST +void sqlite3pager_refdump(Pager*); +int pager3_refinfo_enable; +#endif + +#endif /* _PAGER_H_ */ Added: external/sqlite-source-3.3.4/parse.c ============================================================================== --- (empty file) +++ external/sqlite-source-3.3.4/parse.c Mon Apr 3 07:54:59 2006 @@ -0,0 +1,3303 @@ +/* Driver template for the LEMON parser generator. +** The author disclaims copyright to this source code. +*/ +/* First off, code is include which follows the "include" declaration +** in the input file. */ +#include +#line 51 "parse.y" + +#include "sqliteInt.h" +#include "parse.h" + +/* +** An instance of this structure holds information about the +** LIMIT clause of a SELECT statement. +*/ +struct LimitVal { + Expr *pLimit; /* The LIMIT expression. NULL if there is no limit */ + Expr *pOffset; /* The OFFSET expression. NULL if there is none */ +}; + +/* +** An instance of this structure is used to store the LIKE, +** GLOB, NOT LIKE, and NOT GLOB operators. +*/ +struct LikeOp { + Token eOperator; /* "like" or "glob" or "regexp" */ + int not; /* True if the NOT keyword is present */ +}; + +/* +** An instance of the following structure describes the event of a +** TRIGGER. "a" is the event type, one of TK_UPDATE, TK_INSERT, +** TK_DELETE, or TK_INSTEAD. If the event is of the form +** +** UPDATE ON (a,b,c) +** +** Then the "b" IdList records the list "a,b,c". +*/ +struct TrigEvent { int a; IdList * b; }; + +/* +** An instance of this structure holds the ATTACH key and the key type. +*/ +struct AttachKey { int type; Token key; }; + +#line 48 "parse.c" +/* Next is all token values, in a form suitable for use by makeheaders. +** This section will be null unless lemon is run with the -m switch. +*/ +/* +** These constants (all generated automatically by the parser generator) +** specify the various kinds of tokens (terminals) that the parser +** understands. +** +** Each symbol here is a terminal symbol in the grammar. +*/ +/* Make sure the INTERFACE macro is defined. +*/ +#ifndef INTERFACE +# define INTERFACE 1 +#endif +/* The next thing included is series of defines which control +** various aspects of the generated parser. +** YYCODETYPE is the data type used for storing terminal +** and nonterminal numbers. "unsigned char" is +** used if there are fewer than 250 terminals +** and nonterminals. "int" is used otherwise. +** YYNOCODE is a number of type YYCODETYPE which corresponds +** to no legal terminal or nonterminal number. This +** number is used to fill in empty slots of the hash +** table. +** YYFALLBACK If defined, this indicates that one or more tokens +** have fall-back values which should be used if the +** original value of the token will not parse. +** YYACTIONTYPE is the data type used for storing terminal +** and nonterminal numbers. "unsigned char" is +** used if there are fewer than 250 rules and +** states combined. "int" is used otherwise. +** sqlite3ParserTOKENTYPE is the data type used for minor tokens given +** directly to the parser from the tokenizer. +** YYMINORTYPE is the data type used for all minor tokens. +** This is typically a union of many types, one of +** which is sqlite3ParserTOKENTYPE. The entry in the union +** for base tokens is called "yy0". +** YYSTACKDEPTH is the maximum depth of the parser's stack. +** sqlite3ParserARG_SDECL A static variable declaration for the %extra_argument +** sqlite3ParserARG_PDECL A parameter declaration for the %extra_argument +** sqlite3ParserARG_STORE Code to store %extra_argument into yypParser +** sqlite3ParserARG_FETCH Code to extract %extra_argument from yypParser +** YYNSTATE the combined number of states. +** YYNRULE the number of rules in the grammar +** YYERRORSYMBOL is the code number of the error symbol. If not +** defined, then do no error processing. +*/ +#define YYCODETYPE unsigned char +#define YYNOCODE 240 +#define YYACTIONTYPE unsigned short int +#define sqlite3ParserTOKENTYPE Token +typedef union { + sqlite3ParserTOKENTYPE yy0; + struct {int value; int mask;} yy13; + struct TrigEvent yy132; + IdList* yy160; + Expr* yy178; + int yy230; + Select* yy239; + TriggerStep* yy247; + struct LimitVal yy270; + SrcList* yy285; + Expr * yy292; + Token yy384; + struct LikeOp yy440; + ExprList* yy462; + int yy479; +} YYMINORTYPE; +#define YYSTACKDEPTH 100 +#define sqlite3ParserARG_SDECL Parse *pParse; +#define sqlite3ParserARG_PDECL ,Parse *pParse +#define sqlite3ParserARG_FETCH Parse *pParse = yypParser->pParse +#define sqlite3ParserARG_STORE yypParser->pParse = pParse +#define YYNSTATE 560 +#define YYNRULE 295 +#define YYERRORSYMBOL 137 +#define YYERRSYMDT yy479 +#define YYFALLBACK 1 +#define YY_NO_ACTION (YYNSTATE+YYNRULE+2) +#define YY_ACCEPT_ACTION (YYNSTATE+YYNRULE+1) +#define YY_ERROR_ACTION (YYNSTATE+YYNRULE) + +/* Next are that tables used to determine what action to take based on the +** current state and lookahead token. These tables are used to implement +** functions that take a state number and lookahead value and return an +** action integer. +** +** Suppose the action integer is N. Then the action is determined as +** follows +** +** 0 <= N < YYNSTATE Shift N. That is, push the lookahead +** token onto the stack and goto state N. +** +** YYNSTATE <= N < YYNSTATE+YYNRULE Reduce by rule N-YYNSTATE. +** +** N == YYNSTATE+YYNRULE A syntax error has occurred. +** +** N == YYNSTATE+YYNRULE+1 The parser accepts its input. +** +** N == YYNSTATE+YYNRULE+2 No such action. Denotes unused +** slots in the yy_action[] table. +** +** The action table is constructed as a single large table named yy_action[]. +** Given state S and lookahead X, the action is computed as +** +** yy_action[ yy_shift_ofst[S] + X ] +** +** If the index value yy_shift_ofst[S]+X is out of range or if the value +** yy_lookahead[yy_shift_ofst[S]+X] is not equal to X or if yy_shift_ofst[S] +** is equal to YY_SHIFT_USE_DFLT, it means that the action is not in the table +** and that yy_default[S] should be used instead. +** +** The formula above is for computing the action when the lookahead is +** a terminal symbol. If the lookahead is a non-terminal (as occurs after +** a reduce action) then the yy_reduce_ofst[] array is used in place of +** the yy_shift_ofst[] array and YY_REDUCE_USE_DFLT is used in place of +** YY_SHIFT_USE_DFLT. +** +** The following are the tables generated in this section: +** +** yy_action[] A single table containing all actions. +** yy_lookahead[] A table containing the lookahead for each entry in +** yy_action. Used to detect hash collisions. +** yy_shift_ofst[] For each state, the offset into yy_action for +** shifting terminals. +** yy_reduce_ofst[] For each state, the offset into yy_action for +** shifting non-terminals after a reduce. +** yy_default[] Default action for each state. +*/ +static const YYACTIONTYPE yy_action[] = { + /* 0 */ 279, 68, 283, 70, 148, 166, 546, 419, 62, 62, + /* 10 */ 62, 62, 202, 64, 64, 64, 64, 65, 65, 66, + /* 20 */ 66, 66, 67, 67, 548, 549, 432, 69, 64, 64, + /* 30 */ 64, 64, 65, 65, 66, 66, 66, 67, 68, 454, + /* 40 */ 70, 148, 499, 61, 59, 287, 440, 441, 437, 437, + /* 50 */ 63, 63, 62, 62, 62, 62, 501, 64, 64, 64, + /* 60 */ 64, 65, 65, 66, 66, 66, 67, 279, 371, 283, + /* 70 */ 419, 2, 377, 80, 158, 115, 220, 304, 225, 305, + /* 80 */ 170, 245, 856, 119, 559, 504, 204, 2, 246, 389, + /* 90 */ 496, 219, 22, 432, 514, 21, 419, 58, 493, 171, + /* 100 */ 64, 64, 64, 64, 65, 65, 66, 66, 66, 67, + /* 110 */ 61, 59, 287, 440, 441, 437, 437, 63, 63, 62, + /* 120 */ 62, 62, 62, 512, 64, 64, 64, 64, 65, 65, + /* 130 */ 66, 66, 66, 67, 279, 378, 379, 175, 202, 377, + /* 140 */ 330, 333, 334, 220, 304, 225, 305, 170, 245, 203, + /* 150 */ 146, 357, 335, 281, 377, 246, 55, 301, 373, 419, + /* 160 */ 432, 505, 92, 200, 530, 66, 66, 66, 67, 525, + /* 170 */ 192, 65, 65, 66, 66, 66, 67, 61, 59, 287, + /* 180 */ 440, 441, 437, 437, 63, 63, 62, 62, 62, 62, + /* 190 */ 433, 64, 64, 64, 64, 65, 65, 66, 66, 66, + /* 200 */ 67, 279, 378, 379, 411, 431, 110, 226, 427, 205, + /* 210 */ 435, 436, 308, 358, 261, 260, 175, 378, 379, 330, + /* 220 */ 333, 334, 372, 369, 202, 511, 480, 432, 547, 362, + /* 230 */ 466, 335, 510, 500, 410, 41, 276, 414, 434, 429, + /* 240 */ 503, 162, 233, 527, 61, 59, 287, 440, 441, 437, + /* 250 */ 437, 63, 63, 62, 62, 62, 62, 319, 64, 64, + /* 260 */ 64, 64, 65, 65, 66, 66, 66, 67, 279, 472, + /* 270 */ 416, 416, 416, 308, 322, 236, 308, 68, 308, 70, + /* 280 */ 148, 1, 308, 793, 308, 377, 68, 153, 70, 148, + /* 290 */ 149, 377, 325, 282, 432, 410, 35, 551, 410, 35, + /* 300 */ 410, 36, 427, 205, 410, 35, 410, 35, 286, 422, + /* 310 */ 423, 61, 59, 287, 440, 441, 437, 437, 63, 63, + /* 320 */ 62, 62, 62, 62, 411, 64, 64, 64, 64, 65, + /* 330 */ 65, 66, 66, 66, 67, 308, 504, 466, 290, 255, + /* 340 */ 279, 324, 485, 147, 237, 388, 21, 288, 378, 379, + /* 350 */ 451, 419, 232, 451, 378, 379, 308, 410, 28, 451, + /* 360 */ 175, 450, 486, 330, 333, 334, 432, 215, 347, 145, + /* 370 */ 513, 204, 350, 186, 168, 335, 238, 411, 410, 41, + /* 380 */ 256, 462, 76, 61, 59, 287, 440, 441, 437, 437, + /* 390 */ 63, 63, 62, 62, 62, 62, 309, 64, 64, 64, + /* 400 */ 64, 65, 65, 66, 66, 66, 67, 411, 411, 186, + /* 410 */ 396, 308, 279, 291, 419, 338, 476, 308, 390, 234, + /* 420 */ 169, 154, 397, 475, 396, 327, 493, 311, 422, 423, + /* 430 */ 444, 377, 356, 410, 49, 398, 397, 394, 432, 410, + /* 440 */ 49, 502, 171, 411, 429, 312, 162, 395, 351, 398, + /* 450 */ 497, 318, 470, 352, 79, 61, 59, 287, 440, 441, + /* 460 */ 437, 437, 63, 63, 62, 62, 62, 62, 356, 64, + /* 470 */ 64, 64, 64, 65, 65, 66, 66, 66, 67, 279, + /* 480 */ 298, 445, 376, 479, 532, 405, 299, 11, 504, 352, + /* 490 */ 204, 377, 406, 377, 378, 379, 281, 556, 21, 491, + /* 500 */ 491, 246, 560, 372, 369, 432, 392, 393, 314, 123, + /* 510 */ 443, 443, 166, 289, 419, 314, 116, 443, 443, 251, + /* 520 */ 264, 463, 61, 59, 287, 440, 441, 437, 437, 63, + /* 530 */ 63, 62, 62, 62, 62, 292, 64, 64, 64, 64, + /* 540 */ 65, 65, 66, 66, 66, 67, 279, 459, 328, 474, + /* 550 */ 498, 308, 202, 308, 378, 379, 378, 379, 181, 131, + /* 560 */ 179, 265, 308, 5, 308, 363, 314, 355, 443, 443, + /* 570 */ 410, 3, 432, 410, 29, 410, 24, 419, 243, 244, + /* 580 */ 380, 381, 382, 404, 410, 33, 410, 54, 466, 61, + /* 590 */ 59, 287, 440, 441, 437, 437, 63, 63, 62, 62, + /* 600 */ 62, 62, 308, 64, 64, 64, 64, 65, 65, 66, + /* 610 */ 66, 66, 67, 279, 521, 344, 521, 249, 308, 491, + /* 620 */ 308, 470, 308, 470, 410, 25, 308, 240, 308, 314, + /* 630 */ 308, 443, 443, 213, 172, 173, 174, 142, 266, 432, + /* 640 */ 410, 52, 410, 97, 410, 94, 528, 393, 410, 99, + /* 650 */ 410, 100, 410, 111, 212, 255, 61, 59, 287, 440, + /* 660 */ 441, 437, 437, 63, 63, 62, 62, 62, 62, 308, + /* 670 */ 64, 64, 64, 64, 65, 65, 66, 66, 66, 67, + /* 680 */ 279, 308, 345, 188, 297, 91, 308, 491, 308, 415, + /* 690 */ 308, 410, 112, 308, 428, 308, 537, 308, 244, 165, + /* 700 */ 154, 409, 355, 410, 18, 408, 432, 320, 410, 98, + /* 710 */ 410, 34, 410, 95, 313, 410, 53, 410, 113, 410, + /* 720 */ 114, 255, 293, 61, 59, 287, 440, 441, 437, 437, + /* 730 */ 63, 63, 62, 62, 62, 62, 308, 64, 64, 64, + /* 740 */ 64, 65, 65, 66, 66, 66, 67, 279, 308, 491, + /* 750 */ 491, 523, 308, 452, 308, 522, 308, 461, 410, 26, + /* 760 */ 308, 75, 539, 77, 308, 460, 244, 346, 214, 465, + /* 770 */ 410, 37, 469, 432, 410, 38, 410, 27, 410, 39, + /* 780 */ 242, 82, 410, 40, 294, 296, 410, 42, 438, 329, + /* 790 */ 61, 59, 287, 440, 441, 437, 437, 63, 63, 62, + /* 800 */ 62, 62, 62, 308, 64, 64, 64, 64, 65, 65, + /* 810 */ 66, 66, 66, 67, 279, 308, 409, 190, 221, 308, + /* 820 */ 408, 308, 152, 308, 159, 410, 43, 308, 244, 244, + /* 830 */ 222, 20, 308, 139, 425, 425, 481, 410, 44, 482, + /* 840 */ 432, 410, 30, 410, 31, 410, 45, 487, 461, 410, + /* 850 */ 46, 411, 506, 255, 410, 47, 488, 61, 71, 287, + /* 860 */ 440, 441, 437, 437, 63, 63, 62, 62, 62, 62, + /* 870 */ 308, 64, 64, 64, 64, 65, 65, 66, 66, 66, + /* 880 */ 67, 279, 308, 401, 402, 250, 308, 193, 308, 420, + /* 890 */ 308, 23, 410, 48, 540, 449, 255, 14, 468, 477, + /* 900 */ 167, 14, 484, 483, 410, 32, 252, 432, 410, 12, + /* 910 */ 410, 50, 410, 51, 255, 255, 594, 255, 255, 150, + /* 920 */ 489, 411, 123, 253, 279, 59, 287, 440, 441, 437, + /* 930 */ 437, 63, 63, 62, 62, 62, 62, 541, 64, 64, + /* 940 */ 64, 64, 65, 65, 66, 66, 66, 67, 254, 248, + /* 950 */ 432, 123, 337, 411, 123, 267, 269, 196, 361, 366, + /* 960 */ 183, 177, 180, 519, 520, 526, 534, 123, 167, 287, + /* 970 */ 440, 441, 437, 437, 63, 63, 62, 62, 62, 62, + /* 980 */ 342, 64, 64, 64, 64, 65, 65, 66, 66, 66, + /* 990 */ 67, 72, 315, 259, 4, 411, 411, 535, 285, 89, + /* 1000 */ 544, 349, 89, 353, 354, 19, 310, 72, 315, 368, + /* 1010 */ 4, 386, 262, 263, 285, 223, 545, 270, 364, 273, + /* 1020 */ 274, 141, 310, 317, 227, 316, 555, 424, 426, 480, + /* 1030 */ 455, 458, 490, 431, 332, 492, 533, 157, 543, 317, + /* 1040 */ 375, 383, 384, 385, 8, 302, 303, 391, 284, 431, + /* 1050 */ 404, 399, 74, 73, 224, 403, 407, 82, 323, 321, + /* 1060 */ 72, 306, 307, 400, 231, 414, 81, 206, 74, 73, + /* 1070 */ 473, 57, 78, 164, 453, 412, 72, 306, 307, 72, + /* 1080 */ 315, 414, 4, 228, 202, 229, 285, 235, 230, 456, + /* 1090 */ 457, 413, 207, 120, 310, 83, 326, 102, 416, 416, + /* 1100 */ 416, 417, 418, 13, 239, 495, 467, 241, 277, 208, + /* 1110 */ 471, 317, 494, 210, 416, 416, 416, 417, 418, 13, + /* 1120 */ 211, 431, 156, 278, 339, 507, 508, 216, 217, 218, + /* 1130 */ 106, 509, 515, 178, 343, 84, 341, 182, 517, 456, + /* 1140 */ 74, 73, 86, 198, 518, 271, 257, 184, 72, 306, + /* 1150 */ 307, 348, 272, 414, 118, 529, 187, 127, 536, 359, + /* 1160 */ 128, 136, 129, 542, 195, 130, 530, 133, 300, 552, + /* 1170 */ 553, 194, 137, 197, 431, 90, 554, 557, 96, 209, + /* 1180 */ 101, 374, 387, 117, 201, 56, 416, 416, 416, 417, + /* 1190 */ 418, 13, 93, 143, 144, 595, 596, 109, 160, 161, + /* 1200 */ 60, 439, 500, 421, 430, 442, 414, 138, 446, 151, + /* 1210 */ 6, 447, 155, 448, 163, 360, 268, 260, 15, 7, + /* 1220 */ 14, 280, 121, 464, 122, 478, 202, 103, 104, 331, + /* 1230 */ 247, 85, 105, 336, 222, 176, 340, 140, 516, 416, + /* 1240 */ 416, 416, 124, 295, 125, 167, 524, 258, 107, 185, + /* 1250 */ 365, 9, 531, 10, 126, 189, 16, 538, 191, 132, + /* 1260 */ 134, 87, 88, 135, 17, 108, 275, 550, 367, 199, + /* 1270 */ 370, 536, 558, +}; +static const YYCODETYPE yy_lookahead[] = { + /* 0 */ 16, 216, 16, 218, 219, 21, 146, 23, 68, 69, + /* 10 */ 70, 71, 109, 73, 74, 75, 76, 77, 78, 79, + /* 20 */ 80, 81, 82, 82, 164, 165, 42, 72, 73, 74, + /* 30 */ 75, 76, 77, 78, 79, 80, 81, 82, 216, 217, + /* 40 */ 218, 219, 168, 59, 60, 61, 62, 63, 64, 65, + /* 50 */ 66, 67, 68, 69, 70, 71, 168, 73, 74, 75, + /* 60 */ 76, 77, 78, 79, 80, 81, 82, 16, 140, 16, + /* 70 */ 86, 143, 23, 22, 88, 89, 90, 91, 92, 93, + /* 80 */ 94, 95, 138, 139, 140, 146, 226, 143, 102, 166, + /* 90 */ 167, 152, 19, 42, 155, 156, 23, 46, 175, 43, + /* 100 */ 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, + /* 110 */ 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, + /* 120 */ 69, 70, 71, 180, 73, 74, 75, 76, 77, 78, + /* 130 */ 79, 80, 81, 82, 16, 86, 87, 88, 109, 23, + /* 140 */ 91, 92, 93, 90, 91, 92, 93, 94, 95, 191, + /* 150 */ 22, 122, 103, 97, 23, 102, 198, 141, 142, 86, + /* 160 */ 42, 180, 44, 147, 49, 79, 80, 81, 82, 18, + /* 170 */ 154, 77, 78, 79, 80, 81, 82, 59, 60, 61, + /* 180 */ 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, + /* 190 */ 42, 73, 74, 75, 76, 77, 78, 79, 80, 81, + /* 200 */ 82, 16, 86, 87, 188, 58, 21, 189, 77, 78, + /* 210 */ 62, 63, 146, 98, 99, 100, 88, 86, 87, 91, + /* 220 */ 92, 93, 1, 2, 109, 175, 176, 42, 97, 213, + /* 230 */ 160, 103, 182, 86, 168, 169, 157, 90, 90, 160, + /* 240 */ 161, 162, 146, 92, 59, 60, 61, 62, 63, 64, + /* 250 */ 65, 66, 67, 68, 69, 70, 71, 185, 73, 74, + /* 260 */ 75, 76, 77, 78, 79, 80, 81, 82, 16, 199, + /* 270 */ 123, 124, 125, 146, 208, 209, 146, 216, 146, 218, + /* 280 */ 219, 19, 146, 132, 146, 23, 216, 146, 218, 219, + /* 290 */ 154, 23, 146, 149, 42, 168, 169, 236, 168, 169, + /* 300 */ 168, 169, 77, 78, 168, 169, 168, 169, 163, 164, + /* 310 */ 165, 59, 60, 61, 62, 63, 64, 65, 66, 67, + /* 320 */ 68, 69, 70, 71, 188, 73, 74, 75, 76, 77, + /* 330 */ 78, 79, 80, 81, 82, 146, 146, 160, 211, 146, + /* 340 */ 16, 211, 30, 154, 146, 155, 156, 211, 86, 87, + /* 350 */ 223, 23, 220, 223, 86, 87, 146, 168, 169, 223, + /* 360 */ 88, 223, 50, 91, 92, 93, 42, 144, 224, 179, + /* 370 */ 180, 226, 228, 154, 154, 103, 199, 188, 168, 169, + /* 380 */ 187, 113, 130, 59, 60, 61, 62, 63, 64, 65, + /* 390 */ 66, 67, 68, 69, 70, 71, 146, 73, 74, 75, + /* 400 */ 76, 77, 78, 79, 80, 81, 82, 188, 188, 154, + /* 410 */ 12, 146, 16, 101, 86, 16, 20, 146, 167, 209, + /* 420 */ 200, 201, 24, 20, 12, 205, 175, 163, 164, 165, + /* 430 */ 20, 23, 213, 168, 169, 37, 24, 39, 42, 168, + /* 440 */ 169, 159, 43, 188, 160, 161, 162, 49, 229, 37, + /* 450 */ 168, 39, 146, 234, 130, 59, 60, 61, 62, 63, + /* 460 */ 64, 65, 66, 67, 68, 69, 70, 71, 213, 73, + /* 470 */ 74, 75, 76, 77, 78, 79, 80, 81, 82, 16, + /* 480 */ 215, 20, 146, 20, 229, 27, 215, 19, 146, 234, + /* 490 */ 226, 23, 34, 23, 86, 87, 97, 155, 156, 146, + /* 500 */ 146, 102, 0, 1, 2, 42, 184, 185, 105, 22, + /* 510 */ 107, 108, 21, 207, 23, 105, 146, 107, 108, 14, + /* 520 */ 14, 113, 59, 60, 61, 62, 63, 64, 65, 66, + /* 530 */ 67, 68, 69, 70, 71, 181, 73, 74, 75, 76, + /* 540 */ 77, 78, 79, 80, 81, 82, 16, 22, 146, 79, + /* 550 */ 20, 146, 109, 146, 86, 87, 86, 87, 53, 53, + /* 560 */ 55, 55, 146, 190, 146, 122, 105, 146, 107, 108, + /* 570 */ 168, 169, 42, 168, 169, 168, 169, 86, 225, 225, + /* 580 */ 7, 8, 9, 96, 168, 169, 168, 169, 160, 59, + /* 590 */ 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, + /* 600 */ 70, 71, 146, 73, 74, 75, 76, 77, 78, 79, + /* 610 */ 80, 81, 82, 16, 98, 99, 100, 20, 146, 146, + /* 620 */ 146, 146, 146, 146, 168, 169, 146, 199, 146, 105, + /* 630 */ 146, 107, 108, 212, 98, 99, 100, 112, 132, 42, + /* 640 */ 168, 169, 168, 169, 168, 169, 184, 185, 168, 169, + /* 650 */ 168, 169, 168, 169, 181, 146, 59, 60, 61, 62, + /* 660 */ 63, 64, 65, 66, 67, 68, 69, 70, 71, 146, + /* 670 */ 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, + /* 680 */ 16, 146, 207, 22, 207, 21, 146, 146, 146, 146, + /* 690 */ 146, 168, 169, 146, 160, 146, 187, 146, 225, 200, + /* 700 */ 201, 106, 146, 168, 169, 110, 42, 146, 168, 169, + /* 710 */ 168, 169, 168, 169, 16, 168, 169, 168, 169, 168, + /* 720 */ 169, 146, 181, 59, 60, 61, 62, 63, 64, 65, + /* 730 */ 66, 67, 68, 69, 70, 71, 146, 73, 74, 75, + /* 740 */ 76, 77, 78, 79, 80, 81, 82, 16, 146, 146, + /* 750 */ 146, 25, 146, 146, 146, 29, 146, 22, 168, 169, + /* 760 */ 146, 129, 187, 131, 146, 202, 225, 41, 212, 146, + /* 770 */ 168, 169, 146, 42, 168, 169, 168, 169, 168, 169, + /* 780 */ 146, 120, 168, 169, 181, 181, 168, 169, 90, 79, + /* 790 */ 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, + /* 800 */ 69, 70, 71, 146, 73, 74, 75, 76, 77, 78, + /* 810 */ 79, 80, 81, 82, 16, 146, 106, 154, 90, 146, + /* 820 */ 110, 146, 87, 146, 19, 168, 169, 146, 225, 225, + /* 830 */ 102, 19, 146, 21, 123, 124, 146, 168, 169, 177, + /* 840 */ 42, 168, 169, 168, 169, 168, 169, 177, 113, 168, + /* 850 */ 169, 188, 146, 146, 168, 169, 177, 59, 60, 61, + /* 860 */ 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, + /* 870 */ 146, 73, 74, 75, 76, 77, 78, 79, 80, 81, + /* 880 */ 82, 16, 146, 7, 8, 146, 146, 154, 146, 20, + /* 890 */ 146, 22, 168, 169, 187, 20, 146, 22, 20, 20, + /* 900 */ 22, 22, 89, 90, 168, 169, 146, 42, 168, 169, + /* 910 */ 168, 169, 168, 169, 146, 146, 111, 146, 146, 154, + /* 920 */ 20, 188, 22, 146, 16, 60, 61, 62, 63, 64, + /* 930 */ 65, 66, 67, 68, 69, 70, 71, 187, 73, 74, + /* 940 */ 75, 76, 77, 78, 79, 80, 81, 82, 146, 20, + /* 950 */ 42, 22, 20, 188, 22, 187, 187, 19, 187, 187, + /* 960 */ 230, 154, 154, 51, 52, 20, 20, 22, 22, 61, + /* 970 */ 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, + /* 980 */ 231, 73, 74, 75, 76, 77, 78, 79, 80, 81, + /* 990 */ 82, 16, 17, 146, 19, 188, 188, 20, 23, 22, + /* 1000 */ 20, 146, 22, 146, 146, 67, 31, 16, 17, 237, + /* 1010 */ 19, 148, 146, 146, 23, 171, 146, 146, 146, 146, + /* 1020 */ 146, 190, 31, 48, 192, 222, 146, 227, 227, 176, + /* 1030 */ 171, 171, 171, 58, 172, 171, 193, 6, 193, 48, + /* 1040 */ 145, 145, 145, 145, 22, 153, 97, 170, 40, 58, + /* 1050 */ 96, 170, 77, 78, 170, 172, 170, 120, 117, 115, + /* 1060 */ 85, 86, 87, 178, 196, 90, 118, 221, 77, 78, + /* 1070 */ 79, 119, 129, 111, 151, 188, 85, 86, 87, 16, + /* 1080 */ 17, 90, 19, 193, 109, 194, 23, 95, 195, 23, + /* 1090 */ 159, 197, 210, 151, 31, 97, 114, 19, 123, 124, + /* 1100 */ 125, 126, 127, 128, 203, 178, 204, 203, 173, 210, + /* 1110 */ 204, 48, 159, 210, 123, 124, 125, 126, 127, 128, + /* 1120 */ 210, 58, 5, 173, 15, 170, 170, 10, 11, 12, + /* 1130 */ 13, 170, 151, 150, 38, 19, 151, 151, 151, 23, + /* 1140 */ 77, 78, 129, 26, 233, 28, 232, 150, 85, 86, + /* 1150 */ 87, 151, 35, 90, 59, 183, 183, 19, 193, 15, + /* 1160 */ 186, 214, 186, 193, 47, 186, 49, 183, 151, 33, + /* 1170 */ 151, 54, 214, 56, 58, 235, 151, 136, 158, 174, + /* 1180 */ 174, 1, 20, 32, 44, 19, 123, 124, 125, 126, + /* 1190 */ 127, 128, 235, 77, 78, 111, 111, 238, 111, 111, + /* 1200 */ 19, 90, 86, 20, 20, 106, 90, 19, 11, 19, + /* 1210 */ 116, 20, 111, 20, 22, 98, 99, 100, 22, 116, + /* 1220 */ 22, 104, 19, 113, 20, 20, 109, 19, 19, 44, + /* 1230 */ 20, 19, 19, 44, 102, 94, 16, 21, 17, 123, + /* 1240 */ 124, 125, 97, 36, 45, 22, 45, 132, 19, 97, + /* 1250 */ 133, 5, 11, 1, 101, 121, 19, 17, 112, 112, + /* 1260 */ 101, 67, 67, 121, 19, 14, 135, 20, 57, 134, + /* 1270 */ 3, 239, 4, +}; +#define YY_SHIFT_USE_DFLT (-98) +#define YY_SHIFT_MAX 370 +static const short yy_shift_ofst[] = { + /* 0 */ 221, 975, 1117, -16, 975, 1063, 1063, 1063, 49, 115, + /* 10 */ 115, -97, 118, 1063, 1063, 1063, 1063, 1063, -45, 131, + /* 20 */ 116, 328, 225, 225, 51, 185, 252, 324, 396, 463, + /* 30 */ 530, 597, 664, 731, 798, 731, 731, 731, 731, 731, + /* 40 */ 731, 731, 731, 731, 731, 731, 731, 731, 731, 731, + /* 50 */ 731, 731, 865, 908, 908, 991, 1063, 1063, 1063, 1063, + /* 60 */ 1063, 1063, 1063, 1063, 1063, 1063, 1063, 1063, 1063, 1063, + /* 70 */ 1063, 1063, 1063, 1063, 1063, 1063, 1063, 1063, 1063, 1063, + /* 80 */ 1063, 1063, 1063, 1063, 1063, 1063, 1063, 1063, 1063, 1063, + /* 90 */ 1063, 1063, 1063, 1063, -60, -60, -14, 27, 27, 94, + /* 100 */ 86, 399, 116, 116, 116, 116, 151, 116, 116, 116, + /* 110 */ 328, -59, -98, -98, -98, 1116, 53, 398, 398, 502, + /* 120 */ 491, 116, 491, 116, 116, 116, 116, 116, 116, 116, + /* 130 */ 116, 116, 116, 116, 116, 116, 29, 443, -97, -97, + /* 140 */ -97, -98, -98, 147, 147, 128, 272, 403, 262, 410, + /* 150 */ 461, 412, 268, 408, 468, 470, 573, 116, 116, 710, + /* 160 */ 116, 116, 73, 116, 116, 735, 116, 116, 524, 735, + /* 170 */ 116, 116, 312, 312, 312, 116, 116, 524, 116, 116, + /* 180 */ 524, 116, 726, 516, 116, 116, 524, 116, 116, 116, + /* 190 */ 524, 116, 524, 524, 116, 116, 116, 116, 116, 116, + /* 200 */ 812, 458, 595, 525, 711, 711, 632, 458, 458, 56, + /* 210 */ 458, 458, 487, 661, 661, 1031, 1031, 1031, 1031, 1022, + /* 220 */ 949, 949, 1008, 949, 954, 949, -97, 937, 941, 948, + /* 230 */ 944, 952, 943, 962, 992, 1066, 992, 962, 998, 982, + /* 240 */ 998, 982, 1078, 992, 992, 1066, 1008, 949, 949, 949, + /* 250 */ 1078, 1109, 962, 962, 962, 962, 1096, 1013, 1109, 962, + /* 260 */ 1095, 1095, 1138, 937, 1144, 1144, 1144, 937, 1095, 1138, + /* 270 */ 962, 1136, 1136, 962, 962, 1041, -98, -98, -98, 148, + /* 280 */ 506, 536, 505, 728, 876, 805, 869, 698, 875, 878, + /* 290 */ 879, 813, 900, 929, 932, 912, 945, 946, 977, 980, + /* 300 */ 938, 1180, 1162, 1151, 1140, 1166, 1084, 1085, 1087, 1088, + /* 310 */ 1181, 1183, 1184, 1111, 1099, 1188, 1197, 1190, 1191, 1192, + /* 320 */ 1193, 1094, 1196, 1103, 1198, 1110, 1203, 1204, 1101, 1205, + /* 330 */ 1185, 1208, 1210, 1209, 1212, 1189, 1213, 1141, 1132, 1220, + /* 340 */ 1221, 1216, 1145, 1207, 1199, 1223, 1201, 1115, 1152, 1229, + /* 350 */ 1246, 1241, 1252, 1153, 1194, 1195, 1134, 1237, 1146, 1240, + /* 360 */ 1147, 1159, 1142, 1245, 1247, 1251, 1211, 1135, 1131, 1267, + /* 370 */ 1268, +}; +#define YY_REDUCE_USE_DFLT (-216) +#define YY_REDUCE_MAX 278 +static const short yy_reduce_ofst[] = { + /* 0 */ -56, 136, 16, 70, 189, 127, 66, 130, 190, 219, + /* 10 */ 255, 220, 61, 132, 138, 210, 265, 271, -178, -140, + /* 20 */ -61, 79, 145, 264, -215, -215, -215, -215, -215, -215, + /* 30 */ -215, -215, -215, -215, -215, -215, -215, -215, -215, -215, + /* 40 */ -215, -215, -215, -215, -215, -215, -215, -215, -215, -215, + /* 50 */ -215, -215, -215, -215, -215, 402, 405, 407, 416, 418, + /* 60 */ 456, 472, 474, 476, 480, 482, 484, 523, 535, 540, + /* 70 */ 542, 544, 547, 549, 551, 590, 602, 606, 608, 610, + /* 80 */ 614, 618, 657, 669, 673, 675, 677, 681, 686, 724, + /* 90 */ 736, 740, 742, 744, -215, -215, -77, -215, -215, -215, + /* 100 */ -215, 50, 354, 473, 541, 603, 144, 604, 772, 342, + /* 110 */ 284, -215, -215, -215, -215, 282, 251, 322, 462, -72, + /* 120 */ 177, 306, 428, 353, 193, 475, 421, 477, 509, 575, + /* 130 */ 707, 750, 768, 771, 556, 769, 663, 733, 765, 807, + /* 140 */ 808, -42, 499, -126, -112, -57, -19, 18, 96, 18, + /* 150 */ 18, 72, 141, 146, 198, 250, 223, 336, 370, 373, + /* 160 */ 250, 543, 534, 561, 607, 563, 623, 626, 18, 563, + /* 170 */ 634, 690, 662, 670, 679, 706, 739, 18, 760, 777, + /* 180 */ 18, 802, 730, 749, 847, 855, 18, 857, 858, 866, + /* 190 */ 18, 867, 18, 18, 870, 871, 872, 873, 874, 880, + /* 200 */ 863, 844, 831, 832, 800, 801, 803, 859, 860, 853, + /* 210 */ 861, 864, 862, 843, 845, 895, 896, 897, 898, 892, + /* 220 */ 877, 881, 885, 884, 883, 886, 887, 890, 891, 893, + /* 230 */ 868, 894, 846, 923, 882, 931, 899, 942, 901, 902, + /* 240 */ 904, 906, 935, 903, 910, 953, 927, 955, 956, 961, + /* 250 */ 950, 983, 981, 985, 986, 987, 914, 911, 997, 1000, + /* 260 */ 972, 973, 947, 965, 974, 976, 979, 970, 984, 958, + /* 270 */ 1017, 940, 957, 1019, 1025, 959, 1020, 1005, 1006, +}; +static const YYACTIONTYPE yy_default[] = { + /* 0 */ 566, 790, 855, 681, 855, 790, 855, 790, 855, 828, + /* 10 */ 828, 685, 841, 786, 790, 855, 855, 855, 761, 812, + /* 20 */ 855, 597, 812, 812, 716, 855, 855, 855, 855, 855, + /* 30 */ 855, 855, 855, 717, 855, 789, 785, 781, 783, 782, + /* 40 */ 718, 705, 714, 721, 697, 826, 723, 724, 729, 730, + /* 50 */ 842, 845, 751, 767, 750, 855, 855, 855, 855, 855, + /* 60 */ 855, 855, 855, 855, 855, 855, 855, 855, 855, 855, + /* 70 */ 855, 855, 855, 855, 855, 855, 855, 855, 855, 855, + /* 80 */ 855, 855, 855, 855, 855, 855, 855, 855, 855, 855, + /* 90 */ 855, 855, 855, 855, 753, 772, 590, 752, 760, 754, + /* 100 */ 755, 650, 855, 855, 855, 855, 585, 855, 855, 855, + /* 110 */ 855, 756, 757, 768, 769, 855, 855, 855, 855, 566, + /* 120 */ 681, 855, 681, 855, 855, 855, 855, 855, 855, 855, + /* 130 */ 855, 855, 855, 855, 855, 855, 855, 855, 855, 855, + /* 140 */ 855, 675, 685, 855, 855, 641, 855, 855, 855, 855, + /* 150 */ 855, 855, 855, 855, 855, 855, 573, 571, 855, 673, + /* 160 */ 855, 855, 599, 855, 855, 683, 855, 855, 688, 689, + /* 170 */ 855, 855, 855, 855, 855, 855, 855, 587, 855, 855, + /* 180 */ 662, 855, 818, 855, 855, 855, 833, 855, 855, 855, + /* 190 */ 831, 855, 664, 726, 800, 855, 855, 846, 848, 855, + /* 200 */ 855, 708, 673, 682, 855, 855, 784, 708, 708, 620, + /* 210 */ 708, 708, 623, 720, 720, 570, 570, 570, 570, 640, + /* 220 */ 652, 652, 637, 652, 623, 652, 855, 720, 711, 713, + /* 230 */ 701, 715, 855, 690, 709, 855, 709, 690, 698, 700, + /* 240 */ 698, 700, 794, 709, 709, 855, 637, 652, 652, 652, + /* 250 */ 794, 582, 690, 690, 690, 690, 822, 825, 582, 690, + /* 260 */ 654, 654, 731, 720, 661, 661, 661, 720, 654, 731, + /* 270 */ 690, 844, 844, 690, 690, 853, 607, 625, 625, 855, + /* 280 */ 855, 855, 855, 855, 855, 738, 855, 855, 855, 855, + /* 290 */ 855, 855, 855, 855, 855, 855, 855, 855, 855, 855, + /* 300 */ 807, 855, 855, 855, 855, 855, 743, 739, 855, 740, + /* 310 */ 855, 855, 855, 855, 667, 855, 855, 855, 855, 855, + /* 320 */ 855, 855, 702, 855, 712, 855, 855, 855, 855, 855, + /* 330 */ 855, 855, 855, 855, 855, 855, 855, 855, 855, 855, + /* 340 */ 855, 855, 855, 855, 820, 821, 855, 855, 855, 855, + /* 350 */ 855, 855, 855, 855, 855, 855, 855, 855, 855, 855, + /* 360 */ 855, 855, 855, 855, 855, 855, 852, 855, 855, 567, + /* 370 */ 855, 561, 564, 563, 565, 569, 572, 594, 595, 596, + /* 380 */ 574, 575, 576, 577, 578, 579, 580, 586, 588, 606, + /* 390 */ 608, 615, 653, 656, 657, 658, 836, 837, 838, 616, + /* 400 */ 635, 638, 639, 617, 624, 706, 707, 618, 671, 672, + /* 410 */ 735, 665, 666, 670, 737, 741, 742, 744, 745, 593, + /* 420 */ 600, 601, 604, 605, 808, 810, 809, 811, 603, 602, + /* 430 */ 746, 749, 758, 759, 765, 771, 774, 763, 764, 766, + /* 440 */ 770, 773, 668, 669, 777, 779, 780, 834, 835, 775, + /* 450 */ 787, 788, 691, 778, 762, 703, 592, 710, 704, 674, + /* 460 */ 684, 693, 694, 695, 696, 679, 680, 686, 699, 733, + /* 470 */ 734, 687, 676, 677, 678, 776, 736, 747, 748, 619, + /* 480 */ 626, 627, 628, 631, 632, 633, 634, 629, 630, 795, + /* 490 */ 796, 798, 797, 621, 622, 636, 609, 610, 611, 612, + /* 500 */ 743, 613, 614, 598, 591, 642, 645, 646, 647, 648, + /* 510 */ 649, 651, 643, 644, 589, 581, 583, 692, 814, 823, + /* 520 */ 824, 819, 815, 816, 817, 584, 791, 792, 655, 727, + /* 530 */ 728, 813, 827, 829, 732, 830, 832, 659, 660, 663, + /* 540 */ 799, 839, 719, 722, 725, 801, 802, 803, 804, 805, + /* 550 */ 806, 840, 843, 847, 849, 850, 851, 854, 568, 562, +}; +#define YY_SZ_ACTTAB (sizeof(yy_action)/sizeof(yy_action[0])) + +/* The next table maps tokens into fallback tokens. If a construct +** like the following: +** +** %fallback ID X Y Z. +** +** appears in the grammer, then ID becomes a fallback token for X, Y, +** and Z. Whenever one of the tokens X, Y, or Z is input to the parser +** but it does not parse, the type of the token is changed to ID and +** the parse is retried before an error is thrown. +*/ +#ifdef YYFALLBACK +static const YYCODETYPE yyFallback[] = { + 0, /* $ => nothing */ + 0, /* SEMI => nothing */ + 23, /* EXPLAIN => ID */ + 23, /* QUERY => ID */ + 23, /* PLAN => ID */ + 23, /* BEGIN => ID */ + 0, /* TRANSACTION => nothing */ + 23, /* DEFERRED => ID */ + 23, /* IMMEDIATE => ID */ + 23, /* EXCLUSIVE => ID */ + 0, /* COMMIT => nothing */ + 23, /* END => ID */ + 0, /* ROLLBACK => nothing */ + 0, /* CREATE => nothing */ + 0, /* TABLE => nothing */ + 23, /* IF => ID */ + 0, /* NOT => nothing */ + 0, /* EXISTS => nothing */ + 23, /* TEMP => ID */ + 0, /* LP => nothing */ + 0, /* RP => nothing */ + 0, /* AS => nothing */ + 0, /* COMMA => nothing */ + 0, /* ID => nothing */ + 23, /* ABORT => ID */ + 23, /* AFTER => ID */ + 23, /* ANALYZE => ID */ + 23, /* ASC => ID */ + 23, /* ATTACH => ID */ + 23, /* BEFORE => ID */ + 23, /* CASCADE => ID */ + 23, /* CAST => ID */ + 23, /* CONFLICT => ID */ + 23, /* DATABASE => ID */ + 23, /* DESC => ID */ + 23, /* DETACH => ID */ + 23, /* EACH => ID */ + 23, /* FAIL => ID */ + 23, /* FOR => ID */ + 23, /* IGNORE => ID */ + 23, /* INITIALLY => ID */ + 23, /* INSTEAD => ID */ + 23, /* LIKE_KW => ID */ + 23, /* MATCH => ID */ + 23, /* KEY => ID */ + 23, /* OF => ID */ + 23, /* OFFSET => ID */ + 23, /* PRAGMA => ID */ + 23, /* RAISE => ID */ + 23, /* REPLACE => ID */ + 23, /* RESTRICT => ID */ + 23, /* ROW => ID */ + 23, /* STATEMENT => ID */ + 23, /* TRIGGER => ID */ + 23, /* VACUUM => ID */ + 23, /* VIEW => ID */ + 23, /* REINDEX => ID */ + 23, /* RENAME => ID */ + 23, /* CTIME_KW => ID */ + 0, /* OR => nothing */ + 0, /* AND => nothing */ + 0, /* IS => nothing */ + 0, /* BETWEEN => nothing */ + 0, /* IN => nothing */ + 0, /* ISNULL => nothing */ + 0, /* NOTNULL => nothing */ + 0, /* NE => nothing */ + 0, /* EQ => nothing */ + 0, /* GT => nothing */ + 0, /* LE => nothing */ + 0, /* LT => nothing */ + 0, /* GE => nothing */ + 0, /* ESCAPE => nothing */ + 0, /* BITAND => nothing */ + 0, /* BITOR => nothing */ + 0, /* LSHIFT => nothing */ + 0, /* RSHIFT => nothing */ + 0, /* PLUS => nothing */ + 0, /* MINUS => nothing */ + 0, /* STAR => nothing */ + 0, /* SLASH => nothing */ + 0, /* REM => nothing */ + 0, /* CONCAT => nothing */ + 0, /* UMINUS => nothing */ + 0, /* UPLUS => nothing */ + 0, /* BITNOT => nothing */ + 0, /* STRING => nothing */ + 0, /* JOIN_KW => nothing */ + 0, /* CONSTRAINT => nothing */ + 0, /* DEFAULT => nothing */ + 0, /* NULL => nothing */ + 0, /* PRIMARY => nothing */ + 0, /* UNIQUE => nothing */ + 0, /* CHECK => nothing */ + 0, /* REFERENCES => nothing */ + 0, /* COLLATE => nothing */ + 0, /* AUTOINCR => nothing */ + 0, /* ON => nothing */ + 0, /* DELETE => nothing */ + 0, /* UPDATE => nothing */ + 0, /* INSERT => nothing */ + 0, /* SET => nothing */ + 0, /* DEFERRABLE => nothing */ + 0, /* FOREIGN => nothing */ + 0, /* DROP => nothing */ + 0, /* UNION => nothing */ + 0, /* ALL => nothing */ + 0, /* EXCEPT => nothing */ + 0, /* INTERSECT => nothing */ + 0, /* SELECT => nothing */ + 0, /* DISTINCT => nothing */ + 0, /* DOT => nothing */ + 0, /* FROM => nothing */ + 0, /* JOIN => nothing */ + 0, /* USING => nothing */ + 0, /* ORDER => nothing */ + 0, /* BY => nothing */ + 0, /* GROUP => nothing */ + 0, /* HAVING => nothing */ + 0, /* LIMIT => nothing */ + 0, /* WHERE => nothing */ + 0, /* INTO => nothing */ + 0, /* VALUES => nothing */ + 0, /* INTEGER => nothing */ + 0, /* FLOAT => nothing */ + 0, /* BLOB => nothing */ + 0, /* REGISTER => nothing */ + 0, /* VARIABLE => nothing */ + 0, /* CASE => nothing */ + 0, /* WHEN => nothing */ + 0, /* THEN => nothing */ + 0, /* ELSE => nothing */ + 0, /* INDEX => nothing */ + 0, /* ALTER => nothing */ + 0, /* TO => nothing */ + 0, /* ADD => nothing */ + 0, /* COLUMNKW => nothing */ +}; +#endif /* YYFALLBACK */ + +/* The following structure represents a single element of the +** parser's stack. Information stored includes: +** +** + The state number for the parser at this level of the stack. +** +** + The value of the token stored at this level of the stack. +** (In other words, the "major" token.) +** +** + The semantic value stored at this level of the stack. This is +** the information used by the action routines in the grammar. +** It is sometimes called the "minor" token. +*/ +struct yyStackEntry { + int stateno; /* The state-number */ + int major; /* The major token value. This is the code + ** number for the token at this stack level */ + YYMINORTYPE minor; /* The user-supplied minor token value. This + ** is the value of the token */ +}; +typedef struct yyStackEntry yyStackEntry; + +/* The state of the parser is completely contained in an instance of +** the following structure */ +struct yyParser { + int yyidx; /* Index of top element in stack */ + int yyerrcnt; /* Shifts left before out of the error */ + sqlite3ParserARG_SDECL /* A place to hold %extra_argument */ + yyStackEntry yystack[YYSTACKDEPTH]; /* The parser's stack */ +}; +typedef struct yyParser yyParser; + +#ifndef NDEBUG +#include +static FILE *yyTraceFILE = 0; +static char *yyTracePrompt = 0; +#endif /* NDEBUG */ + +#ifndef NDEBUG +/* +** Turn parser tracing on by giving a stream to which to write the trace +** and a prompt to preface each trace message. Tracing is turned off +** by making either argument NULL +** +** Inputs: +**
    +**
  • A FILE* to which trace output should be written. +** If NULL, then tracing is turned off. +**
  • A prefix string written at the beginning of every +** line of trace output. If NULL, then tracing is +** turned off. +**
+** +** Outputs: +** None. +*/ +void sqlite3ParserTrace(FILE *TraceFILE, char *zTracePrompt){ + yyTraceFILE = TraceFILE; + yyTracePrompt = zTracePrompt; + if( yyTraceFILE==0 ) yyTracePrompt = 0; + else if( yyTracePrompt==0 ) yyTraceFILE = 0; +} +#endif /* NDEBUG */ + +#ifndef NDEBUG +/* For tracing shifts, the names of all terminals and nonterminals +** are required. The following table supplies these names */ +static const char *const yyTokenName[] = { + "$", "SEMI", "EXPLAIN", "QUERY", + "PLAN", "BEGIN", "TRANSACTION", "DEFERRED", + "IMMEDIATE", "EXCLUSIVE", "COMMIT", "END", + "ROLLBACK", "CREATE", "TABLE", "IF", + "NOT", "EXISTS", "TEMP", "LP", + "RP", "AS", "COMMA", "ID", + "ABORT", "AFTER", "ANALYZE", "ASC", + "ATTACH", "BEFORE", "CASCADE", "CAST", + "CONFLICT", "DATABASE", "DESC", "DETACH", + "EACH", "FAIL", "FOR", "IGNORE", + "INITIALLY", "INSTEAD", "LIKE_KW", "MATCH", + "KEY", "OF", "OFFSET", "PRAGMA", + "RAISE", "REPLACE", "RESTRICT", "ROW", + "STATEMENT", "TRIGGER", "VACUUM", "VIEW", + "REINDEX", "RENAME", "CTIME_KW", "OR", + "AND", "IS", "BETWEEN", "IN", + "ISNULL", "NOTNULL", "NE", "EQ", + "GT", "LE", "LT", "GE", + "ESCAPE", "BITAND", "BITOR", "LSHIFT", + "RSHIFT", "PLUS", "MINUS", "STAR", + "SLASH", "REM", "CONCAT", "UMINUS", + "UPLUS", "BITNOT", "STRING", "JOIN_KW", + "CONSTRAINT", "DEFAULT", "NULL", "PRIMARY", + "UNIQUE", "CHECK", "REFERENCES", "COLLATE", + "AUTOINCR", "ON", "DELETE", "UPDATE", + "INSERT", "SET", "DEFERRABLE", "FOREIGN", + "DROP", "UNION", "ALL", "EXCEPT", + "INTERSECT", "SELECT", "DISTINCT", "DOT", + "FROM", "JOIN", "USING", "ORDER", + "BY", "GROUP", "HAVING", "LIMIT", + "WHERE", "INTO", "VALUES", "INTEGER", + "FLOAT", "BLOB", "REGISTER", "VARIABLE", + "CASE", "WHEN", "THEN", "ELSE", + "INDEX", "ALTER", "TO", "ADD", + "COLUMNKW", "error", "input", "cmdlist", + "ecmd", "cmdx", "cmd", "explain", + "transtype", "trans_opt", "nm", "create_table", + "create_table_args", "temp", "ifnotexists", "dbnm", + "columnlist", "conslist_opt", "select", "column", + "columnid", "type", "carglist", "id", + "ids", "typetoken", "typename", "signed", + "plus_num", "minus_num", "carg", "ccons", + "term", "expr", "onconf", "sortorder", + "autoinc", "idxlist_opt", "refargs", "defer_subclause", + "refarg", "refact", "init_deferred_pred_opt", "conslist", + "tcons", "idxlist", "defer_subclause_opt", "orconf", + "resolvetype", "raisetype", "ifexists", "fullname", + "oneselect", "multiselect_op", "distinct", "selcollist", + "from", "where_opt", "groupby_opt", "having_opt", + "orderby_opt", "limit_opt", "sclp", "as", + "seltablist", "stl_prefix", "joinop", "on_opt", + "using_opt", "seltablist_paren", "joinop2", "inscollist", + "sortlist", "sortitem", "collate", "exprlist", + "setlist", "insert_cmd", "inscollist_opt", "itemlist", + "likeop", "escape", "between_op", "in_op", + "case_operand", "case_exprlist", "case_else", "expritem", + "uniqueflag", "idxitem", "plus_opt", "number", + "trigger_decl", "trigger_cmd_list", "trigger_time", "trigger_event", + "foreach_clause", "when_clause", "trigger_cmd", "database_kw_opt", + "key_opt", "add_column_fullname", "kwcolumn_opt", +}; +#endif /* NDEBUG */ + +#ifndef NDEBUG +/* For tracing reduce actions, the names of all rules are required. +*/ +static const char *const yyRuleName[] = { + /* 0 */ "input ::= cmdlist", + /* 1 */ "cmdlist ::= cmdlist ecmd", + /* 2 */ "cmdlist ::= ecmd", + /* 3 */ "cmdx ::= cmd", + /* 4 */ "ecmd ::= SEMI", + /* 5 */ "ecmd ::= explain cmdx SEMI", + /* 6 */ "explain ::=", + /* 7 */ "explain ::= EXPLAIN", + /* 8 */ "explain ::= EXPLAIN QUERY PLAN", + /* 9 */ "cmd ::= BEGIN transtype trans_opt", + /* 10 */ "trans_opt ::=", + /* 11 */ "trans_opt ::= TRANSACTION", + /* 12 */ "trans_opt ::= TRANSACTION nm", + /* 13 */ "transtype ::=", + /* 14 */ "transtype ::= DEFERRED", + /* 15 */ "transtype ::= IMMEDIATE", + /* 16 */ "transtype ::= EXCLUSIVE", + /* 17 */ "cmd ::= COMMIT trans_opt", + /* 18 */ "cmd ::= END trans_opt", + /* 19 */ "cmd ::= ROLLBACK trans_opt", + /* 20 */ "cmd ::= create_table create_table_args", + /* 21 */ "create_table ::= CREATE temp TABLE ifnotexists nm dbnm", + /* 22 */ "ifnotexists ::=", + /* 23 */ "ifnotexists ::= IF NOT EXISTS", + /* 24 */ "temp ::= TEMP", + /* 25 */ "temp ::=", + /* 26 */ "create_table_args ::= LP columnlist conslist_opt RP", + /* 27 */ "create_table_args ::= AS select", + /* 28 */ "columnlist ::= columnlist COMMA column", + /* 29 */ "columnlist ::= column", + /* 30 */ "column ::= columnid type carglist", + /* 31 */ "columnid ::= nm", + /* 32 */ "id ::= ID", + /* 33 */ "ids ::= ID|STRING", + /* 34 */ "nm ::= ID", + /* 35 */ "nm ::= STRING", + /* 36 */ "nm ::= JOIN_KW", + /* 37 */ "type ::=", + /* 38 */ "type ::= typetoken", + /* 39 */ "typetoken ::= typename", + /* 40 */ "typetoken ::= typename LP signed RP", + /* 41 */ "typetoken ::= typename LP signed COMMA signed RP", + /* 42 */ "typename ::= ids", + /* 43 */ "typename ::= typename ids", + /* 44 */ "signed ::= plus_num", + /* 45 */ "signed ::= minus_num", + /* 46 */ "carglist ::= carglist carg", + /* 47 */ "carglist ::=", + /* 48 */ "carg ::= CONSTRAINT nm ccons", + /* 49 */ "carg ::= ccons", + /* 50 */ "carg ::= DEFAULT term", + /* 51 */ "carg ::= DEFAULT LP expr RP", + /* 52 */ "carg ::= DEFAULT PLUS term", + /* 53 */ "carg ::= DEFAULT MINUS term", + /* 54 */ "carg ::= DEFAULT id", + /* 55 */ "ccons ::= NULL onconf", + /* 56 */ "ccons ::= NOT NULL onconf", + /* 57 */ "ccons ::= PRIMARY KEY sortorder onconf autoinc", + /* 58 */ "ccons ::= UNIQUE onconf", + /* 59 */ "ccons ::= CHECK LP expr RP", + /* 60 */ "ccons ::= REFERENCES nm idxlist_opt refargs", + /* 61 */ "ccons ::= defer_subclause", + /* 62 */ "ccons ::= COLLATE id", + /* 63 */ "autoinc ::=", + /* 64 */ "autoinc ::= AUTOINCR", + /* 65 */ "refargs ::=", + /* 66 */ "refargs ::= refargs refarg", + /* 67 */ "refarg ::= MATCH nm", + /* 68 */ "refarg ::= ON DELETE refact", + /* 69 */ "refarg ::= ON UPDATE refact", + /* 70 */ "refarg ::= ON INSERT refact", + /* 71 */ "refact ::= SET NULL", + /* 72 */ "refact ::= SET DEFAULT", + /* 73 */ "refact ::= CASCADE", + /* 74 */ "refact ::= RESTRICT", + /* 75 */ "defer_subclause ::= NOT DEFERRABLE init_deferred_pred_opt", + /* 76 */ "defer_subclause ::= DEFERRABLE init_deferred_pred_opt", + /* 77 */ "init_deferred_pred_opt ::=", + /* 78 */ "init_deferred_pred_opt ::= INITIALLY DEFERRED", + /* 79 */ "init_deferred_pred_opt ::= INITIALLY IMMEDIATE", + /* 80 */ "conslist_opt ::=", + /* 81 */ "conslist_opt ::= COMMA conslist", + /* 82 */ "conslist ::= conslist COMMA tcons", + /* 83 */ "conslist ::= conslist tcons", + /* 84 */ "conslist ::= tcons", + /* 85 */ "tcons ::= CONSTRAINT nm", + /* 86 */ "tcons ::= PRIMARY KEY LP idxlist autoinc RP onconf", + /* 87 */ "tcons ::= UNIQUE LP idxlist RP onconf", + /* 88 */ "tcons ::= CHECK LP expr RP onconf", + /* 89 */ "tcons ::= FOREIGN KEY LP idxlist RP REFERENCES nm idxlist_opt refargs defer_subclause_opt", + /* 90 */ "defer_subclause_opt ::=", + /* 91 */ "defer_subclause_opt ::= defer_subclause", + /* 92 */ "onconf ::=", + /* 93 */ "onconf ::= ON CONFLICT resolvetype", + /* 94 */ "orconf ::=", + /* 95 */ "orconf ::= OR resolvetype", + /* 96 */ "resolvetype ::= raisetype", + /* 97 */ "resolvetype ::= IGNORE", + /* 98 */ "resolvetype ::= REPLACE", + /* 99 */ "cmd ::= DROP TABLE ifexists fullname", + /* 100 */ "ifexists ::= IF EXISTS", + /* 101 */ "ifexists ::=", + /* 102 */ "cmd ::= CREATE temp VIEW nm dbnm AS select", + /* 103 */ "cmd ::= DROP VIEW ifexists fullname", + /* 104 */ "cmd ::= select", + /* 105 */ "select ::= oneselect", + /* 106 */ "select ::= select multiselect_op oneselect", + /* 107 */ "multiselect_op ::= UNION", + /* 108 */ "multiselect_op ::= UNION ALL", + /* 109 */ "multiselect_op ::= EXCEPT|INTERSECT", + /* 110 */ "oneselect ::= SELECT distinct selcollist from where_opt groupby_opt having_opt orderby_opt limit_opt", + /* 111 */ "distinct ::= DISTINCT", + /* 112 */ "distinct ::= ALL", + /* 113 */ "distinct ::=", + /* 114 */ "sclp ::= selcollist COMMA", + /* 115 */ "sclp ::=", + /* 116 */ "selcollist ::= sclp expr as", + /* 117 */ "selcollist ::= sclp STAR", + /* 118 */ "selcollist ::= sclp nm DOT STAR", + /* 119 */ "as ::= AS nm", + /* 120 */ "as ::= ids", + /* 121 */ "as ::=", + /* 122 */ "from ::=", + /* 123 */ "from ::= FROM seltablist", + /* 124 */ "stl_prefix ::= seltablist joinop", + /* 125 */ "stl_prefix ::=", + /* 126 */ "seltablist ::= stl_prefix nm dbnm as on_opt using_opt", + /* 127 */ "seltablist ::= stl_prefix LP seltablist_paren RP as on_opt using_opt", + /* 128 */ "seltablist_paren ::= select", + /* 129 */ "seltablist_paren ::= seltablist", + /* 130 */ "dbnm ::=", + /* 131 */ "dbnm ::= DOT nm", + /* 132 */ "fullname ::= nm dbnm", + /* 133 */ "joinop ::= COMMA|JOIN", + /* 134 */ "joinop ::= JOIN_KW JOIN", + /* 135 */ "joinop ::= JOIN_KW nm JOIN", + /* 136 */ "joinop ::= JOIN_KW nm nm JOIN", + /* 137 */ "on_opt ::= ON expr", + /* 138 */ "on_opt ::=", + /* 139 */ "using_opt ::= USING LP inscollist RP", + /* 140 */ "using_opt ::=", + /* 141 */ "orderby_opt ::=", + /* 142 */ "orderby_opt ::= ORDER BY sortlist", + /* 143 */ "sortlist ::= sortlist COMMA sortitem collate sortorder", + /* 144 */ "sortlist ::= sortitem collate sortorder", + /* 145 */ "sortitem ::= expr", + /* 146 */ "sortorder ::= ASC", + /* 147 */ "sortorder ::= DESC", + /* 148 */ "sortorder ::=", + /* 149 */ "collate ::=", + /* 150 */ "collate ::= COLLATE id", + /* 151 */ "groupby_opt ::=", + /* 152 */ "groupby_opt ::= GROUP BY exprlist", + /* 153 */ "having_opt ::=", + /* 154 */ "having_opt ::= HAVING expr", + /* 155 */ "limit_opt ::=", + /* 156 */ "limit_opt ::= LIMIT expr", + /* 157 */ "limit_opt ::= LIMIT expr OFFSET expr", + /* 158 */ "limit_opt ::= LIMIT expr COMMA expr", + /* 159 */ "cmd ::= DELETE FROM fullname where_opt", + /* 160 */ "where_opt ::=", + /* 161 */ "where_opt ::= WHERE expr", + /* 162 */ "cmd ::= UPDATE orconf fullname SET setlist where_opt", + /* 163 */ "setlist ::= setlist COMMA nm EQ expr", + /* 164 */ "setlist ::= nm EQ expr", + /* 165 */ "cmd ::= insert_cmd INTO fullname inscollist_opt VALUES LP itemlist RP", + /* 166 */ "cmd ::= insert_cmd INTO fullname inscollist_opt select", + /* 167 */ "insert_cmd ::= INSERT orconf", + /* 168 */ "insert_cmd ::= REPLACE", + /* 169 */ "itemlist ::= itemlist COMMA expr", + /* 170 */ "itemlist ::= expr", + /* 171 */ "inscollist_opt ::=", + /* 172 */ "inscollist_opt ::= LP inscollist RP", + /* 173 */ "inscollist ::= inscollist COMMA nm", + /* 174 */ "inscollist ::= nm", + /* 175 */ "expr ::= term", + /* 176 */ "expr ::= LP expr RP", + /* 177 */ "term ::= NULL", + /* 178 */ "expr ::= ID", + /* 179 */ "expr ::= JOIN_KW", + /* 180 */ "expr ::= nm DOT nm", + /* 181 */ "expr ::= nm DOT nm DOT nm", + /* 182 */ "term ::= INTEGER|FLOAT|BLOB", + /* 183 */ "term ::= STRING", + /* 184 */ "expr ::= REGISTER", + /* 185 */ "expr ::= VARIABLE", + /* 186 */ "expr ::= CAST LP expr AS typetoken RP", + /* 187 */ "expr ::= ID LP distinct exprlist RP", + /* 188 */ "expr ::= ID LP STAR RP", + /* 189 */ "term ::= CTIME_KW", + /* 190 */ "expr ::= expr AND expr", + /* 191 */ "expr ::= expr OR expr", + /* 192 */ "expr ::= expr LT|GT|GE|LE expr", + /* 193 */ "expr ::= expr EQ|NE expr", + /* 194 */ "expr ::= expr BITAND|BITOR|LSHIFT|RSHIFT expr", + /* 195 */ "expr ::= expr PLUS|MINUS expr", + /* 196 */ "expr ::= expr STAR|SLASH|REM expr", + /* 197 */ "expr ::= expr CONCAT expr", + /* 198 */ "likeop ::= LIKE_KW", + /* 199 */ "likeop ::= NOT LIKE_KW", + /* 200 */ "escape ::= ESCAPE expr", + /* 201 */ "escape ::=", + /* 202 */ "expr ::= expr likeop expr escape", + /* 203 */ "expr ::= expr ISNULL|NOTNULL", + /* 204 */ "expr ::= expr IS NULL", + /* 205 */ "expr ::= expr NOT NULL", + /* 206 */ "expr ::= expr IS NOT NULL", + /* 207 */ "expr ::= NOT|BITNOT expr", + /* 208 */ "expr ::= MINUS expr", + /* 209 */ "expr ::= PLUS expr", + /* 210 */ "between_op ::= BETWEEN", + /* 211 */ "between_op ::= NOT BETWEEN", + /* 212 */ "expr ::= expr between_op expr AND expr", + /* 213 */ "in_op ::= IN", + /* 214 */ "in_op ::= NOT IN", + /* 215 */ "expr ::= expr in_op LP exprlist RP", + /* 216 */ "expr ::= LP select RP", + /* 217 */ "expr ::= expr in_op LP select RP", + /* 218 */ "expr ::= expr in_op nm dbnm", + /* 219 */ "expr ::= EXISTS LP select RP", + /* 220 */ "expr ::= CASE case_operand case_exprlist case_else END", + /* 221 */ "case_exprlist ::= case_exprlist WHEN expr THEN expr", + /* 222 */ "case_exprlist ::= WHEN expr THEN expr", + /* 223 */ "case_else ::= ELSE expr", + /* 224 */ "case_else ::=", + /* 225 */ "case_operand ::= expr", + /* 226 */ "case_operand ::=", + /* 227 */ "exprlist ::= exprlist COMMA expritem", + /* 228 */ "exprlist ::= expritem", + /* 229 */ "expritem ::= expr", + /* 230 */ "expritem ::=", + /* 231 */ "cmd ::= CREATE uniqueflag INDEX ifnotexists nm dbnm ON nm LP idxlist RP", + /* 232 */ "uniqueflag ::= UNIQUE", + /* 233 */ "uniqueflag ::=", + /* 234 */ "idxlist_opt ::=", + /* 235 */ "idxlist_opt ::= LP idxlist RP", + /* 236 */ "idxlist ::= idxlist COMMA idxitem collate sortorder", + /* 237 */ "idxlist ::= idxitem collate sortorder", + /* 238 */ "idxitem ::= nm", + /* 239 */ "cmd ::= DROP INDEX ifexists fullname", + /* 240 */ "cmd ::= VACUUM", + /* 241 */ "cmd ::= VACUUM nm", + /* 242 */ "cmd ::= PRAGMA nm dbnm EQ nm", + /* 243 */ "cmd ::= PRAGMA nm dbnm EQ ON", + /* 244 */ "cmd ::= PRAGMA nm dbnm EQ plus_num", + /* 245 */ "cmd ::= PRAGMA nm dbnm EQ minus_num", + /* 246 */ "cmd ::= PRAGMA nm dbnm LP nm RP", + /* 247 */ "cmd ::= PRAGMA nm dbnm", + /* 248 */ "plus_num ::= plus_opt number", + /* 249 */ "minus_num ::= MINUS number", + /* 250 */ "number ::= INTEGER|FLOAT", + /* 251 */ "plus_opt ::= PLUS", + /* 252 */ "plus_opt ::=", + /* 253 */ "cmd ::= CREATE trigger_decl BEGIN trigger_cmd_list END", + /* 254 */ "trigger_decl ::= temp TRIGGER nm dbnm trigger_time trigger_event ON fullname foreach_clause when_clause", + /* 255 */ "trigger_time ::= BEFORE", + /* 256 */ "trigger_time ::= AFTER", + /* 257 */ "trigger_time ::= INSTEAD OF", + /* 258 */ "trigger_time ::=", + /* 259 */ "trigger_event ::= DELETE|INSERT", + /* 260 */ "trigger_event ::= UPDATE", + /* 261 */ "trigger_event ::= UPDATE OF inscollist", + /* 262 */ "foreach_clause ::=", + /* 263 */ "foreach_clause ::= FOR EACH ROW", + /* 264 */ "foreach_clause ::= FOR EACH STATEMENT", + /* 265 */ "when_clause ::=", + /* 266 */ "when_clause ::= WHEN expr", + /* 267 */ "trigger_cmd_list ::= trigger_cmd SEMI trigger_cmd_list", + /* 268 */ "trigger_cmd_list ::=", + /* 269 */ "trigger_cmd ::= UPDATE orconf nm SET setlist where_opt", + /* 270 */ "trigger_cmd ::= insert_cmd INTO nm inscollist_opt VALUES LP itemlist RP", + /* 271 */ "trigger_cmd ::= insert_cmd INTO nm inscollist_opt select", + /* 272 */ "trigger_cmd ::= DELETE FROM nm where_opt", + /* 273 */ "trigger_cmd ::= select", + /* 274 */ "expr ::= RAISE LP IGNORE RP", + /* 275 */ "expr ::= RAISE LP raisetype COMMA nm RP", + /* 276 */ "raisetype ::= ROLLBACK", + /* 277 */ "raisetype ::= ABORT", + /* 278 */ "raisetype ::= FAIL", + /* 279 */ "cmd ::= DROP TRIGGER fullname", + /* 280 */ "cmd ::= ATTACH database_kw_opt expr AS expr key_opt", + /* 281 */ "key_opt ::=", + /* 282 */ "key_opt ::= KEY expr", + /* 283 */ "database_kw_opt ::= DATABASE", + /* 284 */ "database_kw_opt ::=", + /* 285 */ "cmd ::= DETACH database_kw_opt expr", + /* 286 */ "cmd ::= REINDEX", + /* 287 */ "cmd ::= REINDEX nm dbnm", + /* 288 */ "cmd ::= ANALYZE", + /* 289 */ "cmd ::= ANALYZE nm dbnm", + /* 290 */ "cmd ::= ALTER TABLE fullname RENAME TO nm", + /* 291 */ "cmd ::= ALTER TABLE add_column_fullname ADD kwcolumn_opt column", + /* 292 */ "add_column_fullname ::= fullname", + /* 293 */ "kwcolumn_opt ::=", + /* 294 */ "kwcolumn_opt ::= COLUMNKW", +}; +#endif /* NDEBUG */ + +/* +** This function returns the symbolic name associated with a token +** value. +*/ +const char *sqlite3ParserTokenName(int tokenType){ +#ifndef NDEBUG + if( tokenType>0 && tokenType<(sizeof(yyTokenName)/sizeof(yyTokenName[0])) ){ + return yyTokenName[tokenType]; + }else{ + return "Unknown"; + } +#else + return ""; +#endif +} + +/* +** This function allocates a new parser. +** The only argument is a pointer to a function which works like +** malloc. +** +** Inputs: +** A pointer to the function used to allocate memory. +** +** Outputs: +** A pointer to a parser. This pointer is used in subsequent calls +** to sqlite3Parser and sqlite3ParserFree. +*/ +void *sqlite3ParserAlloc(void *(*mallocProc)(size_t)){ + yyParser *pParser; + pParser = (yyParser*)(*mallocProc)( (size_t)sizeof(yyParser) ); + if( pParser ){ + pParser->yyidx = -1; + } + return pParser; +} + +/* The following function deletes the value associated with a +** symbol. The symbol can be either a terminal or nonterminal. +** "yymajor" is the symbol code, and "yypminor" is a pointer to +** the value. +*/ +static void yy_destructor(YYCODETYPE yymajor, YYMINORTYPE *yypminor){ + switch( yymajor ){ + /* Here is inserted the actions which take place when a + ** terminal or non-terminal is destroyed. This can happen + ** when the symbol is popped from the stack during a + ** reduce or during error processing or when a parser is + ** being destroyed before it is finished parsing. + ** + ** Note: during a reduce, the only symbols destroyed are those + ** which appear on the RHS of the rule, but which are not used + ** inside the C code. + */ + case 154: + case 188: + case 205: +#line 368 "parse.y" +{sqlite3SelectDelete((yypminor->yy239));} +#line 1216 "parse.c" + break; + case 168: + case 169: + case 193: + case 195: + case 203: + case 209: + case 217: + case 220: + case 222: + case 223: + case 233: +#line 625 "parse.y" +{sqlite3ExprDelete((yypminor->yy178));} +#line 1231 "parse.c" + break; + case 173: + case 181: + case 191: + case 194: + case 196: + case 198: + case 208: + case 211: + case 212: + case 215: + case 221: +#line 855 "parse.y" +{sqlite3ExprListDelete((yypminor->yy462));} +#line 1246 "parse.c" + break; + case 187: + case 192: + case 200: + case 201: +#line 496 "parse.y" +{sqlite3SrcListDelete((yypminor->yy285));} +#line 1254 "parse.c" + break; + case 197: +#line 557 "parse.y" +{ + sqlite3ExprDelete((yypminor->yy270).pLimit); + sqlite3ExprDelete((yypminor->yy270).pOffset); +} +#line 1262 "parse.c" + break; + case 204: + case 207: + case 214: +#line 513 "parse.y" +{sqlite3IdListDelete((yypminor->yy160));} +#line 1269 "parse.c" + break; + case 229: + case 234: +#line 949 "parse.y" +{sqlite3DeleteTriggerStep((yypminor->yy247));} +#line 1275 "parse.c" + break; + case 231: +#line 933 "parse.y" +{sqlite3IdListDelete((yypminor->yy132).b);} +#line 1280 "parse.c" + break; + case 236: +#line 1008 "parse.y" +{sqlite3ExprDelete((yypminor->yy292));} +#line 1285 "parse.c" + break; + default: break; /* If no destructor action specified: do nothing */ + } +} + +/* +** Pop the parser's stack once. +** +** If there is a destructor routine associated with the token which +** is popped from the stack, then call it. +** +** Return the major token number for the symbol popped. +*/ +static int yy_pop_parser_stack(yyParser *pParser){ + YYCODETYPE yymajor; + yyStackEntry *yytos = &pParser->yystack[pParser->yyidx]; + + if( pParser->yyidx<0 ) return 0; +#ifndef NDEBUG + if( yyTraceFILE && pParser->yyidx>=0 ){ + fprintf(yyTraceFILE,"%sPopping %s\n", + yyTracePrompt, + yyTokenName[yytos->major]); + } +#endif + yymajor = yytos->major; + yy_destructor( yymajor, &yytos->minor); + pParser->yyidx--; + return yymajor; +} + +/* +** Deallocate and destroy a parser. Destructors are all called for +** all stack elements before shutting the parser down. +** +** Inputs: +**
    +**
  • A pointer to the parser. This should be a pointer +** obtained from sqlite3ParserAlloc. +**
  • A pointer to a function used to reclaim memory obtained +** from malloc. +**
+*/ +void sqlite3ParserFree( + void *p, /* The parser to be deleted */ + void (*freeProc)(void*) /* Function used to reclaim memory */ +){ + yyParser *pParser = (yyParser*)p; + if( pParser==0 ) return; + while( pParser->yyidx>=0 ) yy_pop_parser_stack(pParser); + (*freeProc)((void*)pParser); +} + +/* +** Find the appropriate action for a parser given the terminal +** look-ahead token iLookAhead. +** +** If the look-ahead token is YYNOCODE, then check to see if the action is +** independent of the look-ahead. If it is, return the action, otherwise +** return YY_NO_ACTION. +*/ +static int yy_find_shift_action( + yyParser *pParser, /* The parser */ + int iLookAhead /* The look-ahead token */ +){ + int i; + int stateno = pParser->yystack[pParser->yyidx].stateno; + + if( stateno>YY_SHIFT_MAX || (i = yy_shift_ofst[stateno])==YY_SHIFT_USE_DFLT ){ + return yy_default[stateno]; + } + if( iLookAhead==YYNOCODE ){ + return YY_NO_ACTION; + } + i += iLookAhead; + if( i<0 || i>=YY_SZ_ACTTAB || yy_lookahead[i]!=iLookAhead ){ +#ifdef YYFALLBACK + int iFallback; /* Fallback token */ + if( iLookAhead %s\n", + yyTracePrompt, yyTokenName[iLookAhead], yyTokenName[iFallback]); + } +#endif + return yy_find_shift_action(pParser, iFallback); + } +#endif + return yy_default[stateno]; + }else{ + return yy_action[i]; + } +} + +/* +** Find the appropriate action for a parser given the non-terminal +** look-ahead token iLookAhead. +** +** If the look-ahead token is YYNOCODE, then check to see if the action is +** independent of the look-ahead. If it is, return the action, otherwise +** return YY_NO_ACTION. +*/ +static int yy_find_reduce_action( + int stateno, /* Current state number */ + int iLookAhead /* The look-ahead token */ +){ + int i; + /* int stateno = pParser->yystack[pParser->yyidx].stateno; */ + + if( stateno>YY_REDUCE_MAX || + (i = yy_reduce_ofst[stateno])==YY_REDUCE_USE_DFLT ){ + return yy_default[stateno]; + } + if( iLookAhead==YYNOCODE ){ + return YY_NO_ACTION; + } + i += iLookAhead; + if( i<0 || i>=YY_SZ_ACTTAB || yy_lookahead[i]!=iLookAhead ){ + return yy_default[stateno]; + }else{ + return yy_action[i]; + } +} + +/* +** Perform a shift action. +*/ +static void yy_shift( + yyParser *yypParser, /* The parser to be shifted */ + int yyNewState, /* The new state to shift in */ + int yyMajor, /* The major token to shift in */ + YYMINORTYPE *yypMinor /* Pointer ot the minor token to shift in */ +){ + yyStackEntry *yytos; + yypParser->yyidx++; + if( yypParser->yyidx>=YYSTACKDEPTH ){ + sqlite3ParserARG_FETCH; + yypParser->yyidx--; +#ifndef NDEBUG + if( yyTraceFILE ){ + fprintf(yyTraceFILE,"%sStack Overflow!\n",yyTracePrompt); + } +#endif + while( yypParser->yyidx>=0 ) yy_pop_parser_stack(yypParser); + /* Here code is inserted which will execute if the parser + ** stack every overflows */ + sqlite3ParserARG_STORE; /* Suppress warning about unused %extra_argument var */ + return; + } + yytos = &yypParser->yystack[yypParser->yyidx]; + yytos->stateno = yyNewState; + yytos->major = yyMajor; + yytos->minor = *yypMinor; +#ifndef NDEBUG + if( yyTraceFILE && yypParser->yyidx>0 ){ + int i; + fprintf(yyTraceFILE,"%sShift %d\n",yyTracePrompt,yyNewState); + fprintf(yyTraceFILE,"%sStack:",yyTracePrompt); + for(i=1; i<=yypParser->yyidx; i++) + fprintf(yyTraceFILE," %s",yyTokenName[yypParser->yystack[i].major]); + fprintf(yyTraceFILE,"\n"); + } +#endif +} + +/* The following table contains information about every rule that +** is used during the reduce. +*/ +static const struct { + YYCODETYPE lhs; /* Symbol on the left-hand side of the rule */ + unsigned char nrhs; /* Number of right-hand side symbols in the rule */ +} yyRuleInfo[] = { + { 138, 1 }, + { 139, 2 }, + { 139, 1 }, + { 141, 1 }, + { 140, 1 }, + { 140, 3 }, + { 143, 0 }, + { 143, 1 }, + { 143, 3 }, + { 142, 3 }, + { 145, 0 }, + { 145, 1 }, + { 145, 2 }, + { 144, 0 }, + { 144, 1 }, + { 144, 1 }, + { 144, 1 }, + { 142, 2 }, + { 142, 2 }, + { 142, 2 }, + { 142, 2 }, + { 147, 6 }, + { 150, 0 }, + { 150, 3 }, + { 149, 1 }, + { 149, 0 }, + { 148, 4 }, + { 148, 2 }, + { 152, 3 }, + { 152, 1 }, + { 155, 3 }, + { 156, 1 }, + { 159, 1 }, + { 160, 1 }, + { 146, 1 }, + { 146, 1 }, + { 146, 1 }, + { 157, 0 }, + { 157, 1 }, + { 161, 1 }, + { 161, 4 }, + { 161, 6 }, + { 162, 1 }, + { 162, 2 }, + { 163, 1 }, + { 163, 1 }, + { 158, 2 }, + { 158, 0 }, + { 166, 3 }, + { 166, 1 }, + { 166, 2 }, + { 166, 4 }, + { 166, 3 }, + { 166, 3 }, + { 166, 2 }, + { 167, 2 }, + { 167, 3 }, + { 167, 5 }, + { 167, 2 }, + { 167, 4 }, + { 167, 4 }, + { 167, 1 }, + { 167, 2 }, + { 172, 0 }, + { 172, 1 }, + { 174, 0 }, + { 174, 2 }, + { 176, 2 }, + { 176, 3 }, + { 176, 3 }, + { 176, 3 }, + { 177, 2 }, + { 177, 2 }, + { 177, 1 }, + { 177, 1 }, + { 175, 3 }, + { 175, 2 }, + { 178, 0 }, + { 178, 2 }, + { 178, 2 }, + { 153, 0 }, + { 153, 2 }, + { 179, 3 }, + { 179, 2 }, + { 179, 1 }, + { 180, 2 }, + { 180, 7 }, + { 180, 5 }, + { 180, 5 }, + { 180, 10 }, + { 182, 0 }, + { 182, 1 }, + { 170, 0 }, + { 170, 3 }, + { 183, 0 }, + { 183, 2 }, + { 184, 1 }, + { 184, 1 }, + { 184, 1 }, + { 142, 4 }, + { 186, 2 }, + { 186, 0 }, + { 142, 7 }, + { 142, 4 }, + { 142, 1 }, + { 154, 1 }, + { 154, 3 }, + { 189, 1 }, + { 189, 2 }, + { 189, 1 }, + { 188, 9 }, + { 190, 1 }, + { 190, 1 }, + { 190, 0 }, + { 198, 2 }, + { 198, 0 }, + { 191, 3 }, + { 191, 2 }, + { 191, 4 }, + { 199, 2 }, + { 199, 1 }, + { 199, 0 }, + { 192, 0 }, + { 192, 2 }, + { 201, 2 }, + { 201, 0 }, + { 200, 6 }, + { 200, 7 }, + { 205, 1 }, + { 205, 1 }, + { 151, 0 }, + { 151, 2 }, + { 187, 2 }, + { 202, 1 }, + { 202, 2 }, + { 202, 3 }, + { 202, 4 }, + { 203, 2 }, + { 203, 0 }, + { 204, 4 }, + { 204, 0 }, + { 196, 0 }, + { 196, 3 }, + { 208, 5 }, + { 208, 3 }, + { 209, 1 }, + { 171, 1 }, + { 171, 1 }, + { 171, 0 }, + { 210, 0 }, + { 210, 2 }, + { 194, 0 }, + { 194, 3 }, + { 195, 0 }, + { 195, 2 }, + { 197, 0 }, + { 197, 2 }, + { 197, 4 }, + { 197, 4 }, + { 142, 4 }, + { 193, 0 }, + { 193, 2 }, + { 142, 6 }, + { 212, 5 }, + { 212, 3 }, + { 142, 8 }, + { 142, 5 }, + { 213, 2 }, + { 213, 1 }, + { 215, 3 }, + { 215, 1 }, + { 214, 0 }, + { 214, 3 }, + { 207, 3 }, + { 207, 1 }, + { 169, 1 }, + { 169, 3 }, + { 168, 1 }, + { 169, 1 }, + { 169, 1 }, + { 169, 3 }, + { 169, 5 }, + { 168, 1 }, + { 168, 1 }, + { 169, 1 }, + { 169, 1 }, + { 169, 6 }, + { 169, 5 }, + { 169, 4 }, + { 168, 1 }, + { 169, 3 }, + { 169, 3 }, + { 169, 3 }, + { 169, 3 }, + { 169, 3 }, + { 169, 3 }, + { 169, 3 }, + { 169, 3 }, + { 216, 1 }, + { 216, 2 }, + { 217, 2 }, + { 217, 0 }, + { 169, 4 }, + { 169, 2 }, + { 169, 3 }, + { 169, 3 }, + { 169, 4 }, + { 169, 2 }, + { 169, 2 }, + { 169, 2 }, + { 218, 1 }, + { 218, 2 }, + { 169, 5 }, + { 219, 1 }, + { 219, 2 }, + { 169, 5 }, + { 169, 3 }, + { 169, 5 }, + { 169, 4 }, + { 169, 4 }, + { 169, 5 }, + { 221, 5 }, + { 221, 4 }, + { 222, 2 }, + { 222, 0 }, + { 220, 1 }, + { 220, 0 }, + { 211, 3 }, + { 211, 1 }, + { 223, 1 }, + { 223, 0 }, + { 142, 11 }, + { 224, 1 }, + { 224, 0 }, + { 173, 0 }, + { 173, 3 }, + { 181, 5 }, + { 181, 3 }, + { 225, 1 }, + { 142, 4 }, + { 142, 1 }, + { 142, 2 }, + { 142, 5 }, + { 142, 5 }, + { 142, 5 }, + { 142, 5 }, + { 142, 6 }, + { 142, 3 }, + { 164, 2 }, + { 165, 2 }, + { 227, 1 }, + { 226, 1 }, + { 226, 0 }, + { 142, 5 }, + { 228, 10 }, + { 230, 1 }, + { 230, 1 }, + { 230, 2 }, + { 230, 0 }, + { 231, 1 }, + { 231, 1 }, + { 231, 3 }, + { 232, 0 }, + { 232, 3 }, + { 232, 3 }, + { 233, 0 }, + { 233, 2 }, + { 229, 3 }, + { 229, 0 }, + { 234, 6 }, + { 234, 8 }, + { 234, 5 }, + { 234, 4 }, + { 234, 1 }, + { 169, 4 }, + { 169, 6 }, + { 185, 1 }, + { 185, 1 }, + { 185, 1 }, + { 142, 3 }, + { 142, 6 }, + { 236, 0 }, + { 236, 2 }, + { 235, 1 }, + { 235, 0 }, + { 142, 3 }, + { 142, 1 }, + { 142, 3 }, + { 142, 1 }, + { 142, 3 }, + { 142, 6 }, + { 142, 6 }, + { 237, 1 }, + { 238, 0 }, + { 238, 1 }, +}; + +static void yy_accept(yyParser*); /* Forward Declaration */ + +/* +** Perform a reduce action and the shift that must immediately +** follow the reduce. +*/ +static void yy_reduce( + yyParser *yypParser, /* The parser */ + int yyruleno /* Number of the rule by which to reduce */ +){ + int yygoto; /* The next state */ + int yyact; /* The next action */ + YYMINORTYPE yygotominor; /* The LHS of the rule reduced */ + yyStackEntry *yymsp; /* The top of the parser's stack */ + int yysize; /* Amount to pop the stack */ + sqlite3ParserARG_FETCH; + yymsp = &yypParser->yystack[yypParser->yyidx]; +#ifndef NDEBUG + if( yyTraceFILE && yyruleno>=0 + && yyruleno + ** { ... } // User supplied code + ** #line + ** break; + */ + case 3: +#line 95 "parse.y" +{ sqlite3FinishCoding(pParse); } +#line 1805 "parse.c" + break; + case 6: +#line 98 "parse.y" +{ sqlite3BeginParse(pParse, 0); } +#line 1810 "parse.c" + break; + case 7: +#line 100 "parse.y" +{ sqlite3BeginParse(pParse, 1); } +#line 1815 "parse.c" + break; + case 8: +#line 101 "parse.y" +{ sqlite3BeginParse(pParse, 2); } +#line 1820 "parse.c" + break; + case 9: +#line 107 "parse.y" +{sqlite3BeginTransaction(pParse, yymsp[-1].minor.yy230);} +#line 1825 "parse.c" + break; + case 13: +#line 112 "parse.y" +{yygotominor.yy230 = TK_DEFERRED;} +#line 1830 "parse.c" + break; + case 14: + case 15: + case 16: + case 107: + case 109: +#line 113 "parse.y" +{yygotominor.yy230 = yymsp[0].major;} +#line 1839 "parse.c" + break; + case 17: + case 18: +#line 116 "parse.y" +{sqlite3CommitTransaction(pParse);} +#line 1845 "parse.c" + break; + case 19: +#line 118 "parse.y" +{sqlite3RollbackTransaction(pParse);} +#line 1850 "parse.c" + break; + case 21: +#line 123 "parse.y" +{ + sqlite3StartTable(pParse,&yymsp[-5].minor.yy0,&yymsp[-1].minor.yy384,&yymsp[0].minor.yy384,yymsp[-4].minor.yy230,0,yymsp[-2].minor.yy230); +} +#line 1857 "parse.c" + break; + case 22: + case 25: + case 63: + case 77: + case 79: + case 90: + case 101: + case 112: + case 113: + case 210: + case 213: +#line 127 "parse.y" +{yygotominor.yy230 = 0;} +#line 1872 "parse.c" + break; + case 23: + case 24: + case 64: + case 78: + case 100: + case 111: + case 211: + case 214: +#line 128 "parse.y" +{yygotominor.yy230 = 1;} +#line 1884 "parse.c" + break; + case 26: +#line 134 "parse.y" +{ + sqlite3EndTable(pParse,&yymsp[-1].minor.yy384,&yymsp[0].minor.yy0,0); +} +#line 1891 "parse.c" + break; + case 27: +#line 137 "parse.y" +{ + sqlite3EndTable(pParse,0,0,yymsp[0].minor.yy239); + sqlite3SelectDelete(yymsp[0].minor.yy239); +} +#line 1899 "parse.c" + break; + case 30: +#line 149 "parse.y" +{ + yygotominor.yy384.z = yymsp[-2].minor.yy384.z; + yygotominor.yy384.n = (pParse->sLastToken.z-yymsp[-2].minor.yy384.z) + pParse->sLastToken.n; +} +#line 1907 "parse.c" + break; + case 31: +#line 153 "parse.y" +{ + sqlite3AddColumn(pParse,&yymsp[0].minor.yy384); + yygotominor.yy384 = yymsp[0].minor.yy384; +} +#line 1915 "parse.c" + break; + case 32: + case 33: + case 34: + case 35: + case 36: + case 250: +#line 163 "parse.y" +{yygotominor.yy384 = yymsp[0].minor.yy0;} +#line 1925 "parse.c" + break; + case 38: +#line 222 "parse.y" +{sqlite3AddColumnType(pParse,&yymsp[0].minor.yy384);} +#line 1930 "parse.c" + break; + case 39: + case 42: + case 119: + case 120: + case 131: + case 150: + case 238: + case 248: + case 249: +#line 223 "parse.y" +{yygotominor.yy384 = yymsp[0].minor.yy384;} +#line 1943 "parse.c" + break; + case 40: +#line 224 "parse.y" +{ + yygotominor.yy384.z = yymsp[-3].minor.yy384.z; + yygotominor.yy384.n = &yymsp[0].minor.yy0.z[yymsp[0].minor.yy0.n] - yymsp[-3].minor.yy384.z; +} +#line 1951 "parse.c" + break; + case 41: +#line 228 "parse.y" +{ + yygotominor.yy384.z = yymsp[-5].minor.yy384.z; + yygotominor.yy384.n = &yymsp[0].minor.yy0.z[yymsp[0].minor.yy0.n] - yymsp[-5].minor.yy384.z; +} +#line 1959 "parse.c" + break; + case 43: +#line 234 "parse.y" +{yygotominor.yy384.z=yymsp[-1].minor.yy384.z; yygotominor.yy384.n=yymsp[0].minor.yy384.n+(yymsp[0].minor.yy384.z-yymsp[-1].minor.yy384.z);} +#line 1964 "parse.c" + break; + case 44: +#line 236 "parse.y" +{ yygotominor.yy230 = atoi((char*)yymsp[0].minor.yy384.z); } +#line 1969 "parse.c" + break; + case 45: +#line 237 "parse.y" +{ yygotominor.yy230 = -atoi((char*)yymsp[0].minor.yy384.z); } +#line 1974 "parse.c" + break; + case 50: + case 52: +#line 246 "parse.y" +{sqlite3AddDefaultValue(pParse,yymsp[0].minor.yy178);} +#line 1980 "parse.c" + break; + case 51: +#line 247 "parse.y" +{sqlite3AddDefaultValue(pParse,yymsp[-1].minor.yy178);} +#line 1985 "parse.c" + break; + case 53: +#line 249 "parse.y" +{ + Expr *p = sqlite3Expr(TK_UMINUS, yymsp[0].minor.yy178, 0, 0); + sqlite3AddDefaultValue(pParse,p); +} +#line 1993 "parse.c" + break; + case 54: +#line 253 "parse.y" +{ + Expr *p = sqlite3Expr(TK_STRING, 0, 0, &yymsp[0].minor.yy384); + sqlite3AddDefaultValue(pParse,p); +} +#line 2001 "parse.c" + break; + case 56: +#line 262 "parse.y" +{sqlite3AddNotNull(pParse, yymsp[0].minor.yy230);} +#line 2006 "parse.c" + break; + case 57: +#line 264 "parse.y" +{sqlite3AddPrimaryKey(pParse,0,yymsp[-1].minor.yy230,yymsp[0].minor.yy230,yymsp[-2].minor.yy230);} +#line 2011 "parse.c" + break; + case 58: +#line 265 "parse.y" +{sqlite3CreateIndex(pParse,0,0,0,0,yymsp[0].minor.yy230,0,0,0,0);} +#line 2016 "parse.c" + break; + case 59: +#line 266 "parse.y" +{sqlite3AddCheckConstraint(pParse,yymsp[-1].minor.yy178);} +#line 2021 "parse.c" + break; + case 60: +#line 268 "parse.y" +{sqlite3CreateForeignKey(pParse,0,&yymsp[-2].minor.yy384,yymsp[-1].minor.yy462,yymsp[0].minor.yy230);} +#line 2026 "parse.c" + break; + case 61: +#line 269 "parse.y" +{sqlite3DeferForeignKey(pParse,yymsp[0].minor.yy230);} +#line 2031 "parse.c" + break; + case 62: +#line 270 "parse.y" +{sqlite3AddCollateType(pParse, (char*)yymsp[0].minor.yy384.z, yymsp[0].minor.yy384.n);} +#line 2036 "parse.c" + break; + case 65: +#line 283 "parse.y" +{ yygotominor.yy230 = OE_Restrict * 0x010101; } +#line 2041 "parse.c" + break; + case 66: +#line 284 "parse.y" +{ yygotominor.yy230 = (yymsp[-1].minor.yy230 & yymsp[0].minor.yy13.mask) | yymsp[0].minor.yy13.value; } +#line 2046 "parse.c" + break; + case 67: +#line 286 "parse.y" +{ yygotominor.yy13.value = 0; yygotominor.yy13.mask = 0x000000; } +#line 2051 "parse.c" + break; + case 68: +#line 287 "parse.y" +{ yygotominor.yy13.value = yymsp[0].minor.yy230; yygotominor.yy13.mask = 0x0000ff; } +#line 2056 "parse.c" + break; + case 69: +#line 288 "parse.y" +{ yygotominor.yy13.value = yymsp[0].minor.yy230<<8; yygotominor.yy13.mask = 0x00ff00; } +#line 2061 "parse.c" + break; + case 70: +#line 289 "parse.y" +{ yygotominor.yy13.value = yymsp[0].minor.yy230<<16; yygotominor.yy13.mask = 0xff0000; } +#line 2066 "parse.c" + break; + case 71: +#line 291 "parse.y" +{ yygotominor.yy230 = OE_SetNull; } +#line 2071 "parse.c" + break; + case 72: +#line 292 "parse.y" +{ yygotominor.yy230 = OE_SetDflt; } +#line 2076 "parse.c" + break; + case 73: +#line 293 "parse.y" +{ yygotominor.yy230 = OE_Cascade; } +#line 2081 "parse.c" + break; + case 74: +#line 294 "parse.y" +{ yygotominor.yy230 = OE_Restrict; } +#line 2086 "parse.c" + break; + case 75: + case 76: + case 91: + case 93: + case 95: + case 96: + case 167: +#line 296 "parse.y" +{yygotominor.yy230 = yymsp[0].minor.yy230;} +#line 2097 "parse.c" + break; + case 80: +#line 306 "parse.y" +{yygotominor.yy384.n = 0; yygotominor.yy384.z = 0;} +#line 2102 "parse.c" + break; + case 81: +#line 307 "parse.y" +{yygotominor.yy384 = yymsp[-1].minor.yy0;} +#line 2107 "parse.c" + break; + case 86: +#line 313 "parse.y" +{sqlite3AddPrimaryKey(pParse,yymsp[-3].minor.yy462,yymsp[0].minor.yy230,yymsp[-2].minor.yy230,0);} +#line 2112 "parse.c" + break; + case 87: +#line 315 "parse.y" +{sqlite3CreateIndex(pParse,0,0,0,yymsp[-2].minor.yy462,yymsp[0].minor.yy230,0,0,0,0);} +#line 2117 "parse.c" + break; + case 88: +#line 316 "parse.y" +{sqlite3AddCheckConstraint(pParse,yymsp[-2].minor.yy178);} +#line 2122 "parse.c" + break; + case 89: +#line 318 "parse.y" +{ + sqlite3CreateForeignKey(pParse, yymsp[-6].minor.yy462, &yymsp[-3].minor.yy384, yymsp[-2].minor.yy462, yymsp[-1].minor.yy230); + sqlite3DeferForeignKey(pParse, yymsp[0].minor.yy230); +} +#line 2130 "parse.c" + break; + case 92: + case 94: +#line 332 "parse.y" +{yygotominor.yy230 = OE_Default;} +#line 2136 "parse.c" + break; + case 97: +#line 337 "parse.y" +{yygotominor.yy230 = OE_Ignore;} +#line 2141 "parse.c" + break; + case 98: + case 168: +#line 338 "parse.y" +{yygotominor.yy230 = OE_Replace;} +#line 2147 "parse.c" + break; + case 99: +#line 342 "parse.y" +{ + sqlite3DropTable(pParse, yymsp[0].minor.yy285, 0, yymsp[-1].minor.yy230); +} +#line 2154 "parse.c" + break; + case 102: +#line 352 "parse.y" +{ + sqlite3CreateView(pParse, &yymsp[-6].minor.yy0, &yymsp[-3].minor.yy384, &yymsp[-2].minor.yy384, yymsp[0].minor.yy239, yymsp[-5].minor.yy230); +} +#line 2161 "parse.c" + break; + case 103: +#line 355 "parse.y" +{ + sqlite3DropTable(pParse, yymsp[0].minor.yy285, 1, yymsp[-1].minor.yy230); +} +#line 2168 "parse.c" + break; + case 104: +#line 362 "parse.y" +{ + sqlite3Select(pParse, yymsp[0].minor.yy239, SRT_Callback, 0, 0, 0, 0, 0); + sqlite3SelectDelete(yymsp[0].minor.yy239); +} +#line 2176 "parse.c" + break; + case 105: + case 128: +#line 372 "parse.y" +{yygotominor.yy239 = yymsp[0].minor.yy239;} +#line 2182 "parse.c" + break; + case 106: +#line 374 "parse.y" +{ + if( yymsp[0].minor.yy239 ){ + yymsp[0].minor.yy239->op = yymsp[-1].minor.yy230; + yymsp[0].minor.yy239->pPrior = yymsp[-2].minor.yy239; + } + yygotominor.yy239 = yymsp[0].minor.yy239; +} +#line 2193 "parse.c" + break; + case 108: +#line 383 "parse.y" +{yygotominor.yy230 = TK_ALL;} +#line 2198 "parse.c" + break; + case 110: +#line 387 "parse.y" +{ + yygotominor.yy239 = sqlite3SelectNew(yymsp[-6].minor.yy462,yymsp[-5].minor.yy285,yymsp[-4].minor.yy178,yymsp[-3].minor.yy462,yymsp[-2].minor.yy178,yymsp[-1].minor.yy462,yymsp[-7].minor.yy230,yymsp[0].minor.yy270.pLimit,yymsp[0].minor.yy270.pOffset); +} +#line 2205 "parse.c" + break; + case 114: + case 235: +#line 408 "parse.y" +{yygotominor.yy462 = yymsp[-1].minor.yy462;} +#line 2211 "parse.c" + break; + case 115: + case 141: + case 151: + case 234: +#line 409 "parse.y" +{yygotominor.yy462 = 0;} +#line 2219 "parse.c" + break; + case 116: +#line 410 "parse.y" +{ + yygotominor.yy462 = sqlite3ExprListAppend(yymsp[-2].minor.yy462,yymsp[-1].minor.yy178,yymsp[0].minor.yy384.n?&yymsp[0].minor.yy384:0); +} +#line 2226 "parse.c" + break; + case 117: +#line 413 "parse.y" +{ + yygotominor.yy462 = sqlite3ExprListAppend(yymsp[-1].minor.yy462, sqlite3Expr(TK_ALL, 0, 0, 0), 0); +} +#line 2233 "parse.c" + break; + case 118: +#line 416 "parse.y" +{ + Expr *pRight = sqlite3Expr(TK_ALL, 0, 0, 0); + Expr *pLeft = sqlite3Expr(TK_ID, 0, 0, &yymsp[-2].minor.yy384); + yygotominor.yy462 = sqlite3ExprListAppend(yymsp[-3].minor.yy462, sqlite3Expr(TK_DOT, pLeft, pRight, 0), 0); +} +#line 2242 "parse.c" + break; + case 121: +#line 428 "parse.y" +{yygotominor.yy384.n = 0;} +#line 2247 "parse.c" + break; + case 122: +#line 440 "parse.y" +{yygotominor.yy285 = sqliteMalloc(sizeof(*yygotominor.yy285));} +#line 2252 "parse.c" + break; + case 123: +#line 441 "parse.y" +{yygotominor.yy285 = yymsp[0].minor.yy285;} +#line 2257 "parse.c" + break; + case 124: +#line 446 "parse.y" +{ + yygotominor.yy285 = yymsp[-1].minor.yy285; + if( yygotominor.yy285 && yygotominor.yy285->nSrc>0 ) yygotominor.yy285->a[yygotominor.yy285->nSrc-1].jointype = yymsp[0].minor.yy230; +} +#line 2265 "parse.c" + break; + case 125: +#line 450 "parse.y" +{yygotominor.yy285 = 0;} +#line 2270 "parse.c" + break; + case 126: +#line 451 "parse.y" +{ + yygotominor.yy285 = sqlite3SrcListAppend(yymsp[-5].minor.yy285,&yymsp[-4].minor.yy384,&yymsp[-3].minor.yy384); + if( yymsp[-2].minor.yy384.n ) sqlite3SrcListAddAlias(yygotominor.yy285,&yymsp[-2].minor.yy384); + if( yymsp[-1].minor.yy178 ){ + if( yygotominor.yy285 && yygotominor.yy285->nSrc>1 ){ yygotominor.yy285->a[yygotominor.yy285->nSrc-2].pOn = yymsp[-1].minor.yy178; } + else { sqlite3ExprDelete(yymsp[-1].minor.yy178); } + } + if( yymsp[0].minor.yy160 ){ + if( yygotominor.yy285 && yygotominor.yy285->nSrc>1 ){ yygotominor.yy285->a[yygotominor.yy285->nSrc-2].pUsing = yymsp[0].minor.yy160; } + else { sqlite3IdListDelete(yymsp[0].minor.yy160); } + } +} +#line 2286 "parse.c" + break; + case 127: +#line 465 "parse.y" +{ + yygotominor.yy285 = sqlite3SrcListAppend(yymsp[-6].minor.yy285,0,0); + yygotominor.yy285->a[yygotominor.yy285->nSrc-1].pSelect = yymsp[-4].minor.yy239; + if( yymsp[-2].minor.yy384.n ) sqlite3SrcListAddAlias(yygotominor.yy285,&yymsp[-2].minor.yy384); + if( yymsp[-1].minor.yy178 ){ + if( yygotominor.yy285 && yygotominor.yy285->nSrc>1 ){ yygotominor.yy285->a[yygotominor.yy285->nSrc-2].pOn = yymsp[-1].minor.yy178; } + else { sqlite3ExprDelete(yymsp[-1].minor.yy178); } + } + if( yymsp[0].minor.yy160 ){ + if( yygotominor.yy285 && yygotominor.yy285->nSrc>1 ){ yygotominor.yy285->a[yygotominor.yy285->nSrc-2].pUsing = yymsp[0].minor.yy160; } + else { sqlite3IdListDelete(yymsp[0].minor.yy160); } + } + } +#line 2303 "parse.c" + break; + case 129: +#line 486 "parse.y" +{ + yygotominor.yy239 = sqlite3SelectNew(0,yymsp[0].minor.yy285,0,0,0,0,0,0,0); + } +#line 2310 "parse.c" + break; + case 130: +#line 492 "parse.y" +{yygotominor.yy384.z=0; yygotominor.yy384.n=0;} +#line 2315 "parse.c" + break; + case 132: +#line 497 "parse.y" +{yygotominor.yy285 = sqlite3SrcListAppend(0,&yymsp[-1].minor.yy384,&yymsp[0].minor.yy384);} +#line 2320 "parse.c" + break; + case 133: +#line 501 "parse.y" +{ yygotominor.yy230 = JT_INNER; } +#line 2325 "parse.c" + break; + case 134: +#line 502 "parse.y" +{ yygotominor.yy230 = sqlite3JoinType(pParse,&yymsp[-1].minor.yy0,0,0); } +#line 2330 "parse.c" + break; + case 135: +#line 503 "parse.y" +{ yygotominor.yy230 = sqlite3JoinType(pParse,&yymsp[-2].minor.yy0,&yymsp[-1].minor.yy384,0); } +#line 2335 "parse.c" + break; + case 136: +#line 505 "parse.y" +{ yygotominor.yy230 = sqlite3JoinType(pParse,&yymsp[-3].minor.yy0,&yymsp[-2].minor.yy384,&yymsp[-1].minor.yy384); } +#line 2340 "parse.c" + break; + case 137: + case 145: + case 154: + case 161: + case 175: + case 200: + case 223: + case 225: + case 229: +#line 509 "parse.y" +{yygotominor.yy178 = yymsp[0].minor.yy178;} +#line 2353 "parse.c" + break; + case 138: + case 153: + case 160: + case 201: + case 224: + case 226: + case 230: +#line 510 "parse.y" +{yygotominor.yy178 = 0;} +#line 2364 "parse.c" + break; + case 139: + case 172: +#line 514 "parse.y" +{yygotominor.yy160 = yymsp[-1].minor.yy160;} +#line 2370 "parse.c" + break; + case 140: + case 171: +#line 515 "parse.y" +{yygotominor.yy160 = 0;} +#line 2376 "parse.c" + break; + case 142: + case 152: +#line 526 "parse.y" +{yygotominor.yy462 = yymsp[0].minor.yy462;} +#line 2382 "parse.c" + break; + case 143: +#line 527 "parse.y" +{ + yygotominor.yy462 = sqlite3ExprListAppend(yymsp[-4].minor.yy462,yymsp[-2].minor.yy178,yymsp[-1].minor.yy384.n>0?&yymsp[-1].minor.yy384:0); + if( yygotominor.yy462 ) yygotominor.yy462->a[yygotominor.yy462->nExpr-1].sortOrder = yymsp[0].minor.yy230; +} +#line 2390 "parse.c" + break; + case 144: +#line 531 "parse.y" +{ + yygotominor.yy462 = sqlite3ExprListAppend(0,yymsp[-2].minor.yy178,yymsp[-1].minor.yy384.n>0?&yymsp[-1].minor.yy384:0); + if( yygotominor.yy462 && yygotominor.yy462->a ) yygotominor.yy462->a[0].sortOrder = yymsp[0].minor.yy230; +} +#line 2398 "parse.c" + break; + case 146: + case 148: +#line 540 "parse.y" +{yygotominor.yy230 = SQLITE_SO_ASC;} +#line 2404 "parse.c" + break; + case 147: +#line 541 "parse.y" +{yygotominor.yy230 = SQLITE_SO_DESC;} +#line 2409 "parse.c" + break; + case 149: +#line 543 "parse.y" +{yygotominor.yy384.z = 0; yygotominor.yy384.n = 0;} +#line 2414 "parse.c" + break; + case 155: +#line 561 "parse.y" +{yygotominor.yy270.pLimit = 0; yygotominor.yy270.pOffset = 0;} +#line 2419 "parse.c" + break; + case 156: +#line 562 "parse.y" +{yygotominor.yy270.pLimit = yymsp[0].minor.yy178; yygotominor.yy270.pOffset = 0;} +#line 2424 "parse.c" + break; + case 157: +#line 564 "parse.y" +{yygotominor.yy270.pLimit = yymsp[-2].minor.yy178; yygotominor.yy270.pOffset = yymsp[0].minor.yy178;} +#line 2429 "parse.c" + break; + case 158: +#line 566 "parse.y" +{yygotominor.yy270.pOffset = yymsp[-2].minor.yy178; yygotominor.yy270.pLimit = yymsp[0].minor.yy178;} +#line 2434 "parse.c" + break; + case 159: +#line 570 "parse.y" +{sqlite3DeleteFrom(pParse,yymsp[-1].minor.yy285,yymsp[0].minor.yy178);} +#line 2439 "parse.c" + break; + case 162: +#line 581 "parse.y" +{sqlite3Update(pParse,yymsp[-3].minor.yy285,yymsp[-1].minor.yy462,yymsp[0].minor.yy178,yymsp[-4].minor.yy230);} +#line 2444 "parse.c" + break; + case 163: +#line 587 "parse.y" +{yygotominor.yy462 = sqlite3ExprListAppend(yymsp[-4].minor.yy462,yymsp[0].minor.yy178,&yymsp[-2].minor.yy384);} +#line 2449 "parse.c" + break; + case 164: +#line 588 "parse.y" +{yygotominor.yy462 = sqlite3ExprListAppend(0,yymsp[0].minor.yy178,&yymsp[-2].minor.yy384);} +#line 2454 "parse.c" + break; + case 165: +#line 594 "parse.y" +{sqlite3Insert(pParse, yymsp[-5].minor.yy285, yymsp[-1].minor.yy462, 0, yymsp[-4].minor.yy160, yymsp[-7].minor.yy230);} +#line 2459 "parse.c" + break; + case 166: +#line 596 "parse.y" +{sqlite3Insert(pParse, yymsp[-2].minor.yy285, 0, yymsp[0].minor.yy239, yymsp[-1].minor.yy160, yymsp[-4].minor.yy230);} +#line 2464 "parse.c" + break; + case 169: + case 227: +#line 606 "parse.y" +{yygotominor.yy462 = sqlite3ExprListAppend(yymsp[-2].minor.yy462,yymsp[0].minor.yy178,0);} +#line 2470 "parse.c" + break; + case 170: + case 228: +#line 607 "parse.y" +{yygotominor.yy462 = sqlite3ExprListAppend(0,yymsp[0].minor.yy178,0);} +#line 2476 "parse.c" + break; + case 173: +#line 616 "parse.y" +{yygotominor.yy160 = sqlite3IdListAppend(yymsp[-2].minor.yy160,&yymsp[0].minor.yy384);} +#line 2481 "parse.c" + break; + case 174: +#line 617 "parse.y" +{yygotominor.yy160 = sqlite3IdListAppend(0,&yymsp[0].minor.yy384);} +#line 2486 "parse.c" + break; + case 176: +#line 628 "parse.y" +{yygotominor.yy178 = yymsp[-1].minor.yy178; sqlite3ExprSpan(yygotominor.yy178,&yymsp[-2].minor.yy0,&yymsp[0].minor.yy0); } +#line 2491 "parse.c" + break; + case 177: + case 182: + case 183: +#line 629 "parse.y" +{yygotominor.yy178 = sqlite3Expr(yymsp[0].major, 0, 0, &yymsp[0].minor.yy0);} +#line 2498 "parse.c" + break; + case 178: + case 179: +#line 630 "parse.y" +{yygotominor.yy178 = sqlite3Expr(TK_ID, 0, 0, &yymsp[0].minor.yy0);} +#line 2504 "parse.c" + break; + case 180: +#line 632 "parse.y" +{ + Expr *temp1 = sqlite3Expr(TK_ID, 0, 0, &yymsp[-2].minor.yy384); + Expr *temp2 = sqlite3Expr(TK_ID, 0, 0, &yymsp[0].minor.yy384); + yygotominor.yy178 = sqlite3Expr(TK_DOT, temp1, temp2, 0); +} +#line 2513 "parse.c" + break; + case 181: +#line 637 "parse.y" +{ + Expr *temp1 = sqlite3Expr(TK_ID, 0, 0, &yymsp[-4].minor.yy384); + Expr *temp2 = sqlite3Expr(TK_ID, 0, 0, &yymsp[-2].minor.yy384); + Expr *temp3 = sqlite3Expr(TK_ID, 0, 0, &yymsp[0].minor.yy384); + Expr *temp4 = sqlite3Expr(TK_DOT, temp2, temp3, 0); + yygotominor.yy178 = sqlite3Expr(TK_DOT, temp1, temp4, 0); +} +#line 2524 "parse.c" + break; + case 184: +#line 646 "parse.y" +{yygotominor.yy178 = sqlite3RegisterExpr(pParse, &yymsp[0].minor.yy0);} +#line 2529 "parse.c" + break; + case 185: +#line 647 "parse.y" +{ + Token *pToken = &yymsp[0].minor.yy0; + Expr *pExpr = yygotominor.yy178 = sqlite3Expr(TK_VARIABLE, 0, 0, pToken); + sqlite3ExprAssignVarNumber(pParse, pExpr); +} +#line 2538 "parse.c" + break; + case 186: +#line 653 "parse.y" +{ + yygotominor.yy178 = sqlite3Expr(TK_CAST, yymsp[-3].minor.yy178, 0, &yymsp[-1].minor.yy384); + sqlite3ExprSpan(yygotominor.yy178,&yymsp[-5].minor.yy0,&yymsp[0].minor.yy0); +} +#line 2546 "parse.c" + break; + case 187: +#line 658 "parse.y" +{ + yygotominor.yy178 = sqlite3ExprFunction(yymsp[-1].minor.yy462, &yymsp[-4].minor.yy0); + sqlite3ExprSpan(yygotominor.yy178,&yymsp[-4].minor.yy0,&yymsp[0].minor.yy0); + if( yymsp[-2].minor.yy230 ){ + yygotominor.yy178->flags |= EP_Distinct; + } +} +#line 2557 "parse.c" + break; + case 188: +#line 665 "parse.y" +{ + yygotominor.yy178 = sqlite3ExprFunction(0, &yymsp[-3].minor.yy0); + sqlite3ExprSpan(yygotominor.yy178,&yymsp[-3].minor.yy0,&yymsp[0].minor.yy0); +} +#line 2565 "parse.c" + break; + case 189: +#line 669 "parse.y" +{ + /* The CURRENT_TIME, CURRENT_DATE, and CURRENT_TIMESTAMP values are + ** treated as functions that return constants */ + yygotominor.yy178 = sqlite3ExprFunction(0,&yymsp[0].minor.yy0); + if( yygotominor.yy178 ) yygotominor.yy178->op = TK_CONST_FUNC; +} +#line 2575 "parse.c" + break; + case 190: + case 191: + case 192: + case 193: + case 194: + case 195: + case 196: + case 197: +#line 675 "parse.y" +{yygotominor.yy178 = sqlite3Expr(yymsp[-1].major, yymsp[-2].minor.yy178, yymsp[0].minor.yy178, 0);} +#line 2587 "parse.c" + break; + case 198: +#line 685 "parse.y" +{yygotominor.yy440.eOperator = yymsp[0].minor.yy0; yygotominor.yy440.not = 0;} +#line 2592 "parse.c" + break; + case 199: +#line 686 "parse.y" +{yygotominor.yy440.eOperator = yymsp[0].minor.yy0; yygotominor.yy440.not = 1;} +#line 2597 "parse.c" + break; + case 202: +#line 691 "parse.y" +{ + ExprList *pList = sqlite3ExprListAppend(0, yymsp[-1].minor.yy178, 0); + pList = sqlite3ExprListAppend(pList, yymsp[-3].minor.yy178, 0); + if( yymsp[0].minor.yy178 ){ + pList = sqlite3ExprListAppend(pList, yymsp[0].minor.yy178, 0); + } + yygotominor.yy178 = sqlite3ExprFunction(pList, &yymsp[-2].minor.yy440.eOperator); + if( yymsp[-2].minor.yy440.not ) yygotominor.yy178 = sqlite3Expr(TK_NOT, yygotominor.yy178, 0, 0); + sqlite3ExprSpan(yygotominor.yy178, &yymsp[-3].minor.yy178->span, &yymsp[-1].minor.yy178->span); +} +#line 2611 "parse.c" + break; + case 203: +#line 702 "parse.y" +{ + yygotominor.yy178 = sqlite3Expr(yymsp[0].major, yymsp[-1].minor.yy178, 0, 0); + sqlite3ExprSpan(yygotominor.yy178,&yymsp[-1].minor.yy178->span,&yymsp[0].minor.yy0); +} +#line 2619 "parse.c" + break; + case 204: +#line 706 "parse.y" +{ + yygotominor.yy178 = sqlite3Expr(TK_ISNULL, yymsp[-2].minor.yy178, 0, 0); + sqlite3ExprSpan(yygotominor.yy178,&yymsp[-2].minor.yy178->span,&yymsp[0].minor.yy0); +} +#line 2627 "parse.c" + break; + case 205: +#line 710 "parse.y" +{ + yygotominor.yy178 = sqlite3Expr(TK_NOTNULL, yymsp[-2].minor.yy178, 0, 0); + sqlite3ExprSpan(yygotominor.yy178,&yymsp[-2].minor.yy178->span,&yymsp[0].minor.yy0); +} +#line 2635 "parse.c" + break; + case 206: +#line 714 "parse.y" +{ + yygotominor.yy178 = sqlite3Expr(TK_NOTNULL, yymsp[-3].minor.yy178, 0, 0); + sqlite3ExprSpan(yygotominor.yy178,&yymsp[-3].minor.yy178->span,&yymsp[0].minor.yy0); +} +#line 2643 "parse.c" + break; + case 207: +#line 718 "parse.y" +{ + yygotominor.yy178 = sqlite3Expr(yymsp[-1].major, yymsp[0].minor.yy178, 0, 0); + sqlite3ExprSpan(yygotominor.yy178,&yymsp[-1].minor.yy0,&yymsp[0].minor.yy178->span); +} +#line 2651 "parse.c" + break; + case 208: +#line 722 "parse.y" +{ + yygotominor.yy178 = sqlite3Expr(TK_UMINUS, yymsp[0].minor.yy178, 0, 0); + sqlite3ExprSpan(yygotominor.yy178,&yymsp[-1].minor.yy0,&yymsp[0].minor.yy178->span); +} +#line 2659 "parse.c" + break; + case 209: +#line 726 "parse.y" +{ + yygotominor.yy178 = sqlite3Expr(TK_UPLUS, yymsp[0].minor.yy178, 0, 0); + sqlite3ExprSpan(yygotominor.yy178,&yymsp[-1].minor.yy0,&yymsp[0].minor.yy178->span); +} +#line 2667 "parse.c" + break; + case 212: +#line 733 "parse.y" +{ + ExprList *pList = sqlite3ExprListAppend(0, yymsp[-2].minor.yy178, 0); + pList = sqlite3ExprListAppend(pList, yymsp[0].minor.yy178, 0); + yygotominor.yy178 = sqlite3Expr(TK_BETWEEN, yymsp[-4].minor.yy178, 0, 0); + if( yygotominor.yy178 ){ + yygotominor.yy178->pList = pList; + }else{ + sqlite3ExprListDelete(pList); + } + if( yymsp[-3].minor.yy230 ) yygotominor.yy178 = sqlite3Expr(TK_NOT, yygotominor.yy178, 0, 0); + sqlite3ExprSpan(yygotominor.yy178,&yymsp[-4].minor.yy178->span,&yymsp[0].minor.yy178->span); +} +#line 2683 "parse.c" + break; + case 215: +#line 749 "parse.y" +{ + yygotominor.yy178 = sqlite3Expr(TK_IN, yymsp[-4].minor.yy178, 0, 0); + if( yygotominor.yy178 ){ + yygotominor.yy178->pList = yymsp[-1].minor.yy462; + }else{ + sqlite3ExprListDelete(yymsp[-1].minor.yy462); + } + if( yymsp[-3].minor.yy230 ) yygotominor.yy178 = sqlite3Expr(TK_NOT, yygotominor.yy178, 0, 0); + sqlite3ExprSpan(yygotominor.yy178,&yymsp[-4].minor.yy178->span,&yymsp[0].minor.yy0); + } +#line 2697 "parse.c" + break; + case 216: +#line 759 "parse.y" +{ + yygotominor.yy178 = sqlite3Expr(TK_SELECT, 0, 0, 0); + if( yygotominor.yy178 ){ + yygotominor.yy178->pSelect = yymsp[-1].minor.yy239; + }else{ + sqlite3SelectDelete(yymsp[-1].minor.yy239); + } + sqlite3ExprSpan(yygotominor.yy178,&yymsp[-2].minor.yy0,&yymsp[0].minor.yy0); + } +#line 2710 "parse.c" + break; + case 217: +#line 768 "parse.y" +{ + yygotominor.yy178 = sqlite3Expr(TK_IN, yymsp[-4].minor.yy178, 0, 0); + if( yygotominor.yy178 ){ + yygotominor.yy178->pSelect = yymsp[-1].minor.yy239; + }else{ + sqlite3SelectDelete(yymsp[-1].minor.yy239); + } + if( yymsp[-3].minor.yy230 ) yygotominor.yy178 = sqlite3Expr(TK_NOT, yygotominor.yy178, 0, 0); + sqlite3ExprSpan(yygotominor.yy178,&yymsp[-4].minor.yy178->span,&yymsp[0].minor.yy0); + } +#line 2724 "parse.c" + break; + case 218: +#line 778 "parse.y" +{ + SrcList *pSrc = sqlite3SrcListAppend(0,&yymsp[-1].minor.yy384,&yymsp[0].minor.yy384); + yygotominor.yy178 = sqlite3Expr(TK_IN, yymsp[-3].minor.yy178, 0, 0); + if( yygotominor.yy178 ){ + yygotominor.yy178->pSelect = sqlite3SelectNew(0,pSrc,0,0,0,0,0,0,0); + }else{ + sqlite3SrcListDelete(pSrc); + } + if( yymsp[-2].minor.yy230 ) yygotominor.yy178 = sqlite3Expr(TK_NOT, yygotominor.yy178, 0, 0); + sqlite3ExprSpan(yygotominor.yy178,&yymsp[-3].minor.yy178->span,yymsp[0].minor.yy384.z?&yymsp[0].minor.yy384:&yymsp[-1].minor.yy384); + } +#line 2739 "parse.c" + break; + case 219: +#line 789 "parse.y" +{ + Expr *p = yygotominor.yy178 = sqlite3Expr(TK_EXISTS, 0, 0, 0); + if( p ){ + p->pSelect = yymsp[-1].minor.yy239; + sqlite3ExprSpan(p,&yymsp[-3].minor.yy0,&yymsp[0].minor.yy0); + }else{ + sqlite3SelectDelete(yymsp[-1].minor.yy239); + } + } +#line 2752 "parse.c" + break; + case 220: +#line 801 "parse.y" +{ + yygotominor.yy178 = sqlite3Expr(TK_CASE, yymsp[-3].minor.yy178, yymsp[-1].minor.yy178, 0); + if( yygotominor.yy178 ){ + yygotominor.yy178->pList = yymsp[-2].minor.yy462; + }else{ + sqlite3ExprListDelete(yymsp[-2].minor.yy462); + } + sqlite3ExprSpan(yygotominor.yy178, &yymsp[-4].minor.yy0, &yymsp[0].minor.yy0); +} +#line 2765 "parse.c" + break; + case 221: +#line 812 "parse.y" +{ + yygotominor.yy462 = sqlite3ExprListAppend(yymsp[-4].minor.yy462, yymsp[-2].minor.yy178, 0); + yygotominor.yy462 = sqlite3ExprListAppend(yygotominor.yy462, yymsp[0].minor.yy178, 0); +} +#line 2773 "parse.c" + break; + case 222: +#line 816 "parse.y" +{ + yygotominor.yy462 = sqlite3ExprListAppend(0, yymsp[-2].minor.yy178, 0); + yygotominor.yy462 = sqlite3ExprListAppend(yygotominor.yy462, yymsp[0].minor.yy178, 0); +} +#line 2781 "parse.c" + break; + case 231: +#line 843 "parse.y" +{ + sqlite3CreateIndex(pParse, &yymsp[-6].minor.yy384, &yymsp[-5].minor.yy384, sqlite3SrcListAppend(0,&yymsp[-3].minor.yy384,0), yymsp[-1].minor.yy462, yymsp[-9].minor.yy230, + &yymsp[-10].minor.yy0, &yymsp[0].minor.yy0, SQLITE_SO_ASC, yymsp[-7].minor.yy230); +} +#line 2789 "parse.c" + break; + case 232: + case 277: +#line 849 "parse.y" +{yygotominor.yy230 = OE_Abort;} +#line 2795 "parse.c" + break; + case 233: +#line 850 "parse.y" +{yygotominor.yy230 = OE_None;} +#line 2800 "parse.c" + break; + case 236: +#line 860 "parse.y" +{ + Expr *p = 0; + if( yymsp[-1].minor.yy384.n>0 ){ + p = sqlite3Expr(TK_COLUMN, 0, 0, 0); + if( p ) p->pColl = sqlite3LocateCollSeq(pParse, (char*)yymsp[-1].minor.yy384.z, yymsp[-1].minor.yy384.n); + } + yygotominor.yy462 = sqlite3ExprListAppend(yymsp[-4].minor.yy462, p, &yymsp[-2].minor.yy384); + if( yygotominor.yy462 ) yygotominor.yy462->a[yygotominor.yy462->nExpr-1].sortOrder = yymsp[0].minor.yy230; +} +#line 2813 "parse.c" + break; + case 237: +#line 869 "parse.y" +{ + Expr *p = 0; + if( yymsp[-1].minor.yy384.n>0 ){ + p = sqlite3Expr(TK_COLUMN, 0, 0, 0); + if( p ) p->pColl = sqlite3LocateCollSeq(pParse, (char*)yymsp[-1].minor.yy384.z, yymsp[-1].minor.yy384.n); + } + yygotominor.yy462 = sqlite3ExprListAppend(0, p, &yymsp[-2].minor.yy384); + if( yygotominor.yy462 ) yygotominor.yy462->a[yygotominor.yy462->nExpr-1].sortOrder = yymsp[0].minor.yy230; +} +#line 2826 "parse.c" + break; + case 239: +#line 883 "parse.y" +{sqlite3DropIndex(pParse, yymsp[0].minor.yy285, yymsp[-1].minor.yy230);} +#line 2831 "parse.c" + break; + case 240: + case 241: +#line 887 "parse.y" +{sqlite3Vacuum(pParse,0);} +#line 2837 "parse.c" + break; + case 242: + case 244: +#line 893 "parse.y" +{sqlite3Pragma(pParse,&yymsp[-3].minor.yy384,&yymsp[-2].minor.yy384,&yymsp[0].minor.yy384,0);} +#line 2843 "parse.c" + break; + case 243: +#line 894 "parse.y" +{sqlite3Pragma(pParse,&yymsp[-3].minor.yy384,&yymsp[-2].minor.yy384,&yymsp[0].minor.yy0,0);} +#line 2848 "parse.c" + break; + case 245: +#line 896 "parse.y" +{ + sqlite3Pragma(pParse,&yymsp[-3].minor.yy384,&yymsp[-2].minor.yy384,&yymsp[0].minor.yy384,1); +} +#line 2855 "parse.c" + break; + case 246: +#line 899 "parse.y" +{sqlite3Pragma(pParse,&yymsp[-4].minor.yy384,&yymsp[-3].minor.yy384,&yymsp[-1].minor.yy384,0);} +#line 2860 "parse.c" + break; + case 247: +#line 900 "parse.y" +{sqlite3Pragma(pParse,&yymsp[-1].minor.yy384,&yymsp[0].minor.yy384,0,0);} +#line 2865 "parse.c" + break; + case 253: +#line 912 "parse.y" +{ + Token all; + all.z = yymsp[-3].minor.yy384.z; + all.n = (yymsp[0].minor.yy0.z - yymsp[-3].minor.yy384.z) + yymsp[0].minor.yy0.n; + sqlite3FinishTrigger(pParse, yymsp[-1].minor.yy247, &all); +} +#line 2875 "parse.c" + break; + case 254: +#line 921 "parse.y" +{ + sqlite3BeginTrigger(pParse, &yymsp[-7].minor.yy384, &yymsp[-6].minor.yy384, yymsp[-5].minor.yy230, yymsp[-4].minor.yy132.a, yymsp[-4].minor.yy132.b, yymsp[-2].minor.yy285, yymsp[-1].minor.yy230, yymsp[0].minor.yy178, yymsp[-9].minor.yy230); + yygotominor.yy384 = (yymsp[-6].minor.yy384.n==0?yymsp[-7].minor.yy384:yymsp[-6].minor.yy384); +} +#line 2883 "parse.c" + break; + case 255: + case 258: +#line 927 "parse.y" +{ yygotominor.yy230 = TK_BEFORE; } +#line 2889 "parse.c" + break; + case 256: +#line 928 "parse.y" +{ yygotominor.yy230 = TK_AFTER; } +#line 2894 "parse.c" + break; + case 257: +#line 929 "parse.y" +{ yygotominor.yy230 = TK_INSTEAD;} +#line 2899 "parse.c" + break; + case 259: + case 260: +#line 934 "parse.y" +{yygotominor.yy132.a = yymsp[0].major; yygotominor.yy132.b = 0;} +#line 2905 "parse.c" + break; + case 261: +#line 936 "parse.y" +{yygotominor.yy132.a = TK_UPDATE; yygotominor.yy132.b = yymsp[0].minor.yy160;} +#line 2910 "parse.c" + break; + case 262: + case 263: +#line 939 "parse.y" +{ yygotominor.yy230 = TK_ROW; } +#line 2916 "parse.c" + break; + case 264: +#line 941 "parse.y" +{ yygotominor.yy230 = TK_STATEMENT; } +#line 2921 "parse.c" + break; + case 265: +#line 945 "parse.y" +{ yygotominor.yy178 = 0; } +#line 2926 "parse.c" + break; + case 266: +#line 946 "parse.y" +{ yygotominor.yy178 = yymsp[0].minor.yy178; } +#line 2931 "parse.c" + break; + case 267: +#line 950 "parse.y" +{ + yymsp[-2].minor.yy247->pNext = yymsp[0].minor.yy247; + yygotominor.yy247 = yymsp[-2].minor.yy247; +} +#line 2939 "parse.c" + break; + case 268: +#line 954 "parse.y" +{ yygotominor.yy247 = 0; } +#line 2944 "parse.c" + break; + case 269: +#line 960 "parse.y" +{ yygotominor.yy247 = sqlite3TriggerUpdateStep(&yymsp[-3].minor.yy384, yymsp[-1].minor.yy462, yymsp[0].minor.yy178, yymsp[-4].minor.yy230); } +#line 2949 "parse.c" + break; + case 270: +#line 965 "parse.y" +{yygotominor.yy247 = sqlite3TriggerInsertStep(&yymsp[-5].minor.yy384, yymsp[-4].minor.yy160, yymsp[-1].minor.yy462, 0, yymsp[-7].minor.yy230);} +#line 2954 "parse.c" + break; + case 271: +#line 968 "parse.y" +{yygotominor.yy247 = sqlite3TriggerInsertStep(&yymsp[-2].minor.yy384, yymsp[-1].minor.yy160, 0, yymsp[0].minor.yy239, yymsp[-4].minor.yy230);} +#line 2959 "parse.c" + break; + case 272: +#line 972 "parse.y" +{yygotominor.yy247 = sqlite3TriggerDeleteStep(&yymsp[-1].minor.yy384, yymsp[0].minor.yy178);} +#line 2964 "parse.c" + break; + case 273: +#line 975 "parse.y" +{yygotominor.yy247 = sqlite3TriggerSelectStep(yymsp[0].minor.yy239); } +#line 2969 "parse.c" + break; + case 274: +#line 978 "parse.y" +{ + yygotominor.yy178 = sqlite3Expr(TK_RAISE, 0, 0, 0); + yygotominor.yy178->iColumn = OE_Ignore; + sqlite3ExprSpan(yygotominor.yy178, &yymsp[-3].minor.yy0, &yymsp[0].minor.yy0); +} +#line 2978 "parse.c" + break; + case 275: +#line 983 "parse.y" +{ + yygotominor.yy178 = sqlite3Expr(TK_RAISE, 0, 0, &yymsp[-1].minor.yy384); + yygotominor.yy178->iColumn = yymsp[-3].minor.yy230; + sqlite3ExprSpan(yygotominor.yy178, &yymsp[-5].minor.yy0, &yymsp[0].minor.yy0); +} +#line 2987 "parse.c" + break; + case 276: +#line 991 "parse.y" +{yygotominor.yy230 = OE_Rollback;} +#line 2992 "parse.c" + break; + case 278: +#line 993 "parse.y" +{yygotominor.yy230 = OE_Fail;} +#line 2997 "parse.c" + break; + case 279: +#line 998 "parse.y" +{ + sqlite3DropTrigger(pParse,yymsp[0].minor.yy285); +} +#line 3004 "parse.c" + break; + case 280: +#line 1004 "parse.y" +{ + sqlite3Attach(pParse, yymsp[-3].minor.yy178, yymsp[-1].minor.yy178, yymsp[0].minor.yy292); +} +#line 3011 "parse.c" + break; + case 281: +#line 1009 "parse.y" +{ yygotominor.yy292 = 0; } +#line 3016 "parse.c" + break; + case 282: +#line 1010 "parse.y" +{ yygotominor.yy292 = yymsp[0].minor.yy178; } +#line 3021 "parse.c" + break; + case 285: +#line 1016 "parse.y" +{ + sqlite3Detach(pParse, yymsp[0].minor.yy178); +} +#line 3028 "parse.c" + break; + case 286: +#line 1022 "parse.y" +{sqlite3Reindex(pParse, 0, 0);} +#line 3033 "parse.c" + break; + case 287: +#line 1023 "parse.y" +{sqlite3Reindex(pParse, &yymsp[-1].minor.yy384, &yymsp[0].minor.yy384);} +#line 3038 "parse.c" + break; + case 288: +#line 1028 "parse.y" +{sqlite3Analyze(pParse, 0, 0);} +#line 3043 "parse.c" + break; + case 289: +#line 1029 "parse.y" +{sqlite3Analyze(pParse, &yymsp[-1].minor.yy384, &yymsp[0].minor.yy384);} +#line 3048 "parse.c" + break; + case 290: +#line 1034 "parse.y" +{ + sqlite3AlterRenameTable(pParse,yymsp[-3].minor.yy285,&yymsp[0].minor.yy384); +} +#line 3055 "parse.c" + break; + case 291: +#line 1037 "parse.y" +{ + sqlite3AlterFinishAddColumn(pParse, &yymsp[0].minor.yy384); +} +#line 3062 "parse.c" + break; + case 292: +#line 1040 "parse.y" +{ + sqlite3AlterBeginAddColumn(pParse, yymsp[0].minor.yy285); +} +#line 3069 "parse.c" + break; + }; + yygoto = yyRuleInfo[yyruleno].lhs; + yysize = yyRuleInfo[yyruleno].nrhs; + yypParser->yyidx -= yysize; + yyact = yy_find_reduce_action(yymsp[-yysize].stateno,yygoto); + if( yyact < YYNSTATE ){ +#ifdef NDEBUG + /* If we are not debugging and the reduce action popped at least + ** one element off the stack, then we can push the new element back + ** onto the stack here, and skip the stack overflow test in yy_shift(). + ** That gives a significant speed improvement. */ + if( yysize ){ + yypParser->yyidx++; + yymsp -= yysize-1; + yymsp->stateno = yyact; + yymsp->major = yygoto; + yymsp->minor = yygotominor; + }else +#endif + { + yy_shift(yypParser,yyact,yygoto,&yygotominor); + } + }else if( yyact == YYNSTATE + YYNRULE + 1 ){ + yy_accept(yypParser); + } +} + +/* +** The following code executes when the parse fails +*/ +static void yy_parse_failed( + yyParser *yypParser /* The parser */ +){ + sqlite3ParserARG_FETCH; +#ifndef NDEBUG + if( yyTraceFILE ){ + fprintf(yyTraceFILE,"%sFail!\n",yyTracePrompt); + } +#endif + while( yypParser->yyidx>=0 ) yy_pop_parser_stack(yypParser); + /* Here code is inserted which will be executed whenever the + ** parser fails */ + sqlite3ParserARG_STORE; /* Suppress warning about unused %extra_argument variable */ +} + +/* +** The following code executes when a syntax error first occurs. +*/ +static void yy_syntax_error( + yyParser *yypParser, /* The parser */ + int yymajor, /* The major type of the error token */ + YYMINORTYPE yyminor /* The minor type of the error token */ +){ + sqlite3ParserARG_FETCH; +#define TOKEN (yyminor.yy0) +#line 34 "parse.y" + + if( pParse->zErrMsg==0 ){ + if( TOKEN.z[0] ){ + sqlite3ErrorMsg(pParse, "near \"%T\": syntax error", &TOKEN); + }else{ + sqlite3ErrorMsg(pParse, "incomplete SQL statement"); + } + } +#line 3136 "parse.c" + sqlite3ParserARG_STORE; /* Suppress warning about unused %extra_argument variable */ +} + +/* +** The following is executed when the parser accepts +*/ +static void yy_accept( + yyParser *yypParser /* The parser */ +){ + sqlite3ParserARG_FETCH; +#ifndef NDEBUG + if( yyTraceFILE ){ + fprintf(yyTraceFILE,"%sAccept!\n",yyTracePrompt); + } +#endif + while( yypParser->yyidx>=0 ) yy_pop_parser_stack(yypParser); + /* Here code is inserted which will be executed whenever the + ** parser accepts */ + sqlite3ParserARG_STORE; /* Suppress warning about unused %extra_argument variable */ +} + +/* The main parser program. +** The first argument is a pointer to a structure obtained from +** "sqlite3ParserAlloc" which describes the current state of the parser. +** The second argument is the major token number. The third is +** the minor token. The fourth optional argument is whatever the +** user wants (and specified in the grammar) and is available for +** use by the action routines. +** +** Inputs: +**
    +**
  • A pointer to the parser (an opaque structure.) +**
  • The major token number. +**
  • The minor token number. +**
  • An option argument of a grammar-specified type. +**
+** +** Outputs: +** None. +*/ +void sqlite3Parser( + void *yyp, /* The parser */ + int yymajor, /* The major token code number */ + sqlite3ParserTOKENTYPE yyminor /* The value for the token */ + sqlite3ParserARG_PDECL /* Optional %extra_argument parameter */ +){ + YYMINORTYPE yyminorunion; + int yyact; /* The parser action. */ + int yyendofinput; /* True if we are at the end of input */ + int yyerrorhit = 0; /* True if yymajor has invoked an error */ + yyParser *yypParser; /* The parser */ + + /* (re)initialize the parser, if necessary */ + yypParser = (yyParser*)yyp; + if( yypParser->yyidx<0 ){ + /* if( yymajor==0 ) return; // not sure why this was here... */ + yypParser->yyidx = 0; + yypParser->yyerrcnt = -1; + yypParser->yystack[0].stateno = 0; + yypParser->yystack[0].major = 0; + } + yyminorunion.yy0 = yyminor; + yyendofinput = (yymajor==0); + sqlite3ParserARG_STORE; + +#ifndef NDEBUG + if( yyTraceFILE ){ + fprintf(yyTraceFILE,"%sInput %s\n",yyTracePrompt,yyTokenName[yymajor]); + } +#endif + + do{ + yyact = yy_find_shift_action(yypParser,yymajor); + if( yyactyyerrcnt--; + if( yyendofinput && yypParser->yyidx>=0 ){ + yymajor = 0; + }else{ + yymajor = YYNOCODE; + } + }else if( yyact < YYNSTATE + YYNRULE ){ + yy_reduce(yypParser,yyact-YYNSTATE); + }else if( yyact == YY_ERROR_ACTION ){ + int yymx; +#ifndef NDEBUG + if( yyTraceFILE ){ + fprintf(yyTraceFILE,"%sSyntax Error!\n",yyTracePrompt); + } +#endif +#ifdef YYERRORSYMBOL + /* A syntax error has occurred. + ** The response to an error depends upon whether or not the + ** grammar defines an error token "ERROR". + ** + ** This is what we do if the grammar does define ERROR: + ** + ** * Call the %syntax_error function. + ** + ** * Begin popping the stack until we enter a state where + ** it is legal to shift the error symbol, then shift + ** the error symbol. + ** + ** * Set the error count to three. + ** + ** * Begin accepting and shifting new tokens. No new error + ** processing will occur until three tokens have been + ** shifted successfully. + ** + */ + if( yypParser->yyerrcnt<0 ){ + yy_syntax_error(yypParser,yymajor,yyminorunion); + } + yymx = yypParser->yystack[yypParser->yyidx].major; + if( yymx==YYERRORSYMBOL || yyerrorhit ){ +#ifndef NDEBUG + if( yyTraceFILE ){ + fprintf(yyTraceFILE,"%sDiscard input token %s\n", + yyTracePrompt,yyTokenName[yymajor]); + } +#endif + yy_destructor(yymajor,&yyminorunion); + yymajor = YYNOCODE; + }else{ + while( + yypParser->yyidx >= 0 && + yymx != YYERRORSYMBOL && + (yyact = yy_find_shift_action(yypParser,YYERRORSYMBOL)) >= YYNSTATE + ){ + yy_pop_parser_stack(yypParser); + } + if( yypParser->yyidx < 0 || yymajor==0 ){ + yy_destructor(yymajor,&yyminorunion); + yy_parse_failed(yypParser); + yymajor = YYNOCODE; + }else if( yymx!=YYERRORSYMBOL ){ + YYMINORTYPE u2; + u2.YYERRSYMDT = 0; + yy_shift(yypParser,yyact,YYERRORSYMBOL,&u2); + } + } + yypParser->yyerrcnt = 3; + yyerrorhit = 1; +#else /* YYERRORSYMBOL is not defined */ + /* This is what we do if the grammar does not define ERROR: + ** + ** * Report an error message, and throw away the input token. + ** + ** * If the input token is $, then fail the parse. + ** + ** As before, subsequent error messages are suppressed until + ** three input tokens have been successfully shifted. + */ + if( yypParser->yyerrcnt<=0 ){ + yy_syntax_error(yypParser,yymajor,yyminorunion); + } + yypParser->yyerrcnt = 3; + yy_destructor(yymajor,&yyminorunion); + if( yyendofinput ){ + yy_parse_failed(yypParser); + } + yymajor = YYNOCODE; +#endif + }else{ + yy_accept(yypParser); + yymajor = YYNOCODE; + } + }while( yymajor!=YYNOCODE && yypParser->yyidx>=0 ); + return; +} Added: external/sqlite-source-3.3.4/parse.h ============================================================================== --- (empty file) +++ external/sqlite-source-3.3.4/parse.h Mon Apr 3 07:54:59 2006 @@ -0,0 +1,151 @@ +#define TK_SEMI 1 +#define TK_EXPLAIN 2 +#define TK_QUERY 3 +#define TK_PLAN 4 +#define TK_BEGIN 5 +#define TK_TRANSACTION 6 +#define TK_DEFERRED 7 +#define TK_IMMEDIATE 8 +#define TK_EXCLUSIVE 9 +#define TK_COMMIT 10 +#define TK_END 11 +#define TK_ROLLBACK 12 +#define TK_CREATE 13 +#define TK_TABLE 14 +#define TK_IF 15 +#define TK_NOT 16 +#define TK_EXISTS 17 +#define TK_TEMP 18 +#define TK_LP 19 +#define TK_RP 20 +#define TK_AS 21 +#define TK_COMMA 22 +#define TK_ID 23 +#define TK_ABORT 24 +#define TK_AFTER 25 +#define TK_ANALYZE 26 +#define TK_ASC 27 +#define TK_ATTACH 28 +#define TK_BEFORE 29 +#define TK_CASCADE 30 +#define TK_CAST 31 +#define TK_CONFLICT 32 +#define TK_DATABASE 33 +#define TK_DESC 34 +#define TK_DETACH 35 +#define TK_EACH 36 +#define TK_FAIL 37 +#define TK_FOR 38 +#define TK_IGNORE 39 +#define TK_INITIALLY 40 +#define TK_INSTEAD 41 +#define TK_LIKE_KW 42 +#define TK_MATCH 43 +#define TK_KEY 44 +#define TK_OF 45 +#define TK_OFFSET 46 +#define TK_PRAGMA 47 +#define TK_RAISE 48 +#define TK_REPLACE 49 +#define TK_RESTRICT 50 +#define TK_ROW 51 +#define TK_STATEMENT 52 +#define TK_TRIGGER 53 +#define TK_VACUUM 54 +#define TK_VIEW 55 +#define TK_REINDEX 56 +#define TK_RENAME 57 +#define TK_CTIME_KW 58 +#define TK_OR 59 +#define TK_AND 60 +#define TK_IS 61 +#define TK_BETWEEN 62 +#define TK_IN 63 +#define TK_ISNULL 64 +#define TK_NOTNULL 65 +#define TK_NE 66 +#define TK_EQ 67 +#define TK_GT 68 +#define TK_LE 69 +#define TK_LT 70 +#define TK_GE 71 +#define TK_ESCAPE 72 +#define TK_BITAND 73 +#define TK_BITOR 74 +#define TK_LSHIFT 75 +#define TK_RSHIFT 76 +#define TK_PLUS 77 +#define TK_MINUS 78 +#define TK_STAR 79 +#define TK_SLASH 80 +#define TK_REM 81 +#define TK_CONCAT 82 +#define TK_UMINUS 83 +#define TK_UPLUS 84 +#define TK_BITNOT 85 +#define TK_STRING 86 +#define TK_JOIN_KW 87 +#define TK_CONSTRAINT 88 +#define TK_DEFAULT 89 +#define TK_NULL 90 +#define TK_PRIMARY 91 +#define TK_UNIQUE 92 +#define TK_CHECK 93 +#define TK_REFERENCES 94 +#define TK_COLLATE 95 +#define TK_AUTOINCR 96 +#define TK_ON 97 +#define TK_DELETE 98 +#define TK_UPDATE 99 +#define TK_INSERT 100 +#define TK_SET 101 +#define TK_DEFERRABLE 102 +#define TK_FOREIGN 103 +#define TK_DROP 104 +#define TK_UNION 105 +#define TK_ALL 106 +#define TK_EXCEPT 107 +#define TK_INTERSECT 108 +#define TK_SELECT 109 +#define TK_DISTINCT 110 +#define TK_DOT 111 +#define TK_FROM 112 +#define TK_JOIN 113 +#define TK_USING 114 +#define TK_ORDER 115 +#define TK_BY 116 +#define TK_GROUP 117 +#define TK_HAVING 118 +#define TK_LIMIT 119 +#define TK_WHERE 120 +#define TK_INTO 121 +#define TK_VALUES 122 +#define TK_INTEGER 123 +#define TK_FLOAT 124 +#define TK_BLOB 125 +#define TK_REGISTER 126 +#define TK_VARIABLE 127 +#define TK_CASE 128 +#define TK_WHEN 129 +#define TK_THEN 130 +#define TK_ELSE 131 +#define TK_INDEX 132 +#define TK_ALTER 133 +#define TK_TO 134 +#define TK_ADD 135 +#define TK_COLUMNKW 136 +#define TK_TO_TEXT 137 +#define TK_TO_BLOB 138 +#define TK_TO_NUMERIC 139 +#define TK_TO_INT 140 +#define TK_TO_REAL 141 +#define TK_END_OF_FILE 142 +#define TK_ILLEGAL 143 +#define TK_SPACE 144 +#define TK_UNCLOSED_STRING 145 +#define TK_COMMENT 146 +#define TK_FUNCTION 147 +#define TK_COLUMN 148 +#define TK_AGG_FUNCTION 149 +#define TK_AGG_COLUMN 150 +#define TK_CONST_FUNC 151 Added: external/sqlite-source-3.3.4/pragma.c ============================================================================== --- (empty file) +++ external/sqlite-source-3.3.4/pragma.c Mon Apr 3 07:54:59 2006 @@ -0,0 +1,961 @@ +/* +** 2003 April 6 +** +** The author disclaims copyright to this source code. In place of +** a legal notice, here is a blessing: +** +** May you do good and not evil. +** May you find forgiveness for yourself and forgive others. +** May you share freely, never taking more than you give. +** +************************************************************************* +** This file contains code used to implement the PRAGMA command. +** +** $Id: pragma.c,v 1.118 2006/02/11 01:25:51 drh Exp $ +*/ +#include "sqliteInt.h" +#include "os.h" +#include + +/* Ignore this whole file if pragmas are disabled +*/ +#if !defined(SQLITE_OMIT_PRAGMA) && !defined(SQLITE_OMIT_PARSER) + +#if defined(SQLITE_DEBUG) || defined(SQLITE_TEST) +# include "pager.h" +# include "btree.h" +#endif + +/* +** Interpret the given string as a safety level. Return 0 for OFF, +** 1 for ON or NORMAL and 2 for FULL. Return 1 for an empty or +** unrecognized string argument. +** +** Note that the values returned are one less that the values that +** should be passed into sqlite3BtreeSetSafetyLevel(). The is done +** to support legacy SQL code. The safety level used to be boolean +** and older scripts may have used numbers 0 for OFF and 1 for ON. +*/ +static int getSafetyLevel(const char *z){ + /* 123456789 123456789 */ + static const char zText[] = "onoffalseyestruefull"; + static const u8 iOffset[] = {0, 1, 2, 4, 9, 12, 16}; + static const u8 iLength[] = {2, 2, 3, 5, 3, 4, 4}; + static const u8 iValue[] = {1, 0, 0, 0, 1, 1, 2}; + int i, n; + if( isdigit(*z) ){ + return atoi(z); + } + n = strlen(z); + for(i=0; i='0' && z[0]<='2' ){ + return z[0] - '0'; + }else if( sqlite3StrICmp(z, "file")==0 ){ + return 1; + }else if( sqlite3StrICmp(z, "memory")==0 ){ + return 2; + }else{ + return 0; + } +} +#endif /* SQLITE_PAGER_PRAGMAS */ + +#ifndef SQLITE_OMIT_PAGER_PRAGMAS +/* +** Invalidate temp storage, either when the temp storage is changed +** from default, or when 'file' and the temp_store_directory has changed +*/ +static int invalidateTempStorage(Parse *pParse){ + sqlite3 *db = pParse->db; + if( db->aDb[1].pBt!=0 ){ + if( db->flags & SQLITE_InTrans ){ + sqlite3ErrorMsg(pParse, "temporary storage cannot be changed " + "from within a transaction"); + return SQLITE_ERROR; + } + sqlite3BtreeClose(db->aDb[1].pBt); + db->aDb[1].pBt = 0; + sqlite3ResetInternalSchema(db, 0); + } + return SQLITE_OK; +} +#endif /* SQLITE_PAGER_PRAGMAS */ + +#ifndef SQLITE_OMIT_PAGER_PRAGMAS +/* +** If the TEMP database is open, close it and mark the database schema +** as needing reloading. This must be done when using the TEMP_STORE +** or DEFAULT_TEMP_STORE pragmas. +*/ +static int changeTempStorage(Parse *pParse, const char *zStorageType){ + int ts = getTempStore(zStorageType); + sqlite3 *db = pParse->db; + if( db->temp_store==ts ) return SQLITE_OK; + if( invalidateTempStorage( pParse ) != SQLITE_OK ){ + return SQLITE_ERROR; + } + db->temp_store = ts; + return SQLITE_OK; +} +#endif /* SQLITE_PAGER_PRAGMAS */ + +/* +** Generate code to return a single integer value. +*/ +static void returnSingleInt(Parse *pParse, const char *zLabel, int value){ + Vdbe *v = sqlite3GetVdbe(pParse); + sqlite3VdbeAddOp(v, OP_Integer, value, 0); + if( pParse->explain==0 ){ + sqlite3VdbeSetNumCols(v, 1); + sqlite3VdbeSetColName(v, 0, COLNAME_NAME, zLabel, P3_STATIC); + } + sqlite3VdbeAddOp(v, OP_Callback, 1, 0); +} + +#ifndef SQLITE_OMIT_FLAG_PRAGMAS +/* +** Check to see if zRight and zLeft refer to a pragma that queries +** or changes one of the flags in db->flags. Return 1 if so and 0 if not. +** Also, implement the pragma. +*/ +static int flagPragma(Parse *pParse, const char *zLeft, const char *zRight){ + static const struct sPragmaType { + const char *zName; /* Name of the pragma */ + int mask; /* Mask for the db->flags value */ + } aPragma[] = { + { "vdbe_trace", SQLITE_VdbeTrace }, + { "sql_trace", SQLITE_SqlTrace }, + { "vdbe_listing", SQLITE_VdbeListing }, + { "full_column_names", SQLITE_FullColNames }, + { "short_column_names", SQLITE_ShortColNames }, + { "count_changes", SQLITE_CountRows }, + { "empty_result_callbacks", SQLITE_NullCallback }, + { "legacy_file_format", SQLITE_LegacyFileFmt }, + { "fullfsync", SQLITE_FullFSync }, +#ifndef SQLITE_OMIT_CHECK + { "ignore_check_constraints", SQLITE_IgnoreChecks }, +#endif + /* The following is VERY experimental */ + { "writable_schema", SQLITE_WriteSchema }, + { "omit_readlock", SQLITE_NoReadlock }, + + /* TODO: Maybe it shouldn't be possible to change the ReadUncommitted + ** flag if there are any active statements. */ + { "read_uncommitted", SQLITE_ReadUncommitted }, + }; + int i; + const struct sPragmaType *p; + for(i=0, p=aPragma; izName)==0 ){ + sqlite3 *db = pParse->db; + Vdbe *v; + v = sqlite3GetVdbe(pParse); + if( v ){ + if( zRight==0 ){ + returnSingleInt(pParse, p->zName, (db->flags & p->mask)!=0 ); + }else{ + if( getBoolean(zRight) ){ + db->flags |= p->mask; + }else{ + db->flags &= ~p->mask; + } + } + } + return 1; + } + } + return 0; +} +#endif /* SQLITE_OMIT_FLAG_PRAGMAS */ + +/* +** Process a pragma statement. +** +** Pragmas are of this form: +** +** PRAGMA [database.]id [= value] +** +** The identifier might also be a string. The value is a string, and +** identifier, or a number. If minusFlag is true, then the value is +** a number that was preceded by a minus sign. +** +** If the left side is "database.id" then pId1 is the database name +** and pId2 is the id. If the left side is just "id" then pId1 is the +** id and pId2 is any empty string. +*/ +void sqlite3Pragma( + Parse *pParse, + Token *pId1, /* First part of [database.]id field */ + Token *pId2, /* Second part of [database.]id field, or NULL */ + Token *pValue, /* Token for , or NULL */ + int minusFlag /* True if a '-' sign preceded */ +){ + char *zLeft = 0; /* Nul-terminated UTF-8 string */ + char *zRight = 0; /* Nul-terminated UTF-8 string , or NULL */ + const char *zDb = 0; /* The database name */ + Token *pId; /* Pointer to token */ + int iDb; /* Database index for */ + sqlite3 *db = pParse->db; + Db *pDb; + Vdbe *v = sqlite3GetVdbe(pParse); + if( v==0 ) return; + + /* Interpret the [database.] part of the pragma statement. iDb is the + ** index of the database this pragma is being applied to in db.aDb[]. */ + iDb = sqlite3TwoPartName(pParse, pId1, pId2, &pId); + if( iDb<0 ) return; + pDb = &db->aDb[iDb]; + + zLeft = sqlite3NameFromToken(pId); + if( !zLeft ) return; + if( minusFlag ){ + zRight = sqlite3MPrintf("-%T", pValue); + }else{ + zRight = sqlite3NameFromToken(pValue); + } + + zDb = ((iDb>0)?pDb->zName:0); + if( sqlite3AuthCheck(pParse, SQLITE_PRAGMA, zLeft, zRight, zDb) ){ + goto pragma_out; + } + +#ifndef SQLITE_OMIT_PAGER_PRAGMAS + /* + ** PRAGMA [database.]default_cache_size + ** PRAGMA [database.]default_cache_size=N + ** + ** The first form reports the current persistent setting for the + ** page cache size. The value returned is the maximum number of + ** pages in the page cache. The second form sets both the current + ** page cache size value and the persistent page cache size value + ** stored in the database file. + ** + ** The default cache size is stored in meta-value 2 of page 1 of the + ** database file. The cache size is actually the absolute value of + ** this memory location. The sign of meta-value 2 determines the + ** synchronous setting. A negative value means synchronous is off + ** and a positive value means synchronous is on. + */ + if( sqlite3StrICmp(zLeft,"default_cache_size")==0 ){ + static const VdbeOpList getCacheSize[] = { + { OP_ReadCookie, 0, 2, 0}, /* 0 */ + { OP_AbsValue, 0, 0, 0}, + { OP_Dup, 0, 0, 0}, + { OP_Integer, 0, 0, 0}, + { OP_Ne, 0, 6, 0}, + { OP_Integer, 0, 0, 0}, /* 5 */ + { OP_Callback, 1, 0, 0}, + }; + int addr; + if( sqlite3ReadSchema(pParse) ) goto pragma_out; + if( !zRight ){ + sqlite3VdbeSetNumCols(v, 1); + sqlite3VdbeSetColName(v, 0, COLNAME_NAME, "cache_size", P3_STATIC); + addr = sqlite3VdbeAddOpList(v, ArraySize(getCacheSize), getCacheSize); + sqlite3VdbeChangeP1(v, addr, iDb); + sqlite3VdbeChangeP1(v, addr+5, MAX_PAGES); + }else{ + int size = atoi(zRight); + if( size<0 ) size = -size; + sqlite3BeginWriteOperation(pParse, 0, iDb); + sqlite3VdbeAddOp(v, OP_Integer, size, 0); + sqlite3VdbeAddOp(v, OP_ReadCookie, iDb, 2); + addr = sqlite3VdbeAddOp(v, OP_Integer, 0, 0); + sqlite3VdbeAddOp(v, OP_Ge, 0, addr+3); + sqlite3VdbeAddOp(v, OP_Negative, 0, 0); + sqlite3VdbeAddOp(v, OP_SetCookie, iDb, 2); + pDb->pSchema->cache_size = size; + sqlite3BtreeSetCacheSize(pDb->pBt, pDb->pSchema->cache_size); + } + }else + + /* + ** PRAGMA [database.]page_size + ** PRAGMA [database.]page_size=N + ** + ** The first form reports the current setting for the + ** database page size in bytes. The second form sets the + ** database page size value. The value can only be set if + ** the database has not yet been created. + */ + if( sqlite3StrICmp(zLeft,"page_size")==0 ){ + Btree *pBt = pDb->pBt; + if( !zRight ){ + int size = pBt ? sqlite3BtreeGetPageSize(pBt) : 0; + returnSingleInt(pParse, "page_size", size); + }else{ + sqlite3BtreeSetPageSize(pBt, atoi(zRight), -1); + } + }else +#endif /* SQLITE_OMIT_PAGER_PRAGMAS */ + + /* + ** PRAGMA [database.]auto_vacuum + ** PRAGMA [database.]auto_vacuum=N + ** + ** Get or set the (boolean) value of the database 'auto-vacuum' parameter. + */ +#ifndef SQLITE_OMIT_AUTOVACUUM + if( sqlite3StrICmp(zLeft,"auto_vacuum")==0 ){ + Btree *pBt = pDb->pBt; + if( !zRight ){ + int auto_vacuum = + pBt ? sqlite3BtreeGetAutoVacuum(pBt) : SQLITE_DEFAULT_AUTOVACUUM; + returnSingleInt(pParse, "auto_vacuum", auto_vacuum); + }else{ + sqlite3BtreeSetAutoVacuum(pBt, getBoolean(zRight)); + } + }else +#endif + +#ifndef SQLITE_OMIT_PAGER_PRAGMAS + /* + ** PRAGMA [database.]cache_size + ** PRAGMA [database.]cache_size=N + ** + ** The first form reports the current local setting for the + ** page cache size. The local setting can be different from + ** the persistent cache size value that is stored in the database + ** file itself. The value returned is the maximum number of + ** pages in the page cache. The second form sets the local + ** page cache size value. It does not change the persistent + ** cache size stored on the disk so the cache size will revert + ** to its default value when the database is closed and reopened. + ** N should be a positive integer. + */ + if( sqlite3StrICmp(zLeft,"cache_size")==0 ){ + if( sqlite3ReadSchema(pParse) ) goto pragma_out; + if( !zRight ){ + returnSingleInt(pParse, "cache_size", pDb->pSchema->cache_size); + }else{ + int size = atoi(zRight); + if( size<0 ) size = -size; + pDb->pSchema->cache_size = size; + sqlite3BtreeSetCacheSize(pDb->pBt, pDb->pSchema->cache_size); + } + }else + + /* + ** PRAGMA temp_store + ** PRAGMA temp_store = "default"|"memory"|"file" + ** + ** Return or set the local value of the temp_store flag. Changing + ** the local value does not make changes to the disk file and the default + ** value will be restored the next time the database is opened. + ** + ** Note that it is possible for the library compile-time options to + ** override this setting + */ + if( sqlite3StrICmp(zLeft, "temp_store")==0 ){ + if( !zRight ){ + returnSingleInt(pParse, "temp_store", db->temp_store); + }else{ + changeTempStorage(pParse, zRight); + } + }else + + /* + ** PRAGMA temp_store_directory + ** PRAGMA temp_store_directory = ""|"directory_name" + ** + ** Return or set the local value of the temp_store_directory flag. Changing + ** the value sets a specific directory to be used for temporary files. + ** Setting to a null string reverts to the default temporary directory search. + ** If temporary directory is changed, then invalidateTempStorage. + ** + */ + if( sqlite3StrICmp(zLeft, "temp_store_directory")==0 ){ + if( !zRight ){ + if( sqlite3_temp_directory ){ + sqlite3VdbeSetNumCols(v, 1); + sqlite3VdbeSetColName(v, 0, COLNAME_NAME, + "temp_store_directory", P3_STATIC); + sqlite3VdbeOp3(v, OP_String8, 0, 0, sqlite3_temp_directory, 0); + sqlite3VdbeAddOp(v, OP_Callback, 1, 0); + } + }else{ + if( zRight[0] && !sqlite3OsIsDirWritable(zRight) ){ + sqlite3ErrorMsg(pParse, "not a writable directory"); + goto pragma_out; + } + if( TEMP_STORE==0 + || (TEMP_STORE==1 && db->temp_store<=1) + || (TEMP_STORE==2 && db->temp_store==1) + ){ + invalidateTempStorage(pParse); + } + sqliteFree(sqlite3_temp_directory); + if( zRight[0] ){ + sqlite3_temp_directory = zRight; + zRight = 0; + }else{ + sqlite3_temp_directory = 0; + } + } + }else + + /* + ** PRAGMA [database.]synchronous + ** PRAGMA [database.]synchronous=OFF|ON|NORMAL|FULL + ** + ** Return or set the local value of the synchronous flag. Changing + ** the local value does not make changes to the disk file and the + ** default value will be restored the next time the database is + ** opened. + */ + if( sqlite3StrICmp(zLeft,"synchronous")==0 ){ + if( sqlite3ReadSchema(pParse) ) goto pragma_out; + if( !zRight ){ + returnSingleInt(pParse, "synchronous", pDb->safety_level-1); + }else{ + if( !db->autoCommit ){ + sqlite3ErrorMsg(pParse, + "Safety level may not be changed inside a transaction"); + }else{ + pDb->safety_level = getSafetyLevel(zRight)+1; + } + } + }else +#endif /* SQLITE_OMIT_PAGER_PRAGMAS */ + +#ifndef SQLITE_OMIT_FLAG_PRAGMAS + if( flagPragma(pParse, zLeft, zRight) ){ + /* The flagPragma() subroutine also generates any necessary code + ** there is nothing more to do here */ + }else +#endif /* SQLITE_OMIT_FLAG_PRAGMAS */ + +#ifndef SQLITE_OMIT_SCHEMA_PRAGMAS + /* + ** PRAGMA table_info(
) + ** + ** Return a single row for each column of the named table. The columns of + ** the returned data set are: + ** + ** cid: Column id (numbered from left to right, starting at 0) + ** name: Column name + ** type: Column declaration type. + ** notnull: True if 'NOT NULL' is part of column declaration + ** dflt_value: The default value for the column, if any. + */ + if( sqlite3StrICmp(zLeft, "table_info")==0 && zRight ){ + Table *pTab; + if( sqlite3ReadSchema(pParse) ) goto pragma_out; + pTab = sqlite3FindTable(db, zRight, zDb); + if( pTab ){ + int i; + Column *pCol; + sqlite3VdbeSetNumCols(v, 6); + sqlite3VdbeSetColName(v, 0, COLNAME_NAME, "cid", P3_STATIC); + sqlite3VdbeSetColName(v, 1, COLNAME_NAME, "name", P3_STATIC); + sqlite3VdbeSetColName(v, 2, COLNAME_NAME, "type", P3_STATIC); + sqlite3VdbeSetColName(v, 3, COLNAME_NAME, "notnull", P3_STATIC); + sqlite3VdbeSetColName(v, 4, COLNAME_NAME, "dflt_value", P3_STATIC); + sqlite3VdbeSetColName(v, 5, COLNAME_NAME, "pk", P3_STATIC); + sqlite3ViewGetColumnNames(pParse, pTab); + for(i=0, pCol=pTab->aCol; inCol; i++, pCol++){ + sqlite3VdbeAddOp(v, OP_Integer, i, 0); + sqlite3VdbeOp3(v, OP_String8, 0, 0, pCol->zName, 0); + sqlite3VdbeOp3(v, OP_String8, 0, 0, + pCol->zType ? pCol->zType : "numeric", 0); + sqlite3VdbeAddOp(v, OP_Integer, pCol->notNull, 0); + sqlite3ExprCode(pParse, pCol->pDflt); + sqlite3VdbeAddOp(v, OP_Integer, pCol->isPrimKey, 0); + sqlite3VdbeAddOp(v, OP_Callback, 6, 0); + } + } + }else + + if( sqlite3StrICmp(zLeft, "index_info")==0 && zRight ){ + Index *pIdx; + Table *pTab; + if( sqlite3ReadSchema(pParse) ) goto pragma_out; + pIdx = sqlite3FindIndex(db, zRight, zDb); + if( pIdx ){ + int i; + pTab = pIdx->pTable; + sqlite3VdbeSetNumCols(v, 3); + sqlite3VdbeSetColName(v, 0, COLNAME_NAME, "seqno", P3_STATIC); + sqlite3VdbeSetColName(v, 1, COLNAME_NAME, "cid", P3_STATIC); + sqlite3VdbeSetColName(v, 2, COLNAME_NAME, "name", P3_STATIC); + for(i=0; inColumn; i++){ + int cnum = pIdx->aiColumn[i]; + sqlite3VdbeAddOp(v, OP_Integer, i, 0); + sqlite3VdbeAddOp(v, OP_Integer, cnum, 0); + assert( pTab->nCol>cnum ); + sqlite3VdbeOp3(v, OP_String8, 0, 0, pTab->aCol[cnum].zName, 0); + sqlite3VdbeAddOp(v, OP_Callback, 3, 0); + } + } + }else + + if( sqlite3StrICmp(zLeft, "index_list")==0 && zRight ){ + Index *pIdx; + Table *pTab; + if( sqlite3ReadSchema(pParse) ) goto pragma_out; + pTab = sqlite3FindTable(db, zRight, zDb); + if( pTab ){ + v = sqlite3GetVdbe(pParse); + pIdx = pTab->pIndex; + if( pIdx ){ + int i = 0; + sqlite3VdbeSetNumCols(v, 3); + sqlite3VdbeSetColName(v, 0, COLNAME_NAME, "seq", P3_STATIC); + sqlite3VdbeSetColName(v, 1, COLNAME_NAME, "name", P3_STATIC); + sqlite3VdbeSetColName(v, 2, COLNAME_NAME, "unique", P3_STATIC); + while(pIdx){ + sqlite3VdbeAddOp(v, OP_Integer, i, 0); + sqlite3VdbeOp3(v, OP_String8, 0, 0, pIdx->zName, 0); + sqlite3VdbeAddOp(v, OP_Integer, pIdx->onError!=OE_None, 0); + sqlite3VdbeAddOp(v, OP_Callback, 3, 0); + ++i; + pIdx = pIdx->pNext; + } + } + } + }else + + if( sqlite3StrICmp(zLeft, "database_list")==0 ){ + int i; + if( sqlite3ReadSchema(pParse) ) goto pragma_out; + sqlite3VdbeSetNumCols(v, 3); + sqlite3VdbeSetColName(v, 0, COLNAME_NAME, "seq", P3_STATIC); + sqlite3VdbeSetColName(v, 1, COLNAME_NAME, "name", P3_STATIC); + sqlite3VdbeSetColName(v, 2, COLNAME_NAME, "file", P3_STATIC); + for(i=0; inDb; i++){ + if( db->aDb[i].pBt==0 ) continue; + assert( db->aDb[i].zName!=0 ); + sqlite3VdbeAddOp(v, OP_Integer, i, 0); + sqlite3VdbeOp3(v, OP_String8, 0, 0, db->aDb[i].zName, 0); + sqlite3VdbeOp3(v, OP_String8, 0, 0, + sqlite3BtreeGetFilename(db->aDb[i].pBt), 0); + sqlite3VdbeAddOp(v, OP_Callback, 3, 0); + } + }else + + if( sqlite3StrICmp(zLeft, "collation_list")==0 ){ + int i = 0; + HashElem *p; + sqlite3VdbeSetNumCols(v, 2); + sqlite3VdbeSetColName(v, 0, COLNAME_NAME, "seq", P3_STATIC); + sqlite3VdbeSetColName(v, 1, COLNAME_NAME, "name", P3_STATIC); + for(p=sqliteHashFirst(&db->aCollSeq); p; p=sqliteHashNext(p)){ + CollSeq *pColl = (CollSeq *)sqliteHashData(p); + sqlite3VdbeAddOp(v, OP_Integer, i++, 0); + sqlite3VdbeOp3(v, OP_String8, 0, 0, pColl->zName, 0); + sqlite3VdbeAddOp(v, OP_Callback, 2, 0); + } + }else +#endif /* SQLITE_OMIT_SCHEMA_PRAGMAS */ + +#ifndef SQLITE_OMIT_FOREIGN_KEY + if( sqlite3StrICmp(zLeft, "foreign_key_list")==0 && zRight ){ + FKey *pFK; + Table *pTab; + if( sqlite3ReadSchema(pParse) ) goto pragma_out; + pTab = sqlite3FindTable(db, zRight, zDb); + if( pTab ){ + v = sqlite3GetVdbe(pParse); + pFK = pTab->pFKey; + if( pFK ){ + int i = 0; + sqlite3VdbeSetNumCols(v, 5); + sqlite3VdbeSetColName(v, 0, COLNAME_NAME, "id", P3_STATIC); + sqlite3VdbeSetColName(v, 1, COLNAME_NAME, "seq", P3_STATIC); + sqlite3VdbeSetColName(v, 2, COLNAME_NAME, "table", P3_STATIC); + sqlite3VdbeSetColName(v, 3, COLNAME_NAME, "from", P3_STATIC); + sqlite3VdbeSetColName(v, 4, COLNAME_NAME, "to", P3_STATIC); + while(pFK){ + int j; + for(j=0; jnCol; j++){ + char *zCol = pFK->aCol[j].zCol; + sqlite3VdbeAddOp(v, OP_Integer, i, 0); + sqlite3VdbeAddOp(v, OP_Integer, j, 0); + sqlite3VdbeOp3(v, OP_String8, 0, 0, pFK->zTo, 0); + sqlite3VdbeOp3(v, OP_String8, 0, 0, + pTab->aCol[pFK->aCol[j].iFrom].zName, 0); + sqlite3VdbeOp3(v, zCol ? OP_String8 : OP_Null, 0, 0, zCol, 0); + sqlite3VdbeAddOp(v, OP_Callback, 5, 0); + } + ++i; + pFK = pFK->pNextFrom; + } + } + } + }else +#endif /* !defined(SQLITE_OMIT_FOREIGN_KEY) */ + +#ifndef NDEBUG + if( sqlite3StrICmp(zLeft, "parser_trace")==0 ){ + extern void sqlite3ParserTrace(FILE*, char *); + if( zRight ){ + if( getBoolean(zRight) ){ + sqlite3ParserTrace(stderr, "parser: "); + }else{ + sqlite3ParserTrace(0, 0); + } + } + }else +#endif + + /* Reinstall the LIKE and GLOB functions. The variant of LIKE + ** used will be case sensitive or not depending on the RHS. + */ + if( sqlite3StrICmp(zLeft, "case_sensitive_like")==0 ){ + if( zRight ){ + sqlite3RegisterLikeFunctions(db, getBoolean(zRight)); + } + }else + +#ifndef SQLITE_OMIT_INTEGRITY_CHECK + if( sqlite3StrICmp(zLeft, "integrity_check")==0 ){ + int i, j, addr; + + /* Code that appears at the end of the integrity check. If no error + ** messages have been generated, output OK. Otherwise output the + ** error message + */ + static const VdbeOpList endCode[] = { + { OP_MemLoad, 0, 0, 0}, + { OP_Integer, 0, 0, 0}, + { OP_Ne, 0, 0, 0}, /* 2 */ + { OP_String8, 0, 0, "ok"}, + { OP_Callback, 1, 0, 0}, + }; + + /* Initialize the VDBE program */ + if( sqlite3ReadSchema(pParse) ) goto pragma_out; + sqlite3VdbeSetNumCols(v, 1); + sqlite3VdbeSetColName(v, 0, COLNAME_NAME, "integrity_check", P3_STATIC); + sqlite3VdbeAddOp(v, OP_MemInt, 0, 0); /* Initialize error count to 0 */ + + /* Do an integrity check on each database file */ + for(i=0; inDb; i++){ + HashElem *x; + Hash *pTbls; + int cnt = 0; + + if( OMIT_TEMPDB && i==1 ) continue; + + sqlite3CodeVerifySchema(pParse, i); + + /* Do an integrity check of the B-Tree + */ + pTbls = &db->aDb[i].pSchema->tblHash; + for(x=sqliteHashFirst(pTbls); x; x=sqliteHashNext(x)){ + Table *pTab = sqliteHashData(x); + Index *pIdx; + sqlite3VdbeAddOp(v, OP_Integer, pTab->tnum, 0); + cnt++; + for(pIdx=pTab->pIndex; pIdx; pIdx=pIdx->pNext){ + sqlite3VdbeAddOp(v, OP_Integer, pIdx->tnum, 0); + cnt++; + } + } + assert( cnt>0 ); + sqlite3VdbeAddOp(v, OP_IntegrityCk, cnt, i); + sqlite3VdbeAddOp(v, OP_Dup, 0, 1); + addr = sqlite3VdbeOp3(v, OP_String8, 0, 0, "ok", P3_STATIC); + sqlite3VdbeAddOp(v, OP_Eq, 0, addr+7); + sqlite3VdbeOp3(v, OP_String8, 0, 0, + sqlite3MPrintf("*** in database %s ***\n", db->aDb[i].zName), + P3_DYNAMIC); + sqlite3VdbeAddOp(v, OP_Pull, 1, 0); + sqlite3VdbeAddOp(v, OP_Concat, 0, 1); + sqlite3VdbeAddOp(v, OP_Callback, 1, 0); + sqlite3VdbeAddOp(v, OP_MemIncr, 1, 0); + + /* Make sure all the indices are constructed correctly. + */ + sqlite3CodeVerifySchema(pParse, i); + for(x=sqliteHashFirst(pTbls); x; x=sqliteHashNext(x)){ + Table *pTab = sqliteHashData(x); + Index *pIdx; + int loopTop; + + if( pTab->pIndex==0 ) continue; + sqlite3OpenTableAndIndices(pParse, pTab, 1, OP_OpenRead); + sqlite3VdbeAddOp(v, OP_MemInt, 0, 1); + loopTop = sqlite3VdbeAddOp(v, OP_Rewind, 1, 0); + sqlite3VdbeAddOp(v, OP_MemIncr, 1, 1); + for(j=0, pIdx=pTab->pIndex; pIdx; pIdx=pIdx->pNext, j++){ + int jmp2; + static const VdbeOpList idxErr[] = { + { OP_MemIncr, 1, 0, 0}, + { OP_String8, 0, 0, "rowid "}, + { OP_Rowid, 1, 0, 0}, + { OP_String8, 0, 0, " missing from index "}, + { OP_String8, 0, 0, 0}, /* 4 */ + { OP_Concat, 2, 0, 0}, + { OP_Callback, 1, 0, 0}, + }; + sqlite3GenerateIndexKey(v, pIdx, 1); + jmp2 = sqlite3VdbeAddOp(v, OP_Found, j+2, 0); + addr = sqlite3VdbeAddOpList(v, ArraySize(idxErr), idxErr); + sqlite3VdbeChangeP3(v, addr+4, pIdx->zName, P3_STATIC); + sqlite3VdbeJumpHere(v, jmp2); + } + sqlite3VdbeAddOp(v, OP_Next, 1, loopTop+1); + sqlite3VdbeJumpHere(v, loopTop); + for(j=0, pIdx=pTab->pIndex; pIdx; pIdx=pIdx->pNext, j++){ + static const VdbeOpList cntIdx[] = { + { OP_MemInt, 0, 2, 0}, + { OP_Rewind, 0, 0, 0}, /* 1 */ + { OP_MemIncr, 1, 2, 0}, + { OP_Next, 0, 0, 0}, /* 3 */ + { OP_MemLoad, 1, 0, 0}, + { OP_MemLoad, 2, 0, 0}, + { OP_Eq, 0, 0, 0}, /* 6 */ + { OP_MemIncr, 1, 0, 0}, + { OP_String8, 0, 0, "wrong # of entries in index "}, + { OP_String8, 0, 0, 0}, /* 9 */ + { OP_Concat, 0, 0, 0}, + { OP_Callback, 1, 0, 0}, + }; + if( pIdx->tnum==0 ) continue; + addr = sqlite3VdbeAddOpList(v, ArraySize(cntIdx), cntIdx); + sqlite3VdbeChangeP1(v, addr+1, j+2); + sqlite3VdbeChangeP2(v, addr+1, addr+4); + sqlite3VdbeChangeP1(v, addr+3, j+2); + sqlite3VdbeChangeP2(v, addr+3, addr+2); + sqlite3VdbeJumpHere(v, addr+6); + sqlite3VdbeChangeP3(v, addr+9, pIdx->zName, P3_STATIC); + } + } + } + addr = sqlite3VdbeAddOpList(v, ArraySize(endCode), endCode); + sqlite3VdbeJumpHere(v, addr+2); + }else +#endif /* SQLITE_OMIT_INTEGRITY_CHECK */ + +#ifndef SQLITE_OMIT_UTF16 + /* + ** PRAGMA encoding + ** PRAGMA encoding = "utf-8"|"utf-16"|"utf-16le"|"utf-16be" + ** + ** In it's first form, this pragma returns the encoding of the main + ** database. If the database is not initialized, it is initialized now. + ** + ** The second form of this pragma is a no-op if the main database file + ** has not already been initialized. In this case it sets the default + ** encoding that will be used for the main database file if a new file + ** is created. If an existing main database file is opened, then the + ** default text encoding for the existing database is used. + ** + ** In all cases new databases created using the ATTACH command are + ** created to use the same default text encoding as the main database. If + ** the main database has not been initialized and/or created when ATTACH + ** is executed, this is done before the ATTACH operation. + ** + ** In the second form this pragma sets the text encoding to be used in + ** new database files created using this database handle. It is only + ** useful if invoked immediately after the main database i + */ + if( sqlite3StrICmp(zLeft, "encoding")==0 ){ + static struct EncName { + char *zName; + u8 enc; + } encnames[] = { + { "UTF-8", SQLITE_UTF8 }, + { "UTF8", SQLITE_UTF8 }, + { "UTF-16le", SQLITE_UTF16LE }, + { "UTF16le", SQLITE_UTF16LE }, + { "UTF-16be", SQLITE_UTF16BE }, + { "UTF16be", SQLITE_UTF16BE }, + { "UTF-16", 0 /* Filled in at run-time */ }, + { "UTF16", 0 /* Filled in at run-time */ }, + { 0, 0 } + }; + struct EncName *pEnc; + encnames[6].enc = encnames[7].enc = SQLITE_UTF16NATIVE; + if( !zRight ){ /* "PRAGMA encoding" */ + if( sqlite3ReadSchema(pParse) ) goto pragma_out; + sqlite3VdbeSetNumCols(v, 1); + sqlite3VdbeSetColName(v, 0, COLNAME_NAME, "encoding", P3_STATIC); + sqlite3VdbeAddOp(v, OP_String8, 0, 0); + for(pEnc=&encnames[0]; pEnc->zName; pEnc++){ + if( pEnc->enc==ENC(pParse->db) ){ + sqlite3VdbeChangeP3(v, -1, pEnc->zName, P3_STATIC); + break; + } + } + sqlite3VdbeAddOp(v, OP_Callback, 1, 0); + }else{ /* "PRAGMA encoding = XXX" */ + /* Only change the value of sqlite.enc if the database handle is not + ** initialized. If the main database exists, the new sqlite.enc value + ** will be overwritten when the schema is next loaded. If it does not + ** already exists, it will be created to use the new encoding value. + */ + if( + !(DbHasProperty(db, 0, DB_SchemaLoaded)) || + DbHasProperty(db, 0, DB_Empty) + ){ + for(pEnc=&encnames[0]; pEnc->zName; pEnc++){ + if( 0==sqlite3StrICmp(zRight, pEnc->zName) ){ + ENC(pParse->db) = pEnc->enc; + break; + } + } + if( !pEnc->zName ){ + sqlite3ErrorMsg(pParse, "unsupported encoding: %s", zRight); + } + } + } + }else +#endif /* SQLITE_OMIT_UTF16 */ + +#ifndef SQLITE_OMIT_SCHEMA_VERSION_PRAGMAS + /* + ** PRAGMA [database.]schema_version + ** PRAGMA [database.]schema_version = + ** + ** PRAGMA [database.]user_version + ** PRAGMA [database.]user_version = + ** + ** The pragma's schema_version and user_version are used to set or get + ** the value of the schema-version and user-version, respectively. Both + ** the schema-version and the user-version are 32-bit signed integers + ** stored in the database header. + ** + ** The schema-cookie is usually only manipulated internally by SQLite. It + ** is incremented by SQLite whenever the database schema is modified (by + ** creating or dropping a table or index). The schema version is used by + ** SQLite each time a query is executed to ensure that the internal cache + ** of the schema used when compiling the SQL query matches the schema of + ** the database against which the compiled query is actually executed. + ** Subverting this mechanism by using "PRAGMA schema_version" to modify + ** the schema-version is potentially dangerous and may lead to program + ** crashes or database corruption. Use with caution! + ** + ** The user-version is not used internally by SQLite. It may be used by + ** applications for any purpose. + */ + if( sqlite3StrICmp(zLeft, "schema_version")==0 || + sqlite3StrICmp(zLeft, "user_version")==0 ){ + + int iCookie; /* Cookie index. 0 for schema-cookie, 6 for user-cookie. */ + if( zLeft[0]=='s' || zLeft[0]=='S' ){ + iCookie = 0; + }else{ + iCookie = 5; + } + + if( zRight ){ + /* Write the specified cookie value */ + static const VdbeOpList setCookie[] = { + { OP_Transaction, 0, 1, 0}, /* 0 */ + { OP_Integer, 0, 0, 0}, /* 1 */ + { OP_SetCookie, 0, 0, 0}, /* 2 */ + }; + int addr = sqlite3VdbeAddOpList(v, ArraySize(setCookie), setCookie); + sqlite3VdbeChangeP1(v, addr, iDb); + sqlite3VdbeChangeP1(v, addr+1, atoi(zRight)); + sqlite3VdbeChangeP1(v, addr+2, iDb); + sqlite3VdbeChangeP2(v, addr+2, iCookie); + }else{ + /* Read the specified cookie value */ + static const VdbeOpList readCookie[] = { + { OP_ReadCookie, 0, 0, 0}, /* 0 */ + { OP_Callback, 1, 0, 0} + }; + int addr = sqlite3VdbeAddOpList(v, ArraySize(readCookie), readCookie); + sqlite3VdbeChangeP1(v, addr, iDb); + sqlite3VdbeChangeP2(v, addr, iCookie); + sqlite3VdbeSetNumCols(v, 1); + } + } +#endif /* SQLITE_OMIT_SCHEMA_VERSION_PRAGMAS */ + +#if defined(SQLITE_DEBUG) || defined(SQLITE_TEST) + /* + ** Report the current state of file logs for all databases + */ + if( sqlite3StrICmp(zLeft, "lock_status")==0 ){ + static const char *const azLockName[] = { + "unlocked", "shared", "reserved", "pending", "exclusive" + }; + int i; + Vdbe *v = sqlite3GetVdbe(pParse); + sqlite3VdbeSetNumCols(v, 2); + sqlite3VdbeSetColName(v, 0, COLNAME_NAME, "database", P3_STATIC); + sqlite3VdbeSetColName(v, 1, COLNAME_NAME, "status", P3_STATIC); + for(i=0; inDb; i++){ + Btree *pBt; + Pager *pPager; + if( db->aDb[i].zName==0 ) continue; + sqlite3VdbeOp3(v, OP_String8, 0, 0, db->aDb[i].zName, P3_STATIC); + pBt = db->aDb[i].pBt; + if( pBt==0 || (pPager = sqlite3BtreePager(pBt))==0 ){ + sqlite3VdbeOp3(v, OP_String8, 0, 0, "closed", P3_STATIC); + }else{ + int j = sqlite3pager_lockstate(pPager); + sqlite3VdbeOp3(v, OP_String8, 0, 0, + (j>=0 && j<=4) ? azLockName[j] : "unknown", P3_STATIC); + } + sqlite3VdbeAddOp(v, OP_Callback, 2, 0); + } + }else +#endif + +#ifdef SQLITE_SSE + /* + ** Check to see if the sqlite_statements table exists. Create it + ** if it does not. + */ + if( sqlite3StrICmp(zLeft, "create_sqlite_statement_table")==0 ){ + extern int sqlite3CreateStatementsTable(Parse*); + sqlite3CreateStatementsTable(pParse); + }else +#endif + +#if SQLITE_HAS_CODEC + if( sqlite3StrICmp(zLeft, "key")==0 ){ + sqlite3_key(db, zRight, strlen(zRight)); + }else +#endif + + {} + + if( v ){ + /* Code an OP_Expire at the end of each PRAGMA program to cause + ** the VDBE implementing the pragma to expire. Most (all?) pragmas + ** are only valid for a single execution. + */ + sqlite3VdbeAddOp(v, OP_Expire, 1, 0); + + /* + ** Reset the safety level, in case the fullfsync flag or synchronous + ** setting changed. + */ + if( db->autoCommit ){ + sqlite3BtreeSetSafetyLevel(pDb->pBt, pDb->safety_level, + (db->flags&SQLITE_FullFSync)!=0); + } + } +pragma_out: + sqliteFree(zLeft); + sqliteFree(zRight); +} + +#endif /* SQLITE_OMIT_PRAGMA || SQLITE_OMIT_PARSER */ Added: external/sqlite-source-3.3.4/prepare.c ============================================================================== --- (empty file) +++ external/sqlite-source-3.3.4/prepare.c Mon Apr 3 07:54:59 2006 @@ -0,0 +1,633 @@ +/* +** 2005 May 25 +** +** The author disclaims copyright to this source code. In place of +** a legal notice, here is a blessing: +** +** May you do good and not evil. +** May you find forgiveness for yourself and forgive others. +** May you share freely, never taking more than you give. +** +************************************************************************* +** This file contains the implementation of the sqlite3_prepare() +** interface, and routines that contribute to loading the database schema +** from disk. +** +** $Id: prepare.c,v 1.31 2006/02/10 02:27:43 danielk1977 Exp $ +*/ +#include "sqliteInt.h" +#include "os.h" +#include + +/* +** Fill the InitData structure with an error message that indicates +** that the database is corrupt. +*/ +static void corruptSchema(InitData *pData, const char *zExtra){ + if( !sqlite3MallocFailed() ){ + sqlite3SetString(pData->pzErrMsg, "malformed database schema", + zExtra!=0 && zExtra[0]!=0 ? " - " : (char*)0, zExtra, (char*)0); + } +} + +/* +** This is the callback routine for the code that initializes the +** database. See sqlite3Init() below for additional information. +** This routine is also called from the OP_ParseSchema opcode of the VDBE. +** +** Each callback contains the following information: +** +** argv[0] = name of thing being created +** argv[1] = root page number for table or index. NULL for trigger or view. +** argv[2] = SQL text for the CREATE statement. +** argv[3] = "1" for temporary files, "0" for main database, "2" or more +** for auxiliary database files. +** +*/ +int sqlite3InitCallback(void *pInit, int argc, char **argv, char **azColName){ + InitData *pData = (InitData*)pInit; + sqlite3 *db = pData->db; + int iDb; + + if( sqlite3MallocFailed() ){ + return SQLITE_NOMEM; + } + + assert( argc==4 ); + if( argv==0 ) return 0; /* Might happen if EMPTY_RESULT_CALLBACKS are on */ + if( argv[1]==0 || argv[3]==0 ){ + corruptSchema(pData, 0); + return 1; + } + iDb = atoi(argv[3]); + assert( iDb>=0 && iDbnDb ); + if( argv[2] && argv[2][0] ){ + /* Call the parser to process a CREATE TABLE, INDEX or VIEW. + ** But because db->init.busy is set to 1, no VDBE code is generated + ** or executed. All the parser does is build the internal data + ** structures that describe the table, index, or view. + */ + char *zErr; + int rc; + assert( db->init.busy ); + db->init.iDb = iDb; + db->init.newTnum = atoi(argv[1]); + rc = sqlite3_exec(db, argv[2], 0, 0, &zErr); + db->init.iDb = 0; + if( SQLITE_OK!=rc ){ + if( rc==SQLITE_NOMEM ){ + sqlite3FailedMalloc(); + }else{ + corruptSchema(pData, zErr); + } + sqlite3_free(zErr); + return rc; + } + }else{ + /* If the SQL column is blank it means this is an index that + ** was created to be the PRIMARY KEY or to fulfill a UNIQUE + ** constraint for a CREATE TABLE. The index should have already + ** been created when we processed the CREATE TABLE. All we have + ** to do here is record the root page number for that index. + */ + Index *pIndex; + pIndex = sqlite3FindIndex(db, argv[0], db->aDb[iDb].zName); + if( pIndex==0 || pIndex->tnum!=0 ){ + /* This can occur if there exists an index on a TEMP table which + ** has the same name as another index on a permanent index. Since + ** the permanent table is hidden by the TEMP table, we can also + ** safely ignore the index on the permanent table. + */ + /* Do Nothing */; + }else{ + pIndex->tnum = atoi(argv[1]); + } + } + return 0; +} + +/* +** Attempt to read the database schema and initialize internal +** data structures for a single database file. The index of the +** database file is given by iDb. iDb==0 is used for the main +** database. iDb==1 should never be used. iDb>=2 is used for +** auxiliary databases. Return one of the SQLITE_ error codes to +** indicate success or failure. +*/ +static int sqlite3InitOne(sqlite3 *db, int iDb, char **pzErrMsg){ + int rc; + BtCursor *curMain; + int size; + Table *pTab; + Db *pDb; + char const *azArg[5]; + char zDbNum[30]; + int meta[10]; + InitData initData; + char const *zMasterSchema; + char const *zMasterName = SCHEMA_TABLE(iDb); + + /* + ** The master database table has a structure like this + */ + static const char master_schema[] = + "CREATE TABLE sqlite_master(\n" + " type text,\n" + " name text,\n" + " tbl_name text,\n" + " rootpage integer,\n" + " sql text\n" + ")" + ; +#ifndef SQLITE_OMIT_TEMPDB + static const char temp_master_schema[] = + "CREATE TEMP TABLE sqlite_temp_master(\n" + " type text,\n" + " name text,\n" + " tbl_name text,\n" + " rootpage integer,\n" + " sql text\n" + ")" + ; +#else + #define temp_master_schema 0 +#endif + + assert( iDb>=0 && iDbnDb ); + assert( db->aDb[iDb].pSchema ); + + /* zMasterSchema and zInitScript are set to point at the master schema + ** and initialisation script appropriate for the database being + ** initialised. zMasterName is the name of the master table. + */ + if( !OMIT_TEMPDB && iDb==1 ){ + zMasterSchema = temp_master_schema; + }else{ + zMasterSchema = master_schema; + } + zMasterName = SCHEMA_TABLE(iDb); + + /* Construct the schema tables. */ + sqlite3SafetyOff(db); + azArg[0] = zMasterName; + azArg[1] = "1"; + azArg[2] = zMasterSchema; + sprintf(zDbNum, "%d", iDb); + azArg[3] = zDbNum; + azArg[4] = 0; + initData.db = db; + initData.pzErrMsg = pzErrMsg; + rc = sqlite3InitCallback(&initData, 4, (char **)azArg, 0); + if( rc!=SQLITE_OK ){ + sqlite3SafetyOn(db); + return rc; + } + pTab = sqlite3FindTable(db, zMasterName, db->aDb[iDb].zName); + if( pTab ){ + pTab->readOnly = 1; + } + sqlite3SafetyOn(db); + + /* Create a cursor to hold the database open + */ + pDb = &db->aDb[iDb]; + if( pDb->pBt==0 ){ + if( !OMIT_TEMPDB && iDb==1 ){ + DbSetProperty(db, 1, DB_SchemaLoaded); + } + return SQLITE_OK; + } + rc = sqlite3BtreeCursor(pDb->pBt, MASTER_ROOT, 0, 0, 0, &curMain); + if( rc!=SQLITE_OK && rc!=SQLITE_EMPTY ){ + sqlite3SetString(pzErrMsg, sqlite3ErrStr(rc), (char*)0); + return rc; + } + + /* Get the database meta information. + ** + ** Meta values are as follows: + ** meta[0] Schema cookie. Changes with each schema change. + ** meta[1] File format of schema layer. + ** meta[2] Size of the page cache. + ** meta[3] Use freelist if 0. Autovacuum if greater than zero. + ** meta[4] Db text encoding. 1:UTF-8 3:UTF-16 LE 4:UTF-16 BE + ** meta[5] The user cookie. Used by the application. + ** meta[6] + ** meta[7] + ** meta[8] + ** meta[9] + ** + ** Note: The #defined SQLITE_UTF* symbols in sqliteInt.h correspond to + ** the possible values of meta[4]. + */ + if( rc==SQLITE_OK ){ + int i; + for(i=0; rc==SQLITE_OK && ipBt, i+1, (u32 *)&meta[i]); + } + if( rc ){ + sqlite3SetString(pzErrMsg, sqlite3ErrStr(rc), (char*)0); + sqlite3BtreeCloseCursor(curMain); + return rc; + } + }else{ + memset(meta, 0, sizeof(meta)); + } + pDb->pSchema->schema_cookie = meta[0]; + + /* If opening a non-empty database, check the text encoding. For the + ** main database, set sqlite3.enc to the encoding of the main database. + ** For an attached db, it is an error if the encoding is not the same + ** as sqlite3.enc. + */ + if( meta[4] ){ /* text encoding */ + if( iDb==0 ){ + /* If opening the main database, set ENC(db). */ + ENC(db) = (u8)meta[4]; + db->pDfltColl = sqlite3FindCollSeq(db, SQLITE_UTF8, "BINARY", 6, 0); + }else{ + /* If opening an attached database, the encoding much match ENC(db) */ + if( meta[4]!=ENC(db) ){ + sqlite3BtreeCloseCursor(curMain); + sqlite3SetString(pzErrMsg, "attached databases must use the same" + " text encoding as main database", (char*)0); + return SQLITE_ERROR; + } + } + }else{ + DbSetProperty(db, iDb, DB_Empty); + } + pDb->pSchema->enc = ENC(db); + + size = meta[2]; + if( size==0 ){ size = MAX_PAGES; } + pDb->pSchema->cache_size = size; + sqlite3BtreeSetCacheSize(pDb->pBt, pDb->pSchema->cache_size); + + /* + ** file_format==1 Version 3.0.0. + ** file_format==2 Version 3.1.3. // ALTER TABLE ADD COLUMN + ** file_format==3 Version 3.1.4. // ditto but with non-NULL defaults + ** file_format==4 Version 3.3.0. // DESC indices. Boolean constants + */ + pDb->pSchema->file_format = meta[1]; + if( pDb->pSchema->file_format==0 ){ + pDb->pSchema->file_format = 1; + } + if( pDb->pSchema->file_format>SQLITE_MAX_FILE_FORMAT ){ + sqlite3BtreeCloseCursor(curMain); + sqlite3SetString(pzErrMsg, "unsupported file format", (char*)0); + return SQLITE_ERROR; + } + + + /* Read the schema information out of the schema tables + */ + assert( db->init.busy ); + if( rc==SQLITE_EMPTY ){ + /* For an empty database, there is nothing to read */ + rc = SQLITE_OK; + }else{ + char *zSql; + zSql = sqlite3MPrintf( + "SELECT name, rootpage, sql, '%s' FROM '%q'.%s", + zDbNum, db->aDb[iDb].zName, zMasterName); + sqlite3SafetyOff(db); + rc = sqlite3_exec(db, zSql, sqlite3InitCallback, &initData, 0); + sqlite3SafetyOn(db); + sqliteFree(zSql); +#ifndef SQLITE_OMIT_ANALYZE + if( rc==SQLITE_OK ){ + sqlite3AnalysisLoad(db, iDb); + } +#endif + sqlite3BtreeCloseCursor(curMain); + } + if( sqlite3MallocFailed() ){ + /* sqlite3SetString(pzErrMsg, "out of memory", (char*)0); */ + rc = SQLITE_NOMEM; + sqlite3ResetInternalSchema(db, 0); + } + if( rc==SQLITE_OK ){ + DbSetProperty(db, iDb, DB_SchemaLoaded); + }else{ + sqlite3ResetInternalSchema(db, iDb); + } + return rc; +} + +/* +** Initialize all database files - the main database file, the file +** used to store temporary tables, and any additional database files +** created using ATTACH statements. Return a success code. If an +** error occurs, write an error message into *pzErrMsg. +** +** After a database is initialized, the DB_SchemaLoaded bit is set +** bit is set in the flags field of the Db structure. If the database +** file was of zero-length, then the DB_Empty flag is also set. +*/ +int sqlite3Init(sqlite3 *db, char **pzErrMsg){ + int i, rc; + int called_initone = 0; + + if( db->init.busy ) return SQLITE_OK; + rc = SQLITE_OK; + db->init.busy = 1; + for(i=0; rc==SQLITE_OK && inDb; i++){ + if( DbHasProperty(db, i, DB_SchemaLoaded) || i==1 ) continue; + rc = sqlite3InitOne(db, i, pzErrMsg); + if( rc ){ + sqlite3ResetInternalSchema(db, i); + } + called_initone = 1; + } + + /* Once all the other databases have been initialised, load the schema + ** for the TEMP database. This is loaded last, as the TEMP database + ** schema may contain references to objects in other databases. + */ +#ifndef SQLITE_OMIT_TEMPDB + if( rc==SQLITE_OK && db->nDb>1 && !DbHasProperty(db, 1, DB_SchemaLoaded) ){ + rc = sqlite3InitOne(db, 1, pzErrMsg); + if( rc ){ + sqlite3ResetInternalSchema(db, 1); + } + called_initone = 1; + } +#endif + + db->init.busy = 0; + if( rc==SQLITE_OK && called_initone ){ + sqlite3CommitInternalChanges(db); + } + + return rc; +} + +/* +** This routine is a no-op if the database schema is already initialised. +** Otherwise, the schema is loaded. An error code is returned. +*/ +int sqlite3ReadSchema(Parse *pParse){ + int rc = SQLITE_OK; + sqlite3 *db = pParse->db; + if( !db->init.busy ){ + rc = sqlite3Init(db, &pParse->zErrMsg); + } + if( rc!=SQLITE_OK ){ + pParse->rc = rc; + pParse->nErr++; + } + return rc; +} + + +/* +** Check schema cookies in all databases. If any cookie is out +** of date, return 0. If all schema cookies are current, return 1. +*/ +static int schemaIsValid(sqlite3 *db){ + int iDb; + int rc; + BtCursor *curTemp; + int cookie; + int allOk = 1; + + for(iDb=0; allOk && iDbnDb; iDb++){ + Btree *pBt; + pBt = db->aDb[iDb].pBt; + if( pBt==0 ) continue; + rc = sqlite3BtreeCursor(pBt, MASTER_ROOT, 0, 0, 0, &curTemp); + if( rc==SQLITE_OK ){ + rc = sqlite3BtreeGetMeta(pBt, 1, (u32 *)&cookie); + if( rc==SQLITE_OK && cookie!=db->aDb[iDb].pSchema->schema_cookie ){ + allOk = 0; + } + sqlite3BtreeCloseCursor(curTemp); + } + } + return allOk; +} + +/* +** Free all resources held by the schema structure. The void* argument points +** at a Schema struct. This function does not call sqliteFree() on the +** pointer itself, it just cleans up subsiduary resources (i.e. the contents +** of the schema hash tables). +*/ +void sqlite3SchemaFree(void *p){ + Hash temp1; + Hash temp2; + HashElem *pElem; + Schema *pSchema = (Schema *)p; + + temp1 = pSchema->tblHash; + temp2 = pSchema->trigHash; + sqlite3HashInit(&pSchema->trigHash, SQLITE_HASH_STRING, 0); + sqlite3HashClear(&pSchema->aFKey); + sqlite3HashClear(&pSchema->idxHash); + for(pElem=sqliteHashFirst(&temp2); pElem; pElem=sqliteHashNext(pElem)){ + sqlite3DeleteTrigger((Trigger*)sqliteHashData(pElem)); + } + sqlite3HashClear(&temp2); + sqlite3HashInit(&pSchema->tblHash, SQLITE_HASH_STRING, 0); + for(pElem=sqliteHashFirst(&temp1); pElem; pElem=sqliteHashNext(pElem)){ + Table *pTab = sqliteHashData(pElem); + sqlite3DeleteTable(0, pTab); + } + sqlite3HashClear(&temp1); + pSchema->pSeqTab = 0; + pSchema->flags &= ~DB_SchemaLoaded; +} + +/* +** Find and return the schema associated with a BTree. Create +** a new one if necessary. +*/ +Schema *sqlite3SchemaGet(Btree *pBt){ + Schema * p; + if( pBt ){ + p = (Schema *)sqlite3BtreeSchema(pBt,sizeof(Schema),sqlite3SchemaFree); + }else{ + p = (Schema *)sqliteMalloc(sizeof(Schema)); + } + if( p && 0==p->file_format ){ + sqlite3HashInit(&p->tblHash, SQLITE_HASH_STRING, 0); + sqlite3HashInit(&p->idxHash, SQLITE_HASH_STRING, 0); + sqlite3HashInit(&p->trigHash, SQLITE_HASH_STRING, 0); + sqlite3HashInit(&p->aFKey, SQLITE_HASH_STRING, 1); + } + return p; +} + +/* +** Convert a schema pointer into the iDb index that indicates +** which database file in db->aDb[] the schema refers to. +** +** If the same database is attached more than once, the first +** attached database is returned. +*/ +int sqlite3SchemaToIndex(sqlite3 *db, Schema *pSchema){ + int i = -1000000; + + /* If pSchema is NULL, then return -1000000. This happens when code in + ** expr.c is trying to resolve a reference to a transient table (i.e. one + ** created by a sub-select). In this case the return value of this + ** function should never be used. + ** + ** We return -1000000 instead of the more usual -1 simply because using + ** -1000000 as incorrectly using -1000000 index into db->aDb[] is much + ** more likely to cause a segfault than -1 (of course there are assert() + ** statements too, but it never hurts to play the odds). + */ + if( pSchema ){ + for(i=0; inDb; i++){ + if( db->aDb[i].pSchema==pSchema ){ + break; + } + } + assert( i>=0 &&i>=0 && inDb ); + } + return i; +} + +/* +** Compile the UTF-8 encoded SQL statement zSql into a statement handle. +*/ +int sqlite3_prepare( + sqlite3 *db, /* Database handle. */ + const char *zSql, /* UTF-8 encoded SQL statement. */ + int nBytes, /* Length of zSql in bytes. */ + sqlite3_stmt **ppStmt, /* OUT: A pointer to the prepared statement */ + const char** pzTail /* OUT: End of parsed string */ +){ + Parse sParse; + char *zErrMsg = 0; + int rc = SQLITE_OK; + int i; + + /* Assert that malloc() has not failed */ + assert( !sqlite3MallocFailed() ); + + assert( ppStmt ); + *ppStmt = 0; + if( sqlite3SafetyOn(db) ){ + return SQLITE_MISUSE; + } + + /* If any attached database schemas are locked, do not proceed with + ** compilation. Instead return SQLITE_LOCKED immediately. + */ + for(i=0; inDb; i++) { + Btree *pBt = db->aDb[i].pBt; + if( pBt && sqlite3BtreeSchemaLocked(pBt) ){ + const char *zDb = db->aDb[i].zName; + sqlite3Error(db, SQLITE_LOCKED, "database schema is locked: %s", zDb); + sqlite3SafetyOff(db); + return SQLITE_LOCKED; + } + } + + memset(&sParse, 0, sizeof(sParse)); + sParse.db = db; + if( nBytes>=0 && zSql[nBytes]!=0 ){ + char *zSqlCopy = sqlite3StrNDup(zSql, nBytes); + sqlite3RunParser(&sParse, zSqlCopy, &zErrMsg); + sParse.zTail += zSql - zSqlCopy; + sqliteFree(zSqlCopy); + }else{ + sqlite3RunParser(&sParse, zSql, &zErrMsg); + } + + if( sqlite3MallocFailed() ){ + sParse.rc = SQLITE_NOMEM; + } + if( sParse.rc==SQLITE_DONE ) sParse.rc = SQLITE_OK; + if( sParse.checkSchema && !schemaIsValid(db) ){ + sParse.rc = SQLITE_SCHEMA; + } + if( sParse.rc==SQLITE_SCHEMA ){ + sqlite3ResetInternalSchema(db, 0); + } + if( pzTail ) *pzTail = sParse.zTail; + rc = sParse.rc; + +#ifndef SQLITE_OMIT_EXPLAIN + if( rc==SQLITE_OK && sParse.pVdbe && sParse.explain ){ + if( sParse.explain==2 ){ + sqlite3VdbeSetNumCols(sParse.pVdbe, 3); + sqlite3VdbeSetColName(sParse.pVdbe, 0, COLNAME_NAME, "order", P3_STATIC); + sqlite3VdbeSetColName(sParse.pVdbe, 1, COLNAME_NAME, "from", P3_STATIC); + sqlite3VdbeSetColName(sParse.pVdbe, 2, COLNAME_NAME, "detail", P3_STATIC); + }else{ + sqlite3VdbeSetNumCols(sParse.pVdbe, 5); + sqlite3VdbeSetColName(sParse.pVdbe, 0, COLNAME_NAME, "addr", P3_STATIC); + sqlite3VdbeSetColName(sParse.pVdbe, 1, COLNAME_NAME, "opcode", P3_STATIC); + sqlite3VdbeSetColName(sParse.pVdbe, 2, COLNAME_NAME, "p1", P3_STATIC); + sqlite3VdbeSetColName(sParse.pVdbe, 3, COLNAME_NAME, "p2", P3_STATIC); + sqlite3VdbeSetColName(sParse.pVdbe, 4, COLNAME_NAME, "p3", P3_STATIC); + } + } +#endif + + if( sqlite3SafetyOff(db) ){ + rc = SQLITE_MISUSE; + } + if( rc==SQLITE_OK ){ + *ppStmt = (sqlite3_stmt*)sParse.pVdbe; + }else if( sParse.pVdbe ){ + sqlite3_finalize((sqlite3_stmt*)sParse.pVdbe); + } + + if( zErrMsg ){ + sqlite3Error(db, rc, "%s", zErrMsg); + sqliteFree(zErrMsg); + }else{ + sqlite3Error(db, rc, 0); + } + + rc = sqlite3ApiExit(db, rc); + sqlite3ReleaseThreadData(); + return rc; +} + +#ifndef SQLITE_OMIT_UTF16 +/* +** Compile the UTF-16 encoded SQL statement zSql into a statement handle. +*/ +int sqlite3_prepare16( + sqlite3 *db, /* Database handle. */ + const void *zSql, /* UTF-8 encoded SQL statement. */ + int nBytes, /* Length of zSql in bytes. */ + sqlite3_stmt **ppStmt, /* OUT: A pointer to the prepared statement */ + const void **pzTail /* OUT: End of parsed string */ +){ + /* This function currently works by first transforming the UTF-16 + ** encoded string to UTF-8, then invoking sqlite3_prepare(). The + ** tricky bit is figuring out the pointer to return in *pzTail. + */ + char *zSql8; + const char *zTail8 = 0; + int rc = SQLITE_OK; + + if( sqlite3SafetyCheck(db) ){ + return SQLITE_MISUSE; + } + zSql8 = sqlite3utf16to8(zSql, nBytes); + if( zSql8 ){ + rc = sqlite3_prepare(db, zSql8, -1, ppStmt, &zTail8); + } + + if( zTail8 && pzTail ){ + /* If sqlite3_prepare returns a tail pointer, we calculate the + ** equivalent pointer into the UTF-16 string by counting the unicode + ** characters between zSql8 and zTail8, and then returning a pointer + ** the same number of characters into the UTF-16 string. + */ + int chars_parsed = sqlite3utf8CharLen(zSql8, zTail8-zSql8); + *pzTail = (u8 *)zSql + sqlite3utf16ByteLen(zSql, chars_parsed); + } + sqliteFree(zSql8); + return sqlite3ApiExit(db, rc); +} +#endif /* SQLITE_OMIT_UTF16 */ Added: external/sqlite-source-3.3.4/printf.c ============================================================================== --- (empty file) +++ external/sqlite-source-3.3.4/printf.c Mon Apr 3 07:54:59 2006 @@ -0,0 +1,868 @@ +/* +** The "printf" code that follows dates from the 1980's. It is in +** the public domain. The original comments are included here for +** completeness. They are very out-of-date but might be useful as +** an historical reference. Most of the "enhancements" have been backed +** out so that the functionality is now the same as standard printf(). +** +************************************************************************** +** +** The following modules is an enhanced replacement for the "printf" subroutines +** found in the standard C library. The following enhancements are +** supported: +** +** + Additional functions. The standard set of "printf" functions +** includes printf, fprintf, sprintf, vprintf, vfprintf, and +** vsprintf. This module adds the following: +** +** * snprintf -- Works like sprintf, but has an extra argument +** which is the size of the buffer written to. +** +** * mprintf -- Similar to sprintf. Writes output to memory +** obtained from malloc. +** +** * xprintf -- Calls a function to dispose of output. +** +** * nprintf -- No output, but returns the number of characters +** that would have been output by printf. +** +** * A v- version (ex: vsnprintf) of every function is also +** supplied. +** +** + A few extensions to the formatting notation are supported: +** +** * The "=" flag (similar to "-") causes the output to be +** be centered in the appropriately sized field. +** +** * The %b field outputs an integer in binary notation. +** +** * The %c field now accepts a precision. The character output +** is repeated by the number of times the precision specifies. +** +** * The %' field works like %c, but takes as its character the +** next character of the format string, instead of the next +** argument. For example, printf("%.78'-") prints 78 minus +** signs, the same as printf("%.78c",'-'). +** +** + When compiled using GCC on a SPARC, this version of printf is +** faster than the library printf for SUN OS 4.1. +** +** + All functions are fully reentrant. +** +*/ +#include "sqliteInt.h" + +/* +** Conversion types fall into various categories as defined by the +** following enumeration. +*/ +#define etRADIX 1 /* Integer types. %d, %x, %o, and so forth */ +#define etFLOAT 2 /* Floating point. %f */ +#define etEXP 3 /* Exponentional notation. %e and %E */ +#define etGENERIC 4 /* Floating or exponential, depending on exponent. %g */ +#define etSIZE 5 /* Return number of characters processed so far. %n */ +#define etSTRING 6 /* Strings. %s */ +#define etDYNSTRING 7 /* Dynamically allocated strings. %z */ +#define etPERCENT 8 /* Percent symbol. %% */ +#define etCHARX 9 /* Characters. %c */ +#define etERROR 10 /* Used to indicate no such conversion type */ +/* The rest are extensions, not normally found in printf() */ +#define etCHARLIT 11 /* Literal characters. %' */ +#define etSQLESCAPE 12 /* Strings with '\'' doubled. %q */ +#define etSQLESCAPE2 13 /* Strings with '\'' doubled and enclosed in '', + NULL pointers replaced by SQL NULL. %Q */ +#define etTOKEN 14 /* a pointer to a Token structure */ +#define etSRCLIST 15 /* a pointer to a SrcList */ +#define etPOINTER 16 /* The %p conversion */ + + +/* +** An "etByte" is an 8-bit unsigned value. +*/ +typedef unsigned char etByte; + +/* +** Each builtin conversion character (ex: the 'd' in "%d") is described +** by an instance of the following structure +*/ +typedef struct et_info { /* Information about each format field */ + char fmttype; /* The format field code letter */ + etByte base; /* The base for radix conversion */ + etByte flags; /* One or more of FLAG_ constants below */ + etByte type; /* Conversion paradigm */ + etByte charset; /* Offset into aDigits[] of the digits string */ + etByte prefix; /* Offset into aPrefix[] of the prefix string */ +} et_info; + +/* +** Allowed values for et_info.flags +*/ +#define FLAG_SIGNED 1 /* True if the value to convert is signed */ +#define FLAG_INTERN 2 /* True if for internal use only */ +#define FLAG_STRING 4 /* Allow infinity precision */ + + +/* +** The following table is searched linearly, so it is good to put the +** most frequently used conversion types first. +*/ +static const char aDigits[] = "0123456789ABCDEF0123456789abcdef"; +static const char aPrefix[] = "-x0\000X0"; +static const et_info fmtinfo[] = { + { 'd', 10, 1, etRADIX, 0, 0 }, + { 's', 0, 4, etSTRING, 0, 0 }, + { 'g', 0, 1, etGENERIC, 30, 0 }, + { 'z', 0, 6, etDYNSTRING, 0, 0 }, + { 'q', 0, 4, etSQLESCAPE, 0, 0 }, + { 'Q', 0, 4, etSQLESCAPE2, 0, 0 }, + { 'c', 0, 0, etCHARX, 0, 0 }, + { 'o', 8, 0, etRADIX, 0, 2 }, + { 'u', 10, 0, etRADIX, 0, 0 }, + { 'x', 16, 0, etRADIX, 16, 1 }, + { 'X', 16, 0, etRADIX, 0, 4 }, +#ifndef SQLITE_OMIT_FLOATING_POINT + { 'f', 0, 1, etFLOAT, 0, 0 }, + { 'e', 0, 1, etEXP, 30, 0 }, + { 'E', 0, 1, etEXP, 14, 0 }, + { 'G', 0, 1, etGENERIC, 14, 0 }, +#endif + { 'i', 10, 1, etRADIX, 0, 0 }, + { 'n', 0, 0, etSIZE, 0, 0 }, + { '%', 0, 0, etPERCENT, 0, 0 }, + { 'p', 16, 0, etPOINTER, 0, 1 }, + { 'T', 0, 2, etTOKEN, 0, 0 }, + { 'S', 0, 2, etSRCLIST, 0, 0 }, +}; +#define etNINFO (sizeof(fmtinfo)/sizeof(fmtinfo[0])) + +/* +** If SQLITE_OMIT_FLOATING_POINT is defined, then none of the floating point +** conversions will work. +*/ +#ifndef SQLITE_OMIT_FLOATING_POINT +/* +** "*val" is a double such that 0.1 <= *val < 10.0 +** Return the ascii code for the leading digit of *val, then +** multiply "*val" by 10.0 to renormalize. +** +** Example: +** input: *val = 3.14159 +** output: *val = 1.4159 function return = '3' +** +** The counter *cnt is incremented each time. After counter exceeds +** 16 (the number of significant digits in a 64-bit float) '0' is +** always returned. +*/ +static int et_getdigit(LONGDOUBLE_TYPE *val, int *cnt){ + int digit; + LONGDOUBLE_TYPE d; + if( (*cnt)++ >= 16 ) return '0'; + digit = (int)*val; + d = digit; + digit += '0'; + *val = (*val - d)*10.0; + return digit; +} +#endif /* SQLITE_OMIT_FLOATING_POINT */ + +/* +** On machines with a small stack size, you can redefine the +** SQLITE_PRINT_BUF_SIZE to be less than 350. But beware - for +** smaller values some %f conversions may go into an infinite loop. +*/ +#ifndef SQLITE_PRINT_BUF_SIZE +# define SQLITE_PRINT_BUF_SIZE 350 +#endif +#define etBUFSIZE SQLITE_PRINT_BUF_SIZE /* Size of the output buffer */ + +/* +** The root program. All variations call this core. +** +** INPUTS: +** func This is a pointer to a function taking three arguments +** 1. A pointer to anything. Same as the "arg" parameter. +** 2. A pointer to the list of characters to be output +** (Note, this list is NOT null terminated.) +** 3. An integer number of characters to be output. +** (Note: This number might be zero.) +** +** arg This is the pointer to anything which will be passed as the +** first argument to "func". Use it for whatever you like. +** +** fmt This is the format string, as in the usual print. +** +** ap This is a pointer to a list of arguments. Same as in +** vfprint. +** +** OUTPUTS: +** The return value is the total number of characters sent to +** the function "func". Returns -1 on a error. +** +** Note that the order in which automatic variables are declared below +** seems to make a big difference in determining how fast this beast +** will run. +*/ +static int vxprintf( + void (*func)(void*,const char*,int), /* Consumer of text */ + void *arg, /* First argument to the consumer */ + int useExtended, /* Allow extended %-conversions */ + const char *fmt, /* Format string */ + va_list ap /* arguments */ +){ + int c; /* Next character in the format string */ + char *bufpt; /* Pointer to the conversion buffer */ + int precision; /* Precision of the current field */ + int length; /* Length of the field */ + int idx; /* A general purpose loop counter */ + int count; /* Total number of characters output */ + int width; /* Width of the current field */ + etByte flag_leftjustify; /* True if "-" flag is present */ + etByte flag_plussign; /* True if "+" flag is present */ + etByte flag_blanksign; /* True if " " flag is present */ + etByte flag_alternateform; /* True if "#" flag is present */ + etByte flag_altform2; /* True if "!" flag is present */ + etByte flag_zeropad; /* True if field width constant starts with zero */ + etByte flag_long; /* True if "l" flag is present */ + etByte flag_longlong; /* True if the "ll" flag is present */ + etByte done; /* Loop termination flag */ + UINT64_TYPE longvalue; /* Value for integer types */ + LONGDOUBLE_TYPE realvalue; /* Value for real types */ + const et_info *infop; /* Pointer to the appropriate info structure */ + char buf[etBUFSIZE]; /* Conversion buffer */ + char prefix; /* Prefix character. "+" or "-" or " " or '\0'. */ + etByte errorflag = 0; /* True if an error is encountered */ + etByte xtype; /* Conversion paradigm */ + char *zExtra; /* Extra memory used for etTCLESCAPE conversions */ + static const char spaces[] = + " "; +#define etSPACESIZE (sizeof(spaces)-1) +#ifndef SQLITE_OMIT_FLOATING_POINT + int exp, e2; /* exponent of real numbers */ + double rounder; /* Used for rounding floating point values */ + etByte flag_dp; /* True if decimal point should be shown */ + etByte flag_rtz; /* True if trailing zeros should be removed */ + etByte flag_exp; /* True to force display of the exponent */ + int nsd; /* Number of significant digits returned */ +#endif + + func(arg,"",0); + count = length = 0; + bufpt = 0; + for(; (c=(*fmt))!=0; ++fmt){ + if( c!='%' ){ + int amt; + bufpt = (char *)fmt; + amt = 1; + while( (c=(*++fmt))!='%' && c!=0 ) amt++; + (*func)(arg,bufpt,amt); + count += amt; + if( c==0 ) break; + } + if( (c=(*++fmt))==0 ){ + errorflag = 1; + (*func)(arg,"%",1); + count++; + break; + } + /* Find out what flags are present */ + flag_leftjustify = flag_plussign = flag_blanksign = + flag_alternateform = flag_altform2 = flag_zeropad = 0; + done = 0; + do{ + switch( c ){ + case '-': flag_leftjustify = 1; break; + case '+': flag_plussign = 1; break; + case ' ': flag_blanksign = 1; break; + case '#': flag_alternateform = 1; break; + case '!': flag_altform2 = 1; break; + case '0': flag_zeropad = 1; break; + default: done = 1; break; + } + }while( !done && (c=(*++fmt))!=0 ); + /* Get the field width */ + width = 0; + if( c=='*' ){ + width = va_arg(ap,int); + if( width<0 ){ + flag_leftjustify = 1; + width = -width; + } + c = *++fmt; + }else{ + while( c>='0' && c<='9' ){ + width = width*10 + c - '0'; + c = *++fmt; + } + } + if( width > etBUFSIZE-10 ){ + width = etBUFSIZE-10; + } + /* Get the precision */ + if( c=='.' ){ + precision = 0; + c = *++fmt; + if( c=='*' ){ + precision = va_arg(ap,int); + if( precision<0 ) precision = -precision; + c = *++fmt; + }else{ + while( c>='0' && c<='9' ){ + precision = precision*10 + c - '0'; + c = *++fmt; + } + } + }else{ + precision = -1; + } + /* Get the conversion type modifier */ + if( c=='l' ){ + flag_long = 1; + c = *++fmt; + if( c=='l' ){ + flag_longlong = 1; + c = *++fmt; + }else{ + flag_longlong = 0; + } + }else{ + flag_long = flag_longlong = 0; + } + /* Fetch the info entry for the field */ + infop = 0; + xtype = etERROR; + for(idx=0; idxflags & FLAG_INTERN)==0 ){ + xtype = infop->type; + } + break; + } + } + zExtra = 0; + + /* Limit the precision to prevent overflowing buf[] during conversion */ + if( precision>etBUFSIZE-40 && (infop->flags & FLAG_STRING)==0 ){ + precision = etBUFSIZE-40; + } + + /* + ** At this point, variables are initialized as follows: + ** + ** flag_alternateform TRUE if a '#' is present. + ** flag_altform2 TRUE if a '!' is present. + ** flag_plussign TRUE if a '+' is present. + ** flag_leftjustify TRUE if a '-' is present or if the + ** field width was negative. + ** flag_zeropad TRUE if the width began with 0. + ** flag_long TRUE if the letter 'l' (ell) prefixed + ** the conversion character. + ** flag_longlong TRUE if the letter 'll' (ell ell) prefixed + ** the conversion character. + ** flag_blanksign TRUE if a ' ' is present. + ** width The specified field width. This is + ** always non-negative. Zero is the default. + ** precision The specified precision. The default + ** is -1. + ** xtype The class of the conversion. + ** infop Pointer to the appropriate info struct. + */ + switch( xtype ){ + case etPOINTER: + flag_longlong = sizeof(char*)==sizeof(i64); + flag_long = sizeof(char*)==sizeof(long int); + /* Fall through into the next case */ + case etRADIX: + if( infop->flags & FLAG_SIGNED ){ + i64 v; + if( flag_longlong ) v = va_arg(ap,i64); + else if( flag_long ) v = va_arg(ap,long int); + else v = va_arg(ap,int); + if( v<0 ){ + longvalue = -v; + prefix = '-'; + }else{ + longvalue = v; + if( flag_plussign ) prefix = '+'; + else if( flag_blanksign ) prefix = ' '; + else prefix = 0; + } + }else{ + if( flag_longlong ) longvalue = va_arg(ap,u64); + else if( flag_long ) longvalue = va_arg(ap,unsigned long int); + else longvalue = va_arg(ap,unsigned int); + prefix = 0; + } + if( longvalue==0 ) flag_alternateform = 0; + if( flag_zeropad && precisioncharset]; + base = infop->base; + do{ /* Convert to ascii */ + *(--bufpt) = cset[longvalue%base]; + longvalue = longvalue/base; + }while( longvalue>0 ); + } + length = &buf[etBUFSIZE-1]-bufpt; + for(idx=precision-length; idx>0; idx--){ + *(--bufpt) = '0'; /* Zero pad */ + } + if( prefix ) *(--bufpt) = prefix; /* Add sign */ + if( flag_alternateform && infop->prefix ){ /* Add "0" or "0x" */ + const char *pre; + char x; + pre = &aPrefix[infop->prefix]; + if( *bufpt!=pre[0] ){ + for(; (x=(*pre))!=0; pre++) *(--bufpt) = x; + } + } + length = &buf[etBUFSIZE-1]-bufpt; + break; + case etFLOAT: + case etEXP: + case etGENERIC: + realvalue = va_arg(ap,double); +#ifndef SQLITE_OMIT_FLOATING_POINT + if( precision<0 ) precision = 6; /* Set default precision */ + if( precision>etBUFSIZE/2-10 ) precision = etBUFSIZE/2-10; + if( realvalue<0.0 ){ + realvalue = -realvalue; + prefix = '-'; + }else{ + if( flag_plussign ) prefix = '+'; + else if( flag_blanksign ) prefix = ' '; + else prefix = 0; + } + if( xtype==etGENERIC && precision>0 ) precision--; +#if 0 + /* Rounding works like BSD when the constant 0.4999 is used. Wierd! */ + for(idx=precision, rounder=0.4999; idx>0; idx--, rounder*=0.1); +#else + /* It makes more sense to use 0.5 */ + for(idx=precision, rounder=0.5; idx>0; idx--, rounder*=0.1); +#endif + if( xtype==etFLOAT ) realvalue += rounder; + /* Normalize realvalue to within 10.0 > realvalue >= 1.0 */ + exp = 0; + if( realvalue>0.0 ){ + while( realvalue>=1e32 && exp<=350 ){ realvalue *= 1e-32; exp+=32; } + while( realvalue>=1e8 && exp<=350 ){ realvalue *= 1e-8; exp+=8; } + while( realvalue>=10.0 && exp<=350 ){ realvalue *= 0.1; exp++; } + while( realvalue<1e-8 && exp>=-350 ){ realvalue *= 1e8; exp-=8; } + while( realvalue<1.0 && exp>=-350 ){ realvalue *= 10.0; exp--; } + if( exp>350 || exp<-350 ){ + bufpt = "NaN"; + length = 3; + break; + } + } + bufpt = buf; + /* + ** If the field type is etGENERIC, then convert to either etEXP + ** or etFLOAT, as appropriate. + */ + flag_exp = xtype==etEXP; + if( xtype!=etFLOAT ){ + realvalue += rounder; + if( realvalue>=10.0 ){ realvalue *= 0.1; exp++; } + } + if( xtype==etGENERIC ){ + flag_rtz = !flag_alternateform; + if( exp<-4 || exp>precision ){ + xtype = etEXP; + }else{ + precision = precision - exp; + xtype = etFLOAT; + } + }else{ + flag_rtz = 0; + } + if( xtype==etEXP ){ + e2 = 0; + }else{ + e2 = exp; + } + nsd = 0; + flag_dp = (precision>0) | flag_alternateform | flag_altform2; + /* The sign in front of the number */ + if( prefix ){ + *(bufpt++) = prefix; + } + /* Digits prior to the decimal point */ + if( e2<0 ){ + *(bufpt++) = '0'; + }else{ + for(; e2>=0; e2--){ + *(bufpt++) = et_getdigit(&realvalue,&nsd); + } + } + /* The decimal point */ + if( flag_dp ){ + *(bufpt++) = '.'; + } + /* "0" digits after the decimal point but before the first + ** significant digit of the number */ + for(e2++; e2<0 && precision>0; precision--, e2++){ + *(bufpt++) = '0'; + } + /* Significant digits after the decimal point */ + while( (precision--)>0 ){ + *(bufpt++) = et_getdigit(&realvalue,&nsd); + } + /* Remove trailing zeros and the "." if no digits follow the "." */ + if( flag_rtz && flag_dp ){ + while( bufpt[-1]=='0' ) *(--bufpt) = 0; + assert( bufpt>buf ); + if( bufpt[-1]=='.' ){ + if( flag_altform2 ){ + *(bufpt++) = '0'; + }else{ + *(--bufpt) = 0; + } + } + } + /* Add the "eNNN" suffix */ + if( flag_exp || (xtype==etEXP && exp) ){ + *(bufpt++) = aDigits[infop->charset]; + if( exp<0 ){ + *(bufpt++) = '-'; exp = -exp; + }else{ + *(bufpt++) = '+'; + } + if( exp>=100 ){ + *(bufpt++) = (exp/100)+'0'; /* 100's digit */ + exp %= 100; + } + *(bufpt++) = exp/10+'0'; /* 10's digit */ + *(bufpt++) = exp%10+'0'; /* 1's digit */ + } + *bufpt = 0; + + /* The converted number is in buf[] and zero terminated. Output it. + ** Note that the number is in the usual order, not reversed as with + ** integer conversions. */ + length = bufpt-buf; + bufpt = buf; + + /* Special case: Add leading zeros if the flag_zeropad flag is + ** set and we are not left justified */ + if( flag_zeropad && !flag_leftjustify && length < width){ + int i; + int nPad = width - length; + for(i=width; i>=nPad; i--){ + bufpt[i] = bufpt[i-nPad]; + } + i = prefix!=0; + while( nPad-- ) bufpt[i++] = '0'; + length = width; + } +#endif + break; + case etSIZE: + *(va_arg(ap,int*)) = count; + length = width = 0; + break; + case etPERCENT: + buf[0] = '%'; + bufpt = buf; + length = 1; + break; + case etCHARLIT: + case etCHARX: + c = buf[0] = (xtype==etCHARX ? va_arg(ap,int) : *++fmt); + if( precision>=0 ){ + for(idx=1; idx=0 && precisionetBUFSIZE ){ + bufpt = zExtra = sqliteMalloc( n ); + if( bufpt==0 ) return -1; + }else{ + bufpt = buf; + } + j = 0; + if( needQuote ) bufpt[j++] = '\''; + for(i=0; (ch=escarg[i])!=0; i++){ + bufpt[j++] = ch; + if( ch=='\'' ) bufpt[j++] = ch; + } + if( needQuote ) bufpt[j++] = '\''; + bufpt[j] = 0; + length = j; + if( precision>=0 && precisionz ){ + (*func)(arg, (char*)pToken->z, pToken->n); + } + length = width = 0; + break; + } + case etSRCLIST: { + SrcList *pSrc = va_arg(ap, SrcList*); + int k = va_arg(ap, int); + struct SrcList_item *pItem = &pSrc->a[k]; + assert( k>=0 && knSrc ); + if( pItem->zDatabase && pItem->zDatabase[0] ){ + (*func)(arg, pItem->zDatabase, strlen(pItem->zDatabase)); + (*func)(arg, ".", 1); + } + (*func)(arg, pItem->zName, strlen(pItem->zName)); + length = width = 0; + break; + } + case etERROR: + buf[0] = '%'; + buf[1] = c; + errorflag = 0; + idx = 1+(c!=0); + (*func)(arg,"%",idx); + count += idx; + if( c==0 ) fmt--; + break; + }/* End switch over the format type */ + /* + ** The text of the conversion is pointed to by "bufpt" and is + ** "length" characters long. The field width is "width". Do + ** the output. + */ + if( !flag_leftjustify ){ + register int nspace; + nspace = width-length; + if( nspace>0 ){ + count += nspace; + while( nspace>=etSPACESIZE ){ + (*func)(arg,spaces,etSPACESIZE); + nspace -= etSPACESIZE; + } + if( nspace>0 ) (*func)(arg,spaces,nspace); + } + } + if( length>0 ){ + (*func)(arg,bufpt,length); + count += length; + } + if( flag_leftjustify ){ + register int nspace; + nspace = width-length; + if( nspace>0 ){ + count += nspace; + while( nspace>=etSPACESIZE ){ + (*func)(arg,spaces,etSPACESIZE); + nspace -= etSPACESIZE; + } + if( nspace>0 ) (*func)(arg,spaces,nspace); + } + } + if( zExtra ){ + sqliteFree(zExtra); + } + }/* End for loop over the format string */ + return errorflag ? -1 : count; +} /* End of function */ + + +/* This structure is used to store state information about the +** write to memory that is currently in progress. +*/ +struct sgMprintf { + char *zBase; /* A base allocation */ + char *zText; /* The string collected so far */ + int nChar; /* Length of the string so far */ + int nTotal; /* Output size if unconstrained */ + int nAlloc; /* Amount of space allocated in zText */ + void *(*xRealloc)(void*,int); /* Function used to realloc memory */ +}; + +/* +** This function implements the callback from vxprintf. +** +** This routine add nNewChar characters of text in zNewText to +** the sgMprintf structure pointed to by "arg". +*/ +static void mout(void *arg, const char *zNewText, int nNewChar){ + struct sgMprintf *pM = (struct sgMprintf*)arg; + pM->nTotal += nNewChar; + if( pM->nChar + nNewChar + 1 > pM->nAlloc ){ + if( pM->xRealloc==0 ){ + nNewChar = pM->nAlloc - pM->nChar - 1; + }else{ + pM->nAlloc = pM->nChar + nNewChar*2 + 1; + if( pM->zText==pM->zBase ){ + pM->zText = pM->xRealloc(0, pM->nAlloc); + if( pM->zText && pM->nChar ){ + memcpy(pM->zText, pM->zBase, pM->nChar); + } + }else{ + char *zNew; + zNew = pM->xRealloc(pM->zText, pM->nAlloc); + if( zNew ){ + pM->zText = zNew; + } + } + } + } + if( pM->zText ){ + if( nNewChar>0 ){ + memcpy(&pM->zText[pM->nChar], zNewText, nNewChar); + pM->nChar += nNewChar; + } + pM->zText[pM->nChar] = 0; + } +} + +/* +** This routine is a wrapper around xprintf() that invokes mout() as +** the consumer. +*/ +static char *base_vprintf( + void *(*xRealloc)(void*,int), /* Routine to realloc memory. May be NULL */ + int useInternal, /* Use internal %-conversions if true */ + char *zInitBuf, /* Initially write here, before mallocing */ + int nInitBuf, /* Size of zInitBuf[] */ + const char *zFormat, /* format string */ + va_list ap /* arguments */ +){ + struct sgMprintf sM; + sM.zBase = sM.zText = zInitBuf; + sM.nChar = sM.nTotal = 0; + sM.nAlloc = nInitBuf; + sM.xRealloc = xRealloc; + vxprintf(mout, &sM, useInternal, zFormat, ap); + if( xRealloc ){ + if( sM.zText==sM.zBase ){ + sM.zText = xRealloc(0, sM.nChar+1); + if( sM.zText ){ + memcpy(sM.zText, sM.zBase, sM.nChar+1); + } + }else if( sM.nAlloc>sM.nChar+10 ){ + char *zNew = xRealloc(sM.zText, sM.nChar+1); + if( zNew ){ + sM.zText = zNew; + } + } + } + return sM.zText; +} + +/* +** Realloc that is a real function, not a macro. +*/ +static void *printf_realloc(void *old, int size){ + return sqliteRealloc(old,size); +} + +/* +** Print into memory obtained from sqliteMalloc(). Use the internal +** %-conversion extensions. +*/ +char *sqlite3VMPrintf(const char *zFormat, va_list ap){ + char zBase[SQLITE_PRINT_BUF_SIZE]; + return base_vprintf(printf_realloc, 1, zBase, sizeof(zBase), zFormat, ap); +} + +/* +** Print into memory obtained from sqliteMalloc(). Use the internal +** %-conversion extensions. +*/ +char *sqlite3MPrintf(const char *zFormat, ...){ + va_list ap; + char *z; + char zBase[SQLITE_PRINT_BUF_SIZE]; + va_start(ap, zFormat); + z = base_vprintf(printf_realloc, 1, zBase, sizeof(zBase), zFormat, ap); + va_end(ap); + return z; +} + +/* +** Print into memory obtained from malloc(). Do not use the internal +** %-conversion extensions. This routine is for use by external users. +*/ +char *sqlite3_mprintf(const char *zFormat, ...){ + va_list ap; + char *z; + char zBuf[200]; + + va_start(ap,zFormat); + z = base_vprintf((void*(*)(void*,int))realloc, 0, + zBuf, sizeof(zBuf), zFormat, ap); + va_end(ap); + return z; +} + +/* This is the varargs version of sqlite3_mprintf. +*/ +char *sqlite3_vmprintf(const char *zFormat, va_list ap){ + char zBuf[200]; + return base_vprintf((void*(*)(void*,int))realloc, 0, + zBuf, sizeof(zBuf), zFormat, ap); +} + +/* +** sqlite3_snprintf() works like snprintf() except that it ignores the +** current locale settings. This is important for SQLite because we +** are not able to use a "," as the decimal point in place of "." as +** specified by some locales. +*/ +char *sqlite3_snprintf(int n, char *zBuf, const char *zFormat, ...){ + char *z; + va_list ap; + + va_start(ap,zFormat); + z = base_vprintf(0, 0, zBuf, n, zFormat, ap); + va_end(ap); + return z; +} + +#if defined(SQLITE_TEST) || defined(SQLITE_DEBUG) +/* +** A version of printf() that understands %lld. Used for debugging. +** The printf() built into some versions of windows does not understand %lld +** and segfaults if you give it a long long int. +*/ +void sqlite3DebugPrintf(const char *zFormat, ...){ + extern int getpid(void); + va_list ap; + char zBuf[500]; + va_start(ap, zFormat); + base_vprintf(0, 0, zBuf, sizeof(zBuf), zFormat, ap); + va_end(ap); + fprintf(stdout,"%d: %s", getpid(), zBuf); + fflush(stdout); +} +#endif Added: external/sqlite-source-3.3.4/random.c ============================================================================== --- (empty file) +++ external/sqlite-source-3.3.4/random.c Mon Apr 3 07:54:59 2006 @@ -0,0 +1,100 @@ +/* +** 2001 September 15 +** +** The author disclaims copyright to this source code. In place of +** a legal notice, here is a blessing: +** +** May you do good and not evil. +** May you find forgiveness for yourself and forgive others. +** May you share freely, never taking more than you give. +** +************************************************************************* +** This file contains code to implement a pseudo-random number +** generator (PRNG) for SQLite. +** +** Random numbers are used by some of the database backends in order +** to generate random integer keys for tables or random filenames. +** +** $Id: random.c,v 1.15 2006/01/06 14:32:20 drh Exp $ +*/ +#include "sqliteInt.h" +#include "os.h" + + +/* +** Get a single 8-bit random value from the RC4 PRNG. The Mutex +** must be held while executing this routine. +** +** Why not just use a library random generator like lrand48() for this? +** Because the OP_NewRowid opcode in the VDBE depends on having a very +** good source of random numbers. The lrand48() library function may +** well be good enough. But maybe not. Or maybe lrand48() has some +** subtle problems on some systems that could cause problems. It is hard +** to know. To minimize the risk of problems due to bad lrand48() +** implementations, SQLite uses this random number generator based +** on RC4, which we know works very well. +** +** (Later): Actually, OP_NewRowid does not depend on a good source of +** randomness any more. But we will leave this code in all the same. +*/ +static int randomByte(){ + unsigned char t; + + /* All threads share a single random number generator. + ** This structure is the current state of the generator. + */ + static struct { + unsigned char isInit; /* True if initialized */ + unsigned char i, j; /* State variables */ + unsigned char s[256]; /* State variables */ + } prng; + + /* Initialize the state of the random number generator once, + ** the first time this routine is called. The seed value does + ** not need to contain a lot of randomness since we are not + ** trying to do secure encryption or anything like that... + ** + ** Nothing in this file or anywhere else in SQLite does any kind of + ** encryption. The RC4 algorithm is being used as a PRNG (pseudo-random + ** number generator) not as an encryption device. + */ + if( !prng.isInit ){ + int i; + char k[256]; + prng.j = 0; + prng.i = 0; + sqlite3OsRandomSeed(k); + for(i=0; i<256; i++){ + prng.s[i] = i; + } + for(i=0; i<256; i++){ + prng.j += prng.s[i] + k[i]; + t = prng.s[prng.j]; + prng.s[prng.j] = prng.s[i]; + prng.s[i] = t; + } + prng.isInit = 1; + } + + /* Generate and return single random byte + */ + prng.i++; + t = prng.s[prng.i]; + prng.j += t; + prng.s[prng.i] = prng.s[prng.j]; + prng.s[prng.j] = t; + t += prng.s[prng.i]; + return prng.s[t]; +} + +/* +** Return N random bytes. +*/ +void sqlite3Randomness(int N, void *pBuf){ + unsigned char *zBuf = pBuf; + sqlite3OsEnterMutex(); + while( N-- ){ + *(zBuf++) = randomByte(); + } + sqlite3OsLeaveMutex(); +} Added: external/sqlite-source-3.3.4/select.c ============================================================================== --- (empty file) +++ external/sqlite-source-3.3.4/select.c Mon Apr 3 07:54:59 2006 @@ -0,0 +1,3297 @@ +/* +** 2001 September 15 +** +** The author disclaims copyright to this source code. In place of +** a legal notice, here is a blessing: +** +** May you do good and not evil. +** May you find forgiveness for yourself and forgive others. +** May you share freely, never taking more than you give. +** +************************************************************************* +** This file contains C code routines that are called by the parser +** to handle SELECT statements in SQLite. +** +** $Id: select.c,v 1.304 2006/02/10 07:07:16 danielk1977 Exp $ +*/ +#include "sqliteInt.h" + + +/* +** Delete all the content of a Select structure but do not deallocate +** the select structure itself. +*/ +static void clearSelect(Select *p){ + sqlite3ExprListDelete(p->pEList); + sqlite3SrcListDelete(p->pSrc); + sqlite3ExprDelete(p->pWhere); + sqlite3ExprListDelete(p->pGroupBy); + sqlite3ExprDelete(p->pHaving); + sqlite3ExprListDelete(p->pOrderBy); + sqlite3SelectDelete(p->pPrior); + sqlite3ExprDelete(p->pLimit); + sqlite3ExprDelete(p->pOffset); +} + + +/* +** Allocate a new Select structure and return a pointer to that +** structure. +*/ +Select *sqlite3SelectNew( + ExprList *pEList, /* which columns to include in the result */ + SrcList *pSrc, /* the FROM clause -- which tables to scan */ + Expr *pWhere, /* the WHERE clause */ + ExprList *pGroupBy, /* the GROUP BY clause */ + Expr *pHaving, /* the HAVING clause */ + ExprList *pOrderBy, /* the ORDER BY clause */ + int isDistinct, /* true if the DISTINCT keyword is present */ + Expr *pLimit, /* LIMIT value. NULL means not used */ + Expr *pOffset /* OFFSET value. NULL means no offset */ +){ + Select *pNew; + Select standin; + pNew = sqliteMalloc( sizeof(*pNew) ); + assert( !pOffset || pLimit ); /* Can't have OFFSET without LIMIT. */ + if( pNew==0 ){ + pNew = &standin; + memset(pNew, 0, sizeof(*pNew)); + } + if( pEList==0 ){ + pEList = sqlite3ExprListAppend(0, sqlite3Expr(TK_ALL,0,0,0), 0); + } + pNew->pEList = pEList; + pNew->pSrc = pSrc; + pNew->pWhere = pWhere; + pNew->pGroupBy = pGroupBy; + pNew->pHaving = pHaving; + pNew->pOrderBy = pOrderBy; + pNew->isDistinct = isDistinct; + pNew->op = TK_SELECT; + pNew->pLimit = pLimit; + pNew->pOffset = pOffset; + pNew->iLimit = -1; + pNew->iOffset = -1; + pNew->addrOpenVirt[0] = -1; + pNew->addrOpenVirt[1] = -1; + pNew->addrOpenVirt[2] = -1; + if( pNew==&standin) { + clearSelect(pNew); + pNew = 0; + } + return pNew; +} + +/* +** Delete the given Select structure and all of its substructures. +*/ +void sqlite3SelectDelete(Select *p){ + if( p ){ + clearSelect(p); + sqliteFree(p); + } +} + +/* +** Given 1 to 3 identifiers preceeding the JOIN keyword, determine the +** type of join. Return an integer constant that expresses that type +** in terms of the following bit values: +** +** JT_INNER +** JT_CROSS +** JT_OUTER +** JT_NATURAL +** JT_LEFT +** JT_RIGHT +** +** A full outer join is the combination of JT_LEFT and JT_RIGHT. +** +** If an illegal or unsupported join type is seen, then still return +** a join type, but put an error in the pParse structure. +*/ +int sqlite3JoinType(Parse *pParse, Token *pA, Token *pB, Token *pC){ + int jointype = 0; + Token *apAll[3]; + Token *p; + static const struct { + const char zKeyword[8]; + u8 nChar; + u8 code; + } keywords[] = { + { "natural", 7, JT_NATURAL }, + { "left", 4, JT_LEFT|JT_OUTER }, + { "right", 5, JT_RIGHT|JT_OUTER }, + { "full", 4, JT_LEFT|JT_RIGHT|JT_OUTER }, + { "outer", 5, JT_OUTER }, + { "inner", 5, JT_INNER }, + { "cross", 5, JT_INNER|JT_CROSS }, + }; + int i, j; + apAll[0] = pA; + apAll[1] = pB; + apAll[2] = pC; + for(i=0; i<3 && apAll[i]; i++){ + p = apAll[i]; + for(j=0; jn==keywords[j].nChar + && sqlite3StrNICmp((char*)p->z, keywords[j].zKeyword, p->n)==0 ){ + jointype |= keywords[j].code; + break; + } + } + if( j>=sizeof(keywords)/sizeof(keywords[0]) ){ + jointype |= JT_ERROR; + break; + } + } + if( + (jointype & (JT_INNER|JT_OUTER))==(JT_INNER|JT_OUTER) || + (jointype & JT_ERROR)!=0 + ){ + const char *zSp1 = " "; + const char *zSp2 = " "; + if( pB==0 ){ zSp1++; } + if( pC==0 ){ zSp2++; } + sqlite3ErrorMsg(pParse, "unknown or unsupported join type: " + "%T%s%T%s%T", pA, zSp1, pB, zSp2, pC); + jointype = JT_INNER; + }else if( jointype & JT_RIGHT ){ + sqlite3ErrorMsg(pParse, + "RIGHT and FULL OUTER JOINs are not currently supported"); + jointype = JT_INNER; + } + return jointype; +} + +/* +** Return the index of a column in a table. Return -1 if the column +** is not contained in the table. +*/ +static int columnIndex(Table *pTab, const char *zCol){ + int i; + for(i=0; inCol; i++){ + if( sqlite3StrICmp(pTab->aCol[i].zName, zCol)==0 ) return i; + } + return -1; +} + +/* +** Set the value of a token to a '\000'-terminated string. +*/ +static void setToken(Token *p, const char *z){ + p->z = (u8*)z; + p->n = z ? strlen(z) : 0; + p->dyn = 0; +} + +/* +** Create an expression node for an identifier with the name of zName +*/ +static Expr *createIdExpr(const char *zName){ + Token dummy; + setToken(&dummy, zName); + return sqlite3Expr(TK_ID, 0, 0, &dummy); +} + + +/* +** Add a term to the WHERE expression in *ppExpr that requires the +** zCol column to be equal in the two tables pTab1 and pTab2. +*/ +static void addWhereTerm( + const char *zCol, /* Name of the column */ + const Table *pTab1, /* First table */ + const char *zAlias1, /* Alias for first table. May be NULL */ + const Table *pTab2, /* Second table */ + const char *zAlias2, /* Alias for second table. May be NULL */ + int iRightJoinTable, /* VDBE cursor for the right table */ + Expr **ppExpr /* Add the equality term to this expression */ +){ + Expr *pE1a, *pE1b, *pE1c; + Expr *pE2a, *pE2b, *pE2c; + Expr *pE; + + pE1a = createIdExpr(zCol); + pE2a = createIdExpr(zCol); + if( zAlias1==0 ){ + zAlias1 = pTab1->zName; + } + pE1b = createIdExpr(zAlias1); + if( zAlias2==0 ){ + zAlias2 = pTab2->zName; + } + pE2b = createIdExpr(zAlias2); + pE1c = sqlite3Expr(TK_DOT, pE1b, pE1a, 0); + pE2c = sqlite3Expr(TK_DOT, pE2b, pE2a, 0); + pE = sqlite3Expr(TK_EQ, pE1c, pE2c, 0); + ExprSetProperty(pE, EP_FromJoin); + pE->iRightJoinTable = iRightJoinTable; + *ppExpr = sqlite3ExprAnd(*ppExpr, pE); +} + +/* +** Set the EP_FromJoin property on all terms of the given expression. +** And set the Expr.iRightJoinTable to iTable for every term in the +** expression. +** +** The EP_FromJoin property is used on terms of an expression to tell +** the LEFT OUTER JOIN processing logic that this term is part of the +** join restriction specified in the ON or USING clause and not a part +** of the more general WHERE clause. These terms are moved over to the +** WHERE clause during join processing but we need to remember that they +** originated in the ON or USING clause. +** +** The Expr.iRightJoinTable tells the WHERE clause processing that the +** expression depends on table iRightJoinTable even if that table is not +** explicitly mentioned in the expression. That information is needed +** for cases like this: +** +** SELECT * FROM t1 LEFT JOIN t2 ON t1.a=t2.b AND t1.x=5 +** +** The where clause needs to defer the handling of the t1.x=5 +** term until after the t2 loop of the join. In that way, a +** NULL t2 row will be inserted whenever t1.x!=5. If we do not +** defer the handling of t1.x=5, it will be processed immediately +** after the t1 loop and rows with t1.x!=5 will never appear in +** the output, which is incorrect. +*/ +static void setJoinExpr(Expr *p, int iTable){ + while( p ){ + ExprSetProperty(p, EP_FromJoin); + p->iRightJoinTable = iTable; + setJoinExpr(p->pLeft, iTable); + p = p->pRight; + } +} + +/* +** This routine processes the join information for a SELECT statement. +** ON and USING clauses are converted into extra terms of the WHERE clause. +** NATURAL joins also create extra WHERE clause terms. +** +** The terms of a FROM clause are contained in the Select.pSrc structure. +** The left most table is the first entry in Select.pSrc. The right-most +** table is the last entry. The join operator is held in the entry to +** the left. Thus entry 0 contains the join operator for the join between +** entries 0 and 1. Any ON or USING clauses associated with the join are +** also attached to the left entry. +** +** This routine returns the number of errors encountered. +*/ +static int sqliteProcessJoin(Parse *pParse, Select *p){ + SrcList *pSrc; /* All tables in the FROM clause */ + int i, j; /* Loop counters */ + struct SrcList_item *pLeft; /* Left table being joined */ + struct SrcList_item *pRight; /* Right table being joined */ + + pSrc = p->pSrc; + pLeft = &pSrc->a[0]; + pRight = &pLeft[1]; + for(i=0; inSrc-1; i++, pRight++, pLeft++){ + Table *pLeftTab = pLeft->pTab; + Table *pRightTab = pRight->pTab; + + if( pLeftTab==0 || pRightTab==0 ) continue; + + /* When the NATURAL keyword is present, add WHERE clause terms for + ** every column that the two tables have in common. + */ + if( pLeft->jointype & JT_NATURAL ){ + if( pLeft->pOn || pLeft->pUsing ){ + sqlite3ErrorMsg(pParse, "a NATURAL join may not have " + "an ON or USING clause", 0); + return 1; + } + for(j=0; jnCol; j++){ + char *zName = pLeftTab->aCol[j].zName; + if( columnIndex(pRightTab, zName)>=0 ){ + addWhereTerm(zName, pLeftTab, pLeft->zAlias, + pRightTab, pRight->zAlias, + pRight->iCursor, &p->pWhere); + + } + } + } + + /* Disallow both ON and USING clauses in the same join + */ + if( pLeft->pOn && pLeft->pUsing ){ + sqlite3ErrorMsg(pParse, "cannot have both ON and USING " + "clauses in the same join"); + return 1; + } + + /* Add the ON clause to the end of the WHERE clause, connected by + ** an AND operator. + */ + if( pLeft->pOn ){ + setJoinExpr(pLeft->pOn, pRight->iCursor); + p->pWhere = sqlite3ExprAnd(p->pWhere, pLeft->pOn); + pLeft->pOn = 0; + } + + /* Create extra terms on the WHERE clause for each column named + ** in the USING clause. Example: If the two tables to be joined are + ** A and B and the USING clause names X, Y, and Z, then add this + ** to the WHERE clause: A.X=B.X AND A.Y=B.Y AND A.Z=B.Z + ** Report an error if any column mentioned in the USING clause is + ** not contained in both tables to be joined. + */ + if( pLeft->pUsing ){ + IdList *pList = pLeft->pUsing; + for(j=0; jnId; j++){ + char *zName = pList->a[j].zName; + if( columnIndex(pLeftTab, zName)<0 || columnIndex(pRightTab, zName)<0 ){ + sqlite3ErrorMsg(pParse, "cannot join using column %s - column " + "not present in both tables", zName); + return 1; + } + addWhereTerm(zName, pLeftTab, pLeft->zAlias, + pRightTab, pRight->zAlias, + pRight->iCursor, &p->pWhere); + } + } + } + return 0; +} + +/* +** Insert code into "v" that will push the record on the top of the +** stack into the sorter. +*/ +static void pushOntoSorter( + Parse *pParse, /* Parser context */ + ExprList *pOrderBy, /* The ORDER BY clause */ + Select *pSelect /* The whole SELECT statement */ +){ + Vdbe *v = pParse->pVdbe; + sqlite3ExprCodeExprList(pParse, pOrderBy); + sqlite3VdbeAddOp(v, OP_Sequence, pOrderBy->iECursor, 0); + sqlite3VdbeAddOp(v, OP_Pull, pOrderBy->nExpr + 1, 0); + sqlite3VdbeAddOp(v, OP_MakeRecord, pOrderBy->nExpr + 2, 0); + sqlite3VdbeAddOp(v, OP_IdxInsert, pOrderBy->iECursor, 0); + if( pSelect->iLimit>=0 ){ + int addr1, addr2; + addr1 = sqlite3VdbeAddOp(v, OP_IfMemZero, pSelect->iLimit+1, 0); + sqlite3VdbeAddOp(v, OP_MemIncr, -1, pSelect->iLimit+1); + addr2 = sqlite3VdbeAddOp(v, OP_Goto, 0, 0); + sqlite3VdbeJumpHere(v, addr1); + sqlite3VdbeAddOp(v, OP_Last, pOrderBy->iECursor, 0); + sqlite3VdbeAddOp(v, OP_Delete, pOrderBy->iECursor, 0); + sqlite3VdbeJumpHere(v, addr2); + pSelect->iLimit = -1; + } +} + +/* +** Add code to implement the OFFSET +*/ +static void codeOffset( + Vdbe *v, /* Generate code into this VM */ + Select *p, /* The SELECT statement being coded */ + int iContinue, /* Jump here to skip the current record */ + int nPop /* Number of times to pop stack when jumping */ +){ + if( p->iOffset>=0 && iContinue!=0 ){ + int addr; + sqlite3VdbeAddOp(v, OP_MemIncr, -1, p->iOffset); + addr = sqlite3VdbeAddOp(v, OP_IfMemNeg, p->iOffset, 0); + if( nPop>0 ){ + sqlite3VdbeAddOp(v, OP_Pop, nPop, 0); + } + sqlite3VdbeAddOp(v, OP_Goto, 0, iContinue); + VdbeComment((v, "# skip OFFSET records")); + sqlite3VdbeJumpHere(v, addr); + } +} + +/* +** Add code that will check to make sure the top N elements of the +** stack are distinct. iTab is a sorting index that holds previously +** seen combinations of the N values. A new entry is made in iTab +** if the current N values are new. +** +** A jump to addrRepeat is made and the K values are popped from the +** stack if the top N elements are not distinct. +*/ +static void codeDistinct( + Vdbe *v, /* Generate code into this VM */ + int iTab, /* A sorting index used to test for distinctness */ + int addrRepeat, /* Jump to here if not distinct */ + int N, /* The top N elements of the stack must be distinct */ + int K /* Pop K elements from the stack if indistinct */ +){ +#if NULL_ALWAYS_DISTINCT + sqlite3VdbeAddOp(v, OP_IsNull, -N, sqlite3VdbeCurrentAddr(v)+6); +#endif + sqlite3VdbeAddOp(v, OP_MakeRecord, -N, 0); + sqlite3VdbeAddOp(v, OP_Distinct, iTab, sqlite3VdbeCurrentAddr(v)+3); + sqlite3VdbeAddOp(v, OP_Pop, K, 0); + sqlite3VdbeAddOp(v, OP_Goto, 0, addrRepeat); + VdbeComment((v, "# skip indistinct records")); + sqlite3VdbeAddOp(v, OP_IdxInsert, iTab, 0); +} + + +/* +** This routine generates the code for the inside of the inner loop +** of a SELECT. +** +** If srcTab and nColumn are both zero, then the pEList expressions +** are evaluated in order to get the data for this row. If nColumn>0 +** then data is pulled from srcTab and pEList is used only to get the +** datatypes for each column. +*/ +static int selectInnerLoop( + Parse *pParse, /* The parser context */ + Select *p, /* The complete select statement being coded */ + ExprList *pEList, /* List of values being extracted */ + int srcTab, /* Pull data from this table */ + int nColumn, /* Number of columns in the source table */ + ExprList *pOrderBy, /* If not NULL, sort results using this key */ + int distinct, /* If >=0, make sure results are distinct */ + int eDest, /* How to dispose of the results */ + int iParm, /* An argument to the disposal method */ + int iContinue, /* Jump here to continue with next row */ + int iBreak, /* Jump here to break out of the inner loop */ + char *aff /* affinity string if eDest is SRT_Union */ +){ + Vdbe *v = pParse->pVdbe; + int i; + int hasDistinct; /* True if the DISTINCT keyword is present */ + + if( v==0 ) return 0; + assert( pEList!=0 ); + + /* If there was a LIMIT clause on the SELECT statement, then do the check + ** to see if this row should be output. + */ + hasDistinct = distinct>=0 && pEList->nExpr>0; + if( pOrderBy==0 && !hasDistinct ){ + codeOffset(v, p, iContinue, 0); + } + + /* Pull the requested columns. + */ + if( nColumn>0 ){ + for(i=0; inExpr; + sqlite3ExprCodeExprList(pParse, pEList); + } + + /* If the DISTINCT keyword was present on the SELECT statement + ** and this row has been seen before, then do not make this row + ** part of the result. + */ + if( hasDistinct ){ + int n = pEList->nExpr; + codeDistinct(v, distinct, iContinue, n, n+1); + if( pOrderBy==0 ){ + codeOffset(v, p, iContinue, nColumn); + } + } + + switch( eDest ){ + /* In this mode, write each query result to the key of the temporary + ** table iParm. + */ +#ifndef SQLITE_OMIT_COMPOUND_SELECT + case SRT_Union: { + sqlite3VdbeAddOp(v, OP_MakeRecord, nColumn, NULL_ALWAYS_DISTINCT); + if( aff ){ + sqlite3VdbeChangeP3(v, -1, aff, P3_STATIC); + } + sqlite3VdbeAddOp(v, OP_IdxInsert, iParm, 0); + break; + } + + /* Construct a record from the query result, but instead of + ** saving that record, use it as a key to delete elements from + ** the temporary table iParm. + */ + case SRT_Except: { + int addr; + addr = sqlite3VdbeAddOp(v, OP_MakeRecord, nColumn, NULL_ALWAYS_DISTINCT); + sqlite3VdbeChangeP3(v, -1, aff, P3_STATIC); + sqlite3VdbeAddOp(v, OP_NotFound, iParm, addr+3); + sqlite3VdbeAddOp(v, OP_Delete, iParm, 0); + break; + } +#endif + + /* Store the result as data using a unique key. + */ + case SRT_Table: + case SRT_VirtualTab: { + sqlite3VdbeAddOp(v, OP_MakeRecord, nColumn, 0); + if( pOrderBy ){ + pushOntoSorter(pParse, pOrderBy, p); + }else{ + sqlite3VdbeAddOp(v, OP_NewRowid, iParm, 0); + sqlite3VdbeAddOp(v, OP_Pull, 1, 0); + sqlite3VdbeAddOp(v, OP_Insert, iParm, 0); + } + break; + } + +#ifndef SQLITE_OMIT_SUBQUERY + /* If we are creating a set for an "expr IN (SELECT ...)" construct, + ** then there should be a single item on the stack. Write this + ** item into the set table with bogus data. + */ + case SRT_Set: { + int addr1 = sqlite3VdbeCurrentAddr(v); + int addr2; + + assert( nColumn==1 ); + sqlite3VdbeAddOp(v, OP_NotNull, -1, addr1+3); + sqlite3VdbeAddOp(v, OP_Pop, 1, 0); + addr2 = sqlite3VdbeAddOp(v, OP_Goto, 0, 0); + if( pOrderBy ){ + /* At first glance you would think we could optimize out the + ** ORDER BY in this case since the order of entries in the set + ** does not matter. But there might be a LIMIT clause, in which + ** case the order does matter */ + pushOntoSorter(pParse, pOrderBy, p); + }else{ + char affinity = (iParm>>16)&0xFF; + affinity = sqlite3CompareAffinity(pEList->a[0].pExpr, affinity); + sqlite3VdbeOp3(v, OP_MakeRecord, 1, 0, &affinity, 1); + sqlite3VdbeAddOp(v, OP_IdxInsert, (iParm&0x0000FFFF), 0); + } + sqlite3VdbeJumpHere(v, addr2); + break; + } + + /* If any row exist in the result set, record that fact and abort. + */ + case SRT_Exists: { + sqlite3VdbeAddOp(v, OP_MemInt, 1, iParm); + sqlite3VdbeAddOp(v, OP_Pop, nColumn, 0); + /* The LIMIT clause will terminate the loop for us */ + break; + } + + /* If this is a scalar select that is part of an expression, then + ** store the results in the appropriate memory cell and break out + ** of the scan loop. + */ + case SRT_Mem: { + assert( nColumn==1 ); + if( pOrderBy ){ + pushOntoSorter(pParse, pOrderBy, p); + }else{ + sqlite3VdbeAddOp(v, OP_MemStore, iParm, 1); + /* The LIMIT clause will jump out of the loop for us */ + } + break; + } +#endif /* #ifndef SQLITE_OMIT_SUBQUERY */ + + /* Send the data to the callback function or to a subroutine. In the + ** case of a subroutine, the subroutine itself is responsible for + ** popping the data from the stack. + */ + case SRT_Subroutine: + case SRT_Callback: { + if( pOrderBy ){ + sqlite3VdbeAddOp(v, OP_MakeRecord, nColumn, 0); + pushOntoSorter(pParse, pOrderBy, p); + }else if( eDest==SRT_Subroutine ){ + sqlite3VdbeAddOp(v, OP_Gosub, 0, iParm); + }else{ + sqlite3VdbeAddOp(v, OP_Callback, nColumn, 0); + } + break; + } + +#if !defined(SQLITE_OMIT_TRIGGER) + /* Discard the results. This is used for SELECT statements inside + ** the body of a TRIGGER. The purpose of such selects is to call + ** user-defined functions that have side effects. We do not care + ** about the actual results of the select. + */ + default: { + assert( eDest==SRT_Discard ); + sqlite3VdbeAddOp(v, OP_Pop, nColumn, 0); + break; + } +#endif + } + + /* Jump to the end of the loop if the LIMIT is reached. + */ + if( p->iLimit>=0 && pOrderBy==0 ){ + sqlite3VdbeAddOp(v, OP_MemIncr, -1, p->iLimit); + sqlite3VdbeAddOp(v, OP_IfMemZero, p->iLimit, iBreak); + } + return 0; +} + +/* +** Given an expression list, generate a KeyInfo structure that records +** the collating sequence for each expression in that expression list. +** +** If the ExprList is an ORDER BY or GROUP BY clause then the resulting +** KeyInfo structure is appropriate for initializing a virtual index to +** implement that clause. If the ExprList is the result set of a SELECT +** then the KeyInfo structure is appropriate for initializing a virtual +** index to implement a DISTINCT test. +** +** Space to hold the KeyInfo structure is obtain from malloc. The calling +** function is responsible for seeing that this structure is eventually +** freed. Add the KeyInfo structure to the P3 field of an opcode using +** P3_KEYINFO_HANDOFF is the usual way of dealing with this. +*/ +static KeyInfo *keyInfoFromExprList(Parse *pParse, ExprList *pList){ + sqlite3 *db = pParse->db; + int nExpr; + KeyInfo *pInfo; + struct ExprList_item *pItem; + int i; + + nExpr = pList->nExpr; + pInfo = sqliteMalloc( sizeof(*pInfo) + nExpr*(sizeof(CollSeq*)+1) ); + if( pInfo ){ + pInfo->aSortOrder = (u8*)&pInfo->aColl[nExpr]; + pInfo->nField = nExpr; + pInfo->enc = ENC(db); + for(i=0, pItem=pList->a; ipExpr); + if( !pColl ){ + pColl = db->pDfltColl; + } + pInfo->aColl[i] = pColl; + pInfo->aSortOrder[i] = pItem->sortOrder; + } + } + return pInfo; +} + + +/* +** If the inner loop was generated using a non-null pOrderBy argument, +** then the results were placed in a sorter. After the loop is terminated +** we need to run the sorter and output the results. The following +** routine generates the code needed to do that. +*/ +static void generateSortTail( + Parse *pParse, /* The parsing context */ + Select *p, /* The SELECT statement */ + Vdbe *v, /* Generate code into this VDBE */ + int nColumn, /* Number of columns of data */ + int eDest, /* Write the sorted results here */ + int iParm /* Optional parameter associated with eDest */ +){ + int brk = sqlite3VdbeMakeLabel(v); + int cont = sqlite3VdbeMakeLabel(v); + int addr; + int iTab; + ExprList *pOrderBy = p->pOrderBy; + + iTab = pOrderBy->iECursor; + addr = 1 + sqlite3VdbeAddOp(v, OP_Sort, iTab, brk); + codeOffset(v, p, cont, 0); + sqlite3VdbeAddOp(v, OP_Column, iTab, pOrderBy->nExpr + 1); + switch( eDest ){ + case SRT_Table: + case SRT_VirtualTab: { + sqlite3VdbeAddOp(v, OP_NewRowid, iParm, 0); + sqlite3VdbeAddOp(v, OP_Pull, 1, 0); + sqlite3VdbeAddOp(v, OP_Insert, iParm, 0); + break; + } +#ifndef SQLITE_OMIT_SUBQUERY + case SRT_Set: { + assert( nColumn==1 ); + sqlite3VdbeAddOp(v, OP_NotNull, -1, sqlite3VdbeCurrentAddr(v)+3); + sqlite3VdbeAddOp(v, OP_Pop, 1, 0); + sqlite3VdbeAddOp(v, OP_Goto, 0, sqlite3VdbeCurrentAddr(v)+3); + sqlite3VdbeOp3(v, OP_MakeRecord, 1, 0, "c", P3_STATIC); + sqlite3VdbeAddOp(v, OP_IdxInsert, (iParm&0x0000FFFF), 0); + break; + } + case SRT_Mem: { + assert( nColumn==1 ); + sqlite3VdbeAddOp(v, OP_MemStore, iParm, 1); + /* The LIMIT clause will terminate the loop for us */ + break; + } +#endif + case SRT_Callback: + case SRT_Subroutine: { + int i; + sqlite3VdbeAddOp(v, OP_Integer, p->pEList->nExpr, 0); + sqlite3VdbeAddOp(v, OP_Pull, 1, 0); + for(i=0; iiLimit>=0 ){ + sqlite3VdbeAddOp(v, OP_MemIncr, -1, p->iLimit); + sqlite3VdbeAddOp(v, OP_IfMemZero, p->iLimit, brk); + } + + /* The bottom of the loop + */ + sqlite3VdbeResolveLabel(v, cont); + sqlite3VdbeAddOp(v, OP_Next, iTab, addr); + sqlite3VdbeResolveLabel(v, brk); +} + +/* +** Return a pointer to a string containing the 'declaration type' of the +** expression pExpr. The string may be treated as static by the caller. +** +** The declaration type is the exact datatype definition extracted from the +** original CREATE TABLE statement if the expression is a column. The +** declaration type for a ROWID field is INTEGER. Exactly when an expression +** is considered a column can be complex in the presence of subqueries. The +** result-set expression in all of the following SELECT statements is +** considered a column by this function. +** +** SELECT col FROM tbl; +** SELECT (SELECT col FROM tbl; +** SELECT (SELECT col FROM tbl); +** SELECT abc FROM (SELECT col AS abc FROM tbl); +** +** The declaration type for any expression other than a column is NULL. +*/ +static const char *columnType( + NameContext *pNC, + Expr *pExpr, + const char **pzOriginDb, + const char **pzOriginTab, + const char **pzOriginCol +){ + char const *zType = 0; + char const *zOriginDb = 0; + char const *zOriginTab = 0; + char const *zOriginCol = 0; + int j; + if( pExpr==0 || pNC->pSrcList==0 ) return 0; + + /* The TK_AS operator can only occur in ORDER BY, GROUP BY, HAVING, + ** and LIMIT clauses. But pExpr originates in the result set of a + ** SELECT. So pExpr can never contain an AS operator. + */ + assert( pExpr->op!=TK_AS ); + + switch( pExpr->op ){ + case TK_COLUMN: { + /* The expression is a column. Locate the table the column is being + ** extracted from in NameContext.pSrcList. This table may be real + ** database table or a subquery. + */ + Table *pTab = 0; /* Table structure column is extracted from */ + Select *pS = 0; /* Select the column is extracted from */ + int iCol = pExpr->iColumn; /* Index of column in pTab */ + while( pNC && !pTab ){ + SrcList *pTabList = pNC->pSrcList; + for(j=0;jnSrc && pTabList->a[j].iCursor!=pExpr->iTable;j++); + if( jnSrc ){ + pTab = pTabList->a[j].pTab; + pS = pTabList->a[j].pSelect; + }else{ + pNC = pNC->pNext; + } + } + + if( pTab==0 ){ + /* FIX ME: + ** This can occurs if you have something like "SELECT new.x;" inside + ** a trigger. In other words, if you reference the special "new" + ** table in the result set of a select. We do not have a good way + ** to find the actual table type, so call it "TEXT". This is really + ** something of a bug, but I do not know how to fix it. + ** + ** This code does not produce the correct answer - it just prevents + ** a segfault. See ticket #1229. + */ + zType = "TEXT"; + break; + } + + assert( pTab ); +#ifndef SQLITE_OMIT_SUBQUERY + if( pS ){ + /* The "table" is actually a sub-select or a view in the FROM clause + ** of the SELECT statement. Return the declaration type and origin + ** data for the result-set column of the sub-select. + */ + if( iCol>=0 && iColpEList->nExpr ){ + /* If iCol is less than zero, then the expression requests the + ** rowid of the sub-select or view. This expression is legal (see + ** test case misc2.2.2) - it always evaluates to NULL. + */ + NameContext sNC; + Expr *p = pS->pEList->a[iCol].pExpr; + sNC.pSrcList = pS->pSrc; + sNC.pNext = 0; + sNC.pParse = pNC->pParse; + zType = columnType(&sNC, p, &zOriginDb, &zOriginTab, &zOriginCol); + } + }else +#endif + if( pTab->pSchema ){ + /* A real table */ + assert( !pS ); + if( iCol<0 ) iCol = pTab->iPKey; + assert( iCol==-1 || (iCol>=0 && iColnCol) ); + if( iCol<0 ){ + zType = "INTEGER"; + zOriginCol = "rowid"; + }else{ + zType = pTab->aCol[iCol].zType; + zOriginCol = pTab->aCol[iCol].zName; + } + zOriginTab = pTab->zName; + if( pNC->pParse ){ + int iDb = sqlite3SchemaToIndex(pNC->pParse->db, pTab->pSchema); + zOriginDb = pNC->pParse->db->aDb[iDb].zName; + } + } + break; + } +#ifndef SQLITE_OMIT_SUBQUERY + case TK_SELECT: { + /* The expression is a sub-select. Return the declaration type and + ** origin info for the single column in the result set of the SELECT + ** statement. + */ + NameContext sNC; + Select *pS = pExpr->pSelect; + Expr *p = pS->pEList->a[0].pExpr; + sNC.pSrcList = pS->pSrc; + sNC.pNext = pNC; + sNC.pParse = pNC->pParse; + zType = columnType(&sNC, p, &zOriginDb, &zOriginTab, &zOriginCol); + break; + } +#endif + } + + if( pzOriginDb ){ + assert( pzOriginTab && pzOriginCol ); + *pzOriginDb = zOriginDb; + *pzOriginTab = zOriginTab; + *pzOriginCol = zOriginCol; + } + return zType; +} + +/* +** Generate code that will tell the VDBE the declaration types of columns +** in the result set. +*/ +static void generateColumnTypes( + Parse *pParse, /* Parser context */ + SrcList *pTabList, /* List of tables */ + ExprList *pEList /* Expressions defining the result set */ +){ + Vdbe *v = pParse->pVdbe; + int i; + NameContext sNC; + sNC.pSrcList = pTabList; + sNC.pParse = pParse; + for(i=0; inExpr; i++){ + Expr *p = pEList->a[i].pExpr; + const char *zOrigDb = 0; + const char *zOrigTab = 0; + const char *zOrigCol = 0; + const char *zType = columnType(&sNC, p, &zOrigDb, &zOrigTab, &zOrigCol); + + /* The vdbe must make it's own copy of the column-type and other + ** column specific strings, in case the schema is reset before this + ** virtual machine is deleted. + */ + sqlite3VdbeSetColName(v, i, COLNAME_DECLTYPE, zType, P3_TRANSIENT); + sqlite3VdbeSetColName(v, i, COLNAME_DATABASE, zOrigDb, P3_TRANSIENT); + sqlite3VdbeSetColName(v, i, COLNAME_TABLE, zOrigTab, P3_TRANSIENT); + sqlite3VdbeSetColName(v, i, COLNAME_COLUMN, zOrigCol, P3_TRANSIENT); + } +} + +/* +** Generate code that will tell the VDBE the names of columns +** in the result set. This information is used to provide the +** azCol[] values in the callback. +*/ +static void generateColumnNames( + Parse *pParse, /* Parser context */ + SrcList *pTabList, /* List of tables */ + ExprList *pEList /* Expressions defining the result set */ +){ + Vdbe *v = pParse->pVdbe; + int i, j; + sqlite3 *db = pParse->db; + int fullNames, shortNames; + +#ifndef SQLITE_OMIT_EXPLAIN + /* If this is an EXPLAIN, skip this step */ + if( pParse->explain ){ + return; + } +#endif + + assert( v!=0 ); + if( pParse->colNamesSet || v==0 || sqlite3MallocFailed() ) return; + pParse->colNamesSet = 1; + fullNames = (db->flags & SQLITE_FullColNames)!=0; + shortNames = (db->flags & SQLITE_ShortColNames)!=0; + sqlite3VdbeSetNumCols(v, pEList->nExpr); + for(i=0; inExpr; i++){ + Expr *p; + p = pEList->a[i].pExpr; + if( p==0 ) continue; + if( pEList->a[i].zName ){ + char *zName = pEList->a[i].zName; + sqlite3VdbeSetColName(v, i, COLNAME_NAME, zName, strlen(zName)); + continue; + } + if( p->op==TK_COLUMN && pTabList ){ + Table *pTab; + char *zCol; + int iCol = p->iColumn; + for(j=0; jnSrc && pTabList->a[j].iCursor!=p->iTable; j++){} + assert( jnSrc ); + pTab = pTabList->a[j].pTab; + if( iCol<0 ) iCol = pTab->iPKey; + assert( iCol==-1 || (iCol>=0 && iColnCol) ); + if( iCol<0 ){ + zCol = "rowid"; + }else{ + zCol = pTab->aCol[iCol].zName; + } + if( !shortNames && !fullNames && p->span.z && p->span.z[0] ){ + sqlite3VdbeSetColName(v, i, COLNAME_NAME, (char*)p->span.z, p->span.n); + }else if( fullNames || (!shortNames && pTabList->nSrc>1) ){ + char *zName = 0; + char *zTab; + + zTab = pTabList->a[j].zAlias; + if( fullNames || zTab==0 ) zTab = pTab->zName; + sqlite3SetString(&zName, zTab, ".", zCol, (char*)0); + sqlite3VdbeSetColName(v, i, COLNAME_NAME, zName, P3_DYNAMIC); + }else{ + sqlite3VdbeSetColName(v, i, COLNAME_NAME, zCol, strlen(zCol)); + } + }else if( p->span.z && p->span.z[0] ){ + sqlite3VdbeSetColName(v, i, COLNAME_NAME, (char*)p->span.z, p->span.n); + /* sqlite3VdbeCompressSpace(v, addr); */ + }else{ + char zName[30]; + assert( p->op!=TK_COLUMN || pTabList==0 ); + sprintf(zName, "column%d", i+1); + sqlite3VdbeSetColName(v, i, COLNAME_NAME, zName, 0); + } + } + generateColumnTypes(pParse, pTabList, pEList); +} + +#ifndef SQLITE_OMIT_COMPOUND_SELECT +/* +** Name of the connection operator, used for error messages. +*/ +static const char *selectOpName(int id){ + char *z; + switch( id ){ + case TK_ALL: z = "UNION ALL"; break; + case TK_INTERSECT: z = "INTERSECT"; break; + case TK_EXCEPT: z = "EXCEPT"; break; + default: z = "UNION"; break; + } + return z; +} +#endif /* SQLITE_OMIT_COMPOUND_SELECT */ + +/* +** Forward declaration +*/ +static int prepSelectStmt(Parse*, Select*); + +/* +** Given a SELECT statement, generate a Table structure that describes +** the result set of that SELECT. +*/ +Table *sqlite3ResultSetOfSelect(Parse *pParse, char *zTabName, Select *pSelect){ + Table *pTab; + int i, j; + ExprList *pEList; + Column *aCol, *pCol; + + if( prepSelectStmt(pParse, pSelect) ){ + return 0; + } + if( sqlite3SelectResolve(pParse, pSelect, 0) ){ + return 0; + } + pTab = sqliteMalloc( sizeof(Table) ); + if( pTab==0 ){ + return 0; + } + pTab->nRef = 1; + pTab->zName = zTabName ? sqliteStrDup(zTabName) : 0; + pEList = pSelect->pEList; + pTab->nCol = pEList->nExpr; + assert( pTab->nCol>0 ); + pTab->aCol = aCol = sqliteMalloc( sizeof(pTab->aCol[0])*pTab->nCol ); + for(i=0, pCol=aCol; inCol; i++, pCol++){ + Expr *p, *pR; + char *zType; + char *zName; + char *zBasename; + CollSeq *pColl; + int cnt; + NameContext sNC; + + /* Get an appropriate name for the column + */ + p = pEList->a[i].pExpr; + assert( p->pRight==0 || p->pRight->token.z==0 || p->pRight->token.z[0]!=0 ); + if( (zName = pEList->a[i].zName)!=0 ){ + /* If the column contains an "AS " phrase, use as the name */ + zName = sqliteStrDup(zName); + }else if( p->op==TK_DOT + && (pR=p->pRight)!=0 && pR->token.z && pR->token.z[0] ){ + /* For columns of the from A.B use B as the name */ + zName = sqlite3MPrintf("%T", &pR->token); + }else if( p->span.z && p->span.z[0] ){ + /* Use the original text of the column expression as its name */ + zName = sqlite3MPrintf("%T", &p->span); + }else{ + /* If all else fails, make up a name */ + zName = sqlite3MPrintf("column%d", i+1); + } + sqlite3Dequote(zName); + if( sqlite3MallocFailed() ){ + sqliteFree(zName); + sqlite3DeleteTable(0, pTab); + return 0; + } + + /* Make sure the column name is unique. If the name is not unique, + ** append a integer to the name so that it becomes unique. + */ + zBasename = zName; + for(j=cnt=0; jzName = zName; + + /* Get the typename, type affinity, and collating sequence for the + ** column. + */ + memset(&sNC, 0, sizeof(sNC)); + sNC.pSrcList = pSelect->pSrc; + zType = sqliteStrDup(columnType(&sNC, p, 0, 0, 0)); + pCol->zType = zType; + pCol->affinity = sqlite3ExprAffinity(p); + pColl = sqlite3ExprCollSeq(pParse, p); + if( pColl ){ + pCol->zColl = sqliteStrDup(pColl->zName); + } + } + pTab->iPKey = -1; + return pTab; +} + +/* +** Prepare a SELECT statement for processing by doing the following +** things: +** +** (1) Make sure VDBE cursor numbers have been assigned to every +** element of the FROM clause. +** +** (2) Fill in the pTabList->a[].pTab fields in the SrcList that +** defines FROM clause. When views appear in the FROM clause, +** fill pTabList->a[].pSelect with a copy of the SELECT statement +** that implements the view. A copy is made of the view's SELECT +** statement so that we can freely modify or delete that statement +** without worrying about messing up the presistent representation +** of the view. +** +** (3) Add terms to the WHERE clause to accomodate the NATURAL keyword +** on joins and the ON and USING clause of joins. +** +** (4) Scan the list of columns in the result set (pEList) looking +** for instances of the "*" operator or the TABLE.* operator. +** If found, expand each "*" to be every column in every table +** and TABLE.* to be every column in TABLE. +** +** Return 0 on success. If there are problems, leave an error message +** in pParse and return non-zero. +*/ +static int prepSelectStmt(Parse *pParse, Select *p){ + int i, j, k, rc; + SrcList *pTabList; + ExprList *pEList; + struct SrcList_item *pFrom; + + if( p==0 || p->pSrc==0 || sqlite3MallocFailed() ){ + return 1; + } + pTabList = p->pSrc; + pEList = p->pEList; + + /* Make sure cursor numbers have been assigned to all entries in + ** the FROM clause of the SELECT statement. + */ + sqlite3SrcListAssignCursors(pParse, p->pSrc); + + /* Look up every table named in the FROM clause of the select. If + ** an entry of the FROM clause is a subquery instead of a table or view, + ** then create a transient table structure to describe the subquery. + */ + for(i=0, pFrom=pTabList->a; inSrc; i++, pFrom++){ + Table *pTab; + if( pFrom->pTab!=0 ){ + /* This statement has already been prepared. There is no need + ** to go further. */ + assert( i==0 ); + return 0; + } + if( pFrom->zName==0 ){ +#ifndef SQLITE_OMIT_SUBQUERY + /* A sub-query in the FROM clause of a SELECT */ + assert( pFrom->pSelect!=0 ); + if( pFrom->zAlias==0 ){ + pFrom->zAlias = + sqlite3MPrintf("sqlite_subquery_%p_", (void*)pFrom->pSelect); + } + assert( pFrom->pTab==0 ); + pFrom->pTab = pTab = + sqlite3ResultSetOfSelect(pParse, pFrom->zAlias, pFrom->pSelect); + if( pTab==0 ){ + return 1; + } + /* The isTransient flag indicates that the Table structure has been + ** dynamically allocated and may be freed at any time. In other words, + ** pTab is not pointing to a persistent table structure that defines + ** part of the schema. */ + pTab->isTransient = 1; +#endif + }else{ + /* An ordinary table or view name in the FROM clause */ + assert( pFrom->pTab==0 ); + pFrom->pTab = pTab = + sqlite3LocateTable(pParse,pFrom->zName,pFrom->zDatabase); + if( pTab==0 ){ + return 1; + } + pTab->nRef++; +#ifndef SQLITE_OMIT_VIEW + if( pTab->pSelect ){ + /* We reach here if the named table is a really a view */ + if( sqlite3ViewGetColumnNames(pParse, pTab) ){ + return 1; + } + /* If pFrom->pSelect!=0 it means we are dealing with a + ** view within a view. The SELECT structure has already been + ** copied by the outer view so we can skip the copy step here + ** in the inner view. + */ + if( pFrom->pSelect==0 ){ + pFrom->pSelect = sqlite3SelectDup(pTab->pSelect); + } + } +#endif + } + } + + /* Process NATURAL keywords, and ON and USING clauses of joins. + */ + if( sqliteProcessJoin(pParse, p) ) return 1; + + /* For every "*" that occurs in the column list, insert the names of + ** all columns in all tables. And for every TABLE.* insert the names + ** of all columns in TABLE. The parser inserted a special expression + ** with the TK_ALL operator for each "*" that it found in the column list. + ** The following code just has to locate the TK_ALL expressions and expand + ** each one to the list of all columns in all tables. + ** + ** The first loop just checks to see if there are any "*" operators + ** that need expanding. + */ + for(k=0; knExpr; k++){ + Expr *pE = pEList->a[k].pExpr; + if( pE->op==TK_ALL ) break; + if( pE->op==TK_DOT && pE->pRight && pE->pRight->op==TK_ALL + && pE->pLeft && pE->pLeft->op==TK_ID ) break; + } + rc = 0; + if( knExpr ){ + /* + ** If we get here it means the result set contains one or more "*" + ** operators that need to be expanded. Loop through each expression + ** in the result set and expand them one by one. + */ + struct ExprList_item *a = pEList->a; + ExprList *pNew = 0; + int flags = pParse->db->flags; + int longNames = (flags & SQLITE_FullColNames)!=0 && + (flags & SQLITE_ShortColNames)==0; + + for(k=0; knExpr; k++){ + Expr *pE = a[k].pExpr; + if( pE->op!=TK_ALL && + (pE->op!=TK_DOT || pE->pRight==0 || pE->pRight->op!=TK_ALL) ){ + /* This particular expression does not need to be expanded. + */ + pNew = sqlite3ExprListAppend(pNew, a[k].pExpr, 0); + if( pNew ){ + pNew->a[pNew->nExpr-1].zName = a[k].zName; + }else{ + rc = 1; + } + a[k].pExpr = 0; + a[k].zName = 0; + }else{ + /* This expression is a "*" or a "TABLE.*" and needs to be + ** expanded. */ + int tableSeen = 0; /* Set to 1 when TABLE matches */ + char *zTName; /* text of name of TABLE */ + if( pE->op==TK_DOT && pE->pLeft ){ + zTName = sqlite3NameFromToken(&pE->pLeft->token); + }else{ + zTName = 0; + } + for(i=0, pFrom=pTabList->a; inSrc; i++, pFrom++){ + Table *pTab = pFrom->pTab; + char *zTabName = pFrom->zAlias; + if( zTabName==0 || zTabName[0]==0 ){ + zTabName = pTab->zName; + } + if( zTName && (zTabName==0 || zTabName[0]==0 || + sqlite3StrICmp(zTName, zTabName)!=0) ){ + continue; + } + tableSeen = 1; + for(j=0; jnCol; j++){ + Expr *pExpr, *pRight; + char *zName = pTab->aCol[j].zName; + + if( i>0 ){ + struct SrcList_item *pLeft = &pTabList->a[i-1]; + if( (pLeft->jointype & JT_NATURAL)!=0 && + columnIndex(pLeft->pTab, zName)>=0 ){ + /* In a NATURAL join, omit the join columns from the + ** table on the right */ + continue; + } + if( sqlite3IdListIndex(pLeft->pUsing, zName)>=0 ){ + /* In a join with a USING clause, omit columns in the + ** using clause from the table on the right. */ + continue; + } + } + pRight = sqlite3Expr(TK_ID, 0, 0, 0); + if( pRight==0 ) break; + setToken(&pRight->token, zName); + if( zTabName && (longNames || pTabList->nSrc>1) ){ + Expr *pLeft = sqlite3Expr(TK_ID, 0, 0, 0); + pExpr = sqlite3Expr(TK_DOT, pLeft, pRight, 0); + if( pExpr==0 ) break; + setToken(&pLeft->token, zTabName); + setToken(&pExpr->span, sqlite3MPrintf("%s.%s", zTabName, zName)); + pExpr->span.dyn = 1; + pExpr->token.z = 0; + pExpr->token.n = 0; + pExpr->token.dyn = 0; + }else{ + pExpr = pRight; + pExpr->span = pExpr->token; + } + if( longNames ){ + pNew = sqlite3ExprListAppend(pNew, pExpr, &pExpr->span); + }else{ + pNew = sqlite3ExprListAppend(pNew, pExpr, &pRight->token); + } + } + } + if( !tableSeen ){ + if( zTName ){ + sqlite3ErrorMsg(pParse, "no such table: %s", zTName); + }else{ + sqlite3ErrorMsg(pParse, "no tables specified"); + } + rc = 1; + } + sqliteFree(zTName); + } + } + sqlite3ExprListDelete(pEList); + p->pEList = pNew; + } + return rc; +} + +#ifndef SQLITE_OMIT_COMPOUND_SELECT +/* +** This routine associates entries in an ORDER BY expression list with +** columns in a result. For each ORDER BY expression, the opcode of +** the top-level node is changed to TK_COLUMN and the iColumn value of +** the top-level node is filled in with column number and the iTable +** value of the top-level node is filled with iTable parameter. +** +** If there are prior SELECT clauses, they are processed first. A match +** in an earlier SELECT takes precedence over a later SELECT. +** +** Any entry that does not match is flagged as an error. The number +** of errors is returned. +*/ +static int matchOrderbyToColumn( + Parse *pParse, /* A place to leave error messages */ + Select *pSelect, /* Match to result columns of this SELECT */ + ExprList *pOrderBy, /* The ORDER BY values to match against columns */ + int iTable, /* Insert this value in iTable */ + int mustComplete /* If TRUE all ORDER BYs must match */ +){ + int nErr = 0; + int i, j; + ExprList *pEList; + + if( pSelect==0 || pOrderBy==0 ) return 1; + if( mustComplete ){ + for(i=0; inExpr; i++){ pOrderBy->a[i].done = 0; } + } + if( prepSelectStmt(pParse, pSelect) ){ + return 1; + } + if( pSelect->pPrior ){ + if( matchOrderbyToColumn(pParse, pSelect->pPrior, pOrderBy, iTable, 0) ){ + return 1; + } + } + pEList = pSelect->pEList; + for(i=0; inExpr; i++){ + Expr *pE = pOrderBy->a[i].pExpr; + int iCol = -1; + if( pOrderBy->a[i].done ) continue; + if( sqlite3ExprIsInteger(pE, &iCol) ){ + if( iCol<=0 || iCol>pEList->nExpr ){ + sqlite3ErrorMsg(pParse, + "ORDER BY position %d should be between 1 and %d", + iCol, pEList->nExpr); + nErr++; + break; + } + if( !mustComplete ) continue; + iCol--; + } + for(j=0; iCol<0 && jnExpr; j++){ + if( pEList->a[j].zName && (pE->op==TK_ID || pE->op==TK_STRING) ){ + char *zName, *zLabel; + zName = pEList->a[j].zName; + zLabel = sqlite3NameFromToken(&pE->token); + assert( zLabel!=0 ); + if( sqlite3StrICmp(zName, zLabel)==0 ){ + iCol = j; + } + sqliteFree(zLabel); + } + if( iCol<0 && sqlite3ExprCompare(pE, pEList->a[j].pExpr) ){ + iCol = j; + } + } + if( iCol>=0 ){ + pE->op = TK_COLUMN; + pE->iColumn = iCol; + pE->iTable = iTable; + pE->iAgg = -1; + pOrderBy->a[i].done = 1; + } + if( iCol<0 && mustComplete ){ + sqlite3ErrorMsg(pParse, + "ORDER BY term number %d does not match any result column", i+1); + nErr++; + break; + } + } + return nErr; +} +#endif /* #ifndef SQLITE_OMIT_COMPOUND_SELECT */ + +/* +** Get a VDBE for the given parser context. Create a new one if necessary. +** If an error occurs, return NULL and leave a message in pParse. +*/ +Vdbe *sqlite3GetVdbe(Parse *pParse){ + Vdbe *v = pParse->pVdbe; + if( v==0 ){ + v = pParse->pVdbe = sqlite3VdbeCreate(pParse->db); + } + return v; +} + + +/* +** Compute the iLimit and iOffset fields of the SELECT based on the +** pLimit and pOffset expressions. pLimit and pOffset hold the expressions +** that appear in the original SQL statement after the LIMIT and OFFSET +** keywords. Or NULL if those keywords are omitted. iLimit and iOffset +** are the integer memory register numbers for counters used to compute +** the limit and offset. If there is no limit and/or offset, then +** iLimit and iOffset are negative. +** +** This routine changes the values of iLimit and iOffset only if +** a limit or offset is defined by pLimit and pOffset. iLimit and +** iOffset should have been preset to appropriate default values +** (usually but not always -1) prior to calling this routine. +** Only if pLimit!=0 or pOffset!=0 do the limit registers get +** redefined. The UNION ALL operator uses this property to force +** the reuse of the same limit and offset registers across multiple +** SELECT statements. +*/ +static void computeLimitRegisters(Parse *pParse, Select *p, int iBreak){ + Vdbe *v = 0; + int iLimit = 0; + int iOffset; + int addr1, addr2; + + /* + ** "LIMIT -1" always shows all rows. There is some + ** contraversy about what the correct behavior should be. + ** The current implementation interprets "LIMIT 0" to mean + ** no rows. + */ + if( p->pLimit ){ + p->iLimit = iLimit = pParse->nMem; + pParse->nMem += 2; + v = sqlite3GetVdbe(pParse); + if( v==0 ) return; + sqlite3ExprCode(pParse, p->pLimit); + sqlite3VdbeAddOp(v, OP_MustBeInt, 0, 0); + sqlite3VdbeAddOp(v, OP_MemStore, iLimit, 0); + VdbeComment((v, "# LIMIT counter")); + sqlite3VdbeAddOp(v, OP_IfMemZero, iLimit, iBreak); + } + if( p->pOffset ){ + p->iOffset = iOffset = pParse->nMem++; + v = sqlite3GetVdbe(pParse); + if( v==0 ) return; + sqlite3ExprCode(pParse, p->pOffset); + sqlite3VdbeAddOp(v, OP_MustBeInt, 0, 0); + sqlite3VdbeAddOp(v, OP_MemStore, iOffset, p->pLimit==0); + VdbeComment((v, "# OFFSET counter")); + addr1 = sqlite3VdbeAddOp(v, OP_IfMemPos, iOffset, 0); + sqlite3VdbeAddOp(v, OP_Pop, 1, 0); + sqlite3VdbeAddOp(v, OP_Integer, 0, 0); + sqlite3VdbeJumpHere(v, addr1); + if( p->pLimit ){ + sqlite3VdbeAddOp(v, OP_Add, 0, 0); + } + } + if( p->pLimit ){ + addr1 = sqlite3VdbeAddOp(v, OP_IfMemPos, iLimit, 0); + sqlite3VdbeAddOp(v, OP_Pop, 1, 0); + sqlite3VdbeAddOp(v, OP_MemInt, -1, iLimit+1); + addr2 = sqlite3VdbeAddOp(v, OP_Goto, 0, 0); + sqlite3VdbeJumpHere(v, addr1); + sqlite3VdbeAddOp(v, OP_MemStore, iLimit+1, 1); + VdbeComment((v, "# LIMIT+OFFSET")); + sqlite3VdbeJumpHere(v, addr2); + } +} + +/* +** Allocate a virtual index to use for sorting. +*/ +static void createSortingIndex(Parse *pParse, Select *p, ExprList *pOrderBy){ + if( pOrderBy ){ + int addr; + assert( pOrderBy->iECursor==0 ); + pOrderBy->iECursor = pParse->nTab++; + addr = sqlite3VdbeAddOp(pParse->pVdbe, OP_OpenVirtual, + pOrderBy->iECursor, pOrderBy->nExpr+1); + assert( p->addrOpenVirt[2] == -1 ); + p->addrOpenVirt[2] = addr; + } +} + +/* +** The opcode at addr is an OP_OpenVirtual that created a sorting +** index tha we ended up not needing. This routine changes that +** opcode to OP_Noop. +*/ +static void uncreateSortingIndex(Parse *pParse, int addr){ + Vdbe *v = pParse->pVdbe; + VdbeOp *pOp = sqlite3VdbeGetOp(v, addr); + sqlite3VdbeChangeP3(v, addr, 0, 0); + pOp->opcode = OP_Noop; + pOp->p1 = 0; + pOp->p2 = 0; +} + +#ifndef SQLITE_OMIT_COMPOUND_SELECT +/* +** Return the appropriate collating sequence for the iCol-th column of +** the result set for the compound-select statement "p". Return NULL if +** the column has no default collating sequence. +** +** The collating sequence for the compound select is taken from the +** left-most term of the select that has a collating sequence. +*/ +static CollSeq *multiSelectCollSeq(Parse *pParse, Select *p, int iCol){ + CollSeq *pRet; + if( p->pPrior ){ + pRet = multiSelectCollSeq(pParse, p->pPrior, iCol); + }else{ + pRet = 0; + } + if( pRet==0 ){ + pRet = sqlite3ExprCollSeq(pParse, p->pEList->a[iCol].pExpr); + } + return pRet; +} +#endif /* SQLITE_OMIT_COMPOUND_SELECT */ + +#ifndef SQLITE_OMIT_COMPOUND_SELECT +/* +** This routine is called to process a query that is really the union +** or intersection of two or more separate queries. +** +** "p" points to the right-most of the two queries. the query on the +** left is p->pPrior. The left query could also be a compound query +** in which case this routine will be called recursively. +** +** The results of the total query are to be written into a destination +** of type eDest with parameter iParm. +** +** Example 1: Consider a three-way compound SQL statement. +** +** SELECT a FROM t1 UNION SELECT b FROM t2 UNION SELECT c FROM t3 +** +** This statement is parsed up as follows: +** +** SELECT c FROM t3 +** | +** `-----> SELECT b FROM t2 +** | +** `------> SELECT a FROM t1 +** +** The arrows in the diagram above represent the Select.pPrior pointer. +** So if this routine is called with p equal to the t3 query, then +** pPrior will be the t2 query. p->op will be TK_UNION in this case. +** +** Notice that because of the way SQLite parses compound SELECTs, the +** individual selects always group from left to right. +*/ +static int multiSelect( + Parse *pParse, /* Parsing context */ + Select *p, /* The right-most of SELECTs to be coded */ + int eDest, /* \___ Store query results as specified */ + int iParm, /* / by these two parameters. */ + char *aff /* If eDest is SRT_Union, the affinity string */ +){ + int rc = SQLITE_OK; /* Success code from a subroutine */ + Select *pPrior; /* Another SELECT immediately to our left */ + Vdbe *v; /* Generate code to this VDBE */ + int nCol; /* Number of columns in the result set */ + ExprList *pOrderBy; /* The ORDER BY clause on p */ + int aSetP2[2]; /* Set P2 value of these op to number of columns */ + int nSetP2 = 0; /* Number of slots in aSetP2[] used */ + + /* Make sure there is no ORDER BY or LIMIT clause on prior SELECTs. Only + ** the last (right-most) SELECT in the series may have an ORDER BY or LIMIT. + */ + if( p==0 || p->pPrior==0 ){ + rc = 1; + goto multi_select_end; + } + pPrior = p->pPrior; + assert( pPrior->pRightmost!=pPrior ); + assert( pPrior->pRightmost==p->pRightmost ); + if( pPrior->pOrderBy ){ + sqlite3ErrorMsg(pParse,"ORDER BY clause should come after %s not before", + selectOpName(p->op)); + rc = 1; + goto multi_select_end; + } + if( pPrior->pLimit ){ + sqlite3ErrorMsg(pParse,"LIMIT clause should come after %s not before", + selectOpName(p->op)); + rc = 1; + goto multi_select_end; + } + + /* Make sure we have a valid query engine. If not, create a new one. + */ + v = sqlite3GetVdbe(pParse); + if( v==0 ){ + rc = 1; + goto multi_select_end; + } + + /* Create the destination temporary table if necessary + */ + if( eDest==SRT_VirtualTab ){ + assert( p->pEList ); + assert( nSetP2pOrderBy; + switch( p->op ){ + case TK_ALL: { + if( pOrderBy==0 ){ + int addr = 0; + assert( !pPrior->pLimit ); + pPrior->pLimit = p->pLimit; + pPrior->pOffset = p->pOffset; + rc = sqlite3Select(pParse, pPrior, eDest, iParm, 0, 0, 0, aff); + p->pLimit = 0; + p->pOffset = 0; + if( rc ){ + goto multi_select_end; + } + p->pPrior = 0; + p->iLimit = pPrior->iLimit; + p->iOffset = pPrior->iOffset; + if( p->iLimit>=0 ){ + addr = sqlite3VdbeAddOp(v, OP_IfMemZero, p->iLimit, 0); + VdbeComment((v, "# Jump ahead if LIMIT reached")); + } + rc = sqlite3Select(pParse, p, eDest, iParm, 0, 0, 0, aff); + p->pPrior = pPrior; + if( rc ){ + goto multi_select_end; + } + if( addr ){ + sqlite3VdbeJumpHere(v, addr); + } + break; + } + /* For UNION ALL ... ORDER BY fall through to the next case */ + } + case TK_EXCEPT: + case TK_UNION: { + int unionTab; /* Cursor number of the temporary table holding result */ + int op = 0; /* One of the SRT_ operations to apply to self */ + int priorOp; /* The SRT_ operation to apply to prior selects */ + Expr *pLimit, *pOffset; /* Saved values of p->nLimit and p->nOffset */ + int addr; + + priorOp = p->op==TK_ALL ? SRT_Table : SRT_Union; + if( eDest==priorOp && pOrderBy==0 && !p->pLimit && !p->pOffset ){ + /* We can reuse a temporary table generated by a SELECT to our + ** right. + */ + unionTab = iParm; + }else{ + /* We will need to create our own temporary table to hold the + ** intermediate results. + */ + unionTab = pParse->nTab++; + if( pOrderBy && matchOrderbyToColumn(pParse, p, pOrderBy, unionTab,1) ){ + rc = 1; + goto multi_select_end; + } + addr = sqlite3VdbeAddOp(v, OP_OpenVirtual, unionTab, 0); + if( priorOp==SRT_Table ){ + assert( nSetP2addrOpenVirt[0] == -1 ); + p->addrOpenVirt[0] = addr; + p->pRightmost->usesVirt = 1; + } + createSortingIndex(pParse, p, pOrderBy); + assert( p->pEList ); + } + + /* Code the SELECT statements to our left + */ + assert( !pPrior->pOrderBy ); + rc = sqlite3Select(pParse, pPrior, priorOp, unionTab, 0, 0, 0, aff); + if( rc ){ + goto multi_select_end; + } + + /* Code the current SELECT statement + */ + switch( p->op ){ + case TK_EXCEPT: op = SRT_Except; break; + case TK_UNION: op = SRT_Union; break; + case TK_ALL: op = SRT_Table; break; + } + p->pPrior = 0; + p->pOrderBy = 0; + p->disallowOrderBy = pOrderBy!=0; + pLimit = p->pLimit; + p->pLimit = 0; + pOffset = p->pOffset; + p->pOffset = 0; + rc = sqlite3Select(pParse, p, op, unionTab, 0, 0, 0, aff); + p->pPrior = pPrior; + p->pOrderBy = pOrderBy; + sqlite3ExprDelete(p->pLimit); + p->pLimit = pLimit; + p->pOffset = pOffset; + p->iLimit = -1; + p->iOffset = -1; + if( rc ){ + goto multi_select_end; + } + + + /* Convert the data in the temporary table into whatever form + ** it is that we currently need. + */ + if( eDest!=priorOp || unionTab!=iParm ){ + int iCont, iBreak, iStart; + assert( p->pEList ); + if( eDest==SRT_Callback ){ + generateColumnNames(pParse, 0, p->pEList); + } + iBreak = sqlite3VdbeMakeLabel(v); + iCont = sqlite3VdbeMakeLabel(v); + computeLimitRegisters(pParse, p, iBreak); + sqlite3VdbeAddOp(v, OP_Rewind, unionTab, iBreak); + iStart = sqlite3VdbeCurrentAddr(v); + rc = selectInnerLoop(pParse, p, p->pEList, unionTab, p->pEList->nExpr, + pOrderBy, -1, eDest, iParm, + iCont, iBreak, 0); + if( rc ){ + rc = 1; + goto multi_select_end; + } + sqlite3VdbeResolveLabel(v, iCont); + sqlite3VdbeAddOp(v, OP_Next, unionTab, iStart); + sqlite3VdbeResolveLabel(v, iBreak); + sqlite3VdbeAddOp(v, OP_Close, unionTab, 0); + } + break; + } + case TK_INTERSECT: { + int tab1, tab2; + int iCont, iBreak, iStart; + Expr *pLimit, *pOffset; + int addr; + + /* INTERSECT is different from the others since it requires + ** two temporary tables. Hence it has its own case. Begin + ** by allocating the tables we will need. + */ + tab1 = pParse->nTab++; + tab2 = pParse->nTab++; + if( pOrderBy && matchOrderbyToColumn(pParse,p,pOrderBy,tab1,1) ){ + rc = 1; + goto multi_select_end; + } + createSortingIndex(pParse, p, pOrderBy); + + addr = sqlite3VdbeAddOp(v, OP_OpenVirtual, tab1, 0); + assert( p->addrOpenVirt[0] == -1 ); + p->addrOpenVirt[0] = addr; + p->pRightmost->usesVirt = 1; + assert( p->pEList ); + + /* Code the SELECTs to our left into temporary table "tab1". + */ + rc = sqlite3Select(pParse, pPrior, SRT_Union, tab1, 0, 0, 0, aff); + if( rc ){ + goto multi_select_end; + } + + /* Code the current SELECT into temporary table "tab2" + */ + addr = sqlite3VdbeAddOp(v, OP_OpenVirtual, tab2, 0); + assert( p->addrOpenVirt[1] == -1 ); + p->addrOpenVirt[1] = addr; + p->pPrior = 0; + pLimit = p->pLimit; + p->pLimit = 0; + pOffset = p->pOffset; + p->pOffset = 0; + rc = sqlite3Select(pParse, p, SRT_Union, tab2, 0, 0, 0, aff); + p->pPrior = pPrior; + sqlite3ExprDelete(p->pLimit); + p->pLimit = pLimit; + p->pOffset = pOffset; + if( rc ){ + goto multi_select_end; + } + + /* Generate code to take the intersection of the two temporary + ** tables. + */ + assert( p->pEList ); + if( eDest==SRT_Callback ){ + generateColumnNames(pParse, 0, p->pEList); + } + iBreak = sqlite3VdbeMakeLabel(v); + iCont = sqlite3VdbeMakeLabel(v); + computeLimitRegisters(pParse, p, iBreak); + sqlite3VdbeAddOp(v, OP_Rewind, tab1, iBreak); + iStart = sqlite3VdbeAddOp(v, OP_RowKey, tab1, 0); + sqlite3VdbeAddOp(v, OP_NotFound, tab2, iCont); + rc = selectInnerLoop(pParse, p, p->pEList, tab1, p->pEList->nExpr, + pOrderBy, -1, eDest, iParm, + iCont, iBreak, 0); + if( rc ){ + rc = 1; + goto multi_select_end; + } + sqlite3VdbeResolveLabel(v, iCont); + sqlite3VdbeAddOp(v, OP_Next, tab1, iStart); + sqlite3VdbeResolveLabel(v, iBreak); + sqlite3VdbeAddOp(v, OP_Close, tab2, 0); + sqlite3VdbeAddOp(v, OP_Close, tab1, 0); + break; + } + } + + /* Make sure all SELECTs in the statement have the same number of elements + ** in their result sets. + */ + assert( p->pEList && pPrior->pEList ); + if( p->pEList->nExpr!=pPrior->pEList->nExpr ){ + sqlite3ErrorMsg(pParse, "SELECTs to the left and right of %s" + " do not have the same number of result columns", selectOpName(p->op)); + rc = 1; + goto multi_select_end; + } + + /* Set the number of columns in temporary tables + */ + nCol = p->pEList->nExpr; + while( nSetP2 ){ + sqlite3VdbeChangeP2(v, aSetP2[--nSetP2], nCol); + } + + /* Compute collating sequences used by either the ORDER BY clause or + ** by any temporary tables needed to implement the compound select. + ** Attach the KeyInfo structure to all temporary tables. Invoke the + ** ORDER BY processing if there is an ORDER BY clause. + ** + ** This section is run by the right-most SELECT statement only. + ** SELECT statements to the left always skip this part. The right-most + ** SELECT might also skip this part if it has no ORDER BY clause and + ** no temp tables are required. + */ + if( pOrderBy || p->usesVirt ){ + int i; /* Loop counter */ + KeyInfo *pKeyInfo; /* Collating sequence for the result set */ + Select *pLoop; /* For looping through SELECT statements */ + CollSeq **apColl; + CollSeq **aCopy; + + assert( p->pRightmost==p ); + pKeyInfo = sqliteMalloc(sizeof(*pKeyInfo)+nCol*2*sizeof(CollSeq*) + nCol); + if( !pKeyInfo ){ + rc = SQLITE_NOMEM; + goto multi_select_end; + } + + pKeyInfo->enc = ENC(pParse->db); + pKeyInfo->nField = nCol; + + for(i=0, apColl=pKeyInfo->aColl; idb->pDfltColl; + } + } + + for(pLoop=p; pLoop; pLoop=pLoop->pPrior){ + for(i=0; i<2; i++){ + int addr = pLoop->addrOpenVirt[i]; + if( addr<0 ){ + /* If [0] is unused then [1] is also unused. So we can + ** always safely abort as soon as the first unused slot is found */ + assert( pLoop->addrOpenVirt[1]<0 ); + break; + } + sqlite3VdbeChangeP2(v, addr, nCol); + sqlite3VdbeChangeP3(v, addr, (char*)pKeyInfo, P3_KEYINFO); + } + } + + if( pOrderBy ){ + struct ExprList_item *pOTerm = pOrderBy->a; + int nOrderByExpr = pOrderBy->nExpr; + int addr; + u8 *pSortOrder; + + aCopy = &pKeyInfo->aColl[nCol]; + pSortOrder = pKeyInfo->aSortOrder = (u8*)&aCopy[nCol]; + memcpy(aCopy, pKeyInfo->aColl, nCol*sizeof(CollSeq*)); + apColl = pKeyInfo->aColl; + for(i=0; ipExpr; + char *zName = pOTerm->zName; + assert( pExpr->op==TK_COLUMN && pExpr->iColumniColumn]; + } + *pSortOrder = pOTerm->sortOrder; + } + assert( p->pRightmost==p ); + assert( p->addrOpenVirt[2]>=0 ); + addr = p->addrOpenVirt[2]; + sqlite3VdbeChangeP2(v, addr, p->pEList->nExpr+2); + pKeyInfo->nField = nOrderByExpr; + sqlite3VdbeChangeP3(v, addr, (char*)pKeyInfo, P3_KEYINFO_HANDOFF); + pKeyInfo = 0; + generateSortTail(pParse, p, v, p->pEList->nExpr, eDest, iParm); + } + + sqliteFree(pKeyInfo); + } + +multi_select_end: + return rc; +} +#endif /* SQLITE_OMIT_COMPOUND_SELECT */ + +#ifndef SQLITE_OMIT_VIEW +/* +** Scan through the expression pExpr. Replace every reference to +** a column in table number iTable with a copy of the iColumn-th +** entry in pEList. (But leave references to the ROWID column +** unchanged.) +** +** This routine is part of the flattening procedure. A subquery +** whose result set is defined by pEList appears as entry in the +** FROM clause of a SELECT such that the VDBE cursor assigned to that +** FORM clause entry is iTable. This routine make the necessary +** changes to pExpr so that it refers directly to the source table +** of the subquery rather the result set of the subquery. +*/ +static void substExprList(ExprList*,int,ExprList*); /* Forward Decl */ +static void substSelect(Select *, int, ExprList *); /* Forward Decl */ +static void substExpr(Expr *pExpr, int iTable, ExprList *pEList){ + if( pExpr==0 ) return; + if( pExpr->op==TK_COLUMN && pExpr->iTable==iTable ){ + if( pExpr->iColumn<0 ){ + pExpr->op = TK_NULL; + }else{ + Expr *pNew; + assert( pEList!=0 && pExpr->iColumnnExpr ); + assert( pExpr->pLeft==0 && pExpr->pRight==0 && pExpr->pList==0 ); + pNew = pEList->a[pExpr->iColumn].pExpr; + assert( pNew!=0 ); + pExpr->op = pNew->op; + assert( pExpr->pLeft==0 ); + pExpr->pLeft = sqlite3ExprDup(pNew->pLeft); + assert( pExpr->pRight==0 ); + pExpr->pRight = sqlite3ExprDup(pNew->pRight); + assert( pExpr->pList==0 ); + pExpr->pList = sqlite3ExprListDup(pNew->pList); + pExpr->iTable = pNew->iTable; + pExpr->iColumn = pNew->iColumn; + pExpr->iAgg = pNew->iAgg; + sqlite3TokenCopy(&pExpr->token, &pNew->token); + sqlite3TokenCopy(&pExpr->span, &pNew->span); + pExpr->pSelect = sqlite3SelectDup(pNew->pSelect); + pExpr->flags = pNew->flags; + } + }else{ + substExpr(pExpr->pLeft, iTable, pEList); + substExpr(pExpr->pRight, iTable, pEList); + substSelect(pExpr->pSelect, iTable, pEList); + substExprList(pExpr->pList, iTable, pEList); + } +} +static void substExprList(ExprList *pList, int iTable, ExprList *pEList){ + int i; + if( pList==0 ) return; + for(i=0; inExpr; i++){ + substExpr(pList->a[i].pExpr, iTable, pEList); + } +} +static void substSelect(Select *p, int iTable, ExprList *pEList){ + if( !p ) return; + substExprList(p->pEList, iTable, pEList); + substExprList(p->pGroupBy, iTable, pEList); + substExprList(p->pOrderBy, iTable, pEList); + substExpr(p->pHaving, iTable, pEList); + substExpr(p->pWhere, iTable, pEList); +} +#endif /* !defined(SQLITE_OMIT_VIEW) */ + +#ifndef SQLITE_OMIT_VIEW +/* +** This routine attempts to flatten subqueries in order to speed +** execution. It returns 1 if it makes changes and 0 if no flattening +** occurs. +** +** To understand the concept of flattening, consider the following +** query: +** +** SELECT a FROM (SELECT x+y AS a FROM t1 WHERE z<100) WHERE a>5 +** +** The default way of implementing this query is to execute the +** subquery first and store the results in a temporary table, then +** run the outer query on that temporary table. This requires two +** passes over the data. Furthermore, because the temporary table +** has no indices, the WHERE clause on the outer query cannot be +** optimized. +** +** This routine attempts to rewrite queries such as the above into +** a single flat select, like this: +** +** SELECT x+y AS a FROM t1 WHERE z<100 AND a>5 +** +** The code generated for this simpification gives the same result +** but only has to scan the data once. And because indices might +** exist on the table t1, a complete scan of the data might be +** avoided. +** +** Flattening is only attempted if all of the following are true: +** +** (1) The subquery and the outer query do not both use aggregates. +** +** (2) The subquery is not an aggregate or the outer query is not a join. +** +** (3) The subquery is not the right operand of a left outer join, or +** the subquery is not itself a join. (Ticket #306) +** +** (4) The subquery is not DISTINCT or the outer query is not a join. +** +** (5) The subquery is not DISTINCT or the outer query does not use +** aggregates. +** +** (6) The subquery does not use aggregates or the outer query is not +** DISTINCT. +** +** (7) The subquery has a FROM clause. +** +** (8) The subquery does not use LIMIT or the outer query is not a join. +** +** (9) The subquery does not use LIMIT or the outer query does not use +** aggregates. +** +** (10) The subquery does not use aggregates or the outer query does not +** use LIMIT. +** +** (11) The subquery and the outer query do not both have ORDER BY clauses. +** +** (12) The subquery is not the right term of a LEFT OUTER JOIN or the +** subquery has no WHERE clause. (added by ticket #350) +** +** (13) The subquery and outer query do not both use LIMIT +** +** (14) The subquery does not use OFFSET +** +** In this routine, the "p" parameter is a pointer to the outer query. +** The subquery is p->pSrc->a[iFrom]. isAgg is true if the outer query +** uses aggregates and subqueryIsAgg is true if the subquery uses aggregates. +** +** If flattening is not attempted, this routine is a no-op and returns 0. +** If flattening is attempted this routine returns 1. +** +** All of the expression analysis must occur on both the outer query and +** the subquery before this routine runs. +*/ +static int flattenSubquery( + Parse *pParse, /* The parsing context */ + Select *p, /* The parent or outer SELECT statement */ + int iFrom, /* Index in p->pSrc->a[] of the inner subquery */ + int isAgg, /* True if outer SELECT uses aggregate functions */ + int subqueryIsAgg /* True if the subquery uses aggregate functions */ +){ + Select *pSub; /* The inner query or "subquery" */ + SrcList *pSrc; /* The FROM clause of the outer query */ + SrcList *pSubSrc; /* The FROM clause of the subquery */ + ExprList *pList; /* The result set of the outer query */ + int iParent; /* VDBE cursor number of the pSub result set temp table */ + int i; /* Loop counter */ + Expr *pWhere; /* The WHERE clause */ + struct SrcList_item *pSubitem; /* The subquery */ + + /* Check to see if flattening is permitted. Return 0 if not. + */ + if( p==0 ) return 0; + pSrc = p->pSrc; + assert( pSrc && iFrom>=0 && iFromnSrc ); + pSubitem = &pSrc->a[iFrom]; + pSub = pSubitem->pSelect; + assert( pSub!=0 ); + if( isAgg && subqueryIsAgg ) return 0; /* Restriction (1) */ + if( subqueryIsAgg && pSrc->nSrc>1 ) return 0; /* Restriction (2) */ + pSubSrc = pSub->pSrc; + assert( pSubSrc ); + /* Prior to version 3.1.2, when LIMIT and OFFSET had to be simple constants, + ** not arbitrary expresssions, we allowed some combining of LIMIT and OFFSET + ** because they could be computed at compile-time. But when LIMIT and OFFSET + ** became arbitrary expressions, we were forced to add restrictions (13) + ** and (14). */ + if( pSub->pLimit && p->pLimit ) return 0; /* Restriction (13) */ + if( pSub->pOffset ) return 0; /* Restriction (14) */ + if( pSubSrc->nSrc==0 ) return 0; /* Restriction (7) */ + if( (pSub->isDistinct || pSub->pLimit) + && (pSrc->nSrc>1 || isAgg) ){ /* Restrictions (4)(5)(8)(9) */ + return 0; + } + if( p->isDistinct && subqueryIsAgg ) return 0; /* Restriction (6) */ + if( (p->disallowOrderBy || p->pOrderBy) && pSub->pOrderBy ){ + return 0; /* Restriction (11) */ + } + + /* Restriction 3: If the subquery is a join, make sure the subquery is + ** not used as the right operand of an outer join. Examples of why this + ** is not allowed: + ** + ** t1 LEFT OUTER JOIN (t2 JOIN t3) + ** + ** If we flatten the above, we would get + ** + ** (t1 LEFT OUTER JOIN t2) JOIN t3 + ** + ** which is not at all the same thing. + */ + if( pSubSrc->nSrc>1 && iFrom>0 && (pSrc->a[iFrom-1].jointype & JT_OUTER)!=0 ){ + return 0; + } + + /* Restriction 12: If the subquery is the right operand of a left outer + ** join, make sure the subquery has no WHERE clause. + ** An examples of why this is not allowed: + ** + ** t1 LEFT OUTER JOIN (SELECT * FROM t2 WHERE t2.x>0) + ** + ** If we flatten the above, we would get + ** + ** (t1 LEFT OUTER JOIN t2) WHERE t2.x>0 + ** + ** But the t2.x>0 test will always fail on a NULL row of t2, which + ** effectively converts the OUTER JOIN into an INNER JOIN. + */ + if( iFrom>0 && (pSrc->a[iFrom-1].jointype & JT_OUTER)!=0 + && pSub->pWhere!=0 ){ + return 0; + } + + /* If we reach this point, it means flattening is permitted for the + ** iFrom-th entry of the FROM clause in the outer query. + */ + + /* Move all of the FROM elements of the subquery into the + ** the FROM clause of the outer query. Before doing this, remember + ** the cursor number for the original outer query FROM element in + ** iParent. The iParent cursor will never be used. Subsequent code + ** will scan expressions looking for iParent references and replace + ** those references with expressions that resolve to the subquery FROM + ** elements we are now copying in. + */ + iParent = pSubitem->iCursor; + { + int nSubSrc = pSubSrc->nSrc; + int jointype = pSubitem->jointype; + + sqlite3DeleteTable(0, pSubitem->pTab); + sqliteFree(pSubitem->zDatabase); + sqliteFree(pSubitem->zName); + sqliteFree(pSubitem->zAlias); + if( nSubSrc>1 ){ + int extra = nSubSrc - 1; + for(i=1; ipSrc = pSrc; + for(i=pSrc->nSrc-1; i-extra>=iFrom; i--){ + pSrc->a[i] = pSrc->a[i-extra]; + } + } + for(i=0; ia[i+iFrom] = pSubSrc->a[i]; + memset(&pSubSrc->a[i], 0, sizeof(pSubSrc->a[i])); + } + pSrc->a[iFrom+nSubSrc-1].jointype = jointype; + } + + /* Now begin substituting subquery result set expressions for + ** references to the iParent in the outer query. + ** + ** Example: + ** + ** SELECT a+5, b*10 FROM (SELECT x*3 AS a, y+10 AS b FROM t1) WHERE a>b; + ** \ \_____________ subquery __________/ / + ** \_____________________ outer query ______________________________/ + ** + ** We look at every expression in the outer query and every place we see + ** "a" we substitute "x*3" and every place we see "b" we substitute "y+10". + */ + substExprList(p->pEList, iParent, pSub->pEList); + pList = p->pEList; + for(i=0; inExpr; i++){ + Expr *pExpr; + if( pList->a[i].zName==0 && (pExpr = pList->a[i].pExpr)->span.z!=0 ){ + pList->a[i].zName = sqliteStrNDup((char*)pExpr->span.z, pExpr->span.n); + } + } + if( isAgg ){ + substExprList(p->pGroupBy, iParent, pSub->pEList); + substExpr(p->pHaving, iParent, pSub->pEList); + } + if( pSub->pOrderBy ){ + assert( p->pOrderBy==0 ); + p->pOrderBy = pSub->pOrderBy; + pSub->pOrderBy = 0; + }else if( p->pOrderBy ){ + substExprList(p->pOrderBy, iParent, pSub->pEList); + } + if( pSub->pWhere ){ + pWhere = sqlite3ExprDup(pSub->pWhere); + }else{ + pWhere = 0; + } + if( subqueryIsAgg ){ + assert( p->pHaving==0 ); + p->pHaving = p->pWhere; + p->pWhere = pWhere; + substExpr(p->pHaving, iParent, pSub->pEList); + p->pHaving = sqlite3ExprAnd(p->pHaving, sqlite3ExprDup(pSub->pHaving)); + assert( p->pGroupBy==0 ); + p->pGroupBy = sqlite3ExprListDup(pSub->pGroupBy); + }else{ + substExpr(p->pWhere, iParent, pSub->pEList); + p->pWhere = sqlite3ExprAnd(p->pWhere, pWhere); + } + + /* The flattened query is distinct if either the inner or the + ** outer query is distinct. + */ + p->isDistinct = p->isDistinct || pSub->isDistinct; + + /* + ** SELECT ... FROM (SELECT ... LIMIT a OFFSET b) LIMIT x OFFSET y; + ** + ** One is tempted to try to add a and b to combine the limits. But this + ** does not work if either limit is negative. + */ + if( pSub->pLimit ){ + p->pLimit = pSub->pLimit; + pSub->pLimit = 0; + } + + /* Finially, delete what is left of the subquery and return + ** success. + */ + sqlite3SelectDelete(pSub); + return 1; +} +#endif /* SQLITE_OMIT_VIEW */ + +/* +** Analyze the SELECT statement passed in as an argument to see if it +** is a simple min() or max() query. If it is and this query can be +** satisfied using a single seek to the beginning or end of an index, +** then generate the code for this SELECT and return 1. If this is not a +** simple min() or max() query, then return 0; +** +** A simply min() or max() query looks like this: +** +** SELECT min(a) FROM table; +** SELECT max(a) FROM table; +** +** The query may have only a single table in its FROM argument. There +** can be no GROUP BY or HAVING or WHERE clauses. The result set must +** be the min() or max() of a single column of the table. The column +** in the min() or max() function must be indexed. +** +** The parameters to this routine are the same as for sqlite3Select(). +** See the header comment on that routine for additional information. +*/ +static int simpleMinMaxQuery(Parse *pParse, Select *p, int eDest, int iParm){ + Expr *pExpr; + int iCol; + Table *pTab; + Index *pIdx; + int base; + Vdbe *v; + int seekOp; + ExprList *pEList, *pList, eList; + struct ExprList_item eListItem; + SrcList *pSrc; + int brk; + int iDb; + + /* Check to see if this query is a simple min() or max() query. Return + ** zero if it is not. + */ + if( p->pGroupBy || p->pHaving || p->pWhere ) return 0; + pSrc = p->pSrc; + if( pSrc->nSrc!=1 ) return 0; + pEList = p->pEList; + if( pEList->nExpr!=1 ) return 0; + pExpr = pEList->a[0].pExpr; + if( pExpr->op!=TK_AGG_FUNCTION ) return 0; + pList = pExpr->pList; + if( pList==0 || pList->nExpr!=1 ) return 0; + if( pExpr->token.n!=3 ) return 0; + if( sqlite3StrNICmp((char*)pExpr->token.z,"min",3)==0 ){ + seekOp = OP_Rewind; + }else if( sqlite3StrNICmp((char*)pExpr->token.z,"max",3)==0 ){ + seekOp = OP_Last; + }else{ + return 0; + } + pExpr = pList->a[0].pExpr; + if( pExpr->op!=TK_COLUMN ) return 0; + iCol = pExpr->iColumn; + pTab = pSrc->a[0].pTab; + + + /* If we get to here, it means the query is of the correct form. + ** Check to make sure we have an index and make pIdx point to the + ** appropriate index. If the min() or max() is on an INTEGER PRIMARY + ** key column, no index is necessary so set pIdx to NULL. If no + ** usable index is found, return 0. + */ + if( iCol<0 ){ + pIdx = 0; + }else{ + CollSeq *pColl = sqlite3ExprCollSeq(pParse, pExpr); + for(pIdx=pTab->pIndex; pIdx; pIdx=pIdx->pNext){ + assert( pIdx->nColumn>=1 ); + if( pIdx->aiColumn[0]==iCol && + 0==sqlite3StrICmp(pIdx->azColl[0], pColl->zName) ){ + break; + } + } + if( pIdx==0 ) return 0; + } + + /* Identify column types if we will be using the callback. This + ** step is skipped if the output is going to a table or a memory cell. + ** The column names have already been generated in the calling function. + */ + v = sqlite3GetVdbe(pParse); + if( v==0 ) return 0; + + /* If the output is destined for a temporary table, open that table. + */ + if( eDest==SRT_VirtualTab ){ + sqlite3VdbeAddOp(v, OP_OpenVirtual, iParm, 1); + } + + /* Generating code to find the min or the max. Basically all we have + ** to do is find the first or the last entry in the chosen index. If + ** the min() or max() is on the INTEGER PRIMARY KEY, then find the first + ** or last entry in the main table. + */ + iDb = sqlite3SchemaToIndex(pParse->db, pTab->pSchema); + assert( iDb>=0 || pTab->isTransient ); + sqlite3CodeVerifySchema(pParse, iDb); + sqlite3TableLock(pParse, iDb, pTab->tnum, 0, pTab->zName); + base = pSrc->a[0].iCursor; + brk = sqlite3VdbeMakeLabel(v); + computeLimitRegisters(pParse, p, brk); + if( pSrc->a[0].pSelect==0 ){ + sqlite3OpenTable(pParse, base, iDb, pTab, OP_OpenRead); + } + if( pIdx==0 ){ + sqlite3VdbeAddOp(v, seekOp, base, 0); + }else{ + /* Even though the cursor used to open the index here is closed + ** as soon as a single value has been read from it, allocate it + ** using (pParse->nTab++) to prevent the cursor id from being + ** reused. This is important for statements of the form + ** "INSERT INTO x SELECT max() FROM x". + */ + int iIdx; + KeyInfo *pKey = sqlite3IndexKeyinfo(pParse, pIdx); + iIdx = pParse->nTab++; + assert( pIdx->pSchema==pTab->pSchema ); + sqlite3VdbeAddOp(v, OP_Integer, iDb, 0); + sqlite3VdbeOp3(v, OP_OpenRead, iIdx, pIdx->tnum, + (char*)pKey, P3_KEYINFO_HANDOFF); + if( seekOp==OP_Rewind ){ + sqlite3VdbeAddOp(v, OP_Null, 0, 0); + sqlite3VdbeAddOp(v, OP_MakeRecord, 1, 0); + seekOp = OP_MoveGt; + } + sqlite3VdbeAddOp(v, seekOp, iIdx, 0); + sqlite3VdbeAddOp(v, OP_IdxRowid, iIdx, 0); + sqlite3VdbeAddOp(v, OP_Close, iIdx, 0); + sqlite3VdbeAddOp(v, OP_MoveGe, base, 0); + } + eList.nExpr = 1; + memset(&eListItem, 0, sizeof(eListItem)); + eList.a = &eListItem; + eList.a[0].pExpr = pExpr; + selectInnerLoop(pParse, p, &eList, 0, 0, 0, -1, eDest, iParm, brk, brk, 0); + sqlite3VdbeResolveLabel(v, brk); + sqlite3VdbeAddOp(v, OP_Close, base, 0); + + return 1; +} + +/* +** Analyze and ORDER BY or GROUP BY clause in a SELECT statement. Return +** the number of errors seen. +** +** An ORDER BY or GROUP BY is a list of expressions. If any expression +** is an integer constant, then that expression is replaced by the +** corresponding entry in the result set. +*/ +static int processOrderGroupBy( + NameContext *pNC, /* Name context of the SELECT statement. */ + ExprList *pOrderBy, /* The ORDER BY or GROUP BY clause to be processed */ + const char *zType /* Either "ORDER" or "GROUP", as appropriate */ +){ + int i; + ExprList *pEList = pNC->pEList; /* The result set of the SELECT */ + Parse *pParse = pNC->pParse; /* The result set of the SELECT */ + assert( pEList ); + + if( pOrderBy==0 ) return 0; + for(i=0; inExpr; i++){ + int iCol; + Expr *pE = pOrderBy->a[i].pExpr; + if( sqlite3ExprIsInteger(pE, &iCol) ){ + if( iCol>0 && iCol<=pEList->nExpr ){ + sqlite3ExprDelete(pE); + pE = pOrderBy->a[i].pExpr = sqlite3ExprDup(pEList->a[iCol-1].pExpr); + }else{ + sqlite3ErrorMsg(pParse, + "%s BY column number %d out of range - should be " + "between 1 and %d", zType, iCol, pEList->nExpr); + return 1; + } + } + if( sqlite3ExprResolveNames(pNC, pE) ){ + return 1; + } + if( sqlite3ExprIsConstant(pE) ){ + sqlite3ErrorMsg(pParse, + "%s BY terms must not be non-integer constants", zType); + return 1; + } + } + return 0; +} + +/* +** This routine resolves any names used in the result set of the +** supplied SELECT statement. If the SELECT statement being resolved +** is a sub-select, then pOuterNC is a pointer to the NameContext +** of the parent SELECT. +*/ +int sqlite3SelectResolve( + Parse *pParse, /* The parser context */ + Select *p, /* The SELECT statement being coded. */ + NameContext *pOuterNC /* The outer name context. May be NULL. */ +){ + ExprList *pEList; /* Result set. */ + int i; /* For-loop variable used in multiple places */ + NameContext sNC; /* Local name-context */ + ExprList *pGroupBy; /* The group by clause */ + + /* If this routine has run before, return immediately. */ + if( p->isResolved ){ + assert( !pOuterNC ); + return SQLITE_OK; + } + p->isResolved = 1; + + /* If there have already been errors, do nothing. */ + if( pParse->nErr>0 ){ + return SQLITE_ERROR; + } + + /* Prepare the select statement. This call will allocate all cursors + ** required to handle the tables and subqueries in the FROM clause. + */ + if( prepSelectStmt(pParse, p) ){ + return SQLITE_ERROR; + } + + /* Resolve the expressions in the LIMIT and OFFSET clauses. These + ** are not allowed to refer to any names, so pass an empty NameContext. + */ + memset(&sNC, 0, sizeof(sNC)); + sNC.pParse = pParse; + if( sqlite3ExprResolveNames(&sNC, p->pLimit) || + sqlite3ExprResolveNames(&sNC, p->pOffset) ){ + return SQLITE_ERROR; + } + + /* Set up the local name-context to pass to ExprResolveNames() to + ** resolve the expression-list. + */ + sNC.allowAgg = 1; + sNC.pSrcList = p->pSrc; + sNC.pNext = pOuterNC; + + /* Resolve names in the result set. */ + pEList = p->pEList; + if( !pEList ) return SQLITE_ERROR; + for(i=0; inExpr; i++){ + Expr *pX = pEList->a[i].pExpr; + if( sqlite3ExprResolveNames(&sNC, pX) ){ + return SQLITE_ERROR; + } + } + + /* If there are no aggregate functions in the result-set, and no GROUP BY + ** expression, do not allow aggregates in any of the other expressions. + */ + assert( !p->isAgg ); + pGroupBy = p->pGroupBy; + if( pGroupBy || sNC.hasAgg ){ + p->isAgg = 1; + }else{ + sNC.allowAgg = 0; + } + + /* If a HAVING clause is present, then there must be a GROUP BY clause. + */ + if( p->pHaving && !pGroupBy ){ + sqlite3ErrorMsg(pParse, "a GROUP BY clause is required before HAVING"); + return SQLITE_ERROR; + } + + /* Add the expression list to the name-context before parsing the + ** other expressions in the SELECT statement. This is so that + ** expressions in the WHERE clause (etc.) can refer to expressions by + ** aliases in the result set. + ** + ** Minor point: If this is the case, then the expression will be + ** re-evaluated for each reference to it. + */ + sNC.pEList = p->pEList; + if( sqlite3ExprResolveNames(&sNC, p->pWhere) || + sqlite3ExprResolveNames(&sNC, p->pHaving) || + processOrderGroupBy(&sNC, p->pOrderBy, "ORDER") || + processOrderGroupBy(&sNC, pGroupBy, "GROUP") + ){ + return SQLITE_ERROR; + } + + /* Make sure the GROUP BY clause does not contain aggregate functions. + */ + if( pGroupBy ){ + struct ExprList_item *pItem; + + for(i=0, pItem=pGroupBy->a; inExpr; i++, pItem++){ + if( ExprHasProperty(pItem->pExpr, EP_Agg) ){ + sqlite3ErrorMsg(pParse, "aggregate functions are not allowed in " + "the GROUP BY clause"); + return SQLITE_ERROR; + } + } + } + + return SQLITE_OK; +} + +/* +** Reset the aggregate accumulator. +** +** The aggregate accumulator is a set of memory cells that hold +** intermediate results while calculating an aggregate. This +** routine simply stores NULLs in all of those memory cells. +*/ +static void resetAccumulator(Parse *pParse, AggInfo *pAggInfo){ + Vdbe *v = pParse->pVdbe; + int i; + struct AggInfo_func *pFunc; + if( pAggInfo->nFunc+pAggInfo->nColumn==0 ){ + return; + } + for(i=0; inColumn; i++){ + sqlite3VdbeAddOp(v, OP_MemNull, pAggInfo->aCol[i].iMem, 0); + } + for(pFunc=pAggInfo->aFunc, i=0; inFunc; i++, pFunc++){ + sqlite3VdbeAddOp(v, OP_MemNull, pFunc->iMem, 0); + if( pFunc->iDistinct>=0 ){ + Expr *pE = pFunc->pExpr; + if( pE->pList==0 || pE->pList->nExpr!=1 ){ + sqlite3ErrorMsg(pParse, "DISTINCT in aggregate must be followed " + "by an expression"); + pFunc->iDistinct = -1; + }else{ + KeyInfo *pKeyInfo = keyInfoFromExprList(pParse, pE->pList); + sqlite3VdbeOp3(v, OP_OpenVirtual, pFunc->iDistinct, 0, + (char*)pKeyInfo, P3_KEYINFO_HANDOFF); + } + } + } +} + +/* +** Invoke the OP_AggFinalize opcode for every aggregate function +** in the AggInfo structure. +*/ +static void finalizeAggFunctions(Parse *pParse, AggInfo *pAggInfo){ + Vdbe *v = pParse->pVdbe; + int i; + struct AggInfo_func *pF; + for(i=0, pF=pAggInfo->aFunc; inFunc; i++, pF++){ + ExprList *pList = pF->pExpr->pList; + sqlite3VdbeOp3(v, OP_AggFinal, pF->iMem, pList ? pList->nExpr : 0, + (void*)pF->pFunc, P3_FUNCDEF); + } +} + +/* +** Update the accumulator memory cells for an aggregate based on +** the current cursor position. +*/ +static void updateAccumulator(Parse *pParse, AggInfo *pAggInfo){ + Vdbe *v = pParse->pVdbe; + int i; + struct AggInfo_func *pF; + struct AggInfo_col *pC; + + pAggInfo->directMode = 1; + for(i=0, pF=pAggInfo->aFunc; inFunc; i++, pF++){ + int nArg; + int addrNext = 0; + ExprList *pList = pF->pExpr->pList; + if( pList ){ + nArg = pList->nExpr; + sqlite3ExprCodeExprList(pParse, pList); + }else{ + nArg = 0; + } + if( pF->iDistinct>=0 ){ + addrNext = sqlite3VdbeMakeLabel(v); + assert( nArg==1 ); + codeDistinct(v, pF->iDistinct, addrNext, 1, 2); + } + if( pF->pFunc->needCollSeq ){ + CollSeq *pColl = 0; + struct ExprList_item *pItem; + int j; + for(j=0, pItem=pList->a; !pColl && jnExpr; j++, pItem++){ + pColl = sqlite3ExprCollSeq(pParse, pItem->pExpr); + } + if( !pColl ){ + pColl = pParse->db->pDfltColl; + } + sqlite3VdbeOp3(v, OP_CollSeq, 0, 0, (char *)pColl, P3_COLLSEQ); + } + sqlite3VdbeOp3(v, OP_AggStep, pF->iMem, nArg, (void*)pF->pFunc, P3_FUNCDEF); + if( addrNext ){ + sqlite3VdbeResolveLabel(v, addrNext); + } + } + for(i=0, pC=pAggInfo->aCol; inAccumulator; i++, pC++){ + sqlite3ExprCode(pParse, pC->pExpr); + sqlite3VdbeAddOp(v, OP_MemStore, pC->iMem, 1); + } + pAggInfo->directMode = 0; +} + + +/* +** Generate code for the given SELECT statement. +** +** The results are distributed in various ways depending on the +** value of eDest and iParm. +** +** eDest Value Result +** ------------ ------------------------------------------- +** SRT_Callback Invoke the callback for each row of the result. +** +** SRT_Mem Store first result in memory cell iParm +** +** SRT_Set Store results as keys of table iParm. +** +** SRT_Union Store results as a key in a temporary table iParm +** +** SRT_Except Remove results from the temporary table iParm. +** +** SRT_Table Store results in temporary table iParm +** +** The table above is incomplete. Additional eDist value have be added +** since this comment was written. See the selectInnerLoop() function for +** a complete listing of the allowed values of eDest and their meanings. +** +** This routine returns the number of errors. If any errors are +** encountered, then an appropriate error message is left in +** pParse->zErrMsg. +** +** This routine does NOT free the Select structure passed in. The +** calling function needs to do that. +** +** The pParent, parentTab, and *pParentAgg fields are filled in if this +** SELECT is a subquery. This routine may try to combine this SELECT +** with its parent to form a single flat query. In so doing, it might +** change the parent query from a non-aggregate to an aggregate query. +** For that reason, the pParentAgg flag is passed as a pointer, so it +** can be changed. +** +** Example 1: The meaning of the pParent parameter. +** +** SELECT * FROM t1 JOIN (SELECT x, count(*) FROM t2) JOIN t3; +** \ \_______ subquery _______/ / +** \ / +** \____________________ outer query ___________________/ +** +** This routine is called for the outer query first. For that call, +** pParent will be NULL. During the processing of the outer query, this +** routine is called recursively to handle the subquery. For the recursive +** call, pParent will point to the outer query. Because the subquery is +** the second element in a three-way join, the parentTab parameter will +** be 1 (the 2nd value of a 0-indexed array.) +*/ +int sqlite3Select( + Parse *pParse, /* The parser context */ + Select *p, /* The SELECT statement being coded. */ + int eDest, /* How to dispose of the results */ + int iParm, /* A parameter used by the eDest disposal method */ + Select *pParent, /* Another SELECT for which this is a sub-query */ + int parentTab, /* Index in pParent->pSrc of this query */ + int *pParentAgg, /* True if pParent uses aggregate functions */ + char *aff /* If eDest is SRT_Union, the affinity string */ +){ + int i, j; /* Loop counters */ + WhereInfo *pWInfo; /* Return from sqlite3WhereBegin() */ + Vdbe *v; /* The virtual machine under construction */ + int isAgg; /* True for select lists like "count(*)" */ + ExprList *pEList; /* List of columns to extract. */ + SrcList *pTabList; /* List of tables to select from */ + Expr *pWhere; /* The WHERE clause. May be NULL */ + ExprList *pOrderBy; /* The ORDER BY clause. May be NULL */ + ExprList *pGroupBy; /* The GROUP BY clause. May be NULL */ + Expr *pHaving; /* The HAVING clause. May be NULL */ + int isDistinct; /* True if the DISTINCT keyword is present */ + int distinct; /* Table to use for the distinct set */ + int rc = 1; /* Value to return from this function */ + int addrSortIndex; /* Address of an OP_OpenVirtual instruction */ + AggInfo sAggInfo; /* Information used by aggregate queries */ + int iEnd; /* Address of the end of the query */ + + if( p==0 || sqlite3MallocFailed() || pParse->nErr ){ + return 1; + } + if( sqlite3AuthCheck(pParse, SQLITE_SELECT, 0, 0, 0) ) return 1; + memset(&sAggInfo, 0, sizeof(sAggInfo)); + +#ifndef SQLITE_OMIT_COMPOUND_SELECT + /* If there is are a sequence of queries, do the earlier ones first. + */ + if( p->pPrior ){ + if( p->pRightmost==0 ){ + Select *pLoop; + for(pLoop=p; pLoop; pLoop=pLoop->pPrior){ + pLoop->pRightmost = p; + } + } + return multiSelect(pParse, p, eDest, iParm, aff); + } +#endif + + pOrderBy = p->pOrderBy; + if( IgnorableOrderby(eDest) ){ + p->pOrderBy = 0; + } + if( sqlite3SelectResolve(pParse, p, 0) ){ + goto select_end; + } + p->pOrderBy = pOrderBy; + + /* Make local copies of the parameters for this query. + */ + pTabList = p->pSrc; + pWhere = p->pWhere; + pGroupBy = p->pGroupBy; + pHaving = p->pHaving; + isAgg = p->isAgg; + isDistinct = p->isDistinct; + pEList = p->pEList; + if( pEList==0 ) goto select_end; + + /* + ** Do not even attempt to generate any code if we have already seen + ** errors before this routine starts. + */ + if( pParse->nErr>0 ) goto select_end; + + /* If writing to memory or generating a set + ** only a single column may be output. + */ +#ifndef SQLITE_OMIT_SUBQUERY + if( (eDest==SRT_Mem || eDest==SRT_Set) && pEList->nExpr>1 ){ + sqlite3ErrorMsg(pParse, "only a single result allowed for " + "a SELECT that is part of an expression"); + goto select_end; + } +#endif + + /* ORDER BY is ignored for some destinations. + */ + if( IgnorableOrderby(eDest) ){ + pOrderBy = 0; + } + + /* Begin generating code. + */ + v = sqlite3GetVdbe(pParse); + if( v==0 ) goto select_end; + + /* Generate code for all sub-queries in the FROM clause + */ +#if !defined(SQLITE_OMIT_SUBQUERY) || !defined(SQLITE_OMIT_VIEW) + for(i=0; inSrc; i++){ + const char *zSavedAuthContext = 0; + int needRestoreContext; + struct SrcList_item *pItem = &pTabList->a[i]; + + if( pItem->pSelect==0 || pItem->isPopulated ) continue; + if( pItem->zName!=0 ){ + zSavedAuthContext = pParse->zAuthContext; + pParse->zAuthContext = pItem->zName; + needRestoreContext = 1; + }else{ + needRestoreContext = 0; + } + sqlite3Select(pParse, pItem->pSelect, SRT_VirtualTab, + pItem->iCursor, p, i, &isAgg, 0); + if( needRestoreContext ){ + pParse->zAuthContext = zSavedAuthContext; + } + pTabList = p->pSrc; + pWhere = p->pWhere; + if( !IgnorableOrderby(eDest) ){ + pOrderBy = p->pOrderBy; + } + pGroupBy = p->pGroupBy; + pHaving = p->pHaving; + isDistinct = p->isDistinct; + } +#endif + + /* Check for the special case of a min() or max() function by itself + ** in the result set. + */ + if( simpleMinMaxQuery(pParse, p, eDest, iParm) ){ + rc = 0; + goto select_end; + } + + /* Check to see if this is a subquery that can be "flattened" into its parent. + ** If flattening is a possiblity, do so and return immediately. + */ +#ifndef SQLITE_OMIT_VIEW + if( pParent && pParentAgg && + flattenSubquery(pParse, pParent, parentTab, *pParentAgg, isAgg) ){ + if( isAgg ) *pParentAgg = 1; + goto select_end; + } +#endif + + /* If there is an ORDER BY clause, resolve any collation sequences + ** names that have been explicitly specified and create a sorting index. + ** + ** This sorting index might end up being unused if the data can be + ** extracted in pre-sorted order. If that is the case, then the + ** OP_OpenVirtual instruction will be changed to an OP_Noop once + ** we figure out that the sorting index is not needed. The addrSortIndex + ** variable is used to facilitate that change. + */ + if( pOrderBy ){ + struct ExprList_item *pTerm; + KeyInfo *pKeyInfo; + for(i=0, pTerm=pOrderBy->a; inExpr; i++, pTerm++){ + if( pTerm->zName ){ + pTerm->pExpr->pColl = sqlite3LocateCollSeq(pParse, pTerm->zName, -1); + } + } + if( pParse->nErr ){ + goto select_end; + } + pKeyInfo = keyInfoFromExprList(pParse, pOrderBy); + pOrderBy->iECursor = pParse->nTab++; + p->addrOpenVirt[2] = addrSortIndex = + sqlite3VdbeOp3(v, OP_OpenVirtual, pOrderBy->iECursor, pOrderBy->nExpr+2, + (char*)pKeyInfo, P3_KEYINFO_HANDOFF); + }else{ + addrSortIndex = -1; + } + + /* Set the limiter. + */ + iEnd = sqlite3VdbeMakeLabel(v); + computeLimitRegisters(pParse, p, iEnd); + + /* If the output is destined for a temporary table, open that table. + */ + if( eDest==SRT_VirtualTab ){ + sqlite3VdbeAddOp(v, OP_OpenVirtual, iParm, pEList->nExpr); + } + + /* Open a virtual index to use for the distinct set. + */ + if( isDistinct ){ + KeyInfo *pKeyInfo; + distinct = pParse->nTab++; + pKeyInfo = keyInfoFromExprList(pParse, p->pEList); + sqlite3VdbeOp3(v, OP_OpenVirtual, distinct, 0, + (char*)pKeyInfo, P3_KEYINFO_HANDOFF); + }else{ + distinct = -1; + } + + /* Aggregate and non-aggregate queries are handled differently */ + if( !isAgg && pGroupBy==0 ){ + /* This case is for non-aggregate queries + ** Begin the database scan + */ + pWInfo = sqlite3WhereBegin(pParse, pTabList, pWhere, &pOrderBy); + if( pWInfo==0 ) goto select_end; + + /* If sorting index that was created by a prior OP_OpenVirtual + ** instruction ended up not being needed, then change the OP_OpenVirtual + ** into an OP_Noop. + */ + if( addrSortIndex>=0 && pOrderBy==0 ){ + uncreateSortingIndex(pParse, addrSortIndex); + p->addrOpenVirt[2] = -1; + } + + /* Use the standard inner loop + */ + if( selectInnerLoop(pParse, p, pEList, 0, 0, pOrderBy, distinct, eDest, + iParm, pWInfo->iContinue, pWInfo->iBreak, aff) ){ + goto select_end; + } + + /* End the database scan loop. + */ + sqlite3WhereEnd(pWInfo); + }else{ + /* This is the processing for aggregate queries */ + NameContext sNC; /* Name context for processing aggregate information */ + int iAMem; /* First Mem address for storing current GROUP BY */ + int iBMem; /* First Mem address for previous GROUP BY */ + int iUseFlag; /* Mem address holding flag indicating that at least + ** one row of the input to the aggregator has been + ** processed */ + int iAbortFlag; /* Mem address which causes query abort if positive */ + int groupBySort; /* Rows come from source in GROUP BY order */ + + + /* The following variables hold addresses or labels for parts of the + ** virtual machine program we are putting together */ + int addrOutputRow; /* Start of subroutine that outputs a result row */ + int addrSetAbort; /* Set the abort flag and return */ + int addrInitializeLoop; /* Start of code that initializes the input loop */ + int addrTopOfLoop; /* Top of the input loop */ + int addrGroupByChange; /* Code that runs when any GROUP BY term changes */ + int addrProcessRow; /* Code to process a single input row */ + int addrEnd; /* End of all processing */ + int addrSortingIdx; /* The OP_OpenVirtual for the sorting index */ + int addrReset; /* Subroutine for resetting the accumulator */ + + addrEnd = sqlite3VdbeMakeLabel(v); + + /* Convert TK_COLUMN nodes into TK_AGG_COLUMN and make entries in + ** sAggInfo for all TK_AGG_FUNCTION nodes in expressions of the + ** SELECT statement. + */ + memset(&sNC, 0, sizeof(sNC)); + sNC.pParse = pParse; + sNC.pSrcList = pTabList; + sNC.pAggInfo = &sAggInfo; + sAggInfo.nSortingColumn = pGroupBy ? pGroupBy->nExpr+1 : 0; + sAggInfo.pGroupBy = pGroupBy; + if( sqlite3ExprAnalyzeAggList(&sNC, pEList) ){ + goto select_end; + } + if( sqlite3ExprAnalyzeAggList(&sNC, pOrderBy) ){ + goto select_end; + } + if( pHaving && sqlite3ExprAnalyzeAggregates(&sNC, pHaving) ){ + goto select_end; + } + sAggInfo.nAccumulator = sAggInfo.nColumn; + for(i=0; ipList) ){ + goto select_end; + } + } + if( sqlite3MallocFailed() ) goto select_end; + + /* Processing for aggregates with GROUP BY is very different and + ** much more complex tha aggregates without a GROUP BY. + */ + if( pGroupBy ){ + KeyInfo *pKeyInfo; /* Keying information for the group by clause */ + + /* Create labels that we will be needing + */ + + addrInitializeLoop = sqlite3VdbeMakeLabel(v); + addrGroupByChange = sqlite3VdbeMakeLabel(v); + addrProcessRow = sqlite3VdbeMakeLabel(v); + + /* If there is a GROUP BY clause we might need a sorting index to + ** implement it. Allocate that sorting index now. If it turns out + ** that we do not need it after all, the OpenVirtual instruction + ** will be converted into a Noop. + */ + sAggInfo.sortingIdx = pParse->nTab++; + pKeyInfo = keyInfoFromExprList(pParse, pGroupBy); + addrSortingIdx = + sqlite3VdbeOp3(v, OP_OpenVirtual, sAggInfo.sortingIdx, + sAggInfo.nSortingColumn, + (char*)pKeyInfo, P3_KEYINFO_HANDOFF); + + /* Initialize memory locations used by GROUP BY aggregate processing + */ + iUseFlag = pParse->nMem++; + iAbortFlag = pParse->nMem++; + iAMem = pParse->nMem; + pParse->nMem += pGroupBy->nExpr; + iBMem = pParse->nMem; + pParse->nMem += pGroupBy->nExpr; + sqlite3VdbeAddOp(v, OP_MemInt, 0, iAbortFlag); + VdbeComment((v, "# clear abort flag")); + sqlite3VdbeAddOp(v, OP_MemInt, 0, iUseFlag); + VdbeComment((v, "# indicate accumulator empty")); + sqlite3VdbeAddOp(v, OP_Goto, 0, addrInitializeLoop); + + /* Generate a subroutine that outputs a single row of the result + ** set. This subroutine first looks at the iUseFlag. If iUseFlag + ** is less than or equal to zero, the subroutine is a no-op. If + ** the processing calls for the query to abort, this subroutine + ** increments the iAbortFlag memory location before returning in + ** order to signal the caller to abort. + */ + addrSetAbort = sqlite3VdbeCurrentAddr(v); + sqlite3VdbeAddOp(v, OP_MemInt, 1, iAbortFlag); + VdbeComment((v, "# set abort flag")); + sqlite3VdbeAddOp(v, OP_Return, 0, 0); + addrOutputRow = sqlite3VdbeCurrentAddr(v); + sqlite3VdbeAddOp(v, OP_IfMemPos, iUseFlag, addrOutputRow+2); + VdbeComment((v, "# Groupby result generator entry point")); + sqlite3VdbeAddOp(v, OP_Return, 0, 0); + finalizeAggFunctions(pParse, &sAggInfo); + if( pHaving ){ + sqlite3ExprIfFalse(pParse, pHaving, addrOutputRow+1, 1); + } + rc = selectInnerLoop(pParse, p, p->pEList, 0, 0, pOrderBy, + distinct, eDest, iParm, + addrOutputRow+1, addrSetAbort, aff); + if( rc ){ + goto select_end; + } + sqlite3VdbeAddOp(v, OP_Return, 0, 0); + VdbeComment((v, "# end groupby result generator")); + + /* Generate a subroutine that will reset the group-by accumulator + */ + addrReset = sqlite3VdbeCurrentAddr(v); + resetAccumulator(pParse, &sAggInfo); + sqlite3VdbeAddOp(v, OP_Return, 0, 0); + + /* Begin a loop that will extract all source rows in GROUP BY order. + ** This might involve two separate loops with an OP_Sort in between, or + ** it might be a single loop that uses an index to extract information + ** in the right order to begin with. + */ + sqlite3VdbeResolveLabel(v, addrInitializeLoop); + sqlite3VdbeAddOp(v, OP_Gosub, 0, addrReset); + pWInfo = sqlite3WhereBegin(pParse, pTabList, pWhere, &pGroupBy); + if( pWInfo==0 ) goto select_end; + if( pGroupBy==0 ){ + /* The optimizer is able to deliver rows in group by order so + ** we do not have to sort. The OP_OpenVirtual table will be + ** cancelled later because we still need to use the pKeyInfo + */ + pGroupBy = p->pGroupBy; + groupBySort = 0; + }else{ + /* Rows are coming out in undetermined order. We have to push + ** each row into a sorting index, terminate the first loop, + ** then loop over the sorting index in order to get the output + ** in sorted order + */ + groupBySort = 1; + sqlite3ExprCodeExprList(pParse, pGroupBy); + sqlite3VdbeAddOp(v, OP_Sequence, sAggInfo.sortingIdx, 0); + j = pGroupBy->nExpr+1; + for(i=0; iiSorterColumniColumn<0 ){ + sqlite3VdbeAddOp(v, OP_Rowid, pCol->iTable, 0); + }else{ + sqlite3VdbeAddOp(v, OP_Column, pCol->iTable, pCol->iColumn); + } + j++; + } + sqlite3VdbeAddOp(v, OP_MakeRecord, j, 0); + sqlite3VdbeAddOp(v, OP_IdxInsert, sAggInfo.sortingIdx, 0); + sqlite3WhereEnd(pWInfo); + sqlite3VdbeAddOp(v, OP_Sort, sAggInfo.sortingIdx, addrEnd); + VdbeComment((v, "# GROUP BY sort")); + sAggInfo.useSortingIdx = 1; + } + + /* Evaluate the current GROUP BY terms and store in b0, b1, b2... + ** (b0 is memory location iBMem+0, b1 is iBMem+1, and so forth) + ** Then compare the current GROUP BY terms against the GROUP BY terms + ** from the previous row currently stored in a0, a1, a2... + */ + addrTopOfLoop = sqlite3VdbeCurrentAddr(v); + for(j=0; jnExpr; j++){ + if( groupBySort ){ + sqlite3VdbeAddOp(v, OP_Column, sAggInfo.sortingIdx, j); + }else{ + sAggInfo.directMode = 1; + sqlite3ExprCode(pParse, pGroupBy->a[j].pExpr); + } + sqlite3VdbeAddOp(v, OP_MemStore, iBMem+j, jnExpr-1); + } + for(j=pGroupBy->nExpr-1; j>=0; j--){ + if( jnExpr-1 ){ + sqlite3VdbeAddOp(v, OP_MemLoad, iBMem+j, 0); + } + sqlite3VdbeAddOp(v, OP_MemLoad, iAMem+j, 0); + if( j==0 ){ + sqlite3VdbeAddOp(v, OP_Eq, 0x200, addrProcessRow); + }else{ + sqlite3VdbeAddOp(v, OP_Ne, 0x200, addrGroupByChange); + } + sqlite3VdbeChangeP3(v, -1, (void*)pKeyInfo->aColl[j], P3_COLLSEQ); + } + + /* Generate code that runs whenever the GROUP BY changes. + ** Change in the GROUP BY are detected by the previous code + ** block. If there were no changes, this block is skipped. + ** + ** This code copies current group by terms in b0,b1,b2,... + ** over to a0,a1,a2. It then calls the output subroutine + ** and resets the aggregate accumulator registers in preparation + ** for the next GROUP BY batch. + */ + sqlite3VdbeResolveLabel(v, addrGroupByChange); + for(j=0; jnExpr; j++){ + sqlite3VdbeAddOp(v, OP_MemMove, iAMem+j, iBMem+j); + } + sqlite3VdbeAddOp(v, OP_Gosub, 0, addrOutputRow); + VdbeComment((v, "# output one row")); + sqlite3VdbeAddOp(v, OP_IfMemPos, iAbortFlag, addrEnd); + VdbeComment((v, "# check abort flag")); + sqlite3VdbeAddOp(v, OP_Gosub, 0, addrReset); + VdbeComment((v, "# reset accumulator")); + + /* Update the aggregate accumulators based on the content of + ** the current row + */ + sqlite3VdbeResolveLabel(v, addrProcessRow); + updateAccumulator(pParse, &sAggInfo); + sqlite3VdbeAddOp(v, OP_MemInt, 1, iUseFlag); + VdbeComment((v, "# indicate data in accumulator")); + + /* End of the loop + */ + if( groupBySort ){ + sqlite3VdbeAddOp(v, OP_Next, sAggInfo.sortingIdx, addrTopOfLoop); + }else{ + sqlite3WhereEnd(pWInfo); + uncreateSortingIndex(pParse, addrSortingIdx); + } + + /* Output the final row of result + */ + sqlite3VdbeAddOp(v, OP_Gosub, 0, addrOutputRow); + VdbeComment((v, "# output final row")); + + } /* endif pGroupBy */ + else { + /* This case runs if the aggregate has no GROUP BY clause. The + ** processing is much simpler since there is only a single row + ** of output. + */ + resetAccumulator(pParse, &sAggInfo); + pWInfo = sqlite3WhereBegin(pParse, pTabList, pWhere, 0); + if( pWInfo==0 ) goto select_end; + updateAccumulator(pParse, &sAggInfo); + sqlite3WhereEnd(pWInfo); + finalizeAggFunctions(pParse, &sAggInfo); + pOrderBy = 0; + if( pHaving ){ + sqlite3ExprIfFalse(pParse, pHaving, addrEnd, 1); + } + selectInnerLoop(pParse, p, p->pEList, 0, 0, 0, -1, + eDest, iParm, addrEnd, addrEnd, aff); + } + sqlite3VdbeResolveLabel(v, addrEnd); + + } /* endif aggregate query */ + + /* If there is an ORDER BY clause, then we need to sort the results + ** and send them to the callback one by one. + */ + if( pOrderBy ){ + generateSortTail(pParse, p, v, pEList->nExpr, eDest, iParm); + } + +#ifndef SQLITE_OMIT_SUBQUERY + /* If this was a subquery, we have now converted the subquery into a + ** temporary table. So set the SrcList_item.isPopulated flag to prevent + ** this subquery from being evaluated again and to force the use of + ** the temporary table. + */ + if( pParent ){ + assert( pParent->pSrc->nSrc>parentTab ); + assert( pParent->pSrc->a[parentTab].pSelect==p ); + pParent->pSrc->a[parentTab].isPopulated = 1; + } +#endif + + /* Jump here to skip this query + */ + sqlite3VdbeResolveLabel(v, iEnd); + + /* The SELECT was successfully coded. Set the return code to 0 + ** to indicate no errors. + */ + rc = 0; + + /* Control jumps to here if an error is encountered above, or upon + ** successful coding of the SELECT. + */ +select_end: + + /* Identify column names if we will be using them in a callback. This + ** step is skipped if the output is going to some other destination. + */ + if( rc==SQLITE_OK && eDest==SRT_Callback ){ + generateColumnNames(pParse, pTabList, pEList); + } + + sqliteFree(sAggInfo.aCol); + sqliteFree(sAggInfo.aFunc); + return rc; +} Added: external/sqlite-source-3.3.4/shell.c ============================================================================== --- (empty file) +++ external/sqlite-source-3.3.4/shell.c Mon Apr 3 07:54:59 2006 @@ -0,0 +1,1789 @@ +/* +** 2001 September 15 +** +** The author disclaims copyright to this source code. In place of +** a legal notice, here is a blessing: +** +** May you do good and not evil. +** May you find forgiveness for yourself and forgive others. +** May you share freely, never taking more than you give. +** +************************************************************************* +** This file contains code to implement the "sqlite" command line +** utility for accessing SQLite databases. +** +** $Id: shell.c,v 1.133 2006/01/31 19:31:44 drh Exp $ +*/ +#include +#include +#include +#include +#include "sqlite3.h" +#include + +#if !defined(_WIN32) && !defined(WIN32) && !defined(__MACOS__) +# include +# include +# include +# include +#endif + +#ifdef __MACOS__ +# include +# include +# include +# include +# include +# include +#endif + +#if defined(HAVE_READLINE) && HAVE_READLINE==1 +# include +# include +#else +# define readline(p) local_getline(p,stdin) +# define add_history(X) +# define read_history(X) +# define write_history(X) +# define stifle_history(X) +#endif + +/* Make sure isatty() has a prototype. +*/ +extern int isatty(); + +/* +** The following is the open SQLite database. We make a pointer +** to this database a static variable so that it can be accessed +** by the SIGINT handler to interrupt database processing. +*/ +static sqlite3 *db = 0; + +/* +** True if an interrupt (Control-C) has been received. +*/ +static int seenInterrupt = 0; + +/* +** This is the name of our program. It is set in main(), used +** in a number of other places, mostly for error messages. +*/ +static char *Argv0; + +/* +** Prompt strings. Initialized in main. Settable with +** .prompt main continue +*/ +static char mainPrompt[20]; /* First line prompt. default: "sqlite> "*/ +static char continuePrompt[20]; /* Continuation prompt. default: " ...> " */ + + +/* +** Determines if a string is a number of not. +*/ +static int isNumber(const char *z, int *realnum){ + if( *z=='-' || *z=='+' ) z++; + if( !isdigit(*z) ){ + return 0; + } + z++; + if( realnum ) *realnum = 0; + while( isdigit(*z) ){ z++; } + if( *z=='.' ){ + z++; + if( !isdigit(*z) ) return 0; + while( isdigit(*z) ){ z++; } + if( realnum ) *realnum = 1; + } + if( *z=='e' || *z=='E' ){ + z++; + if( *z=='+' || *z=='-' ) z++; + if( !isdigit(*z) ) return 0; + while( isdigit(*z) ){ z++; } + if( realnum ) *realnum = 1; + } + return *z==0; +} + +/* +** A global char* and an SQL function to access its current value +** from within an SQL statement. This program used to use the +** sqlite_exec_printf() API to substitue a string into an SQL statement. +** The correct way to do this with sqlite3 is to use the bind API, but +** since the shell is built around the callback paradigm it would be a lot +** of work. Instead just use this hack, which is quite harmless. +*/ +static const char *zShellStatic = 0; +static void shellstaticFunc( + sqlite3_context *context, + int argc, + sqlite3_value **argv +){ + assert( 0==argc ); + assert( zShellStatic ); + sqlite3_result_text(context, zShellStatic, -1, SQLITE_STATIC); +} + + +/* +** This routine reads a line of text from FILE in, stores +** the text in memory obtained from malloc() and returns a pointer +** to the text. NULL is returned at end of file, or if malloc() +** fails. +** +** The interface is like "readline" but no command-line editing +** is done. +*/ +static char *local_getline(char *zPrompt, FILE *in){ + char *zLine; + int nLine; + int n; + int eol; + + if( zPrompt && *zPrompt ){ + printf("%s",zPrompt); + fflush(stdout); + } + nLine = 100; + zLine = malloc( nLine ); + if( zLine==0 ) return 0; + n = 0; + eol = 0; + while( !eol ){ + if( n+100>nLine ){ + nLine = nLine*2 + 100; + zLine = realloc(zLine, nLine); + if( zLine==0 ) return 0; + } + if( fgets(&zLine[n], nLine - n, in)==0 ){ + if( n==0 ){ + free(zLine); + return 0; + } + zLine[n] = 0; + eol = 1; + break; + } + while( zLine[n] ){ n++; } + if( n>0 && zLine[n-1]=='\n' ){ + n--; + zLine[n] = 0; + eol = 1; + } + } + zLine = realloc( zLine, n+1 ); + return zLine; +} + +/* +** Retrieve a single line of input text. "isatty" is true if text +** is coming from a terminal. In that case, we issue a prompt and +** attempt to use "readline" for command-line editing. If "isatty" +** is false, use "local_getline" instead of "readline" and issue no prompt. +** +** zPrior is a string of prior text retrieved. If not the empty +** string, then issue a continuation prompt. +*/ +static char *one_input_line(const char *zPrior, FILE *in){ + char *zPrompt; + char *zResult; + if( in!=0 ){ + return local_getline(0, in); + } + if( zPrior && zPrior[0] ){ + zPrompt = continuePrompt; + }else{ + zPrompt = mainPrompt; + } + zResult = readline(zPrompt); +#if defined(HAVE_READLINE) && HAVE_READLINE==1 + if( zResult ) add_history(zResult); +#endif + return zResult; +} + +struct previous_mode_data { + int valid; /* Is there legit data in here? */ + int mode; + int showHeader; + int colWidth[100]; +}; +/* +** An pointer to an instance of this structure is passed from +** the main program to the callback. This is used to communicate +** state and mode information. +*/ +struct callback_data { + sqlite3 *db; /* The database */ + int echoOn; /* True to echo input commands */ + int cnt; /* Number of records displayed so far */ + FILE *out; /* Write results here */ + int mode; /* An output mode setting */ + int showHeader; /* True to show column names in List or Column mode */ + char *zDestTable; /* Name of destination table when MODE_Insert */ + char separator[20]; /* Separator character for MODE_List */ + int colWidth[100]; /* Requested width of each column when in column mode*/ + int actualWidth[100]; /* Actual width of each column */ + char nullvalue[20]; /* The text to print when a NULL comes back from + ** the database */ + struct previous_mode_data explainPrev; + /* Holds the mode information just before + ** .explain ON */ + char outfile[FILENAME_MAX]; /* Filename for *out */ + const char *zDbFilename; /* name of the database file */ + char *zKey; /* Encryption key */ +}; + +/* +** These are the allowed modes. +*/ +#define MODE_Line 0 /* One column per line. Blank line between records */ +#define MODE_Column 1 /* One record per line in neat columns */ +#define MODE_List 2 /* One record per line with a separator */ +#define MODE_Semi 3 /* Same as MODE_List but append ";" to each line */ +#define MODE_Html 4 /* Generate an XHTML table */ +#define MODE_Insert 5 /* Generate SQL "insert" statements */ +#define MODE_Tcl 6 /* Generate ANSI-C or TCL quoted elements */ +#define MODE_Csv 7 /* Quote strings, numbers are plain */ +#define MODE_NUM_OF 8 /* The number of modes (not a mode itself) */ + +static const char *modeDescr[MODE_NUM_OF] = { + "line", + "column", + "list", + "semi", + "html", + "insert", + "tcl", + "csv", +}; + +/* +** Number of elements in an array +*/ +#define ArraySize(X) (sizeof(X)/sizeof(X[0])) + +/* +** Output the given string as a quoted string using SQL quoting conventions. +*/ +static void output_quoted_string(FILE *out, const char *z){ + int i; + int nSingle = 0; + for(i=0; z[i]; i++){ + if( z[i]=='\'' ) nSingle++; + } + if( nSingle==0 ){ + fprintf(out,"'%s'",z); + }else{ + fprintf(out,"'"); + while( *z ){ + for(i=0; z[i] && z[i]!='\''; i++){} + if( i==0 ){ + fprintf(out,"''"); + z++; + }else if( z[i]=='\'' ){ + fprintf(out,"%.*s''",i,z); + z += i+1; + }else{ + fprintf(out,"%s",z); + break; + } + } + fprintf(out,"'"); + } +} + +/* +** Output the given string as a quoted according to C or TCL quoting rules. +*/ +static void output_c_string(FILE *out, const char *z){ + unsigned int c; + fputc('"', out); + while( (c = *(z++))!=0 ){ + if( c=='\\' ){ + fputc(c, out); + fputc(c, out); + }else if( c=='\t' ){ + fputc('\\', out); + fputc('t', out); + }else if( c=='\n' ){ + fputc('\\', out); + fputc('n', out); + }else if( c=='\r' ){ + fputc('\\', out); + fputc('r', out); + }else if( !isprint(c) ){ + fprintf(out, "\\%03o", c&0xff); + }else{ + fputc(c, out); + } + } + fputc('"', out); +} + +/* +** Output the given string with characters that are special to +** HTML escaped. +*/ +static void output_html_string(FILE *out, const char *z){ + int i; + while( *z ){ + for(i=0; z[i] && z[i]!='<' && z[i]!='&'; i++){} + if( i>0 ){ + fprintf(out,"%.*s",i,z); + } + if( z[i]=='<' ){ + fprintf(out,"<"); + }else if( z[i]=='&' ){ + fprintf(out,"&"); + }else{ + break; + } + z += i + 1; + } +} + +/* +** Output a single term of CSV. Actually, p->separator is used for +** the separator, which may or may not be a comma. p->nullvalue is +** the null value. Strings are quoted using ANSI-C rules. Numbers +** appear outside of quotes. +*/ +static void output_csv(struct callback_data *p, const char *z, int bSep){ + if( z==0 ){ + fprintf(p->out,"%s",p->nullvalue); + }else if( isNumber(z, 0) ){ + fprintf(p->out,"%s",z); + }else{ + output_c_string(p->out, z); + } + if( bSep ){ + fprintf(p->out, p->separator); + } +} + +#ifdef SIGINT +/* +** This routine runs when the user presses Ctrl-C +*/ +static void interrupt_handler(int NotUsed){ + seenInterrupt = 1; + if( db ) sqlite3_interrupt(db); +} +#endif + +/* +** This is the callback routine that the SQLite library +** invokes for each row of a query result. +*/ +static int callback(void *pArg, int nArg, char **azArg, char **azCol){ + int i; + struct callback_data *p = (struct callback_data*)pArg; + switch( p->mode ){ + case MODE_Line: { + int w = 5; + if( azArg==0 ) break; + for(i=0; iw ) w = len; + } + if( p->cnt++>0 ) fprintf(p->out,"\n"); + for(i=0; iout,"%*s = %s\n", w, azCol[i], + azArg[i] ? azArg[i] : p->nullvalue); + } + break; + } + case MODE_Column: { + if( p->cnt++==0 ){ + for(i=0; icolWidth) ){ + w = p->colWidth[i]; + }else{ + w = 0; + } + if( w<=0 ){ + w = strlen(azCol[i] ? azCol[i] : ""); + if( w<10 ) w = 10; + n = strlen(azArg && azArg[i] ? azArg[i] : p->nullvalue); + if( wactualWidth) ){ + p->actualWidth[i] = w; + } + if( p->showHeader ){ + fprintf(p->out,"%-*.*s%s",w,w,azCol[i], i==nArg-1 ? "\n": " "); + } + } + if( p->showHeader ){ + for(i=0; iactualWidth) ){ + w = p->actualWidth[i]; + }else{ + w = 10; + } + fprintf(p->out,"%-*.*s%s",w,w,"-----------------------------------" + "----------------------------------------------------------", + i==nArg-1 ? "\n": " "); + } + } + } + if( azArg==0 ) break; + for(i=0; iactualWidth) ){ + w = p->actualWidth[i]; + }else{ + w = 10; + } + fprintf(p->out,"%-*.*s%s",w,w, + azArg[i] ? azArg[i] : p->nullvalue, i==nArg-1 ? "\n": " "); + } + break; + } + case MODE_Semi: + case MODE_List: { + if( p->cnt++==0 && p->showHeader ){ + for(i=0; iout,"%s%s",azCol[i], i==nArg-1 ? "\n" : p->separator); + } + } + if( azArg==0 ) break; + for(i=0; inullvalue; + fprintf(p->out, "%s", z); + if( iout, "%s", p->separator); + }else if( p->mode==MODE_Semi ){ + fprintf(p->out, ";\n"); + }else{ + fprintf(p->out, "\n"); + } + } + break; + } + case MODE_Html: { + if( p->cnt++==0 && p->showHeader ){ + fprintf(p->out,""); + for(i=0; iout,"",azCol[i]); + } + fprintf(p->out,"\n"); + } + if( azArg==0 ) break; + fprintf(p->out,""); + for(i=0; iout,"\n"); + } + fprintf(p->out,"\n"); + break; + } + case MODE_Tcl: { + if( p->cnt++==0 && p->showHeader ){ + for(i=0; iout,azCol[i]); + fprintf(p->out, "%s", p->separator); + } + fprintf(p->out,"\n"); + } + if( azArg==0 ) break; + for(i=0; iout, azArg[i] ? azArg[i] : p->nullvalue); + fprintf(p->out, "%s", p->separator); + } + fprintf(p->out,"\n"); + break; + } + case MODE_Csv: { + if( p->cnt++==0 && p->showHeader ){ + for(i=0; iout,"\n"); + } + if( azArg==0 ) break; + for(i=0; iout,"\n"); + break; + } + case MODE_Insert: { + if( azArg==0 ) break; + fprintf(p->out,"INSERT INTO %s VALUES(",p->zDestTable); + for(i=0; i0 ? ",": ""; + if( azArg[i]==0 ){ + fprintf(p->out,"%sNULL",zSep); + }else if( isNumber(azArg[i], 0) ){ + fprintf(p->out,"%s%s",zSep, azArg[i]); + }else{ + if( zSep[0] ) fprintf(p->out,"%s",zSep); + output_quoted_string(p->out, azArg[i]); + } + } + fprintf(p->out,");\n"); + break; + } + } + return 0; +} + +/* +** Set the destination table field of the callback_data structure to +** the name of the table given. Escape any quote characters in the +** table name. +*/ +static void set_table_name(struct callback_data *p, const char *zName){ + int i, n; + int needQuote; + char *z; + + if( p->zDestTable ){ + free(p->zDestTable); + p->zDestTable = 0; + } + if( zName==0 ) return; + needQuote = !isalpha((unsigned char)*zName) && *zName!='_'; + for(i=n=0; zName[i]; i++, n++){ + if( !isalnum((unsigned char)zName[i]) && zName[i]!='_' ){ + needQuote = 1; + if( zName[i]=='\'' ) n++; + } + } + if( needQuote ) n += 2; + z = p->zDestTable = malloc( n+1 ); + if( z==0 ){ + fprintf(stderr,"Out of memory!\n"); + exit(1); + } + n = 0; + if( needQuote ) z[n++] = '\''; + for(i=0; zName[i]; i++){ + z[n++] = zName[i]; + if( zName[i]=='\'' ) z[n++] = '\''; + } + if( needQuote ) z[n++] = '\''; + z[n] = 0; +} + +/* zIn is either a pointer to a NULL-terminated string in memory obtained +** from malloc(), or a NULL pointer. The string pointed to by zAppend is +** added to zIn, and the result returned in memory obtained from malloc(). +** zIn, if it was not NULL, is freed. +** +** If the third argument, quote, is not '\0', then it is used as a +** quote character for zAppend. +*/ +static char * appendText(char *zIn, char const *zAppend, char quote){ + int len; + int i; + int nAppend = strlen(zAppend); + int nIn = (zIn?strlen(zIn):0); + + len = nAppend+nIn+1; + if( quote ){ + len += 2; + for(i=0; iout, "DELETE FROM sqlite_sequence;\n"); + }else if( strcmp(zTable, "sqlite_stat1")==0 ){ + fprintf(p->out, "ANALYZE sqlite_master;\n"); + }else if( strncmp(zTable, "sqlite_", 7)==0 ){ + return 0; + }else{ + fprintf(p->out, "%s;\n", zSql); + } + + if( strcmp(zType, "table")==0 ){ + sqlite3_stmt *pTableInfo = 0; + char *zSelect = 0; + char *zTableInfo = 0; + char *zTmp = 0; + + zTableInfo = appendText(zTableInfo, "PRAGMA table_info(", 0); + zTableInfo = appendText(zTableInfo, zTable, '"'); + zTableInfo = appendText(zTableInfo, ");", 0); + + rc = sqlite3_prepare(p->db, zTableInfo, -1, &pTableInfo, 0); + if( zTableInfo ) free(zTableInfo); + if( rc!=SQLITE_OK || !pTableInfo ){ + return 1; + } + + zSelect = appendText(zSelect, "SELECT 'INSERT INTO ' || ", 0); + zTmp = appendText(zTmp, zTable, '"'); + if( zTmp ){ + zSelect = appendText(zSelect, zTmp, '\''); + } + zSelect = appendText(zSelect, " || ' VALUES(' || ", 0); + rc = sqlite3_step(pTableInfo); + while( rc==SQLITE_ROW ){ + const char *zText = (const char *)sqlite3_column_text(pTableInfo, 1); + zSelect = appendText(zSelect, "quote(", 0); + zSelect = appendText(zSelect, zText, '"'); + rc = sqlite3_step(pTableInfo); + if( rc==SQLITE_ROW ){ + zSelect = appendText(zSelect, ") || ', ' || ", 0); + }else{ + zSelect = appendText(zSelect, ") ", 0); + } + } + rc = sqlite3_finalize(pTableInfo); + if( rc!=SQLITE_OK ){ + if( zSelect ) free(zSelect); + return 1; + } + zSelect = appendText(zSelect, "|| ')' FROM ", 0); + zSelect = appendText(zSelect, zTable, '"'); + + rc = run_table_dump_query(p->out, p->db, zSelect); + if( rc==SQLITE_CORRUPT ){ + zSelect = appendText(zSelect, " ORDER BY rowid DESC", 0); + rc = run_table_dump_query(p->out, p->db, zSelect); + } + if( zSelect ) free(zSelect); + if( rc!=SQLITE_OK ){ + return 1; + } + } + return 0; +} + +/* +** Run zQuery. Update dump_callback() as the callback routine. +** If we get a SQLITE_CORRUPT error, rerun the query after appending +** "ORDER BY rowid DESC" to the end. +*/ +static int run_schema_dump_query( + struct callback_data *p, + const char *zQuery, + char **pzErrMsg +){ + int rc; + rc = sqlite3_exec(p->db, zQuery, dump_callback, p, pzErrMsg); + if( rc==SQLITE_CORRUPT ){ + char *zQ2; + int len = strlen(zQuery); + if( pzErrMsg ) sqlite3_free(*pzErrMsg); + zQ2 = malloc( len+100 ); + if( zQ2==0 ) return rc; + sprintf(zQ2, "%s ORDER BY rowid DESC", zQuery); + rc = sqlite3_exec(p->db, zQ2, dump_callback, p, pzErrMsg); + free(zQ2); + } + return rc; +} + +/* +** Text of a help message +*/ +static char zHelp[] = + ".databases List names and files of attached databases\n" + ".dump ?TABLE? ... Dump the database in an SQL text format\n" + ".echo ON|OFF Turn command echo on or off\n" + ".exit Exit this program\n" + ".explain ON|OFF Turn output mode suitable for EXPLAIN on or off.\n" + ".header(s) ON|OFF Turn display of headers on or off\n" + ".help Show this message\n" + ".import FILE TABLE Import data from FILE into TABLE\n" + ".indices TABLE Show names of all indices on TABLE\n" + ".mode MODE ?TABLE? Set output mode where MODE is one of:\n" + " csv Comma-separated values\n" + " column Left-aligned columns. (See .width)\n" + " html HTML
%s
"); + output_html_string(p->out, azArg[i] ? azArg[i] : p->nullvalue); + fprintf(p->out,"
code\n" + " insert SQL insert statements for TABLE\n" + " line One value per line\n" + " list Values delimited by .separator string\n" + " tabs Tab-separated values\n" + " tcl TCL list elements\n" + ".nullvalue STRING Print STRING in place of NULL values\n" + ".output FILENAME Send output to FILENAME\n" + ".output stdout Send output to the screen\n" + ".prompt MAIN CONTINUE Replace the standard prompts\n" + ".quit Exit this program\n" + ".read FILENAME Execute SQL in FILENAME\n" + ".schema ?TABLE? Show the CREATE statements\n" + ".separator STRING Change separator used by output mode and .import\n" + ".show Show the current values for various settings\n" + ".tables ?PATTERN? List names of tables matching a LIKE pattern\n" + ".timeout MS Try opening locked tables for MS milliseconds\n" + ".width NUM NUM ... Set column widths for \"column\" mode\n" +; + +/* Forward reference */ +static void process_input(struct callback_data *p, FILE *in); + +/* +** Make sure the database is open. If it is not, then open it. If +** the database fails to open, print an error message and exit. +*/ +static void open_db(struct callback_data *p){ + if( p->db==0 ){ + sqlite3_open(p->zDbFilename, &p->db); + db = p->db; + sqlite3_create_function(db, "shellstatic", 0, SQLITE_UTF8, 0, + shellstaticFunc, 0, 0); + if( SQLITE_OK!=sqlite3_errcode(db) ){ + fprintf(stderr,"Unable to open database \"%s\": %s\n", + p->zDbFilename, sqlite3_errmsg(db)); + exit(1); + } + } +} + +/* +** Do C-language style dequoting. +** +** \t -> tab +** \n -> newline +** \r -> carriage return +** \NNN -> ascii character NNN in octal +** \\ -> backslash +*/ +static void resolve_backslashes(char *z){ + int i, j, c; + for(i=j=0; (c = z[i])!=0; i++, j++){ + if( c=='\\' ){ + c = z[++i]; + if( c=='n' ){ + c = '\n'; + }else if( c=='t' ){ + c = '\t'; + }else if( c=='r' ){ + c = '\r'; + }else if( c>='0' && c<='7' ){ + c -= '0'; + if( z[i+1]>='0' && z[i+1]<='7' ){ + i++; + c = (c<<3) + z[i] - '0'; + if( z[i+1]>='0' && z[i+1]<='7' ){ + i++; + c = (c<<3) + z[i] - '0'; + } + } + } + } + z[j] = c; + } + z[j] = 0; +} + +/* +** If an input line begins with "." then invoke this routine to +** process that line. +** +** Return 1 to exit and 0 to continue. +*/ +static int do_meta_command(char *zLine, struct callback_data *p){ + int i = 1; + int nArg = 0; + int n, c; + int rc = 0; + char *azArg[50]; + + /* Parse the input line into tokens. + */ + while( zLine[i] && nArg1 && strncmp(azArg[0], "databases", n)==0 ){ + struct callback_data data; + char *zErrMsg = 0; + open_db(p); + memcpy(&data, p, sizeof(data)); + data.showHeader = 1; + data.mode = MODE_Column; + data.colWidth[0] = 3; + data.colWidth[1] = 15; + data.colWidth[2] = 58; + data.cnt = 0; + sqlite3_exec(p->db, "PRAGMA database_list; ", callback, &data, &zErrMsg); + if( zErrMsg ){ + fprintf(stderr,"Error: %s\n", zErrMsg); + sqlite3_free(zErrMsg); + } + }else + + if( c=='d' && strncmp(azArg[0], "dump", n)==0 ){ + char *zErrMsg = 0; + open_db(p); + fprintf(p->out, "BEGIN TRANSACTION;\n"); + if( nArg==1 ){ + run_schema_dump_query(p, + "SELECT name, type, sql FROM sqlite_master " + "WHERE sql NOT NULL AND type=='table'", 0 + ); + run_schema_dump_query(p, + "SELECT name, type, sql FROM sqlite_master " + "WHERE sql NOT NULL AND type!='table' AND type!='meta'", 0 + ); + }else{ + int i; + for(i=1; iout, "COMMIT;\n"); + } + }else + + if( c=='e' && strncmp(azArg[0], "echo", n)==0 && nArg>1 ){ + int j; + char *z = azArg[1]; + int val = atoi(azArg[1]); + for(j=0; z[j]; j++){ + z[j] = tolower((unsigned char)z[j]); + } + if( strcmp(z,"on")==0 ){ + val = 1; + }else if( strcmp(z,"yes")==0 ){ + val = 1; + } + p->echoOn = val; + }else + + if( c=='e' && strncmp(azArg[0], "exit", n)==0 ){ + rc = 1; + }else + + if( c=='e' && strncmp(azArg[0], "explain", n)==0 ){ + int j; + static char zOne[] = "1"; + char *z = nArg>=2 ? azArg[1] : zOne; + int val = atoi(z); + for(j=0; z[j]; j++){ + z[j] = tolower((unsigned char)z[j]); + } + if( strcmp(z,"on")==0 ){ + val = 1; + }else if( strcmp(z,"yes")==0 ){ + val = 1; + } + if(val == 1) { + if(!p->explainPrev.valid) { + p->explainPrev.valid = 1; + p->explainPrev.mode = p->mode; + p->explainPrev.showHeader = p->showHeader; + memcpy(p->explainPrev.colWidth,p->colWidth,sizeof(p->colWidth)); + } + /* We could put this code under the !p->explainValid + ** condition so that it does not execute if we are already in + ** explain mode. However, always executing it allows us an easy + ** was to reset to explain mode in case the user previously + ** did an .explain followed by a .width, .mode or .header + ** command. + */ + p->mode = MODE_Column; + p->showHeader = 1; + memset(p->colWidth,0,ArraySize(p->colWidth)); + p->colWidth[0] = 4; + p->colWidth[1] = 14; + p->colWidth[2] = 10; + p->colWidth[3] = 10; + p->colWidth[4] = 33; + }else if (p->explainPrev.valid) { + p->explainPrev.valid = 0; + p->mode = p->explainPrev.mode; + p->showHeader = p->explainPrev.showHeader; + memcpy(p->colWidth,p->explainPrev.colWidth,sizeof(p->colWidth)); + } + }else + + if( c=='h' && (strncmp(azArg[0], "header", n)==0 + || + strncmp(azArg[0], "headers", n)==0 )&& nArg>1 ){ + int j; + char *z = azArg[1]; + int val = atoi(azArg[1]); + for(j=0; z[j]; j++){ + z[j] = tolower((unsigned char)z[j]); + } + if( strcmp(z,"on")==0 ){ + val = 1; + }else if( strcmp(z,"yes")==0 ){ + val = 1; + } + p->showHeader = val; + }else + + if( c=='h' && strncmp(azArg[0], "help", n)==0 ){ + fprintf(stderr,zHelp); + }else + + if( c=='i' && strncmp(azArg[0], "import", n)==0 && nArg>=3 ){ + char *zTable = azArg[2]; /* Insert data into this table */ + char *zFile = azArg[1]; /* The file from which to extract data */ + sqlite3_stmt *pStmt; /* A statement */ + int rc; /* Result code */ + int nCol; /* Number of columns in the table */ + int nByte; /* Number of bytes in an SQL string */ + int i, j; /* Loop counters */ + int nSep; /* Number of bytes in p->separator[] */ + char *zSql; /* An SQL statement */ + char *zLine; /* A single line of input from the file */ + char **azCol; /* zLine[] broken up into columns */ + char *zCommit; /* How to commit changes */ + FILE *in; /* The input file */ + int lineno = 0; /* Line number of input file */ + + nSep = strlen(p->separator); + if( nSep==0 ){ + fprintf(stderr, "non-null separator required for import\n"); + return 0; + } + zSql = sqlite3_mprintf("SELECT * FROM '%q'", zTable); + if( zSql==0 ) return 0; + nByte = strlen(zSql); + rc = sqlite3_prepare(p->db, zSql, -1, &pStmt, 0); + sqlite3_free(zSql); + if( rc ){ + fprintf(stderr,"Error: %s\n", sqlite3_errmsg(db)); + nCol = 0; + }else{ + nCol = sqlite3_column_count(pStmt); + } + sqlite3_finalize(pStmt); + if( nCol==0 ) return 0; + zSql = malloc( nByte + 20 + nCol*2 ); + if( zSql==0 ) return 0; + sqlite3_snprintf(nByte+20, zSql, "INSERT INTO '%q' VALUES(?", zTable); + j = strlen(zSql); + for(i=1; idb, zSql, -1, &pStmt, 0); + free(zSql); + if( rc ){ + fprintf(stderr, "Error: %s\n", sqlite3_errmsg(db)); + sqlite3_finalize(pStmt); + return 0; + } + in = fopen(zFile, "rb"); + if( in==0 ){ + fprintf(stderr, "cannot open file: %s\n", zFile); + sqlite3_finalize(pStmt); + return 0; + } + azCol = malloc( sizeof(azCol[0])*(nCol+1) ); + if( azCol==0 ) return 0; + sqlite3_exec(p->db, "BEGIN", 0, 0, 0); + zCommit = "COMMIT"; + while( (zLine = local_getline(0, in))!=0 ){ + char *z; + i = 0; + lineno++; + azCol[0] = zLine; + for(i=0, z=zLine; *z && *z!='\n' && *z!='\r'; z++){ + if( *z==p->separator[0] && strncmp(z, p->separator, nSep)==0 ){ + *z = 0; + i++; + if( idb, zCommit, 0, 0, 0); + }else + + if( c=='i' && strncmp(azArg[0], "indices", n)==0 && nArg>1 ){ + struct callback_data data; + char *zErrMsg = 0; + open_db(p); + memcpy(&data, p, sizeof(data)); + data.showHeader = 0; + data.mode = MODE_List; + zShellStatic = azArg[1]; + sqlite3_exec(p->db, + "SELECT name FROM sqlite_master " + "WHERE type='index' AND tbl_name LIKE shellstatic() " + "UNION ALL " + "SELECT name FROM sqlite_temp_master " + "WHERE type='index' AND tbl_name LIKE shellstatic() " + "ORDER BY 1", + callback, &data, &zErrMsg + ); + zShellStatic = 0; + if( zErrMsg ){ + fprintf(stderr,"Error: %s\n", zErrMsg); + sqlite3_free(zErrMsg); + } + }else + + if( c=='m' && strncmp(azArg[0], "mode", n)==0 && nArg>=2 ){ + int n2 = strlen(azArg[1]); + if( strncmp(azArg[1],"line",n2)==0 + || + strncmp(azArg[1],"lines",n2)==0 ){ + p->mode = MODE_Line; + }else if( strncmp(azArg[1],"column",n2)==0 + || + strncmp(azArg[1],"columns",n2)==0 ){ + p->mode = MODE_Column; + }else if( strncmp(azArg[1],"list",n2)==0 ){ + p->mode = MODE_List; + }else if( strncmp(azArg[1],"html",n2)==0 ){ + p->mode = MODE_Html; + }else if( strncmp(azArg[1],"tcl",n2)==0 ){ + p->mode = MODE_Tcl; + }else if( strncmp(azArg[1],"csv",n2)==0 ){ + p->mode = MODE_Csv; + strcpy(p->separator, ","); + }else if( strncmp(azArg[1],"tabs",n2)==0 ){ + p->mode = MODE_List; + strcpy(p->separator, "\t"); + }else if( strncmp(azArg[1],"insert",n2)==0 ){ + p->mode = MODE_Insert; + if( nArg>=3 ){ + set_table_name(p, azArg[2]); + }else{ + set_table_name(p, "table"); + } + }else { + fprintf(stderr,"mode should be on of: " + "column csv html insert line list tabs tcl\n"); + } + }else + + if( c=='n' && strncmp(azArg[0], "nullvalue", n)==0 && nArg==2 ) { + sprintf(p->nullvalue, "%.*s", (int)ArraySize(p->nullvalue)-1, azArg[1]); + }else + + if( c=='o' && strncmp(azArg[0], "output", n)==0 && nArg==2 ){ + if( p->out!=stdout ){ + fclose(p->out); + } + if( strcmp(azArg[1],"stdout")==0 ){ + p->out = stdout; + strcpy(p->outfile,"stdout"); + }else{ + p->out = fopen(azArg[1], "wb"); + if( p->out==0 ){ + fprintf(stderr,"can't write to \"%s\"\n", azArg[1]); + p->out = stdout; + } else { + strcpy(p->outfile,azArg[1]); + } + } + }else + + if( c=='p' && strncmp(azArg[0], "prompt", n)==0 && (nArg==2 || nArg==3)){ + if( nArg >= 2) { + strncpy(mainPrompt,azArg[1],(int)ArraySize(mainPrompt)-1); + } + if( nArg >= 3) { + strncpy(continuePrompt,azArg[2],(int)ArraySize(continuePrompt)-1); + } + }else + + if( c=='q' && strncmp(azArg[0], "quit", n)==0 ){ + rc = 1; + }else + + if( c=='r' && strncmp(azArg[0], "read", n)==0 && nArg==2 ){ + FILE *alt = fopen(azArg[1], "rb"); + if( alt==0 ){ + fprintf(stderr,"can't open \"%s\"\n", azArg[1]); + }else{ + process_input(p, alt); + fclose(alt); + } + }else + + if( c=='s' && strncmp(azArg[0], "schema", n)==0 ){ + struct callback_data data; + char *zErrMsg = 0; + open_db(p); + memcpy(&data, p, sizeof(data)); + data.showHeader = 0; + data.mode = MODE_Semi; + if( nArg>1 ){ + int i; + for(i=0; azArg[1][i]; i++) azArg[1][i] = tolower(azArg[1][i]); + if( strcmp(azArg[1],"sqlite_master")==0 ){ + char *new_argv[2], *new_colv[2]; + new_argv[0] = "CREATE TABLE sqlite_master (\n" + " type text,\n" + " name text,\n" + " tbl_name text,\n" + " rootpage integer,\n" + " sql text\n" + ")"; + new_argv[1] = 0; + new_colv[0] = "sql"; + new_colv[1] = 0; + callback(&data, 1, new_argv, new_colv); + }else if( strcmp(azArg[1],"sqlite_temp_master")==0 ){ + char *new_argv[2], *new_colv[2]; + new_argv[0] = "CREATE TEMP TABLE sqlite_temp_master (\n" + " type text,\n" + " name text,\n" + " tbl_name text,\n" + " rootpage integer,\n" + " sql text\n" + ")"; + new_argv[1] = 0; + new_colv[0] = "sql"; + new_colv[1] = 0; + callback(&data, 1, new_argv, new_colv); + }else{ + zShellStatic = azArg[1]; + sqlite3_exec(p->db, + "SELECT sql FROM " + " (SELECT * FROM sqlite_master UNION ALL" + " SELECT * FROM sqlite_temp_master) " + "WHERE tbl_name LIKE shellstatic() AND type!='meta' AND sql NOTNULL " + "ORDER BY substr(type,2,1), name", + callback, &data, &zErrMsg); + zShellStatic = 0; + } + }else{ + sqlite3_exec(p->db, + "SELECT sql FROM " + " (SELECT * FROM sqlite_master UNION ALL" + " SELECT * FROM sqlite_temp_master) " + "WHERE type!='meta' AND sql NOTNULL AND name NOT LIKE 'sqlite_%'" + "ORDER BY substr(type,2,1), name", + callback, &data, &zErrMsg + ); + } + if( zErrMsg ){ + fprintf(stderr,"Error: %s\n", zErrMsg); + sqlite3_free(zErrMsg); + } + }else + + if( c=='s' && strncmp(azArg[0], "separator", n)==0 && nArg==2 ){ + sprintf(p->separator, "%.*s", (int)ArraySize(p->separator)-1, azArg[1]); + }else + + if( c=='s' && strncmp(azArg[0], "show", n)==0){ + int i; + fprintf(p->out,"%9.9s: %s\n","echo", p->echoOn ? "on" : "off"); + fprintf(p->out,"%9.9s: %s\n","explain", p->explainPrev.valid ? "on" :"off"); + fprintf(p->out,"%9.9s: %s\n","headers", p->showHeader ? "on" : "off"); + fprintf(p->out,"%9.9s: %s\n","mode", modeDescr[p->mode]); + fprintf(p->out,"%9.9s: ", "nullvalue"); + output_c_string(p->out, p->nullvalue); + fprintf(p->out, "\n"); + fprintf(p->out,"%9.9s: %s\n","output", + strlen(p->outfile) ? p->outfile : "stdout"); + fprintf(p->out,"%9.9s: ", "separator"); + output_c_string(p->out, p->separator); + fprintf(p->out, "\n"); + fprintf(p->out,"%9.9s: ","width"); + for (i=0;i<(int)ArraySize(p->colWidth) && p->colWidth[i] != 0;i++) { + fprintf(p->out,"%d ",p->colWidth[i]); + } + fprintf(p->out,"\n"); + }else + + if( c=='t' && n>1 && strncmp(azArg[0], "tables", n)==0 ){ + char **azResult; + int nRow, rc; + char *zErrMsg; + open_db(p); + if( nArg==1 ){ + rc = sqlite3_get_table(p->db, + "SELECT name FROM sqlite_master " + "WHERE type IN ('table','view') AND name NOT LIKE 'sqlite_%'" + "UNION ALL " + "SELECT name FROM sqlite_temp_master " + "WHERE type IN ('table','view') " + "ORDER BY 1", + &azResult, &nRow, 0, &zErrMsg + ); + }else{ + zShellStatic = azArg[1]; + rc = sqlite3_get_table(p->db, + "SELECT name FROM sqlite_master " + "WHERE type IN ('table','view') AND name LIKE '%'||shellstatic()||'%' " + "UNION ALL " + "SELECT name FROM sqlite_temp_master " + "WHERE type IN ('table','view') AND name LIKE '%'||shellstatic()||'%' " + "ORDER BY 1", + &azResult, &nRow, 0, &zErrMsg + ); + zShellStatic = 0; + } + if( zErrMsg ){ + fprintf(stderr,"Error: %s\n", zErrMsg); + sqlite3_free(zErrMsg); + } + if( rc==SQLITE_OK ){ + int len, maxlen = 0; + int i, j; + int nPrintCol, nPrintRow; + for(i=1; i<=nRow; i++){ + if( azResult[i]==0 ) continue; + len = strlen(azResult[i]); + if( len>maxlen ) maxlen = len; + } + nPrintCol = 80/(maxlen+2); + if( nPrintCol<1 ) nPrintCol = 1; + nPrintRow = (nRow + nPrintCol - 1)/nPrintCol; + for(i=0; i1 && strncmp(azArg[0], "timeout", n)==0 && nArg>=2 ){ + open_db(p); + sqlite3_busy_timeout(p->db, atoi(azArg[1])); + }else + + if( c=='w' && strncmp(azArg[0], "width", n)==0 ){ + int j; + for(j=1; jcolWidth); j++){ + p->colWidth[j-1] = atoi(azArg[j]); + } + }else + + { + fprintf(stderr, "unknown command or invalid arguments: " + " \"%s\". Enter \".help\" for help\n", azArg[0]); + } + + return rc; +} + +/* +** Return TRUE if the last non-whitespace character in z[] is a semicolon. +** z[] is N characters long. +*/ +static int _ends_with_semicolon(const char *z, int N){ + while( N>0 && isspace((unsigned char)z[N-1]) ){ N--; } + return N>0 && z[N-1]==';'; +} + +/* +** Test to see if a line consists entirely of whitespace. +*/ +static int _all_whitespace(const char *z){ + for(; *z; z++){ + if( isspace(*(unsigned char*)z) ) continue; + if( *z=='/' && z[1]=='*' ){ + z += 2; + while( *z && (*z!='*' || z[1]!='/') ){ z++; } + if( *z==0 ) return 0; + z++; + continue; + } + if( *z=='-' && z[1]=='-' ){ + z += 2; + while( *z && *z!='\n' ){ z++; } + if( *z==0 ) return 1; + continue; + } + return 0; + } + return 1; +} + +/* +** Return TRUE if the line typed in is an SQL command terminator other +** than a semi-colon. The SQL Server style "go" command is understood +** as is the Oracle "/". +*/ +static int _is_command_terminator(const char *zLine){ + while( isspace(*(unsigned char*)zLine) ){ zLine++; }; + if( zLine[0]=='/' && _all_whitespace(&zLine[1]) ) return 1; /* Oracle */ + if( tolower(zLine[0])=='g' && tolower(zLine[1])=='o' + && _all_whitespace(&zLine[2]) ){ + return 1; /* SQL Server */ + } + return 0; +} + +/* +** Read input from *in and process it. If *in==0 then input +** is interactive - the user is typing it it. Otherwise, input +** is coming from a file or device. A prompt is issued and history +** is saved only if input is interactive. An interrupt signal will +** cause this routine to exit immediately, unless input is interactive. +*/ +static void process_input(struct callback_data *p, FILE *in){ + char *zLine; + char *zSql = 0; + int nSql = 0; + char *zErrMsg; + int rc; + while( fflush(p->out), (zLine = one_input_line(zSql, in))!=0 ){ + if( seenInterrupt ){ + if( in!=0 ) break; + seenInterrupt = 0; + } + if( p->echoOn ) printf("%s\n", zLine); + if( (zSql==0 || zSql[0]==0) && _all_whitespace(zLine) ) continue; + if( zLine && zLine[0]=='.' && nSql==0 ){ + int rc = do_meta_command(zLine, p); + free(zLine); + if( rc ) break; + continue; + } + if( _is_command_terminator(zLine) ){ + strcpy(zLine,";"); + } + if( zSql==0 ){ + int i; + for(i=0; zLine[i] && isspace((unsigned char)zLine[i]); i++){} + if( zLine[i]!=0 ){ + nSql = strlen(zLine); + zSql = malloc( nSql+1 ); + strcpy(zSql, zLine); + } + }else{ + int len = strlen(zLine); + zSql = realloc( zSql, nSql + len + 2 ); + if( zSql==0 ){ + fprintf(stderr,"%s: out of memory!\n", Argv0); + exit(1); + } + strcpy(&zSql[nSql++], "\n"); + strcpy(&zSql[nSql], zLine); + nSql += len; + } + free(zLine); + if( zSql && _ends_with_semicolon(zSql, nSql) && sqlite3_complete(zSql) ){ + p->cnt = 0; + open_db(p); + rc = sqlite3_exec(p->db, zSql, callback, p, &zErrMsg); + if( rc || zErrMsg ){ + if( in!=0 && !p->echoOn ) printf("%s\n",zSql); + if( zErrMsg!=0 ){ + printf("SQL error: %s\n", zErrMsg); + sqlite3_free(zErrMsg); + zErrMsg = 0; + }else{ + printf("SQL error: %s\n", sqlite3_errmsg(p->db)); + } + } + free(zSql); + zSql = 0; + nSql = 0; + } + } + if( zSql ){ + if( !_all_whitespace(zSql) ) printf("Incomplete SQL: %s\n", zSql); + free(zSql); + } +} + +/* +** Return a pathname which is the user's home directory. A +** 0 return indicates an error of some kind. Space to hold the +** resulting string is obtained from malloc(). The calling +** function should free the result. +*/ +static char *find_home_dir(void){ + char *home_dir = NULL; + +#if !defined(_WIN32) && !defined(WIN32) && !defined(__MACOS__) + struct passwd *pwent; + uid_t uid = getuid(); + if( (pwent=getpwuid(uid)) != NULL) { + home_dir = pwent->pw_dir; + } +#endif + +#ifdef __MACOS__ + char home_path[_MAX_PATH+1]; + home_dir = getcwd(home_path, _MAX_PATH); +#endif + + if (!home_dir) { + home_dir = getenv("HOME"); + if (!home_dir) { + home_dir = getenv("HOMEPATH"); /* Windows? */ + } + } + +#if defined(_WIN32) || defined(WIN32) + if (!home_dir) { + home_dir = "c:"; + } +#endif + + if( home_dir ){ + char *z = malloc( strlen(home_dir)+1 ); + if( z ) strcpy(z, home_dir); + home_dir = z; + } + + return home_dir; +} + +/* +** Read input from the file given by sqliterc_override. Or if that +** parameter is NULL, take input from ~/.sqliterc +*/ +static void process_sqliterc( + struct callback_data *p, /* Configuration data */ + const char *sqliterc_override /* Name of config file. NULL to use default */ +){ + char *home_dir = NULL; + const char *sqliterc = sqliterc_override; + char *zBuf; + FILE *in = NULL; + + if (sqliterc == NULL) { + home_dir = find_home_dir(); + if( home_dir==0 ){ + fprintf(stderr,"%s: cannot locate your home directory!\n", Argv0); + return; + } + zBuf = malloc(strlen(home_dir) + 15); + if( zBuf==0 ){ + fprintf(stderr,"%s: out of memory!\n", Argv0); + exit(1); + } + sprintf(zBuf,"%s/.sqliterc",home_dir); + free(home_dir); + sqliterc = (const char*)zBuf; + } + in = fopen(sqliterc,"rb"); + if( in ){ + if( isatty(fileno(stdout)) ){ + printf("Loading resources from %s\n",sqliterc); + } + process_input(p,in); + fclose(in); + } + return; +} + +/* +** Show available command line options +*/ +static const char zOptions[] = + " -init filename read/process named file\n" + " -echo print commands before execution\n" + " -[no]header turn headers on or off\n" + " -column set output mode to 'column'\n" + " -html set output mode to HTML\n" + " -line set output mode to 'line'\n" + " -list set output mode to 'list'\n" + " -separator 'x' set output field separator (|)\n" + " -nullvalue 'text' set text string for NULL values\n" + " -version show SQLite version\n" + " -help show this text, also show dot-commands\n" +; +static void usage(int showDetail){ + fprintf(stderr, "Usage: %s [OPTIONS] FILENAME [SQL]\n", Argv0); + if( showDetail ){ + fprintf(stderr, "Options are:\n%s", zOptions); + }else{ + fprintf(stderr, "Use the -help option for additional information\n"); + } + exit(1); +} + +/* +** Initialize the state information in data +*/ +static void main_init(struct callback_data *data) { + memset(data, 0, sizeof(*data)); + data->mode = MODE_List; + strcpy(data->separator,"|"); + data->showHeader = 0; + strcpy(mainPrompt,"sqlite> "); + strcpy(continuePrompt," ...> "); +} + +int main(int argc, char **argv){ + char *zErrMsg = 0; + struct callback_data data; + const char *zInitFile = 0; + char *zFirstCmd = 0; + int i; + +#ifdef __MACOS__ + argc = ccommand(&argv); +#endif + + Argv0 = argv[0]; + main_init(&data); + + /* Make sure we have a valid signal handler early, before anything + ** else is done. + */ +#ifdef SIGINT + signal(SIGINT, interrupt_handler); +#endif + + /* Do an initial pass through the command-line argument to locate + ** the name of the database file, the name of the initialization file, + ** and the first command to execute. + */ + for(i=1; i /* Needed for the definition of va_list */ + +/* +** Make sure we can call this stuff from C++. +*/ +#ifdef __cplusplus +extern "C" { +#endif + +/* +** The version of the SQLite library. +*/ +#ifdef SQLITE_VERSION +# undef SQLITE_VERSION +#endif +#define SQLITE_VERSION "3.3.4" + +/* +** The format of the version string is "X.Y.Z", where +** X is the major version number, Y is the minor version number and Z +** is the release number. The trailing string is often "alpha" or "beta". +** For example "3.1.1beta". +** +** The SQLITE_VERSION_NUMBER is an integer with the value +** (X*100000 + Y*1000 + Z). For example, for version "3.1.1beta", +** SQLITE_VERSION_NUMBER is set to 3001001. To detect if they are using +** version 3.1.1 or greater at compile time, programs may use the test +** (SQLITE_VERSION_NUMBER>=3001001). +*/ +#ifdef SQLITE_VERSION_NUMBER +# undef SQLITE_VERSION_NUMBER +#endif +#define SQLITE_VERSION_NUMBER 3003004 + +/* +** The version string is also compiled into the library so that a program +** can check to make sure that the lib*.a file and the *.h file are from +** the same version. The sqlite3_libversion() function returns a pointer +** to the sqlite3_version variable - useful in DLLs which cannot access +** global variables. +*/ +extern const char sqlite3_version[]; +const char *sqlite3_libversion(void); + +/* +** Return the value of the SQLITE_VERSION_NUMBER macro when the +** library was compiled. +*/ +int sqlite3_libversion_number(void); + +/* +** Each open sqlite database is represented by an instance of the +** following opaque structure. +*/ +typedef struct sqlite3 sqlite3; + + +/* +** Some compilers do not support the "long long" datatype. So we have +** to do a typedef that for 64-bit integers that depends on what compiler +** is being used. +*/ +#if defined(_MSC_VER) || defined(__BORLANDC__) + typedef __int64 sqlite_int64; + typedef unsigned __int64 sqlite_uint64; +#else + typedef long long int sqlite_int64; + typedef unsigned long long int sqlite_uint64; +#endif + +/* +** If compiling for a processor that lacks floating point support, +** substitute integer for floating-point +*/ +#ifdef SQLITE_OMIT_FLOATING_POINT +# define double sqlite_int64 +#endif + +/* +** A function to close the database. +** +** Call this function with a pointer to a structure that was previously +** returned from sqlite3_open() and the corresponding database will by closed. +** +** All SQL statements prepared using sqlite3_prepare() or +** sqlite3_prepare16() must be deallocated using sqlite3_finalize() before +** this routine is called. Otherwise, SQLITE_BUSY is returned and the +** database connection remains open. +*/ +int sqlite3_close(sqlite3 *); + +/* +** The type for a callback function. +*/ +typedef int (*sqlite3_callback)(void*,int,char**, char**); + +/* +** A function to executes one or more statements of SQL. +** +** If one or more of the SQL statements are queries, then +** the callback function specified by the 3rd parameter is +** invoked once for each row of the query result. This callback +** should normally return 0. If the callback returns a non-zero +** value then the query is aborted, all subsequent SQL statements +** are skipped and the sqlite3_exec() function returns the SQLITE_ABORT. +** +** The 4th parameter is an arbitrary pointer that is passed +** to the callback function as its first parameter. +** +** The 2nd parameter to the callback function is the number of +** columns in the query result. The 3rd parameter to the callback +** is an array of strings holding the values for each column. +** The 4th parameter to the callback is an array of strings holding +** the names of each column. +** +** The callback function may be NULL, even for queries. A NULL +** callback is not an error. It just means that no callback +** will be invoked. +** +** If an error occurs while parsing or evaluating the SQL (but +** not while executing the callback) then an appropriate error +** message is written into memory obtained from malloc() and +** *errmsg is made to point to that message. The calling function +** is responsible for freeing the memory that holds the error +** message. Use sqlite3_free() for this. If errmsg==NULL, +** then no error message is ever written. +** +** The return value is is SQLITE_OK if there are no errors and +** some other return code if there is an error. The particular +** return value depends on the type of error. +** +** If the query could not be executed because a database file is +** locked or busy, then this function returns SQLITE_BUSY. (This +** behavior can be modified somewhat using the sqlite3_busy_handler() +** and sqlite3_busy_timeout() functions below.) +*/ +int sqlite3_exec( + sqlite3*, /* An open database */ + const char *sql, /* SQL to be executed */ + sqlite3_callback, /* Callback function */ + void *, /* 1st argument to callback function */ + char **errmsg /* Error msg written here */ +); + +/* +** Return values for sqlite3_exec() and sqlite3_step() +*/ +#define SQLITE_OK 0 /* Successful result */ +/* beginning-of-error-codes */ +#define SQLITE_ERROR 1 /* SQL error or missing database */ +#define SQLITE_INTERNAL 2 /* NOT USED. Internal logic error in SQLite */ +#define SQLITE_PERM 3 /* Access permission denied */ +#define SQLITE_ABORT 4 /* Callback routine requested an abort */ +#define SQLITE_BUSY 5 /* The database file is locked */ +#define SQLITE_LOCKED 6 /* A table in the database is locked */ +#define SQLITE_NOMEM 7 /* A malloc() failed */ +#define SQLITE_READONLY 8 /* Attempt to write a readonly database */ +#define SQLITE_INTERRUPT 9 /* Operation terminated by sqlite3_interrupt()*/ +#define SQLITE_IOERR 10 /* Some kind of disk I/O error occurred */ +#define SQLITE_CORRUPT 11 /* The database disk image is malformed */ +#define SQLITE_NOTFOUND 12 /* NOT USED. Table or record not found */ +#define SQLITE_FULL 13 /* Insertion failed because database is full */ +#define SQLITE_CANTOPEN 14 /* Unable to open the database file */ +#define SQLITE_PROTOCOL 15 /* Database lock protocol error */ +#define SQLITE_EMPTY 16 /* Database is empty */ +#define SQLITE_SCHEMA 17 /* The database schema changed */ +#define SQLITE_TOOBIG 18 /* NOT USED. Too much data for one row */ +#define SQLITE_CONSTRAINT 19 /* Abort due to contraint violation */ +#define SQLITE_MISMATCH 20 /* Data type mismatch */ +#define SQLITE_MISUSE 21 /* Library used incorrectly */ +#define SQLITE_NOLFS 22 /* Uses OS features not supported on host */ +#define SQLITE_AUTH 23 /* Authorization denied */ +#define SQLITE_FORMAT 24 /* Auxiliary database format error */ +#define SQLITE_RANGE 25 /* 2nd parameter to sqlite3_bind out of range */ +#define SQLITE_NOTADB 26 /* File opened that is not a database file */ +#define SQLITE_ROW 100 /* sqlite3_step() has another row ready */ +#define SQLITE_DONE 101 /* sqlite3_step() has finished executing */ +/* end-of-error-codes */ + +/* +** Each entry in an SQLite table has a unique integer key. (The key is +** the value of the INTEGER PRIMARY KEY column if there is such a column, +** otherwise the key is generated at random. The unique key is always +** available as the ROWID, OID, or _ROWID_ column.) The following routine +** returns the integer key of the most recent insert in the database. +** +** This function is similar to the mysql_insert_id() function from MySQL. +*/ +sqlite_int64 sqlite3_last_insert_rowid(sqlite3*); + +/* +** This function returns the number of database rows that were changed +** (or inserted or deleted) by the most recent called sqlite3_exec(). +** +** All changes are counted, even if they were later undone by a +** ROLLBACK or ABORT. Except, changes associated with creating and +** dropping tables are not counted. +** +** If a callback invokes sqlite3_exec() recursively, then the changes +** in the inner, recursive call are counted together with the changes +** in the outer call. +** +** SQLite implements the command "DELETE FROM table" without a WHERE clause +** by dropping and recreating the table. (This is much faster than going +** through and deleting individual elements form the table.) Because of +** this optimization, the change count for "DELETE FROM table" will be +** zero regardless of the number of elements that were originally in the +** table. To get an accurate count of the number of rows deleted, use +** "DELETE FROM table WHERE 1" instead. +*/ +int sqlite3_changes(sqlite3*); + +/* +** This function returns the number of database rows that have been +** modified by INSERT, UPDATE or DELETE statements since the database handle +** was opened. This includes UPDATE, INSERT and DELETE statements executed +** as part of trigger programs. All changes are counted as soon as the +** statement that makes them is completed (when the statement handle is +** passed to sqlite3_reset() or sqlite_finalise()). +** +** SQLite implements the command "DELETE FROM table" without a WHERE clause +** by dropping and recreating the table. (This is much faster than going +** through and deleting individual elements form the table.) Because of +** this optimization, the change count for "DELETE FROM table" will be +** zero regardless of the number of elements that were originally in the +** table. To get an accurate count of the number of rows deleted, use +** "DELETE FROM table WHERE 1" instead. +*/ +int sqlite3_total_changes(sqlite3*); + +/* This function causes any pending database operation to abort and +** return at its earliest opportunity. This routine is typically +** called in response to a user action such as pressing "Cancel" +** or Ctrl-C where the user wants a long query operation to halt +** immediately. +*/ +void sqlite3_interrupt(sqlite3*); + + +/* These functions return true if the given input string comprises +** one or more complete SQL statements. For the sqlite3_complete() call, +** the parameter must be a nul-terminated UTF-8 string. For +** sqlite3_complete16(), a nul-terminated machine byte order UTF-16 string +** is required. +** +** The algorithm is simple. If the last token other than spaces +** and comments is a semicolon, then return true. otherwise return +** false. +*/ +int sqlite3_complete(const char *sql); +int sqlite3_complete16(const void *sql); + +/* +** This routine identifies a callback function that is invoked +** whenever an attempt is made to open a database table that is +** currently locked by another process or thread. If the busy callback +** is NULL, then sqlite3_exec() returns SQLITE_BUSY immediately if +** it finds a locked table. If the busy callback is not NULL, then +** sqlite3_exec() invokes the callback with three arguments. The +** second argument is the name of the locked table and the third +** argument is the number of times the table has been busy. If the +** busy callback returns 0, then sqlite3_exec() immediately returns +** SQLITE_BUSY. If the callback returns non-zero, then sqlite3_exec() +** tries to open the table again and the cycle repeats. +** +** The default busy callback is NULL. +** +** Sqlite is re-entrant, so the busy handler may start a new query. +** (It is not clear why anyone would every want to do this, but it +** is allowed, in theory.) But the busy handler may not close the +** database. Closing the database from a busy handler will delete +** data structures out from under the executing query and will +** probably result in a coredump. +*/ +int sqlite3_busy_handler(sqlite3*, int(*)(void*,int), void*); + +/* +** This routine sets a busy handler that sleeps for a while when a +** table is locked. The handler will sleep multiple times until +** at least "ms" milleseconds of sleeping have been done. After +** "ms" milleseconds of sleeping, the handler returns 0 which +** causes sqlite3_exec() to return SQLITE_BUSY. +** +** Calling this routine with an argument less than or equal to zero +** turns off all busy handlers. +*/ +int sqlite3_busy_timeout(sqlite3*, int ms); + +/* +** This next routine is really just a wrapper around sqlite3_exec(). +** Instead of invoking a user-supplied callback for each row of the +** result, this routine remembers each row of the result in memory +** obtained from malloc(), then returns all of the result after the +** query has finished. +** +** As an example, suppose the query result where this table: +** +** Name | Age +** ----------------------- +** Alice | 43 +** Bob | 28 +** Cindy | 21 +** +** If the 3rd argument were &azResult then after the function returns +** azResult will contain the following data: +** +** azResult[0] = "Name"; +** azResult[1] = "Age"; +** azResult[2] = "Alice"; +** azResult[3] = "43"; +** azResult[4] = "Bob"; +** azResult[5] = "28"; +** azResult[6] = "Cindy"; +** azResult[7] = "21"; +** +** Notice that there is an extra row of data containing the column +** headers. But the *nrow return value is still 3. *ncolumn is +** set to 2. In general, the number of values inserted into azResult +** will be ((*nrow) + 1)*(*ncolumn). +** +** After the calling function has finished using the result, it should +** pass the result data pointer to sqlite3_free_table() in order to +** release the memory that was malloc-ed. Because of the way the +** malloc() happens, the calling function must not try to call +** free() directly. Only sqlite3_free_table() is able to release +** the memory properly and safely. +** +** The return value of this routine is the same as from sqlite3_exec(). +*/ +int sqlite3_get_table( + sqlite3*, /* An open database */ + const char *sql, /* SQL to be executed */ + char ***resultp, /* Result written to a char *[] that this points to */ + int *nrow, /* Number of result rows written here */ + int *ncolumn, /* Number of result columns written here */ + char **errmsg /* Error msg written here */ +); + +/* +** Call this routine to free the memory that sqlite3_get_table() allocated. +*/ +void sqlite3_free_table(char **result); + +/* +** The following routines are variants of the "sprintf()" from the +** standard C library. The resulting string is written into memory +** obtained from malloc() so that there is never a possiblity of buffer +** overflow. These routines also implement some additional formatting +** options that are useful for constructing SQL statements. +** +** The strings returned by these routines should be freed by calling +** sqlite3_free(). +** +** All of the usual printf formatting options apply. In addition, there +** is a "%q" option. %q works like %s in that it substitutes a null-terminated +** string from the argument list. But %q also doubles every '\'' character. +** %q is designed for use inside a string literal. By doubling each '\'' +** character it escapes that character and allows it to be inserted into +** the string. +** +** For example, so some string variable contains text as follows: +** +** char *zText = "It's a happy day!"; +** +** We can use this text in an SQL statement as follows: +** +** char *z = sqlite3_mprintf("INSERT INTO TABLES('%q')", zText); +** sqlite3_exec(db, z, callback1, 0, 0); +** sqlite3_free(z); +** +** Because the %q format string is used, the '\'' character in zText +** is escaped and the SQL generated is as follows: +** +** INSERT INTO table1 VALUES('It''s a happy day!') +** +** This is correct. Had we used %s instead of %q, the generated SQL +** would have looked like this: +** +** INSERT INTO table1 VALUES('It's a happy day!'); +** +** This second example is an SQL syntax error. As a general rule you +** should always use %q instead of %s when inserting text into a string +** literal. +*/ +char *sqlite3_mprintf(const char*,...); +char *sqlite3_vmprintf(const char*, va_list); +void sqlite3_free(char *z); +char *sqlite3_snprintf(int,char*,const char*, ...); + +#ifndef SQLITE_OMIT_AUTHORIZATION +/* +** This routine registers a callback with the SQLite library. The +** callback is invoked (at compile-time, not at run-time) for each +** attempt to access a column of a table in the database. The callback +** returns SQLITE_OK if access is allowed, SQLITE_DENY if the entire +** SQL statement should be aborted with an error and SQLITE_IGNORE +** if the column should be treated as a NULL value. +*/ +int sqlite3_set_authorizer( + sqlite3*, + int (*xAuth)(void*,int,const char*,const char*,const char*,const char*), + void *pUserData +); +#endif + +/* +** The second parameter to the access authorization function above will +** be one of the values below. These values signify what kind of operation +** is to be authorized. The 3rd and 4th parameters to the authorization +** function will be parameters or NULL depending on which of the following +** codes is used as the second parameter. The 5th parameter is the name +** of the database ("main", "temp", etc.) if applicable. The 6th parameter +** is the name of the inner-most trigger or view that is responsible for +** the access attempt or NULL if this access attempt is directly from +** input SQL code. +** +** Arg-3 Arg-4 +*/ +#define SQLITE_COPY 0 /* Table Name File Name */ +#define SQLITE_CREATE_INDEX 1 /* Index Name Table Name */ +#define SQLITE_CREATE_TABLE 2 /* Table Name NULL */ +#define SQLITE_CREATE_TEMP_INDEX 3 /* Index Name Table Name */ +#define SQLITE_CREATE_TEMP_TABLE 4 /* Table Name NULL */ +#define SQLITE_CREATE_TEMP_TRIGGER 5 /* Trigger Name Table Name */ +#define SQLITE_CREATE_TEMP_VIEW 6 /* View Name NULL */ +#define SQLITE_CREATE_TRIGGER 7 /* Trigger Name Table Name */ +#define SQLITE_CREATE_VIEW 8 /* View Name NULL */ +#define SQLITE_DELETE 9 /* Table Name NULL */ +#define SQLITE_DROP_INDEX 10 /* Index Name Table Name */ +#define SQLITE_DROP_TABLE 11 /* Table Name NULL */ +#define SQLITE_DROP_TEMP_INDEX 12 /* Index Name Table Name */ +#define SQLITE_DROP_TEMP_TABLE 13 /* Table Name NULL */ +#define SQLITE_DROP_TEMP_TRIGGER 14 /* Trigger Name Table Name */ +#define SQLITE_DROP_TEMP_VIEW 15 /* View Name NULL */ +#define SQLITE_DROP_TRIGGER 16 /* Trigger Name Table Name */ +#define SQLITE_DROP_VIEW 17 /* View Name NULL */ +#define SQLITE_INSERT 18 /* Table Name NULL */ +#define SQLITE_PRAGMA 19 /* Pragma Name 1st arg or NULL */ +#define SQLITE_READ 20 /* Table Name Column Name */ +#define SQLITE_SELECT 21 /* NULL NULL */ +#define SQLITE_TRANSACTION 22 /* NULL NULL */ +#define SQLITE_UPDATE 23 /* Table Name Column Name */ +#define SQLITE_ATTACH 24 /* Filename NULL */ +#define SQLITE_DETACH 25 /* Database Name NULL */ +#define SQLITE_ALTER_TABLE 26 /* Database Name Table Name */ +#define SQLITE_REINDEX 27 /* Index Name NULL */ +#define SQLITE_ANALYZE 28 /* Table Name NULL */ + + +/* +** The return value of the authorization function should be one of the +** following constants: +*/ +/* #define SQLITE_OK 0 // Allow access (This is actually defined above) */ +#define SQLITE_DENY 1 /* Abort the SQL statement with an error */ +#define SQLITE_IGNORE 2 /* Don't allow access, but don't generate an error */ + +/* +** Register a function for tracing SQL command evaluation. The function +** registered by sqlite3_trace() is invoked at the first sqlite3_step() +** for the evaluation of an SQL statement. The function registered by +** sqlite3_profile() runs at the end of each SQL statement and includes +** information on how long that statement ran. +** +** The sqlite3_profile() API is currently considered experimental and +** is subject to change. +*/ +void *sqlite3_trace(sqlite3*, void(*xTrace)(void*,const char*), void*); +void *sqlite3_profile(sqlite3*, + void(*xProfile)(void*,const char*,sqlite_uint64), void*); + +/* +** This routine configures a callback function - the progress callback - that +** is invoked periodically during long running calls to sqlite3_exec(), +** sqlite3_step() and sqlite3_get_table(). An example use for this API is to +** keep a GUI updated during a large query. +** +** The progress callback is invoked once for every N virtual machine opcodes, +** where N is the second argument to this function. The progress callback +** itself is identified by the third argument to this function. The fourth +** argument to this function is a void pointer passed to the progress callback +** function each time it is invoked. +** +** If a call to sqlite3_exec(), sqlite3_step() or sqlite3_get_table() results +** in less than N opcodes being executed, then the progress callback is not +** invoked. +** +** To remove the progress callback altogether, pass NULL as the third +** argument to this function. +** +** If the progress callback returns a result other than 0, then the current +** query is immediately terminated and any database changes rolled back. If the +** query was part of a larger transaction, then the transaction is not rolled +** back and remains active. The sqlite3_exec() call returns SQLITE_ABORT. +** +******* THIS IS AN EXPERIMENTAL API AND IS SUBJECT TO CHANGE ****** +*/ +void sqlite3_progress_handler(sqlite3*, int, int(*)(void*), void*); + +/* +** Register a callback function to be invoked whenever a new transaction +** is committed. The pArg argument is passed through to the callback. +** callback. If the callback function returns non-zero, then the commit +** is converted into a rollback. +** +** If another function was previously registered, its pArg value is returned. +** Otherwise NULL is returned. +** +** Registering a NULL function disables the callback. +** +******* THIS IS AN EXPERIMENTAL API AND IS SUBJECT TO CHANGE ****** +*/ +void *sqlite3_commit_hook(sqlite3*, int(*)(void*), void*); + +/* +** Open the sqlite database file "filename". The "filename" is UTF-8 +** encoded for sqlite3_open() and UTF-16 encoded in the native byte order +** for sqlite3_open16(). An sqlite3* handle is returned in *ppDb, even +** if an error occurs. If the database is opened (or created) successfully, +** then SQLITE_OK is returned. Otherwise an error code is returned. The +** sqlite3_errmsg() or sqlite3_errmsg16() routines can be used to obtain +** an English language description of the error. +** +** If the database file does not exist, then a new database is created. +** The encoding for the database is UTF-8 if sqlite3_open() is called and +** UTF-16 if sqlite3_open16 is used. +** +** Whether or not an error occurs when it is opened, resources associated +** with the sqlite3* handle should be released by passing it to +** sqlite3_close() when it is no longer required. +*/ +int sqlite3_open( + const char *filename, /* Database filename (UTF-8) */ + sqlite3 **ppDb /* OUT: SQLite db handle */ +); +int sqlite3_open16( + const void *filename, /* Database filename (UTF-16) */ + sqlite3 **ppDb /* OUT: SQLite db handle */ +); + +/* +** Return the error code for the most recent sqlite3_* API call associated +** with sqlite3 handle 'db'. SQLITE_OK is returned if the most recent +** API call was successful. +** +** Calls to many sqlite3_* functions set the error code and string returned +** by sqlite3_errcode(), sqlite3_errmsg() and sqlite3_errmsg16() +** (overwriting the previous values). Note that calls to sqlite3_errcode(), +** sqlite3_errmsg() and sqlite3_errmsg16() themselves do not affect the +** results of future invocations. +** +** Assuming no other intervening sqlite3_* API calls are made, the error +** code returned by this function is associated with the same error as +** the strings returned by sqlite3_errmsg() and sqlite3_errmsg16(). +*/ +int sqlite3_errcode(sqlite3 *db); + +/* +** Return a pointer to a UTF-8 encoded string describing in english the +** error condition for the most recent sqlite3_* API call. The returned +** string is always terminated by an 0x00 byte. +** +** The string "not an error" is returned when the most recent API call was +** successful. +*/ +const char *sqlite3_errmsg(sqlite3*); + +/* +** Return a pointer to a UTF-16 native byte order encoded string describing +** in english the error condition for the most recent sqlite3_* API call. +** The returned string is always terminated by a pair of 0x00 bytes. +** +** The string "not an error" is returned when the most recent API call was +** successful. +*/ +const void *sqlite3_errmsg16(sqlite3*); + +/* +** An instance of the following opaque structure is used to represent +** a compiled SQL statment. +*/ +typedef struct sqlite3_stmt sqlite3_stmt; + +/* +** To execute an SQL query, it must first be compiled into a byte-code +** program using one of the following routines. The only difference between +** them is that the second argument, specifying the SQL statement to +** compile, is assumed to be encoded in UTF-8 for the sqlite3_prepare() +** function and UTF-16 for sqlite3_prepare16(). +** +** The first parameter "db" is an SQLite database handle. The second +** parameter "zSql" is the statement to be compiled, encoded as either +** UTF-8 or UTF-16 (see above). If the next parameter, "nBytes", is less +** than zero, then zSql is read up to the first nul terminator. If +** "nBytes" is not less than zero, then it is the length of the string zSql +** in bytes (not characters). +** +** *pzTail is made to point to the first byte past the end of the first +** SQL statement in zSql. This routine only compiles the first statement +** in zSql, so *pzTail is left pointing to what remains uncompiled. +** +** *ppStmt is left pointing to a compiled SQL statement that can be +** executed using sqlite3_step(). Or if there is an error, *ppStmt may be +** set to NULL. If the input text contained no SQL (if the input is and +** empty string or a comment) then *ppStmt is set to NULL. +** +** On success, SQLITE_OK is returned. Otherwise an error code is returned. +*/ +int sqlite3_prepare( + sqlite3 *db, /* Database handle */ + const char *zSql, /* SQL statement, UTF-8 encoded */ + int nBytes, /* Length of zSql in bytes. */ + sqlite3_stmt **ppStmt, /* OUT: Statement handle */ + const char **pzTail /* OUT: Pointer to unused portion of zSql */ +); +int sqlite3_prepare16( + sqlite3 *db, /* Database handle */ + const void *zSql, /* SQL statement, UTF-16 encoded */ + int nBytes, /* Length of zSql in bytes. */ + sqlite3_stmt **ppStmt, /* OUT: Statement handle */ + const void **pzTail /* OUT: Pointer to unused portion of zSql */ +); + +/* +** Pointers to the following two opaque structures are used to communicate +** with the implementations of user-defined functions. +*/ +typedef struct sqlite3_context sqlite3_context; +typedef struct Mem sqlite3_value; + +/* +** In the SQL strings input to sqlite3_prepare() and sqlite3_prepare16(), +** one or more literals can be replace by parameters "?" or ":AAA" or +** "$VVV" where AAA is an identifer and VVV is a variable name according +** to the syntax rules of the TCL programming language. +** The value of these parameters (also called "host parameter names") can +** be set using the routines listed below. +** +** In every case, the first parameter is a pointer to the sqlite3_stmt +** structure returned from sqlite3_prepare(). The second parameter is the +** index of the parameter. The first parameter as an index of 1. For +** named parameters (":AAA" or "$VVV") you can use +** sqlite3_bind_parameter_index() to get the correct index value given +** the parameters name. If the same named parameter occurs more than +** once, it is assigned the same index each time. +** +** The fifth parameter to sqlite3_bind_blob(), sqlite3_bind_text(), and +** sqlite3_bind_text16() is a destructor used to dispose of the BLOB or +** text after SQLite has finished with it. If the fifth argument is the +** special value SQLITE_STATIC, then the library assumes that the information +** is in static, unmanaged space and does not need to be freed. If the +** fifth argument has the value SQLITE_TRANSIENT, then SQLite makes its +** own private copy of the data. +** +** The sqlite3_bind_* routine must be called before sqlite3_step() after +** an sqlite3_prepare() or sqlite3_reset(). Unbound parameterss are +** interpreted as NULL. +*/ +int sqlite3_bind_blob(sqlite3_stmt*, int, const void*, int n, void(*)(void*)); +int sqlite3_bind_double(sqlite3_stmt*, int, double); +int sqlite3_bind_int(sqlite3_stmt*, int, int); +int sqlite3_bind_int64(sqlite3_stmt*, int, sqlite_int64); +int sqlite3_bind_null(sqlite3_stmt*, int); +int sqlite3_bind_text(sqlite3_stmt*, int, const char*, int n, void(*)(void*)); +int sqlite3_bind_text16(sqlite3_stmt*, int, const void*, int, void(*)(void*)); +int sqlite3_bind_value(sqlite3_stmt*, int, const sqlite3_value*); + +/* +** Return the number of parameters in a compiled SQL statement. This +** routine was added to support DBD::SQLite. +*/ +int sqlite3_bind_parameter_count(sqlite3_stmt*); + +/* +** Return the name of the i-th parameter. Ordinary parameters "?" are +** nameless and a NULL is returned. For parameters of the form :AAA or +** $VVV the complete text of the parameter name is returned, including +** the initial ":" or "$". NULL is returned if the index is out of range. +*/ +const char *sqlite3_bind_parameter_name(sqlite3_stmt*, int); + +/* +** Return the index of a parameter with the given name. The name +** must match exactly. If no parameter with the given name is found, +** return 0. +*/ +int sqlite3_bind_parameter_index(sqlite3_stmt*, const char *zName); + +/* +** Set all the parameters in the compiled SQL statement to NULL. +*/ +int sqlite3_clear_bindings(sqlite3_stmt*); + +/* +** Return the number of columns in the result set returned by the compiled +** SQL statement. This routine returns 0 if pStmt is an SQL statement +** that does not return data (for example an UPDATE). +*/ +int sqlite3_column_count(sqlite3_stmt *pStmt); + +/* +** The first parameter is a compiled SQL statement. This function returns +** the column heading for the Nth column of that statement, where N is the +** second function parameter. The string returned is UTF-8 for +** sqlite3_column_name() and UTF-16 for sqlite3_column_name16(). +*/ +const char *sqlite3_column_name(sqlite3_stmt*,int); +const void *sqlite3_column_name16(sqlite3_stmt*,int); + +/* +** The first parameter to the following calls is a compiled SQL statement. +** These functions return information about the Nth column returned by +** the statement, where N is the second function argument. +** +** If the Nth column returned by the statement is not a column value, +** then all of the functions return NULL. Otherwise, the return the +** name of the attached database, table and column that the expression +** extracts a value from. +** +** As with all other SQLite APIs, those postfixed with "16" return UTF-16 +** encoded strings, the other functions return UTF-8. The memory containing +** the returned strings is valid until the statement handle is finalized(). +** +** These APIs are only available if the library was compiled with the +** SQLITE_ENABLE_COLUMN_METADATA preprocessor symbol defined. +*/ +const char *sqlite3_column_database_name(sqlite3_stmt*,int); +const void *sqlite3_column_database_name16(sqlite3_stmt*,int); +const char *sqlite3_column_table_name(sqlite3_stmt*,int); +const void *sqlite3_column_table_name16(sqlite3_stmt*,int); +const char *sqlite3_column_origin_name(sqlite3_stmt*,int); +const void *sqlite3_column_origin_name16(sqlite3_stmt*,int); + +/* +** The first parameter is a compiled SQL statement. If this statement +** is a SELECT statement, the Nth column of the returned result set +** of the SELECT is a table column then the declared type of the table +** column is returned. If the Nth column of the result set is not at table +** column, then a NULL pointer is returned. The returned string is always +** UTF-8 encoded. For example, in the database schema: +** +** CREATE TABLE t1(c1 VARIANT); +** +** And the following statement compiled: +** +** SELECT c1 + 1, c1 FROM t1; +** +** Then this routine would return the string "VARIANT" for the second +** result column (i==1), and a NULL pointer for the first result column +** (i==0). +*/ +const char *sqlite3_column_decltype(sqlite3_stmt *, int i); + +/* +** The first parameter is a compiled SQL statement. If this statement +** is a SELECT statement, the Nth column of the returned result set +** of the SELECT is a table column then the declared type of the table +** column is returned. If the Nth column of the result set is not at table +** column, then a NULL pointer is returned. The returned string is always +** UTF-16 encoded. For example, in the database schema: +** +** CREATE TABLE t1(c1 INTEGER); +** +** And the following statement compiled: +** +** SELECT c1 + 1, c1 FROM t1; +** +** Then this routine would return the string "INTEGER" for the second +** result column (i==1), and a NULL pointer for the first result column +** (i==0). +*/ +const void *sqlite3_column_decltype16(sqlite3_stmt*,int); + +/* +** After an SQL query has been compiled with a call to either +** sqlite3_prepare() or sqlite3_prepare16(), then this function must be +** called one or more times to execute the statement. +** +** The return value will be either SQLITE_BUSY, SQLITE_DONE, +** SQLITE_ROW, SQLITE_ERROR, or SQLITE_MISUSE. +** +** SQLITE_BUSY means that the database engine attempted to open +** a locked database and there is no busy callback registered. +** Call sqlite3_step() again to retry the open. +** +** SQLITE_DONE means that the statement has finished executing +** successfully. sqlite3_step() should not be called again on this virtual +** machine. +** +** If the SQL statement being executed returns any data, then +** SQLITE_ROW is returned each time a new row of data is ready +** for processing by the caller. The values may be accessed using +** the sqlite3_column_*() functions described below. sqlite3_step() +** is called again to retrieve the next row of data. +** +** SQLITE_ERROR means that a run-time error (such as a constraint +** violation) has occurred. sqlite3_step() should not be called again on +** the VM. More information may be found by calling sqlite3_errmsg(). +** +** SQLITE_MISUSE means that the this routine was called inappropriately. +** Perhaps it was called on a virtual machine that had already been +** finalized or on one that had previously returned SQLITE_ERROR or +** SQLITE_DONE. Or it could be the case the the same database connection +** is being used simulataneously by two or more threads. +*/ +int sqlite3_step(sqlite3_stmt*); + +/* +** Return the number of values in the current row of the result set. +** +** After a call to sqlite3_step() that returns SQLITE_ROW, this routine +** will return the same value as the sqlite3_column_count() function. +** After sqlite3_step() has returned an SQLITE_DONE, SQLITE_BUSY or +** error code, or before sqlite3_step() has been called on a +** compiled SQL statement, this routine returns zero. +*/ +int sqlite3_data_count(sqlite3_stmt *pStmt); + +/* +** Values are stored in the database in one of the following fundamental +** types. +*/ +#define SQLITE_INTEGER 1 +#define SQLITE_FLOAT 2 +/* #define SQLITE_TEXT 3 // See below */ +#define SQLITE_BLOB 4 +#define SQLITE_NULL 5 + +/* +** SQLite version 2 defines SQLITE_TEXT differently. To allow both +** version 2 and version 3 to be included, undefine them both if a +** conflict is seen. Define SQLITE3_TEXT to be the version 3 value. +*/ +#ifdef SQLITE_TEXT +# undef SQLITE_TEXT +#else +# define SQLITE_TEXT 3 +#endif +#define SQLITE3_TEXT 3 + +/* +** The next group of routines returns information about the information +** in a single column of the current result row of a query. In every +** case the first parameter is a pointer to the SQL statement that is being +** executed (the sqlite_stmt* that was returned from sqlite3_prepare()) and +** the second argument is the index of the column for which information +** should be returned. iCol is zero-indexed. The left-most column as an +** index of 0. +** +** If the SQL statement is not currently point to a valid row, or if the +** the colulmn index is out of range, the result is undefined. +** +** These routines attempt to convert the value where appropriate. For +** example, if the internal representation is FLOAT and a text result +** is requested, sprintf() is used internally to do the conversion +** automatically. The following table details the conversions that +** are applied: +** +** Internal Type Requested Type Conversion +** ------------- -------------- -------------------------- +** NULL INTEGER Result is 0 +** NULL FLOAT Result is 0.0 +** NULL TEXT Result is an empty string +** NULL BLOB Result is a zero-length BLOB +** INTEGER FLOAT Convert from integer to float +** INTEGER TEXT ASCII rendering of the integer +** INTEGER BLOB Same as for INTEGER->TEXT +** FLOAT INTEGER Convert from float to integer +** FLOAT TEXT ASCII rendering of the float +** FLOAT BLOB Same as FLOAT->TEXT +** TEXT INTEGER Use atoi() +** TEXT FLOAT Use atof() +** TEXT BLOB No change +** BLOB INTEGER Convert to TEXT then use atoi() +** BLOB FLOAT Convert to TEXT then use atof() +** BLOB TEXT Add a \000 terminator if needed +** +** The following access routines are provided: +** +** _type() Return the datatype of the result. This is one of +** SQLITE_INTEGER, SQLITE_FLOAT, SQLITE_TEXT, SQLITE_BLOB, +** or SQLITE_NULL. +** _blob() Return the value of a BLOB. +** _bytes() Return the number of bytes in a BLOB value or the number +** of bytes in a TEXT value represented as UTF-8. The \000 +** terminator is included in the byte count for TEXT values. +** _bytes16() Return the number of bytes in a BLOB value or the number +** of bytes in a TEXT value represented as UTF-16. The \u0000 +** terminator is included in the byte count for TEXT values. +** _double() Return a FLOAT value. +** _int() Return an INTEGER value in the host computer's native +** integer representation. This might be either a 32- or 64-bit +** integer depending on the host. +** _int64() Return an INTEGER value as a 64-bit signed integer. +** _text() Return the value as UTF-8 text. +** _text16() Return the value as UTF-16 text. +*/ +const void *sqlite3_column_blob(sqlite3_stmt*, int iCol); +int sqlite3_column_bytes(sqlite3_stmt*, int iCol); +int sqlite3_column_bytes16(sqlite3_stmt*, int iCol); +double sqlite3_column_double(sqlite3_stmt*, int iCol); +int sqlite3_column_int(sqlite3_stmt*, int iCol); +sqlite_int64 sqlite3_column_int64(sqlite3_stmt*, int iCol); +const unsigned char *sqlite3_column_text(sqlite3_stmt*, int iCol); +const void *sqlite3_column_text16(sqlite3_stmt*, int iCol); +int sqlite3_column_type(sqlite3_stmt*, int iCol); +int sqlite3_column_numeric_type(sqlite3_stmt*, int iCol); + +/* +** The sqlite3_finalize() function is called to delete a compiled +** SQL statement obtained by a previous call to sqlite3_prepare() +** or sqlite3_prepare16(). If the statement was executed successfully, or +** not executed at all, then SQLITE_OK is returned. If execution of the +** statement failed then an error code is returned. +** +** This routine can be called at any point during the execution of the +** virtual machine. If the virtual machine has not completed execution +** when this routine is called, that is like encountering an error or +** an interrupt. (See sqlite3_interrupt().) Incomplete updates may be +** rolled back and transactions cancelled, depending on the circumstances, +** and the result code returned will be SQLITE_ABORT. +*/ +int sqlite3_finalize(sqlite3_stmt *pStmt); + +/* +** The sqlite3_reset() function is called to reset a compiled SQL +** statement obtained by a previous call to sqlite3_prepare() or +** sqlite3_prepare16() back to it's initial state, ready to be re-executed. +** Any SQL statement variables that had values bound to them using +** the sqlite3_bind_*() API retain their values. +*/ +int sqlite3_reset(sqlite3_stmt *pStmt); + +/* +** The following two functions are used to add user functions or aggregates +** implemented in C to the SQL langauge interpreted by SQLite. The +** difference only between the two is that the second parameter, the +** name of the (scalar) function or aggregate, is encoded in UTF-8 for +** sqlite3_create_function() and UTF-16 for sqlite3_create_function16(). +** +** The first argument is the database handle that the new function or +** aggregate is to be added to. If a single program uses more than one +** database handle internally, then user functions or aggregates must +** be added individually to each database handle with which they will be +** used. +** +** The third parameter is the number of arguments that the function or +** aggregate takes. If this parameter is negative, then the function or +** aggregate may take any number of arguments. +** +** The fourth parameter is one of SQLITE_UTF* values defined below, +** indicating the encoding that the function is most likely to handle +** values in. This does not change the behaviour of the programming +** interface. However, if two versions of the same function are registered +** with different encoding values, SQLite invokes the version likely to +** minimize conversions between text encodings. +** +** The seventh, eighth and ninth parameters, xFunc, xStep and xFinal, are +** pointers to user implemented C functions that implement the user +** function or aggregate. A scalar function requires an implementation of +** the xFunc callback only, NULL pointers should be passed as the xStep +** and xFinal parameters. An aggregate function requires an implementation +** of xStep and xFinal, but NULL should be passed for xFunc. To delete an +** existing user function or aggregate, pass NULL for all three function +** callback. Specifying an inconstent set of callback values, such as an +** xFunc and an xFinal, or an xStep but no xFinal, SQLITE_ERROR is +** returned. +*/ +int sqlite3_create_function( + sqlite3 *, + const char *zFunctionName, + int nArg, + int eTextRep, + void*, + void (*xFunc)(sqlite3_context*,int,sqlite3_value**), + void (*xStep)(sqlite3_context*,int,sqlite3_value**), + void (*xFinal)(sqlite3_context*) +); +int sqlite3_create_function16( + sqlite3*, + const void *zFunctionName, + int nArg, + int eTextRep, + void*, + void (*xFunc)(sqlite3_context*,int,sqlite3_value**), + void (*xStep)(sqlite3_context*,int,sqlite3_value**), + void (*xFinal)(sqlite3_context*) +); + +/* +** This function is deprecated. Do not use it. It continues to exist +** so as not to break legacy code. But new code should avoid using it. +*/ +int sqlite3_aggregate_count(sqlite3_context*); + +/* +** The next group of routines returns information about parameters to +** a user-defined function. Function implementations use these routines +** to access their parameters. These routines are the same as the +** sqlite3_column_* routines except that these routines take a single +** sqlite3_value* pointer instead of an sqlite3_stmt* and an integer +** column number. +*/ +const void *sqlite3_value_blob(sqlite3_value*); +int sqlite3_value_bytes(sqlite3_value*); +int sqlite3_value_bytes16(sqlite3_value*); +double sqlite3_value_double(sqlite3_value*); +int sqlite3_value_int(sqlite3_value*); +sqlite_int64 sqlite3_value_int64(sqlite3_value*); +const unsigned char *sqlite3_value_text(sqlite3_value*); +const void *sqlite3_value_text16(sqlite3_value*); +const void *sqlite3_value_text16le(sqlite3_value*); +const void *sqlite3_value_text16be(sqlite3_value*); +int sqlite3_value_type(sqlite3_value*); +int sqlite3_value_numeric_type(sqlite3_value*); + +/* +** Aggregate functions use the following routine to allocate +** a structure for storing their state. The first time this routine +** is called for a particular aggregate, a new structure of size nBytes +** is allocated, zeroed, and returned. On subsequent calls (for the +** same aggregate instance) the same buffer is returned. The implementation +** of the aggregate can use the returned buffer to accumulate data. +** +** The buffer allocated is freed automatically by SQLite. +*/ +void *sqlite3_aggregate_context(sqlite3_context*, int nBytes); + +/* +** The pUserData parameter to the sqlite3_create_function() +** routine used to register user functions is available to +** the implementation of the function using this call. +*/ +void *sqlite3_user_data(sqlite3_context*); + +/* +** The following two functions may be used by scalar user functions to +** associate meta-data with argument values. If the same value is passed to +** multiple invocations of the user-function during query execution, under +** some circumstances the associated meta-data may be preserved. This may +** be used, for example, to add a regular-expression matching scalar +** function. The compiled version of the regular expression is stored as +** meta-data associated with the SQL value passed as the regular expression +** pattern. +** +** Calling sqlite3_get_auxdata() returns a pointer to the meta data +** associated with the Nth argument value to the current user function +** call, where N is the second parameter. If no meta-data has been set for +** that value, then a NULL pointer is returned. +** +** The sqlite3_set_auxdata() is used to associate meta data with a user +** function argument. The third parameter is a pointer to the meta data +** to be associated with the Nth user function argument value. The fourth +** parameter specifies a 'delete function' that will be called on the meta +** data pointer to release it when it is no longer required. If the delete +** function pointer is NULL, it is not invoked. +** +** In practice, meta-data is preserved between function calls for +** expressions that are constant at compile time. This includes literal +** values and SQL variables. +*/ +void *sqlite3_get_auxdata(sqlite3_context*, int); +void sqlite3_set_auxdata(sqlite3_context*, int, void*, void (*)(void*)); + + +/* +** These are special value for the destructor that is passed in as the +** final argument to routines like sqlite3_result_blob(). If the destructor +** argument is SQLITE_STATIC, it means that the content pointer is constant +** and will never change. It does not need to be destroyed. The +** SQLITE_TRANSIENT value means that the content will likely change in +** the near future and that SQLite should make its own private copy of +** the content before returning. +*/ +#define SQLITE_STATIC ((void(*)(void *))0) +#define SQLITE_TRANSIENT ((void(*)(void *))-1) + +/* +** User-defined functions invoke the following routines in order to +** set their return value. +*/ +void sqlite3_result_blob(sqlite3_context*, const void*, int, void(*)(void*)); +void sqlite3_result_double(sqlite3_context*, double); +void sqlite3_result_error(sqlite3_context*, const char*, int); +void sqlite3_result_error16(sqlite3_context*, const void*, int); +void sqlite3_result_int(sqlite3_context*, int); +void sqlite3_result_int64(sqlite3_context*, sqlite_int64); +void sqlite3_result_null(sqlite3_context*); +void sqlite3_result_text(sqlite3_context*, const char*, int, void(*)(void*)); +void sqlite3_result_text16(sqlite3_context*, const void*, int, void(*)(void*)); +void sqlite3_result_text16le(sqlite3_context*, const void*, int,void(*)(void*)); +void sqlite3_result_text16be(sqlite3_context*, const void*, int,void(*)(void*)); +void sqlite3_result_value(sqlite3_context*, sqlite3_value*); + +/* +** These are the allowed values for the eTextRep argument to +** sqlite3_create_collation and sqlite3_create_function. +*/ +#define SQLITE_UTF8 1 +#define SQLITE_UTF16LE 2 +#define SQLITE_UTF16BE 3 +#define SQLITE_UTF16 4 /* Use native byte order */ +#define SQLITE_ANY 5 /* sqlite3_create_function only */ + +/* +** These two functions are used to add new collation sequences to the +** sqlite3 handle specified as the first argument. +** +** The name of the new collation sequence is specified as a UTF-8 string +** for sqlite3_create_collation() and a UTF-16 string for +** sqlite3_create_collation16(). In both cases the name is passed as the +** second function argument. +** +** The third argument must be one of the constants SQLITE_UTF8, +** SQLITE_UTF16LE or SQLITE_UTF16BE, indicating that the user-supplied +** routine expects to be passed pointers to strings encoded using UTF-8, +** UTF-16 little-endian or UTF-16 big-endian respectively. +** +** A pointer to the user supplied routine must be passed as the fifth +** argument. If it is NULL, this is the same as deleting the collation +** sequence (so that SQLite cannot call it anymore). Each time the user +** supplied function is invoked, it is passed a copy of the void* passed as +** the fourth argument to sqlite3_create_collation() or +** sqlite3_create_collation16() as its first parameter. +** +** The remaining arguments to the user-supplied routine are two strings, +** each represented by a [length, data] pair and encoded in the encoding +** that was passed as the third argument when the collation sequence was +** registered. The user routine should return negative, zero or positive if +** the first string is less than, equal to, or greater than the second +** string. i.e. (STRING1 - STRING2). +*/ +int sqlite3_create_collation( + sqlite3*, + const char *zName, + int eTextRep, + void*, + int(*xCompare)(void*,int,const void*,int,const void*) +); +int sqlite3_create_collation16( + sqlite3*, + const char *zName, + int eTextRep, + void*, + int(*xCompare)(void*,int,const void*,int,const void*) +); + +/* +** To avoid having to register all collation sequences before a database +** can be used, a single callback function may be registered with the +** database handle to be called whenever an undefined collation sequence is +** required. +** +** If the function is registered using the sqlite3_collation_needed() API, +** then it is passed the names of undefined collation sequences as strings +** encoded in UTF-8. If sqlite3_collation_needed16() is used, the names +** are passed as UTF-16 in machine native byte order. A call to either +** function replaces any existing callback. +** +** When the user-function is invoked, the first argument passed is a copy +** of the second argument to sqlite3_collation_needed() or +** sqlite3_collation_needed16(). The second argument is the database +** handle. The third argument is one of SQLITE_UTF8, SQLITE_UTF16BE or +** SQLITE_UTF16LE, indicating the most desirable form of the collation +** sequence function required. The fourth parameter is the name of the +** required collation sequence. +** +** The collation sequence is returned to SQLite by a collation-needed +** callback using the sqlite3_create_collation() or +** sqlite3_create_collation16() APIs, described above. +*/ +int sqlite3_collation_needed( + sqlite3*, + void*, + void(*)(void*,sqlite3*,int eTextRep,const char*) +); +int sqlite3_collation_needed16( + sqlite3*, + void*, + void(*)(void*,sqlite3*,int eTextRep,const void*) +); + +/* +** Specify the key for an encrypted database. This routine should be +** called right after sqlite3_open(). +** +** The code to implement this API is not available in the public release +** of SQLite. +*/ +int sqlite3_key( + sqlite3 *db, /* Database to be rekeyed */ + const void *pKey, int nKey /* The key */ +); + +/* +** Change the key on an open database. If the current database is not +** encrypted, this routine will encrypt it. If pNew==0 or nNew==0, the +** database is decrypted. +** +** The code to implement this API is not available in the public release +** of SQLite. +*/ +int sqlite3_rekey( + sqlite3 *db, /* Database to be rekeyed */ + const void *pKey, int nKey /* The new key */ +); + +/* +** Sleep for a little while. The second parameter is the number of +** miliseconds to sleep for. +** +** If the operating system does not support sleep requests with +** milisecond time resolution, then the time will be rounded up to +** the nearest second. The number of miliseconds of sleep actually +** requested from the operating system is returned. +*/ +int sqlite3_sleep(int); + +/* +** Return TRUE (non-zero) if the statement supplied as an argument needs +** to be recompiled. A statement needs to be recompiled whenever the +** execution environment changes in a way that would alter the program +** that sqlite3_prepare() generates. For example, if new functions or +** collating sequences are registered or if an authorizer function is +** added or changed. +** +*/ +int sqlite3_expired(sqlite3_stmt*); + +/* +** Move all bindings from the first prepared statement over to the second. +** This routine is useful, for example, if the first prepared statement +** fails with an SQLITE_SCHEMA error. The same SQL can be prepared into +** the second prepared statement then all of the bindings transfered over +** to the second statement before the first statement is finalized. +*/ +int sqlite3_transfer_bindings(sqlite3_stmt*, sqlite3_stmt*); + +/* +** If the following global variable is made to point to a +** string which is the name of a directory, then all temporary files +** created by SQLite will be placed in that directory. If this variable +** is NULL pointer, then SQLite does a search for an appropriate temporary +** file directory. +** +** Once sqlite3_open() has been called, changing this variable will invalidate +** the current temporary database, if any. +*/ +extern char *sqlite3_temp_directory; + +/* +** This function is called to recover from a malloc() failure that occured +** within the SQLite library. Normally, after a single malloc() fails the +** library refuses to function (all major calls return SQLITE_NOMEM). +** This function restores the library state so that it can be used again. +** +** All existing statements (sqlite3_stmt pointers) must be finalized or +** reset before this call is made. Otherwise, SQLITE_BUSY is returned. +** If any in-memory databases are in use, either as a main or TEMP +** database, SQLITE_ERROR is returned. In either of these cases, the +** library is not reset and remains unusable. +** +** This function is *not* threadsafe. Calling this from within a threaded +** application when threads other than the caller have used SQLite is +** dangerous and will almost certainly result in malfunctions. +** +** This functionality can be omitted from a build by defining the +** SQLITE_OMIT_GLOBALRECOVER at compile time. +*/ +int sqlite3_global_recover(void); + +/* +** Test to see whether or not the database connection is in autocommit +** mode. Return TRUE if it is and FALSE if not. Autocommit mode is on +** by default. Autocommit is disabled by a BEGIN statement and reenabled +** by the next COMMIT or ROLLBACK. +*/ +int sqlite3_get_autocommit(sqlite3*); + +/* +** Return the sqlite3* database handle to which the prepared statement given +** in the argument belongs. This is the same database handle that was +** the first argument to the sqlite3_prepare() that was used to create +** the statement in the first place. +*/ +sqlite3 *sqlite3_db_handle(sqlite3_stmt*); + +/* +** Register a callback function with the database connection identified by the +** first argument to be invoked whenever a row is updated, inserted or deleted. +** Any callback set by a previous call to this function for the same +** database connection is overridden. +** +** The second argument is a pointer to the function to invoke when a +** row is updated, inserted or deleted. The first argument to the callback is +** a copy of the third argument to sqlite3_update_hook. The second callback +** argument is one of SQLITE_INSERT, SQLITE_DELETE or SQLITE_UPDATE, depending +** on the operation that caused the callback to be invoked. The third and +** fourth arguments to the callback contain pointers to the database and +** table name containing the affected row. The final callback parameter is +** the rowid of the row. In the case of an update, this is the rowid after +** the update takes place. +** +** The update hook is not invoked when internal system tables are +** modified (i.e. sqlite_master and sqlite_sequence). +** +** If another function was previously registered, its pArg value is returned. +** Otherwise NULL is returned. +*/ +void *sqlite3_update_hook( + sqlite3*, + void(*)(void *,int ,char const *,char const *,sqlite_int64), + void* +); + +/* +** Register a callback to be invoked whenever a transaction is rolled +** back. +** +** The new callback function overrides any existing rollback-hook +** callback. If there was an existing callback, then it's pArg value +** (the third argument to sqlite3_rollback_hook() when it was registered) +** is returned. Otherwise, NULL is returned. +** +** For the purposes of this API, a transaction is said to have been +** rolled back if an explicit "ROLLBACK" statement is executed, or +** an error or constraint causes an implicit rollback to occur. The +** callback is not invoked if a transaction is automatically rolled +** back because the database connection is closed. +*/ +void *sqlite3_rollback_hook(sqlite3*, void(*)(void *), void*); + +/* +** This function is only available if the library is compiled without +** the SQLITE_OMIT_SHARED_CACHE macro defined. It is used to enable or +** disable (if the argument is true or false, respectively) the +** "shared pager" feature. +*/ +int sqlite3_enable_shared_cache(int); + +/* +** Attempt to free N bytes of heap memory by deallocating non-essential +** memory allocations held by the database library (example: memory +** used to cache database pages to improve performance). +** +** This function is not a part of standard builds. It is only created +** if SQLite is compiled with the SQLITE_ENABLE_MEMORY_MANAGEMENT macro. +*/ +int sqlite3_release_memory(int); + +/* +** Place a "soft" limit on the amount of heap memory that may be allocated by +** SQLite within the current thread. If an internal allocation is requested +** that would exceed the specified limit, sqlite3_release_memory() is invoked +** one or more times to free up some space before the allocation is made. +** +** The limit is called "soft", because if sqlite3_release_memory() cannot free +** sufficient memory to prevent the limit from being exceeded, the memory is +** allocated anyway and the current operation proceeds. +** +** This function is only available if the library was compiled with the +** SQLITE_ENABLE_MEMORY_MANAGEMENT option set. +** memory-management has been enabled. +*/ +void sqlite3_soft_heap_limit(int); + +/* +** This routine makes sure that all thread-local storage has been +** deallocated for the current thread. +** +** This routine is not technically necessary. All thread-local storage +** will be automatically deallocated once memory-management and +** shared-cache are disabled and the soft heap limit has been set +** to zero. This routine is provided as a convenience for users who +** want to make absolutely sure they have not forgotten something +** prior to killing off a thread. +*/ +void sqlite3_thread_cleanup(void); + +/* +** Return meta information about a specific column of a specific database +** table accessible using the connection handle passed as the first function +** argument. +** +** The column is identified by the second, third and fourth parameters to +** this function. The second parameter is either the name of the database +** (i.e. "main", "temp" or an attached database) containing the specified +** table or NULL. If it is NULL, then all attached databases are searched +** for the table using the same algorithm as the database engine uses to +** resolve unqualified table references. +** +** The third and fourth parameters to this function are the table and column +** name of the desired column, respectively. Neither of these parameters +** may be NULL. +** +** Meta information is returned by writing to the memory locations passed as +** the 5th and subsequent parameters to this function. Any of these +** arguments may be NULL, in which case the corresponding element of meta +** information is ommitted. +** +** Parameter Output Type Description +** ----------------------------------- +** +** 5th const char* Data type +** 6th const char* Name of the default collation sequence +** 7th int True if the column has a NOT NULL constraint +** 8th int True if the column is part of the PRIMARY KEY +** 9th int True if the column is AUTOINCREMENT +** +** +** The memory pointed to by the character pointers returned for the +** declaration type and collation sequence is valid only until the next +** call to any sqlite API function. +** +** If the specified table is actually a view, then an error is returned. +** +** If the specified column is "rowid", "oid" or "_rowid_" and an +** INTEGER PRIMARY KEY column has been explicitly declared, then the output +** parameters are set for the explicitly declared column. If there is no +** explicitly declared IPK column, then the output parameters are set as +** follows: +** +** data type: "INTEGER" +** collation sequence: "BINARY" +** not null: 0 +** primary key: 1 +** auto increment: 0 +** +** This function may load one or more schemas from database files. If an +** error occurs during this process, or if the requested table or column +** cannot be found, an SQLITE error code is returned and an error message +** left in the database handle (to be retrieved using sqlite3_errmsg()). +** +** This API is only available if the library was compiled with the +** SQLITE_ENABLE_COLUMN_METADATA preprocessor symbol defined. +*/ +int sqlite3_table_column_metadata( + sqlite3 *db, /* Connection handle */ + const char *zDbName, /* Database name or NULL */ + const char *zTableName, /* Table name */ + const char *zColumnName, /* Column name */ + char const **pzDataType, /* OUTPUT: Declared data type */ + char const **pzCollSeq, /* OUTPUT: Collation sequence name */ + int *pNotNull, /* OUTPUT: True if NOT NULL constraint exists */ + int *pPrimaryKey, /* OUTPUT: True if column part of PK */ + int *pAutoinc /* OUTPUT: True if colums is auto-increment */ +); + +/* +** Undo the hack that converts floating point types to integer for +** builds on processors without floating point support. +*/ +#ifdef SQLITE_OMIT_FLOATING_POINT +# undef double +#endif + +#ifdef __cplusplus +} /* End of the 'extern "C"' block */ +#endif +#endif Added: external/sqlite-source-3.3.4/sqliteInt.h ============================================================================== --- (empty file) +++ external/sqlite-source-3.3.4/sqliteInt.h Mon Apr 3 07:54:59 2006 @@ -0,0 +1,1774 @@ +/* +** 2001 September 15 +** +** The author disclaims copyright to this source code. In place of +** a legal notice, here is a blessing: +** +** May you do good and not evil. +** May you find forgiveness for yourself and forgive others. +** May you share freely, never taking more than you give. +** +************************************************************************* +** Internal interface definitions for SQLite. +** +** @(#) $Id: sqliteInt.h,v 1.481 2006/02/11 01:25:51 drh Exp $ +*/ +#ifndef _SQLITEINT_H_ +#define _SQLITEINT_H_ + +/* +** Extra interface definitions for those who need them +*/ +#ifdef SQLITE_EXTRA +# include "sqliteExtra.h" +#endif + +/* +** Many people are failing to set -DNDEBUG=1 when compiling SQLite. +** Setting NDEBUG makes the code smaller and run faster. So the following +** lines are added to automatically set NDEBUG unless the -DSQLITE_DEBUG=1 +** option is set. Thus NDEBUG becomes an opt-in rather than an opt-out +** feature. +*/ +#if !defined(NDEBUG) && !defined(SQLITE_DEBUG) +# define NDEBUG 1 +#endif + +/* +** These #defines should enable >2GB file support on Posix if the +** underlying operating system supports it. If the OS lacks +** large file support, or if the OS is windows, these should be no-ops. +** +** Large file support can be disabled using the -DSQLITE_DISABLE_LFS switch +** on the compiler command line. This is necessary if you are compiling +** on a recent machine (ex: RedHat 7.2) but you want your code to work +** on an older machine (ex: RedHat 6.0). If you compile on RedHat 7.2 +** without this option, LFS is enable. But LFS does not exist in the kernel +** in RedHat 6.0, so the code won't work. Hence, for maximum binary +** portability you should omit LFS. +** +** Similar is true for MacOS. LFS is only supported on MacOS 9 and later. +*/ +#ifndef SQLITE_DISABLE_LFS +# define _LARGE_FILE 1 +# ifndef _FILE_OFFSET_BITS +# define _FILE_OFFSET_BITS 64 +# endif +# define _LARGEFILE_SOURCE 1 +#endif + +#include "sqlite3.h" +#include "hash.h" +#include "parse.h" +#include +#include +#include +#include +#include + +/* +** If compiling for a processor that lacks floating point support, +** substitute integer for floating-point +*/ +#ifdef SQLITE_OMIT_FLOATING_POINT +# define double sqlite_int64 +# define LONGDOUBLE_TYPE sqlite_int64 +# ifndef SQLITE_BIG_DBL +# define SQLITE_BIG_DBL (0x7fffffffffffffff) +# endif +# define SQLITE_OMIT_DATETIME_FUNCS 1 +# define SQLITE_OMIT_TRACE 1 +#endif +#ifndef SQLITE_BIG_DBL +# define SQLITE_BIG_DBL (1e99) +#endif + +/* +** The maximum number of in-memory pages to use for the main database +** table and for temporary tables. Internally, the MAX_PAGES and +** TEMP_PAGES macros are used. To override the default values at +** compilation time, the SQLITE_DEFAULT_CACHE_SIZE and +** SQLITE_DEFAULT_TEMP_CACHE_SIZE macros should be set. +*/ +#ifdef SQLITE_DEFAULT_CACHE_SIZE +# define MAX_PAGES SQLITE_DEFAULT_CACHE_SIZE +#else +# define MAX_PAGES 2000 +#endif +#ifdef SQLITE_DEFAULT_TEMP_CACHE_SIZE +# define TEMP_PAGES SQLITE_DEFAULT_TEMP_CACHE_SIZE +#else +# define TEMP_PAGES 500 +#endif + +/* +** OMIT_TEMPDB is set to 1 if SQLITE_OMIT_TEMPDB is defined, or 0 +** afterward. Having this macro allows us to cause the C compiler +** to omit code used by TEMP tables without messy #ifndef statements. +*/ +#ifdef SQLITE_OMIT_TEMPDB +#define OMIT_TEMPDB 1 +#else +#define OMIT_TEMPDB 0 +#endif + +/* +** If the following macro is set to 1, then NULL values are considered +** distinct for the SELECT DISTINCT statement and for UNION or EXCEPT +** compound queries. No other SQL database engine (among those tested) +** works this way except for OCELOT. But the SQL92 spec implies that +** this is how things should work. +** +** If the following macro is set to 0, then NULLs are indistinct for +** SELECT DISTINCT and for UNION. +*/ +#define NULL_ALWAYS_DISTINCT 0 + +/* +** If the following macro is set to 1, then NULL values are considered +** distinct when determining whether or not two entries are the same +** in a UNIQUE index. This is the way PostgreSQL, Oracle, DB2, MySQL, +** OCELOT, and Firebird all work. The SQL92 spec explicitly says this +** is the way things are suppose to work. +** +** If the following macro is set to 0, the NULLs are indistinct for +** a UNIQUE index. In this mode, you can only have a single NULL entry +** for a column declared UNIQUE. This is the way Informix and SQL Server +** work. +*/ +#define NULL_DISTINCT_FOR_UNIQUE 1 + +/* +** The maximum number of attached databases. This must be at least 2 +** in order to support the main database file (0) and the file used to +** hold temporary tables (1). And it must be less than 32 because +** we use a bitmask of databases with a u32 in places (for example +** the Parse.cookieMask field). +*/ +#define MAX_ATTACHED 10 + +/* +** The maximum value of a ?nnn wildcard that the parser will accept. +*/ +#define SQLITE_MAX_VARIABLE_NUMBER 999 + +/* +** The "file format" number is an integer that is incremented whenever +** the VDBE-level file format changes. The following macros define the +** the default file format for new databases and the maximum file format +** that the library can read. +*/ +#define SQLITE_MAX_FILE_FORMAT 4 +#ifndef SQLITE_DEFAULT_FILE_FORMAT +# define SQLITE_DEFAULT_FILE_FORMAT 4 +#endif + +/* +** Provide a default value for TEMP_STORE in case it is not specified +** on the command-line +*/ +#ifndef TEMP_STORE +# define TEMP_STORE 1 +#endif + +/* +** GCC does not define the offsetof() macro so we'll have to do it +** ourselves. +*/ +#ifndef offsetof +#define offsetof(STRUCTURE,FIELD) ((int)((char*)&((STRUCTURE*)0)->FIELD)) +#endif + +/* +** Integers of known sizes. These typedefs might change for architectures +** where the sizes very. Preprocessor macros are available so that the +** types can be conveniently redefined at compile-type. Like this: +** +** cc '-DUINTPTR_TYPE=long long int' ... +*/ +#ifndef UINT64_TYPE +# if defined(_MSC_VER) || defined(__BORLANDC__) +# define UINT64_TYPE unsigned __int64 +# else +# define UINT64_TYPE unsigned long long int +# endif +#endif +#ifndef UINT32_TYPE +# define UINT32_TYPE unsigned int +#endif +#ifndef UINT16_TYPE +# define UINT16_TYPE unsigned short int +#endif +#ifndef INT16_TYPE +# define INT16_TYPE short int +#endif +#ifndef UINT8_TYPE +# define UINT8_TYPE unsigned char +#endif +#ifndef INT8_TYPE +# define INT8_TYPE signed char +#endif +#ifndef LONGDOUBLE_TYPE +# define LONGDOUBLE_TYPE long double +#endif +typedef sqlite_int64 i64; /* 8-byte signed integer */ +typedef UINT64_TYPE u64; /* 8-byte unsigned integer */ +typedef UINT32_TYPE u32; /* 4-byte unsigned integer */ +typedef UINT16_TYPE u16; /* 2-byte unsigned integer */ +typedef INT16_TYPE i16; /* 2-byte signed integer */ +typedef UINT8_TYPE u8; /* 1-byte unsigned integer */ +typedef UINT8_TYPE i8; /* 1-byte signed integer */ + +/* +** Macros to determine whether the machine is big or little endian, +** evaluated at runtime. +*/ +extern const int sqlite3one; +#define SQLITE_BIGENDIAN (*(char *)(&sqlite3one)==0) +#define SQLITE_LITTLEENDIAN (*(char *)(&sqlite3one)==1) + +/* +** An instance of the following structure is used to store the busy-handler +** callback for a given sqlite handle. +** +** The sqlite.busyHandler member of the sqlite struct contains the busy +** callback for the database handle. Each pager opened via the sqlite +** handle is passed a pointer to sqlite.busyHandler. The busy-handler +** callback is currently invoked only from within pager.c. +*/ +typedef struct BusyHandler BusyHandler; +struct BusyHandler { + int (*xFunc)(void *,int); /* The busy callback */ + void *pArg; /* First arg to busy callback */ + int nBusy; /* Incremented with each busy call */ +}; + +/* +** Defer sourcing vdbe.h and btree.h until after the "u8" and +** "BusyHandler typedefs. +*/ +#include "vdbe.h" +#include "btree.h" +#include "pager.h" + +/* +** This macro casts a pointer to an integer. Useful for doing +** pointer arithmetic. +*/ +#define Addr(X) ((uptr)X) + +#ifdef SQLITE_MEMDEBUG +/* +** The following global variables are used for testing and debugging +** only. They only work if SQLITE_MEMDEBUG is defined. +*/ +extern int sqlite3_nMalloc; /* Number of sqliteMalloc() calls */ +extern int sqlite3_nFree; /* Number of sqliteFree() calls */ +extern int sqlite3_iMallocFail; /* Fail sqliteMalloc() after this many calls */ +extern int sqlite3_iMallocReset; /* Set iMallocFail to this when it reaches 0 */ + + +extern void *sqlite3_pFirst; /* Pointer to linked list of allocations */ +extern int sqlite3_nMaxAlloc; /* High water mark of ThreadData.nAlloc */ +extern int sqlite3_mallocDisallowed; /* assert() in sqlite3Malloc() if set */ +extern int sqlite3_isFail; /* True if all malloc calls should fail */ +extern const char *sqlite3_zFile; /* Filename to associate debug info with */ +extern int sqlite3_iLine; /* Line number for debug info */ + +#define ENTER_MALLOC (sqlite3_zFile = __FILE__, sqlite3_iLine = __LINE__) +#define sqliteMalloc(x) (ENTER_MALLOC, sqlite3Malloc(x)) +#define sqliteMallocRaw(x) (ENTER_MALLOC, sqlite3MallocRaw(x)) +#define sqliteRealloc(x,y) (ENTER_MALLOC, sqlite3Realloc(x,y)) +#define sqliteStrDup(x) (ENTER_MALLOC, sqlite3StrDup(x)) +#define sqliteStrNDup(x,y) (ENTER_MALLOC, sqlite3StrNDup(x,y)) +#define sqliteReallocOrFree(x,y) (ENTER_MALLOC, sqlite3ReallocOrFree(x,y)) + +#else + +#define sqliteMalloc(x) sqlite3Malloc(x) +#define sqliteMallocRaw(x) sqlite3MallocRaw(x) +#define sqliteRealloc(x,y) sqlite3Realloc(x,y) +#define sqliteStrDup(x) sqlite3StrDup(x) +#define sqliteStrNDup(x,y) sqlite3StrNDup(x,y) +#define sqliteReallocOrFree(x,y) sqlite3ReallocOrFree(x,y) + +#endif + +#define sqliteFree(x) sqlite3FreeX(x) +#define sqliteAllocSize(x) sqlite3AllocSize(x) + + +/* +** An instance of this structure might be allocated to store information +** specific to a single thread. +*/ +struct ThreadData { + int dummy; /* So that this structure is never empty */ + +#ifdef SQLITE_ENABLE_MEMORY_MANAGEMENT + int nSoftHeapLimit; /* Suggested max mem allocation. No limit if <0 */ + int nAlloc; /* Number of bytes currently allocated */ + Pager *pPager; /* Linked list of all pagers in this thread */ +#endif + +#ifndef SQLITE_OMIT_SHARED_CACHE + u8 useSharedData; /* True if shared pagers and schemas are enabled */ + BtShared *pBtree; /* Linked list of all currently open BTrees */ +#endif +}; + +/* +** Name of the master database table. The master database table +** is a special table that holds the names and attributes of all +** user tables and indices. +*/ +#define MASTER_NAME "sqlite_master" +#define TEMP_MASTER_NAME "sqlite_temp_master" + +/* +** The root-page of the master database table. +*/ +#define MASTER_ROOT 1 + +/* +** The name of the schema table. +*/ +#define SCHEMA_TABLE(x) ((!OMIT_TEMPDB)&&(x==1)?TEMP_MASTER_NAME:MASTER_NAME) + +/* +** A convenience macro that returns the number of elements in +** an array. +*/ +#define ArraySize(X) (sizeof(X)/sizeof(X[0])) + +/* +** Forward references to structures +*/ +typedef struct AggInfo AggInfo; +typedef struct AuthContext AuthContext; +typedef struct CollSeq CollSeq; +typedef struct Column Column; +typedef struct Db Db; +typedef struct Schema Schema; +typedef struct Expr Expr; +typedef struct ExprList ExprList; +typedef struct FKey FKey; +typedef struct FuncDef FuncDef; +typedef struct IdList IdList; +typedef struct Index Index; +typedef struct KeyClass KeyClass; +typedef struct KeyInfo KeyInfo; +typedef struct NameContext NameContext; +typedef struct Parse Parse; +typedef struct Select Select; +typedef struct SrcList SrcList; +typedef struct ThreadData ThreadData; +typedef struct Table Table; +typedef struct TableLock TableLock; +typedef struct Token Token; +typedef struct TriggerStack TriggerStack; +typedef struct TriggerStep TriggerStep; +typedef struct Trigger Trigger; +typedef struct WhereInfo WhereInfo; +typedef struct WhereLevel WhereLevel; + +/* +** Each database file to be accessed by the system is an instance +** of the following structure. There are normally two of these structures +** in the sqlite.aDb[] array. aDb[0] is the main database file and +** aDb[1] is the database file used to hold temporary tables. Additional +** databases may be attached. +*/ +struct Db { + char *zName; /* Name of this database */ + Btree *pBt; /* The B*Tree structure for this database file */ + u8 inTrans; /* 0: not writable. 1: Transaction. 2: Checkpoint */ + u8 safety_level; /* How aggressive at synching data to disk */ + void *pAux; /* Auxiliary data. Usually NULL */ + void (*xFreeAux)(void*); /* Routine to free pAux */ + Schema *pSchema; /* Pointer to database schema (possibly shared) */ +}; + +/* +** An instance of the following structure stores a database schema. +*/ +struct Schema { + int schema_cookie; /* Database schema version number for this file */ + Hash tblHash; /* All tables indexed by name */ + Hash idxHash; /* All (named) indices indexed by name */ + Hash trigHash; /* All triggers indexed by name */ + Hash aFKey; /* Foreign keys indexed by to-table */ + Table *pSeqTab; /* The sqlite_sequence table used by AUTOINCREMENT */ + u8 file_format; /* Schema format version for this file */ + u8 enc; /* Text encoding used by this database */ + u16 flags; /* Flags associated with this schema */ + int cache_size; /* Number of pages to use in the cache */ +}; + +/* +** These macros can be used to test, set, or clear bits in the +** Db.flags field. +*/ +#define DbHasProperty(D,I,P) (((D)->aDb[I].pSchema->flags&(P))==(P)) +#define DbHasAnyProperty(D,I,P) (((D)->aDb[I].pSchema->flags&(P))!=0) +#define DbSetProperty(D,I,P) (D)->aDb[I].pSchema->flags|=(P) +#define DbClearProperty(D,I,P) (D)->aDb[I].pSchema->flags&=~(P) + +/* +** Allowed values for the DB.flags field. +** +** The DB_SchemaLoaded flag is set after the database schema has been +** read into internal hash tables. +** +** DB_UnresetViews means that one or more views have column names that +** have been filled out. If the schema changes, these column names might +** changes and so the view will need to be reset. +*/ +#define DB_SchemaLoaded 0x0001 /* The schema has been loaded */ +#define DB_UnresetViews 0x0002 /* Some views have defined column names */ +#define DB_Empty 0x0004 /* The file is empty (length 0 bytes) */ + +#define SQLITE_UTF16NATIVE (SQLITE_BIGENDIAN?SQLITE_UTF16BE:SQLITE_UTF16LE) + +/* +** Each database is an instance of the following structure. +** +** The sqlite.lastRowid records the last insert rowid generated by an +** insert statement. Inserts on views do not affect its value. Each +** trigger has its own context, so that lastRowid can be updated inside +** triggers as usual. The previous value will be restored once the trigger +** exits. Upon entering a before or instead of trigger, lastRowid is no +** longer (since after version 2.8.12) reset to -1. +** +** The sqlite.nChange does not count changes within triggers and keeps no +** context. It is reset at start of sqlite3_exec. +** The sqlite.lsChange represents the number of changes made by the last +** insert, update, or delete statement. It remains constant throughout the +** length of a statement and is then updated by OP_SetCounts. It keeps a +** context stack just like lastRowid so that the count of changes +** within a trigger is not seen outside the trigger. Changes to views do not +** affect the value of lsChange. +** The sqlite.csChange keeps track of the number of current changes (since +** the last statement) and is used to update sqlite_lsChange. +** +** The member variables sqlite.errCode, sqlite.zErrMsg and sqlite.zErrMsg16 +** store the most recent error code and, if applicable, string. The +** internal function sqlite3Error() is used to set these variables +** consistently. +*/ +struct sqlite3 { + int nDb; /* Number of backends currently in use */ + Db *aDb; /* All backends */ + int flags; /* Miscellanous flags. See below */ + int errCode; /* Most recent error code (SQLITE_*) */ + u8 autoCommit; /* The auto-commit flag. */ + u8 temp_store; /* 1: file 2: memory 0: default */ + int nTable; /* Number of tables in the database */ + CollSeq *pDfltColl; /* The default collating sequence (BINARY) */ + i64 lastRowid; /* ROWID of most recent insert (see above) */ + i64 priorNewRowid; /* Last randomly generated ROWID */ + int magic; /* Magic number for detect library misuse */ + int nChange; /* Value returned by sqlite3_changes() */ + int nTotalChange; /* Value returned by sqlite3_total_changes() */ + struct sqlite3InitInfo { /* Information used during initialization */ + int iDb; /* When back is being initialized */ + int newTnum; /* Rootpage of table being initialized */ + u8 busy; /* TRUE if currently initializing */ + } init; + struct Vdbe *pVdbe; /* List of active virtual machines */ + int activeVdbeCnt; /* Number of vdbes currently executing */ + void (*xTrace)(void*,const char*); /* Trace function */ + void *pTraceArg; /* Argument to the trace function */ + void (*xProfile)(void*,const char*,u64); /* Profiling function */ + void *pProfileArg; /* Argument to profile function */ + void *pCommitArg; /* Argument to xCommitCallback() */ + int (*xCommitCallback)(void*); /* Invoked at every commit. */ + void *pRollbackArg; /* Argument to xRollbackCallback() */ + void (*xRollbackCallback)(void*); /* Invoked at every commit. */ + void *pUpdateArg; + void (*xUpdateCallback)(void*,int, const char*,const char*,sqlite_int64); + void(*xCollNeeded)(void*,sqlite3*,int eTextRep,const char*); + void(*xCollNeeded16)(void*,sqlite3*,int eTextRep,const void*); + void *pCollNeededArg; + sqlite3_value *pErr; /* Most recent error message */ + char *zErrMsg; /* Most recent error message (UTF-8 encoded) */ + char *zErrMsg16; /* Most recent error message (UTF-16 encoded) */ +#ifndef SQLITE_OMIT_AUTHORIZATION + int (*xAuth)(void*,int,const char*,const char*,const char*,const char*); + /* Access authorization function */ + void *pAuthArg; /* 1st argument to the access auth function */ +#endif +#ifndef SQLITE_OMIT_PROGRESS_CALLBACK + int (*xProgress)(void *); /* The progress callback */ + void *pProgressArg; /* Argument to the progress callback */ + int nProgressOps; /* Number of opcodes for progress callback */ +#endif +#ifndef SQLITE_OMIT_GLOBALRECOVER + sqlite3 *pNext; /* Linked list of open db handles. */ +#endif + Hash aFunc; /* All functions that can be in SQL exprs */ + Hash aCollSeq; /* All collating sequences */ + BusyHandler busyHandler; /* Busy callback */ + int busyTimeout; /* Busy handler timeout, in msec */ + Db aDbStatic[2]; /* Static space for the 2 default backends */ +#ifdef SQLITE_SSE + sqlite3_stmt *pFetch; /* Used by SSE to fetch stored statements */ +#endif +}; + +#define ENC(db) ((db)->aDb[0].pSchema->enc) + +/* +** Possible values for the sqlite.flags and or Db.flags fields. +** +** On sqlite.flags, the SQLITE_InTrans value means that we have +** executed a BEGIN. On Db.flags, SQLITE_InTrans means a statement +** transaction is active on that particular database file. +*/ +#define SQLITE_VdbeTrace 0x00000001 /* True to trace VDBE execution */ +#define SQLITE_Interrupt 0x00000004 /* Cancel current operation */ +#define SQLITE_InTrans 0x00000008 /* True if in a transaction */ +#define SQLITE_InternChanges 0x00000010 /* Uncommitted Hash table changes */ +#define SQLITE_FullColNames 0x00000020 /* Show full column names on SELECT */ +#define SQLITE_ShortColNames 0x00000040 /* Show short columns names */ +#define SQLITE_CountRows 0x00000080 /* Count rows changed by INSERT, */ + /* DELETE, or UPDATE and return */ + /* the count using a callback. */ +#define SQLITE_NullCallback 0x00000100 /* Invoke the callback once if the */ + /* result set is empty */ +#define SQLITE_SqlTrace 0x00000200 /* Debug print SQL as it executes */ +#define SQLITE_VdbeListing 0x00000400 /* Debug listings of VDBE programs */ +#define SQLITE_WriteSchema 0x00000800 /* OK to update SQLITE_MASTER */ +#define SQLITE_NoReadlock 0x00001000 /* Readlocks are omitted when + ** accessing read-only databases */ +#define SQLITE_IgnoreChecks 0x00002000 /* Do not enforce check constraints */ +#define SQLITE_ReadUncommitted 0x00004000 /* For shared-cache mode */ +#define SQLITE_LegacyFileFmt 0x00008000 /* Create new databases in format 1 */ +#define SQLITE_FullFSync 0x00010000 /* Use full fsync on the backend */ + +/* +** Possible values for the sqlite.magic field. +** The numbers are obtained at random and have no special meaning, other +** than being distinct from one another. +*/ +#define SQLITE_MAGIC_OPEN 0xa029a697 /* Database is open */ +#define SQLITE_MAGIC_CLOSED 0x9f3c2d33 /* Database is closed */ +#define SQLITE_MAGIC_BUSY 0xf03b7906 /* Database currently in use */ +#define SQLITE_MAGIC_ERROR 0xb5357930 /* An SQLITE_MISUSE error occurred */ + +/* +** Each SQL function is defined by an instance of the following +** structure. A pointer to this structure is stored in the sqlite.aFunc +** hash table. When multiple functions have the same name, the hash table +** points to a linked list of these structures. +*/ +struct FuncDef { + i16 nArg; /* Number of arguments. -1 means unlimited */ + u8 iPrefEnc; /* Preferred text encoding (SQLITE_UTF8, 16LE, 16BE) */ + u8 needCollSeq; /* True if sqlite3GetFuncCollSeq() might be called */ + u8 flags; /* Some combination of SQLITE_FUNC_* */ + void *pUserData; /* User data parameter */ + FuncDef *pNext; /* Next function with same name */ + void (*xFunc)(sqlite3_context*,int,sqlite3_value**); /* Regular function */ + void (*xStep)(sqlite3_context*,int,sqlite3_value**); /* Aggregate step */ + void (*xFinalize)(sqlite3_context*); /* Aggregate finializer */ + char zName[1]; /* SQL name of the function. MUST BE LAST */ +}; + +/* +** Possible values for FuncDef.flags +*/ +#define SQLITE_FUNC_LIKE 0x01 /* Candidate for the LIKE optimization */ +#define SQLITE_FUNC_CASE 0x02 /* Case-sensitive LIKE-type function */ + +/* +** information about each column of an SQL table is held in an instance +** of this structure. +*/ +struct Column { + char *zName; /* Name of this column */ + Expr *pDflt; /* Default value of this column */ + char *zType; /* Data type for this column */ + char *zColl; /* Collating sequence. If NULL, use the default */ + u8 notNull; /* True if there is a NOT NULL constraint */ + u8 isPrimKey; /* True if this column is part of the PRIMARY KEY */ + char affinity; /* One of the SQLITE_AFF_... values */ +}; + +/* +** A "Collating Sequence" is defined by an instance of the following +** structure. Conceptually, a collating sequence consists of a name and +** a comparison routine that defines the order of that sequence. +** +** There may two seperate implementations of the collation function, one +** that processes text in UTF-8 encoding (CollSeq.xCmp) and another that +** processes text encoded in UTF-16 (CollSeq.xCmp16), using the machine +** native byte order. When a collation sequence is invoked, SQLite selects +** the version that will require the least expensive encoding +** translations, if any. +** +** The CollSeq.pUser member variable is an extra parameter that passed in +** as the first argument to the UTF-8 comparison function, xCmp. +** CollSeq.pUser16 is the equivalent for the UTF-16 comparison function, +** xCmp16. +** +** If both CollSeq.xCmp and CollSeq.xCmp16 are NULL, it means that the +** collating sequence is undefined. Indices built on an undefined +** collating sequence may not be read or written. +*/ +struct CollSeq { + char *zName; /* Name of the collating sequence, UTF-8 encoded */ + u8 enc; /* Text encoding handled by xCmp() */ + u8 type; /* One of the SQLITE_COLL_... values below */ + void *pUser; /* First argument to xCmp() */ + int (*xCmp)(void*,int, const void*, int, const void*); +}; + +/* +** Allowed values of CollSeq flags: +*/ +#define SQLITE_COLL_BINARY 1 /* The default memcmp() collating sequence */ +#define SQLITE_COLL_NOCASE 2 /* The built-in NOCASE collating sequence */ +#define SQLITE_COLL_REVERSE 3 /* The built-in REVERSE collating sequence */ +#define SQLITE_COLL_USER 0 /* Any other user-defined collating sequence */ + +/* +** A sort order can be either ASC or DESC. +*/ +#define SQLITE_SO_ASC 0 /* Sort in ascending order */ +#define SQLITE_SO_DESC 1 /* Sort in ascending order */ + +/* +** Column affinity types. +** +** These used to have mnemonic name like 'i' for SQLITE_AFF_INTEGER and +** 't' for SQLITE_AFF_TEXT. But we can save a little space and improve +** the speed a little by number the values consecutively. +** +** But rather than start with 0 or 1, we begin with 'a'. That way, +** when multiple affinity types are concatenated into a string and +** used as the P3 operand, they will be more readable. +** +** Note also that the numeric types are grouped together so that testing +** for a numeric type is a single comparison. +*/ +#define SQLITE_AFF_TEXT 'a' +#define SQLITE_AFF_NONE 'b' +#define SQLITE_AFF_NUMERIC 'c' +#define SQLITE_AFF_INTEGER 'd' +#define SQLITE_AFF_REAL 'e' + +#define sqlite3IsNumericAffinity(X) ((X)>=SQLITE_AFF_NUMERIC) + +/* +** Each SQL table is represented in memory by an instance of the +** following structure. +** +** Table.zName is the name of the table. The case of the original +** CREATE TABLE statement is stored, but case is not significant for +** comparisons. +** +** Table.nCol is the number of columns in this table. Table.aCol is a +** pointer to an array of Column structures, one for each column. +** +** If the table has an INTEGER PRIMARY KEY, then Table.iPKey is the index of +** the column that is that key. Otherwise Table.iPKey is negative. Note +** that the datatype of the PRIMARY KEY must be INTEGER for this field to +** be set. An INTEGER PRIMARY KEY is used as the rowid for each row of +** the table. If a table has no INTEGER PRIMARY KEY, then a random rowid +** is generated for each row of the table. Table.hasPrimKey is true if +** the table has any PRIMARY KEY, INTEGER or otherwise. +** +** Table.tnum is the page number for the root BTree page of the table in the +** database file. If Table.iDb is the index of the database table backend +** in sqlite.aDb[]. 0 is for the main database and 1 is for the file that +** holds temporary tables and indices. If Table.isTransient +** is true, then the table is stored in a file that is automatically deleted +** when the VDBE cursor to the table is closed. In this case Table.tnum +** refers VDBE cursor number that holds the table open, not to the root +** page number. Transient tables are used to hold the results of a +** sub-query that appears instead of a real table name in the FROM clause +** of a SELECT statement. +*/ +struct Table { + char *zName; /* Name of the table */ + int nCol; /* Number of columns in this table */ + Column *aCol; /* Information about each column */ + int iPKey; /* If not less then 0, use aCol[iPKey] as the primary key */ + Index *pIndex; /* List of SQL indexes on this table. */ + int tnum; /* Root BTree node for this table (see note above) */ + Select *pSelect; /* NULL for tables. Points to definition if a view. */ + u8 readOnly; /* True if this table should not be written by the user */ + u8 isTransient; /* True if automatically deleted when VDBE finishes */ + u8 hasPrimKey; /* True if there exists a primary key */ + u8 keyConf; /* What to do in case of uniqueness conflict on iPKey */ + u8 autoInc; /* True if the integer primary key is autoincrement */ + int nRef; /* Number of pointers to this Table */ + Trigger *pTrigger; /* List of SQL triggers on this table */ + FKey *pFKey; /* Linked list of all foreign keys in this table */ + char *zColAff; /* String defining the affinity of each column */ +#ifndef SQLITE_OMIT_CHECK + Expr *pCheck; /* The AND of all CHECK constraints */ +#endif +#ifndef SQLITE_OMIT_ALTERTABLE + int addColOffset; /* Offset in CREATE TABLE statement to add a new column */ +#endif + Schema *pSchema; +}; + +/* +** Each foreign key constraint is an instance of the following structure. +** +** A foreign key is associated with two tables. The "from" table is +** the table that contains the REFERENCES clause that creates the foreign +** key. The "to" table is the table that is named in the REFERENCES clause. +** Consider this example: +** +** CREATE TABLE ex1( +** a INTEGER PRIMARY KEY, +** b INTEGER CONSTRAINT fk1 REFERENCES ex2(x) +** ); +** +** For foreign key "fk1", the from-table is "ex1" and the to-table is "ex2". +** +** Each REFERENCES clause generates an instance of the following structure +** which is attached to the from-table. The to-table need not exist when +** the from-table is created. The existance of the to-table is not checked +** until an attempt is made to insert data into the from-table. +** +** The sqlite.aFKey hash table stores pointers to this structure +** given the name of a to-table. For each to-table, all foreign keys +** associated with that table are on a linked list using the FKey.pNextTo +** field. +*/ +struct FKey { + Table *pFrom; /* The table that constains the REFERENCES clause */ + FKey *pNextFrom; /* Next foreign key in pFrom */ + char *zTo; /* Name of table that the key points to */ + FKey *pNextTo; /* Next foreign key that points to zTo */ + int nCol; /* Number of columns in this key */ + struct sColMap { /* Mapping of columns in pFrom to columns in zTo */ + int iFrom; /* Index of column in pFrom */ + char *zCol; /* Name of column in zTo. If 0 use PRIMARY KEY */ + } *aCol; /* One entry for each of nCol column s */ + u8 isDeferred; /* True if constraint checking is deferred till COMMIT */ + u8 updateConf; /* How to resolve conflicts that occur on UPDATE */ + u8 deleteConf; /* How to resolve conflicts that occur on DELETE */ + u8 insertConf; /* How to resolve conflicts that occur on INSERT */ +}; + +/* +** SQLite supports many different ways to resolve a contraint +** error. ROLLBACK processing means that a constraint violation +** causes the operation in process to fail and for the current transaction +** to be rolled back. ABORT processing means the operation in process +** fails and any prior changes from that one operation are backed out, +** but the transaction is not rolled back. FAIL processing means that +** the operation in progress stops and returns an error code. But prior +** changes due to the same operation are not backed out and no rollback +** occurs. IGNORE means that the particular row that caused the constraint +** error is not inserted or updated. Processing continues and no error +** is returned. REPLACE means that preexisting database rows that caused +** a UNIQUE constraint violation are removed so that the new insert or +** update can proceed. Processing continues and no error is reported. +** +** RESTRICT, SETNULL, and CASCADE actions apply only to foreign keys. +** RESTRICT is the same as ABORT for IMMEDIATE foreign keys and the +** same as ROLLBACK for DEFERRED keys. SETNULL means that the foreign +** key is set to NULL. CASCADE means that a DELETE or UPDATE of the +** referenced table row is propagated into the row that holds the +** foreign key. +** +** The following symbolic values are used to record which type +** of action to take. +*/ +#define OE_None 0 /* There is no constraint to check */ +#define OE_Rollback 1 /* Fail the operation and rollback the transaction */ +#define OE_Abort 2 /* Back out changes but do no rollback transaction */ +#define OE_Fail 3 /* Stop the operation but leave all prior changes */ +#define OE_Ignore 4 /* Ignore the error. Do not do the INSERT or UPDATE */ +#define OE_Replace 5 /* Delete existing record, then do INSERT or UPDATE */ + +#define OE_Restrict 6 /* OE_Abort for IMMEDIATE, OE_Rollback for DEFERRED */ +#define OE_SetNull 7 /* Set the foreign key value to NULL */ +#define OE_SetDflt 8 /* Set the foreign key value to its default */ +#define OE_Cascade 9 /* Cascade the changes */ + +#define OE_Default 99 /* Do whatever the default action is */ + + +/* +** An instance of the following structure is passed as the first +** argument to sqlite3VdbeKeyCompare and is used to control the +** comparison of the two index keys. +** +** If the KeyInfo.incrKey value is true and the comparison would +** otherwise be equal, then return a result as if the second key +** were larger. +*/ +struct KeyInfo { + u8 enc; /* Text encoding - one of the TEXT_Utf* values */ + u8 incrKey; /* Increase 2nd key by epsilon before comparison */ + int nField; /* Number of entries in aColl[] */ + u8 *aSortOrder; /* If defined an aSortOrder[i] is true, sort DESC */ + CollSeq *aColl[1]; /* Collating sequence for each term of the key */ +}; + +/* +** Each SQL index is represented in memory by an +** instance of the following structure. +** +** The columns of the table that are to be indexed are described +** by the aiColumn[] field of this structure. For example, suppose +** we have the following table and index: +** +** CREATE TABLE Ex1(c1 int, c2 int, c3 text); +** CREATE INDEX Ex2 ON Ex1(c3,c1); +** +** In the Table structure describing Ex1, nCol==3 because there are +** three columns in the table. In the Index structure describing +** Ex2, nColumn==2 since 2 of the 3 columns of Ex1 are indexed. +** The value of aiColumn is {2, 0}. aiColumn[0]==2 because the +** first column to be indexed (c3) has an index of 2 in Ex1.aCol[]. +** The second column to be indexed (c1) has an index of 0 in +** Ex1.aCol[], hence Ex2.aiColumn[1]==0. +** +** The Index.onError field determines whether or not the indexed columns +** must be unique and what to do if they are not. When Index.onError=OE_None, +** it means this is not a unique index. Otherwise it is a unique index +** and the value of Index.onError indicate the which conflict resolution +** algorithm to employ whenever an attempt is made to insert a non-unique +** element. +*/ +struct Index { + char *zName; /* Name of this index */ + int nColumn; /* Number of columns in the table used by this index */ + int *aiColumn; /* Which columns are used by this index. 1st is 0 */ + unsigned *aiRowEst; /* Result of ANALYZE: Est. rows selected by each column */ + Table *pTable; /* The SQL table being indexed */ + int tnum; /* Page containing root of this index in database file */ + u8 onError; /* OE_Abort, OE_Ignore, OE_Replace, or OE_None */ + u8 autoIndex; /* True if is automatically created (ex: by UNIQUE) */ + char *zColAff; /* String defining the affinity of each column */ + Index *pNext; /* The next index associated with the same table */ + Schema *pSchema; /* Schema containing this index */ + u8 *aSortOrder; /* Array of size Index.nColumn. True==DESC, False==ASC */ + char **azColl; /* Array of collation sequence names for index */ +}; + +/* +** Each token coming out of the lexer is an instance of +** this structure. Tokens are also used as part of an expression. +** +** Note if Token.z==0 then Token.dyn and Token.n are undefined and +** may contain random values. Do not make any assuptions about Token.dyn +** and Token.n when Token.z==0. +*/ +struct Token { + const unsigned char *z; /* Text of the token. Not NULL-terminated! */ + unsigned dyn : 1; /* True for malloced memory, false for static */ + unsigned n : 31; /* Number of characters in this token */ +}; + +/* +** An instance of this structure contains information needed to generate +** code for a SELECT that contains aggregate functions. +** +** If Expr.op==TK_AGG_COLUMN or TK_AGG_FUNCTION then Expr.pAggInfo is a +** pointer to this structure. The Expr.iColumn field is the index in +** AggInfo.aCol[] or AggInfo.aFunc[] of information needed to generate +** code for that node. +** +** AggInfo.pGroupBy and AggInfo.aFunc.pExpr point to fields within the +** original Select structure that describes the SELECT statement. These +** fields do not need to be freed when deallocating the AggInfo structure. +*/ +struct AggInfo { + u8 directMode; /* Direct rendering mode means take data directly + ** from source tables rather than from accumulators */ + u8 useSortingIdx; /* In direct mode, reference the sorting index rather + ** than the source table */ + int sortingIdx; /* Cursor number of the sorting index */ + ExprList *pGroupBy; /* The group by clause */ + int nSortingColumn; /* Number of columns in the sorting index */ + struct AggInfo_col { /* For each column used in source tables */ + int iTable; /* Cursor number of the source table */ + int iColumn; /* Column number within the source table */ + int iSorterColumn; /* Column number in the sorting index */ + int iMem; /* Memory location that acts as accumulator */ + Expr *pExpr; /* The original expression */ + } *aCol; + int nColumn; /* Number of used entries in aCol[] */ + int nColumnAlloc; /* Number of slots allocated for aCol[] */ + int nAccumulator; /* Number of columns that show through to the output. + ** Additional columns are used only as parameters to + ** aggregate functions */ + struct AggInfo_func { /* For each aggregate function */ + Expr *pExpr; /* Expression encoding the function */ + FuncDef *pFunc; /* The aggregate function implementation */ + int iMem; /* Memory location that acts as accumulator */ + int iDistinct; /* Virtual table used to enforce DISTINCT */ + } *aFunc; + int nFunc; /* Number of entries in aFunc[] */ + int nFuncAlloc; /* Number of slots allocated for aFunc[] */ +}; + +/* +** Each node of an expression in the parse tree is an instance +** of this structure. +** +** Expr.op is the opcode. The integer parser token codes are reused +** as opcodes here. For example, the parser defines TK_GE to be an integer +** code representing the ">=" operator. This same integer code is reused +** to represent the greater-than-or-equal-to operator in the expression +** tree. +** +** Expr.pRight and Expr.pLeft are subexpressions. Expr.pList is a list +** of argument if the expression is a function. +** +** Expr.token is the operator token for this node. For some expressions +** that have subexpressions, Expr.token can be the complete text that gave +** rise to the Expr. In the latter case, the token is marked as being +** a compound token. +** +** An expression of the form ID or ID.ID refers to a column in a table. +** For such expressions, Expr.op is set to TK_COLUMN and Expr.iTable is +** the integer cursor number of a VDBE cursor pointing to that table and +** Expr.iColumn is the column number for the specific column. If the +** expression is used as a result in an aggregate SELECT, then the +** value is also stored in the Expr.iAgg column in the aggregate so that +** it can be accessed after all aggregates are computed. +** +** If the expression is a function, the Expr.iTable is an integer code +** representing which function. If the expression is an unbound variable +** marker (a question mark character '?' in the original SQL) then the +** Expr.iTable holds the index number for that variable. +** +** If the expression is a subquery then Expr.iColumn holds an integer +** register number containing the result of the subquery. If the +** subquery gives a constant result, then iTable is -1. If the subquery +** gives a different answer at different times during statement processing +** then iTable is the address of a subroutine that computes the subquery. +** +** The Expr.pSelect field points to a SELECT statement. The SELECT might +** be the right operand of an IN operator. Or, if a scalar SELECT appears +** in an expression the opcode is TK_SELECT and Expr.pSelect is the only +** operand. +** +** If the Expr is of type OP_Column, and the table it is selecting from +** is a disk table or the "old.*" pseudo-table, then pTab points to the +** corresponding table definition. +*/ +struct Expr { + u8 op; /* Operation performed by this node */ + char affinity; /* The affinity of the column or 0 if not a column */ + u8 flags; /* Various flags. See below */ + CollSeq *pColl; /* The collation type of the column or 0 */ + Expr *pLeft, *pRight; /* Left and right subnodes */ + ExprList *pList; /* A list of expressions used as function arguments + ** or in " IN (aCol[] or ->aFunc[] */ + int iRightJoinTable; /* If EP_FromJoin, the right table of the join */ + Select *pSelect; /* When the expression is a sub-select. Also the + ** right side of " IN (

From python-checkins at python.org Tue Apr 11 00:28:11 2006 From: python-checkins at python.org (andrew.kuchling) Date: Tue, 11 Apr 2006 00:28:11 +0200 (CEST) Subject: [Python-checkins] r45245 - python/trunk/Doc/whatsnew/whatsnew25.tex Message-ID: <20060410222811.AA28C1E4007@bag.python.org> Author: andrew.kuchling Date: Tue Apr 11 00:28:11 2006 New Revision: 45245 Modified: python/trunk/Doc/whatsnew/whatsnew25.tex Log: Write part of ElementTree section Modified: python/trunk/Doc/whatsnew/whatsnew25.tex ============================================================================== --- python/trunk/Doc/whatsnew/whatsnew25.tex (original) +++ python/trunk/Doc/whatsnew/whatsnew25.tex Tue Apr 11 00:28:11 2006 @@ -5,8 +5,6 @@ % Fix XXX comments % Distutils upload (PEP 243) % The easy_install stuff -% xml.etree section -% added sqlite3 \title{What's New in Python 2.5} \release{0.1} @@ -1087,19 +1085,88 @@ \subsection{The ElementTree package} A subset of Fredrik Lundh's ElementTree library for processing XML has -been added to the standard library as \module{xml.etree}. The +been added to the standard library as \module{xmlcore.etree}. The vailable modules are \module{ElementTree}, \module{ElementPath}, and \module{ElementInclude} from ElementTree 1.2.6. The \module{cElementTree} accelerator module is also included. -In subsequent alpha releases of Python 2.5, I'll add a brief -introduction that will provide a page-long overview of using -ElementTree. Full documentation for -ElementTree is available at \url{http://effbot.org/zone/element-index.htm}. +The rest of this section will provide a brief overview of using +ElementTree. Full documentation for ElementTree is available at +\url{http://effbot.org/zone/element-index.htm}. + +ElementTree represents an XML document as a tree of element nodes. +The text content of the document is stored as the \member{.text} +and \member{.tail} attributes of +(This is one of the major differences between ElementTree and +the Document Object Model; in the DOM there are many different +types of node, including \class{TextNode}.) + +The most commonly used parsing function is \function{parse()}, that +takes either a string (assumed to contain a filename) or a file-like +object and returns an \class{ElementTree} instance: + +\begin{verbatim} +from xmlcore.etree import ElementTree as ET + +tree = ET.parse('ex-1.xml') + +feed = urllib.urlopen( + 'http://planet.python.org/rss10.xml') +tree = ET.parse(feed) +\end{verbatim} + +Once you have an \class{ElementTree} instance, you +can call its \method{getroot()} method to get the root \class{Element} node. + +There's also an \function{XML()} function that takes a string literal +and returns an \class{Element} node (not an \class{ElementTree}). +This function provides a tidy way to incorporate XML fragments, +approaching the convenience of an XML literal: + +\begin{verbatim} +svg = et.XML(""" + """) +svg.set('height', '320px') +svg.append(elem1) +\end{verbatim} + +Each XML element supports some dictionary-like and some list-like +access methods. Dictionary-like methods are used to access attribute +values, and list-like methods are used to access child nodes. + +% XXX finish this + +To generate XML output, you should call the +\method{ElementTree.write()} method. Like \function{parse()}, +it can take either a string or a file-like object: + +\begin{verbatim} +# Encoding is US-ASCII +tree.write('output.xml') + +# Encoding is UTF-8 +f = open('output.xml', 'w') +tree.write(f, 'utf-8') +\end{verbatim} + +(Caution: the default encoding used for output is ASCII, which isn't +very useful for general XML work, raising an exception if there are +any characters with values greater than 127. You should always +specify a different encoding such as UTF-8 that can handle any Unicode +character.) + % XXX write introduction +\begin{seealso} + +\seeurl{http://effbot.org/zone/element-index.htm} +{Official documentation for ElementTree.} + + +\end{seealso} + \subsection{The hashlib package} @@ -1373,6 +1440,6 @@ The author would like to thank the following people for offering suggestions, corrections and assistance with various drafts of this -article: Thomas Wouters. +article: Mike Rovner, Thomas Wouters. \end{document} From python-checkins at python.org Tue Apr 11 02:43:34 2006 From: python-checkins at python.org (tim.peters) Date: Tue, 11 Apr 2006 02:43:34 +0200 (CEST) Subject: [Python-checkins] r45246 - python/trunk/Lib/test/test_set.py Message-ID: <20060411004334.A046F1E400F@bag.python.org> Author: tim.peters Date: Tue Apr 11 02:43:27 2006 New Revision: 45246 Modified: python/trunk/Lib/test/test_set.py Log: Try to repair more new buildbot failures in "x86 OpenBSD trunk", due to that id() can now return a Python long on a 32-bit box that allocates addresses "with the sign bit set". test_set.py test_subclass_with_custom_hash(): it's never been portably legal for a __hash__() method to return id(self), but on 32-bit boxes that never caused a problem before it became possible for id() to return a Python long. Changed __hash__ here to return a Python int regardless of platform. test_descr.py specials(): vereq(hash(c1), id(c1)) has never been a correct test -- just removed it (hash() is always a Python int; id() may be a Python long). Modified: python/trunk/Lib/test/test_set.py ============================================================================== --- python/trunk/Lib/test/test_set.py (original) +++ python/trunk/Lib/test/test_set.py Tue Apr 11 02:43:27 2006 @@ -224,7 +224,7 @@ # Bug #1257731 class H(self.thetype): def __hash__(self): - return id(self) + return int(id(self) & 0x7fffffff) s=H() f=set() f.add(s) From python-checkins at python.org Tue Apr 11 02:44:30 2006 From: python-checkins at python.org (tim.peters) Date: Tue, 11 Apr 2006 02:44:30 +0200 (CEST) Subject: [Python-checkins] r45247 - python/trunk/Lib/test/test_descr.py Message-ID: <20060411004430.9D2C91E4007@bag.python.org> Author: tim.peters Date: Tue Apr 11 02:44:27 2006 New Revision: 45247 Modified: python/trunk/Lib/test/test_descr.py Log: Huh. This belonged with the last checkin -- no idea why svn didn't commit it. Modified: python/trunk/Lib/test/test_descr.py ============================================================================== --- python/trunk/Lib/test/test_descr.py (original) +++ python/trunk/Lib/test/test_descr.py Tue Apr 11 02:44:27 2006 @@ -1763,7 +1763,6 @@ c1 = C() c2 = C() verify(not not c1) - vereq(hash(c1), id(c1)) vereq(cmp(c1, c2), cmp(id(c1), id(c2))) vereq(c1, c1) verify(c1 != c2) From python-checkins at python.org Tue Apr 11 03:07:45 2006 From: python-checkins at python.org (phillip.eby) Date: Tue, 11 Apr 2006 03:07:45 +0200 (CEST) Subject: [Python-checkins] r45248 - in python/trunk: Doc/lib/liblinecache.tex Doc/lib/libwarnings.tex Lib/doctest.py Lib/inspect.py Lib/linecache.py Lib/site.py Lib/test/test_zipimport.py Lib/traceback.py Lib/warnings.py Misc/NEWS Message-ID: <20060411010745.C26921E4007@bag.python.org> Author: phillip.eby Date: Tue Apr 11 03:07:43 2006 New Revision: 45248 Modified: python/trunk/Doc/lib/liblinecache.tex python/trunk/Doc/lib/libwarnings.tex python/trunk/Lib/doctest.py python/trunk/Lib/inspect.py python/trunk/Lib/linecache.py python/trunk/Lib/site.py python/trunk/Lib/test/test_zipimport.py python/trunk/Lib/traceback.py python/trunk/Lib/warnings.py python/trunk/Misc/NEWS Log: Updated the warnings, linecache, inspect, traceback, site, and doctest modules to work correctly with modules imported from zipfiles or via other PEP 302 __loader__ objects. Tests and doc updates are included. Modified: python/trunk/Doc/lib/liblinecache.tex ============================================================================== --- python/trunk/Doc/lib/liblinecache.tex (original) +++ python/trunk/Doc/lib/liblinecache.tex Tue Apr 11 03:07:43 2006 @@ -15,7 +15,7 @@ The \module{linecache} module defines the following functions: -\begin{funcdesc}{getline}{filename, lineno} +\begin{funcdesc}{getline}{filename, lineno\optional{, module_globals}} Get line \var{lineno} from file named \var{filename}. This function will never throw an exception --- it will return \code{''} on errors (the terminating newline character will be included for lines that are @@ -23,7 +23,11 @@ If a file named \var{filename} is not found, the function will look for it in the module\indexiii{module}{search}{path} search path, -\code{sys.path}. +\code{sys.path}, after first checking for a PEP 302 \code{__loader__} +in \var{module_globals}, in case the module was imported from a zipfile +or other non-filesystem import source. + +\versionadded[The \var{module_globals} parameter was added]{2.5} \end{funcdesc} \begin{funcdesc}{clearcache}{} Modified: python/trunk/Doc/lib/libwarnings.tex ============================================================================== --- python/trunk/Doc/lib/libwarnings.tex (original) +++ python/trunk/Doc/lib/libwarnings.tex Tue Apr 11 03:07:43 2006 @@ -169,7 +169,8 @@ \end{funcdesc} \begin{funcdesc}{warn_explicit}{message, category, filename, - lineno\optional{, module\optional{, registry}}} + lineno\optional{, module\optional{, registry\optional{, + module_globals}}}} This is a low-level interface to the functionality of \function{warn()}, passing in explicitly the message, category, filename and line number, and optionally the module name and the @@ -179,6 +180,11 @@ \var{message} must be a string and \var{category} a subclass of \exception{Warning} or \var{message} may be a \exception{Warning} instance, in which case \var{category} will be ignored. + +\var{module_globals}, if supplied, should be the global namespace in use +by the code for which the warning is issued. (This argument is used to +support displaying source for modules found in zipfiles or other +non-filesystem import sources, and was added in Python 2.5.) \end{funcdesc} \begin{funcdesc}{showwarning}{message, category, filename, Modified: python/trunk/Lib/doctest.py ============================================================================== --- python/trunk/Lib/doctest.py (original) +++ python/trunk/Lib/doctest.py Tue Apr 11 03:07:43 2006 @@ -236,6 +236,15 @@ else: raise TypeError("Expected a module, string, or None") +def _load_testfile(filename, package, module_relative): + if module_relative: + package = _normalize_module(package, 3) + filename = _module_relative_path(package, filename) + if hasattr(package, '__loader__'): + if hasattr(package.__loader__, 'get_data'): + return package.__loader__.get_data(filename), filename + return open(filename).read(), filename + def _indent(s, indent=4): """ Add the given number of space characters to the beginning every @@ -1319,13 +1328,13 @@ __LINECACHE_FILENAME_RE = re.compile(r'[\w\.]+)' r'\[(?P\d+)\]>$') - def __patched_linecache_getlines(self, filename): + def __patched_linecache_getlines(self, filename, module_globals=None): m = self.__LINECACHE_FILENAME_RE.match(filename) if m and m.group('name') == self.test.name: example = self.test.examples[int(m.group('examplenum'))] return example.source.splitlines(True) else: - return self.save_linecache_getlines(filename) + return self.save_linecache_getlines(filename, module_globals) def run(self, test, compileflags=None, out=None, clear_globs=True): """ @@ -1933,9 +1942,7 @@ "relative paths.") # Relativize the path - if module_relative: - package = _normalize_module(package) - filename = _module_relative_path(package, filename) + text, filename = _load_testfile(filename, package, module_relative) # If no name was given, then use the file's name. if name is None: @@ -1955,8 +1962,7 @@ runner = DocTestRunner(verbose=verbose, optionflags=optionflags) # Read the file, convert it to a test, and run it. - s = open(filename).read() - test = parser.get_doctest(s, globs, name, filename, 0) + test = parser.get_doctest(text, globs, name, filename, 0) runner.run(test) if report: @@ -2336,15 +2342,13 @@ "relative paths.") # Relativize the path. - if module_relative: - package = _normalize_module(package) - path = _module_relative_path(package, path) + doc, path = _load_testfile(path, package, module_relative) + if "__file__" not in globs: globs["__file__"] = path # Find the file and read it. name = os.path.basename(path) - doc = open(path).read() # Convert it to a test, and wrap it in a DocFileCase. test = parser.get_doctest(doc, globs, name, path, 0) Modified: python/trunk/Lib/inspect.py ============================================================================== --- python/trunk/Lib/inspect.py (original) +++ python/trunk/Lib/inspect.py Tue Apr 11 03:07:43 2006 @@ -353,7 +353,7 @@ if 'b' in mode and string.lower(filename[-len(suffix):]) == suffix: # Looks like a binary file. We want to only return a text file. return None - if os.path.exists(filename): + if os.path.exists(filename) or hasattr(getmodule(object),'__loader__'): return filename def getabsfile(object): @@ -379,7 +379,7 @@ if file in modulesbyfile: return sys.modules.get(modulesbyfile[file]) for module in sys.modules.values(): - if hasattr(module, '__file__'): + if ismodule(module) and hasattr(module, '__file__'): modulesbyfile[ os.path.realpath( getabsfile(module))] = module.__name__ @@ -406,7 +406,7 @@ in the file and the line number indexes a line in that list. An IOError is raised if the source code cannot be retrieved.""" file = getsourcefile(object) or getfile(object) - lines = linecache.getlines(file) + lines = linecache.getlines(file, getmodule(object).__dict__) if not lines: raise IOError('could not get source code') Modified: python/trunk/Lib/linecache.py ============================================================================== --- python/trunk/Lib/linecache.py (original) +++ python/trunk/Lib/linecache.py Tue Apr 11 03:07:43 2006 @@ -10,8 +10,8 @@ __all__ = ["getline", "clearcache", "checkcache"] -def getline(filename, lineno): - lines = getlines(filename) +def getline(filename, lineno, module_globals=None): + lines = getlines(filename, module_globals) if 1 <= lineno <= len(lines): return lines[lineno-1] else: @@ -30,14 +30,14 @@ cache = {} -def getlines(filename): +def getlines(filename, module_globals=None): """Get the lines for a file from the cache. Update the cache if it doesn't contain an entry for this file already.""" if filename in cache: return cache[filename][2] else: - return updatecache(filename) + return updatecache(filename,module_globals) def checkcache(filename=None): @@ -54,6 +54,8 @@ for filename in filenames: size, mtime, lines, fullname = cache[filename] + if mtime is None: + continue # no-op for files loaded via a __loader__ try: stat = os.stat(fullname) except os.error: @@ -63,7 +65,7 @@ del cache[filename] -def updatecache(filename): +def updatecache(filename, module_globals=None): """Update a cache entry and return its list of lines. If something's wrong, print a message, discard the cache entry, and return an empty list.""" @@ -72,12 +74,34 @@ del cache[filename] if not filename or filename[0] + filename[-1] == '<>': return [] + fullname = filename try: stat = os.stat(fullname) except os.error, msg: - # Try looking through the module search path. basename = os.path.split(filename)[1] + + # Try for a __loader__, if available + if module_globals and '__loader__' in module_globals: + name = module_globals.get('__name__') + loader = module_globals['__loader__'] + get_source = getattr(loader, 'get_source' ,None) + + if name and get_source: + if basename.startswith(name.split('.')[-1]+'.'): + try: + data = get_source(name) + except (ImportError,IOError): + pass + else: + cache[filename] = ( + len(data), None, + [line+'\n' for line in data.splitlines()], fullname + ) + return cache[filename][2] + + # Try looking through the module search path. + for dirname in sys.path: # When using imputil, sys.path may contain things other than # strings; ignore them when it happens. Modified: python/trunk/Lib/site.py ============================================================================== --- python/trunk/Lib/site.py (original) +++ python/trunk/Lib/site.py Tue Apr 11 03:07:43 2006 @@ -69,6 +69,8 @@ def abs__file__(): """Set all module' __file__ attribute to an absolute path""" for m in sys.modules.values(): + if hasattr(m,'__loader__'): + continue # don't mess with a PEP 302-supplied __file__ try: m.__file__ = os.path.abspath(m.__file__) except AttributeError: Modified: python/trunk/Lib/test/test_zipimport.py ============================================================================== --- python/trunk/Lib/test/test_zipimport.py (original) +++ python/trunk/Lib/test/test_zipimport.py Tue Apr 11 03:07:43 2006 @@ -12,7 +12,12 @@ from test.test_importhooks import ImportHooksBaseTestCase, test_src, test_co import zipimport - +import linecache +import doctest +import inspect +import StringIO +from traceback import extract_tb, extract_stack, print_tb +raise_src = 'def do_raise(): raise TypeError\n' # so we only run testAFakeZlib once if this test is run repeatedly # which happens when we look for ref leaks @@ -54,7 +59,8 @@ def setUp(self): # We're reusing the zip archive path, so we must clear the - # cached directory info. + # cached directory info and linecache + linecache.clearcache() zipimport._zip_directory_cache.clear() ImportHooksBaseTestCase.setUp(self) @@ -83,6 +89,11 @@ mod = __import__(".".join(modules), globals(), locals(), ["__dummy__"]) + + call = kw.get('call') + if call is not None: + call(mod) + if expected_ext: file = mod.get_file() self.assertEquals(file, os.path.join(TEMP_ZIP, @@ -249,6 +260,74 @@ self.doTest(".py", files, TESTMOD, stuff="Some Stuff"*31) + def assertModuleSource(self, module): + self.assertEqual(inspect.getsource(module), test_src) + + def testGetSource(self): + files = {TESTMOD + ".py": (NOW, test_src)} + self.doTest(".py", files, TESTMOD, call=self.assertModuleSource) + + def testGetCompiledSource(self): + pyc = make_pyc(compile(test_src, "", "exec"), NOW) + files = {TESTMOD + ".py": (NOW, test_src), + TESTMOD + pyc_ext: (NOW, pyc)} + self.doTest(pyc_ext, files, TESTMOD, call=self.assertModuleSource) + + def runDoctest(self, callback): + files = {TESTMOD + ".py": (NOW, test_src), + "xyz.txt": (NOW, ">>> log.append(True)\n")} + self.doTest(".py", files, TESTMOD, call=callback) + + def doDoctestFile(self, module): + log = [] + old_master, doctest.master = doctest.master, None + try: + doctest.testfile( + 'xyz.txt', package=module, module_relative=True, + globs=locals() + ) + finally: + doctest.master = old_master + self.assertEqual(log,[True]) + + def testDoctestFile(self): + self.runDoctest(self.doDoctestFile) + + def doDoctestSuite(self, module): + log = [] + doctest.DocFileTest( + 'xyz.txt', package=module, module_relative=True, + globs=locals() + ).run() + self.assertEqual(log,[True]) + + def testDoctestSuite(self): + self.runDoctest(self.doDoctestSuite) + + + def doTraceback(self, module): + try: + module.do_raise() + except: + tb = sys.exc_info()[2].tb_next + + f,lno,n,line = extract_tb(tb, 1)[0] + self.assertEqual(line, raise_src.strip()) + + f,lno,n,line = extract_stack(tb.tb_frame, 1)[0] + self.assertEqual(line, raise_src.strip()) + + s = StringIO.StringIO() + print_tb(tb, 1, s) + self.failUnless(s.getvalue().endswith(raise_src)) + else: + raise AssertionError("This ought to be impossible") + + def testTraceback(self): + files = {TESTMOD + ".py": (NOW, raise_src)} + self.doTest(None, files, TESTMOD, call=self.doTraceback) + + class CompressedZipImportTestCase(UncompressedZipImportTestCase): compression = ZIP_DEFLATED Modified: python/trunk/Lib/traceback.py ============================================================================== --- python/trunk/Lib/traceback.py (original) +++ python/trunk/Lib/traceback.py Tue Apr 11 03:07:43 2006 @@ -66,7 +66,7 @@ _print(file, ' File "%s", line %d, in %s' % (filename,lineno,name)) linecache.checkcache(filename) - line = linecache.getline(filename, lineno) + line = linecache.getline(filename, lineno, f.f_globals) if line: _print(file, ' ' + line.strip()) tb = tb.tb_next n = n+1 @@ -98,7 +98,7 @@ filename = co.co_filename name = co.co_name linecache.checkcache(filename) - line = linecache.getline(filename, lineno) + line = linecache.getline(filename, lineno, f.f_globals) if line: line = line.strip() else: line = None list.append((filename, lineno, name, line)) @@ -281,7 +281,7 @@ filename = co.co_filename name = co.co_name linecache.checkcache(filename) - line = linecache.getline(filename, lineno) + line = linecache.getline(filename, lineno, f.f_globals) if line: line = line.strip() else: line = None list.append((filename, lineno, name, line)) Modified: python/trunk/Lib/warnings.py ============================================================================== --- python/trunk/Lib/warnings.py (original) +++ python/trunk/Lib/warnings.py Tue Apr 11 03:07:43 2006 @@ -58,10 +58,11 @@ if not filename: filename = module registry = globals.setdefault("__warningregistry__", {}) - warn_explicit(message, category, filename, lineno, module, registry) + warn_explicit(message, category, filename, lineno, module, registry, + globals) def warn_explicit(message, category, filename, lineno, - module=None, registry=None): + module=None, registry=None, module_globals=None): if module is None: module = filename or "" if module[-3:].lower() == ".py": @@ -92,6 +93,11 @@ if action == "ignore": registry[key] = 1 return + + # Prime the linecache for formatting, in case the + # "file" is actually in a zipfile or something. + linecache.getlines(filename, module_globals) + if action == "error": raise message # Other actions Modified: python/trunk/Misc/NEWS ============================================================================== --- python/trunk/Misc/NEWS (original) +++ python/trunk/Misc/NEWS Tue Apr 11 03:07:43 2006 @@ -30,6 +30,10 @@ Library ------- +- The warnings, linecache, inspect, traceback, site, and doctest modules + were updated to work correctly with modules imported from zipfiles or + via other PEP 302 __loader__ objects. + - Patch #1467770: Reduce usage of subprocess._active to processes which the application hasn't waited on. From python-checkins at python.org Tue Apr 11 03:15:29 2006 From: python-checkins at python.org (phillip.eby) Date: Tue, 11 Apr 2006 03:15:29 +0200 (CEST) Subject: [Python-checkins] r45249 - python/trunk/Doc/lib/liblinecache.tex Message-ID: <20060411011529.3CC701E4007@bag.python.org> Author: phillip.eby Date: Tue Apr 11 03:15:28 2006 New Revision: 45249 Modified: python/trunk/Doc/lib/liblinecache.tex Log: Forgot to mark up a PEP reference Modified: python/trunk/Doc/lib/liblinecache.tex ============================================================================== --- python/trunk/Doc/lib/liblinecache.tex (original) +++ python/trunk/Doc/lib/liblinecache.tex Tue Apr 11 03:15:28 2006 @@ -23,7 +23,7 @@ If a file named \var{filename} is not found, the function will look for it in the module\indexiii{module}{search}{path} search path, -\code{sys.path}, after first checking for a PEP 302 \code{__loader__} +\code{sys.path}, after first checking for a \pep{302} \code{__loader__} in \var{module_globals}, in case the module was imported from a zipfile or other non-filesystem import source. From python-checkins at python.org Tue Apr 11 03:21:01 2006 From: python-checkins at python.org (tim.peters) Date: Tue, 11 Apr 2006 03:21:01 +0200 (CEST) Subject: [Python-checkins] r45250 - python/trunk/Lib/test/test_descr.py Message-ID: <20060411012101.14A641E4007@bag.python.org> Author: tim.peters Date: Tue Apr 11 03:21:00 2006 New Revision: 45250 Modified: python/trunk/Lib/test/test_descr.py Log: specials(): squash another incorrect hash(x) == id(x) test. Add some lines that at least invoke the default __hash__, although there's nothing to check there beyond that they don't blow up. Modified: python/trunk/Lib/test/test_descr.py ============================================================================== --- python/trunk/Lib/test/test_descr.py (original) +++ python/trunk/Lib/test/test_descr.py Tue Apr 11 03:21:00 2006 @@ -1763,6 +1763,9 @@ c1 = C() c2 = C() verify(not not c1) + verify(id(c1) != id(c2)) + hash(c1) + hash(c2) vereq(cmp(c1, c2), cmp(id(c1), id(c2))) vereq(c1, c1) verify(c1 != c2) @@ -1784,7 +1787,9 @@ d1 = D() d2 = D() verify(not not d1) - vereq(hash(d1), id(d1)) + verify(id(d1) != id(d2)) + hash(d1) + hash(d2) vereq(cmp(d1, d2), cmp(id(d1), id(d2))) vereq(d1, d1) verify(d1 != d2) From python-checkins at python.org Tue Apr 11 03:21:31 2006 From: python-checkins at python.org (phillip.eby) Date: Tue, 11 Apr 2006 03:21:31 +0200 (CEST) Subject: [Python-checkins] r45251 - python/trunk/Misc/NEWS Message-ID: <20060411012131.5C9711E4007@bag.python.org> Author: phillip.eby Date: Tue Apr 11 03:21:31 2006 New Revision: 45251 Modified: python/trunk/Misc/NEWS Log: Add notes to NEWS for other work today. Modified: python/trunk/Misc/NEWS ============================================================================== --- python/trunk/Misc/NEWS (original) +++ python/trunk/Misc/NEWS Tue Apr 11 03:21:31 2006 @@ -21,6 +21,11 @@ sys.setdlopenflags() now works correctly on these systems. (SF patch #1454844) +- Patch #1463867: enhanced garbage collection to allow cleanup of cycles + involving generators that have paused outside of any ``try`` or ``with`` + blocks. (In 2.5a1, a paused generator that was part of a reference + cycle could not be garbage collected, regardless of whether it was + paused in a ``try`` or ``with`` block.) Extension Modules ----------------- @@ -52,6 +57,11 @@ Tests ----- +- The test_contextlib test in 2.5a1 wasn't actually run unless you ran + it separately and by hand. It also wasn't cleaning up its changes to + the current Decimal context. + + Documentation ------------- From buildbot at python.org Tue Apr 11 03:26:47 2006 From: buildbot at python.org (buildbot at python.org) Date: Tue, 11 Apr 2006 01:26:47 +0000 Subject: [Python-checkins] buildbot warnings in x86 XP-2 trunk Message-ID: <20060411012648.105871E4007@bag.python.org> The Buildbot has detected a new failure of x86 XP-2 trunk. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520XP-2%2520trunk/builds/226 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: phillip.eby Build Had Warnings: warnings failed slave lost sincerely, -The Buildbot From python-checkins at python.org Tue Apr 11 03:44:08 2006 From: python-checkins at python.org (tim.peters) Date: Tue, 11 Apr 2006 03:44:08 +0200 (CEST) Subject: [Python-checkins] r45252 - python/trunk/Misc/NEWS Message-ID: <20060411014408.75D4C1E4007@bag.python.org> Author: tim.peters Date: Tue Apr 11 03:44:07 2006 New Revision: 45252 Modified: python/trunk/Misc/NEWS Log: More words on patch #837242, since 4 or 5 tests started failing on one of the 32-bit buildbot boxes because of it, due to tempting but always-wrong Python code. Users probably have code like this too (I know I did ...). Modified: python/trunk/Misc/NEWS ============================================================================== --- python/trunk/Misc/NEWS (original) +++ python/trunk/Misc/NEWS Tue Apr 11 03:44:07 2006 @@ -12,13 +12,24 @@ Core and builtins ----------------- -- Patch #837242: id() of any Python object always gives a positive - number, which might be a long integer. PyLong_FromVoidPtr and - PyLong_AsVoidPtr have been changed accordingly. +- Patch #837242: ``id()`` of any Python object always gives a positive + number now, which might be a long integer. ``PyLong_FromVoidPtr`` and + ``PyLong_AsVoidPtr`` have been changed accordingly. Note that it has + never been correct to implement a ``hash()`` method that returns the + ``id()`` of an object: + + def __hash__(self): + return id(self) # WRONG + + because a hash result must be a (short) Python int but it was always + possible for ``id()`` to return a Python long. However, because ``id()`` + cuold return negative values before, on a 32-bit box an ``id()`` result + was always usable as a hash value before this patch. That's no longer + necessarily so. -- Python on OS X 10.3 and above now uses dlopen() (via dynload_shlib.c) +- Python on OS X 10.3 and above now uses dlopen() (via dynload_shlib.c) to load extension modules and now provides the dl module. As a result, - sys.setdlopenflags() now works correctly on these systems. (SF patch + sys.setdlopenflags() now works correctly on these systems. (SF patch #1454844) - Patch #1463867: enhanced garbage collection to allow cleanup of cycles @@ -197,7 +208,7 @@ and long longs. - SF Bug #1350188, "setdlopenflags" leads to crash upon "import" - It was possible for dlerror() to return a NULL pointer, so + It was possible for dlerror() to return a NULL pointer, so it will now use a default error message in this case. - Replaced most Unicode charmap codecs with new ones using the @@ -572,7 +583,7 @@ - Added the sqlite3 package. This is based on pysqlite2.1.3, and provides a DB-API interface in the standard library. You'll need sqlite 3.0.8 or - later to build this - if you have an earlier version, the C extension + later to build this - if you have an earlier version, the C extension module will not be built. - Bug #1460340: ``random.sample(dict)`` failed in various ways. Dicts From python-checkins at python.org Tue Apr 11 03:44:27 2006 From: python-checkins at python.org (tim.peters) Date: Tue, 11 Apr 2006 03:44:27 +0200 (CEST) Subject: [Python-checkins] r45253 - in python/trunk/Lib: linecache.py test/test_zipimport.py Message-ID: <20060411014427.201C91E4007@bag.python.org> Author: tim.peters Date: Tue Apr 11 03:44:26 2006 New Revision: 45253 Modified: python/trunk/Lib/linecache.py python/trunk/Lib/test/test_zipimport.py Log: Whitespace normalization. Modified: python/trunk/Lib/linecache.py ============================================================================== --- python/trunk/Lib/linecache.py (original) +++ python/trunk/Lib/linecache.py Tue Apr 11 03:44:26 2006 @@ -81,7 +81,7 @@ except os.error, msg: basename = os.path.split(filename)[1] - # Try for a __loader__, if available + # Try for a __loader__, if available if module_globals and '__loader__' in module_globals: name = module_globals.get('__name__') loader = module_globals['__loader__'] @@ -95,7 +95,7 @@ pass else: cache[filename] = ( - len(data), None, + len(data), None, [line+'\n' for line in data.splitlines()], fullname ) return cache[filename][2] Modified: python/trunk/Lib/test/test_zipimport.py ============================================================================== --- python/trunk/Lib/test/test_zipimport.py (original) +++ python/trunk/Lib/test/test_zipimport.py Tue Apr 11 03:44:26 2006 @@ -269,7 +269,7 @@ def testGetCompiledSource(self): pyc = make_pyc(compile(test_src, "", "exec"), NOW) - files = {TESTMOD + ".py": (NOW, test_src), + files = {TESTMOD + ".py": (NOW, test_src), TESTMOD + pyc_ext: (NOW, pyc)} self.doTest(pyc_ext, files, TESTMOD, call=self.assertModuleSource) @@ -310,7 +310,7 @@ module.do_raise() except: tb = sys.exc_info()[2].tb_next - + f,lno,n,line = extract_tb(tb, 1)[0] self.assertEqual(line, raise_src.strip()) From python-checkins at python.org Tue Apr 11 03:47:17 2006 From: python-checkins at python.org (tim.peters) Date: Tue, 11 Apr 2006 03:47:17 +0200 (CEST) Subject: [Python-checkins] r45254 - python/trunk/Misc/NEWS Message-ID: <20060411014717.591A71E4007@bag.python.org> Author: tim.peters Date: Tue Apr 11 03:47:17 2006 New Revision: 45254 Modified: python/trunk/Misc/NEWS Log: Typo repair. Modified: python/trunk/Misc/NEWS ============================================================================== --- python/trunk/Misc/NEWS (original) +++ python/trunk/Misc/NEWS Tue Apr 11 03:47:17 2006 @@ -15,7 +15,7 @@ - Patch #837242: ``id()`` of any Python object always gives a positive number now, which might be a long integer. ``PyLong_FromVoidPtr`` and ``PyLong_AsVoidPtr`` have been changed accordingly. Note that it has - never been correct to implement a ``hash()`` method that returns the + never been correct to implement a ``__hash()__`` method that returns the ``id()`` of an object: def __hash__(self): @@ -23,7 +23,7 @@ because a hash result must be a (short) Python int but it was always possible for ``id()`` to return a Python long. However, because ``id()`` - cuold return negative values before, on a 32-bit box an ``id()`` result + could return negative values before, on a 32-bit box an ``id()`` result was always usable as a hash value before this patch. That's no longer necessarily so. From python-checkins at python.org Tue Apr 11 03:59:35 2006 From: python-checkins at python.org (tim.peters) Date: Tue, 11 Apr 2006 03:59:35 +0200 (CEST) Subject: [Python-checkins] r45255 - python/trunk/Lib/test/test_descr.py Message-ID: <20060411015935.3EC2C1E4007@bag.python.org> Author: tim.peters Date: Tue Apr 11 03:59:34 2006 New Revision: 45255 Modified: python/trunk/Lib/test/test_descr.py Log: subclasspropagation(): Squash two more bogus hash(x) == id(x) tests. Alas, because only the "x86 OpenBSD trunk" buildbot fails these tests, and test_descr stops after the first failure, there's no sane way for me to fix these short of fixing one and then waiting for the buildbot to reveal the next one. Modified: python/trunk/Lib/test/test_descr.py ============================================================================== --- python/trunk/Lib/test/test_descr.py (original) +++ python/trunk/Lib/test/test_descr.py Tue Apr 11 03:59:34 2006 @@ -3043,7 +3043,7 @@ class D(B, C): pass d = D() - vereq(hash(d), id(d)) + orig_hash = hash(d) # related to id(d) in platform-dependent ways A.__hash__ = lambda self: 42 vereq(hash(d), 42) C.__hash__ = lambda self: 314 @@ -3059,7 +3059,7 @@ del C.__hash__ vereq(hash(d), 42) del A.__hash__ - vereq(hash(d), id(d)) + vereq(hash(d), orig_hash) d.foo = 42 d.bar = 42 vereq(d.foo, 42) From python-checkins at python.org Tue Apr 11 04:59:48 2006 From: python-checkins at python.org (tim.peters) Date: Tue, 11 Apr 2006 04:59:48 +0200 (CEST) Subject: [Python-checkins] r45256 - python/trunk/Lib/ctypes/test/test_prototypes.py Message-ID: <20060411025948.CA2001E4007@bag.python.org> Author: tim.peters Date: Tue Apr 11 04:59:48 2006 New Revision: 45256 Modified: python/trunk/Lib/ctypes/test/test_prototypes.py Log: Try to repair what may be the last new test failure on the "x86 OpenBSD trunk" buildbot due to changing Python so that Python-exposed addresses are always non-negative. test_int_pointer_arg(): This line failed now whenever the box happened to assign an address to `ci` "with the sign bit set": self.failUnlessEqual(addressof(ci), func(byref(ci))) The problem is that the ctypes addressof() inherited "all addresses are non-negative now" from changes to PyLong_FromVoidPtr(), but byref() did not inherit that change and can still return a negative int. I don't know whether, or what, the ctypes implementation wants to do about that (possibly nothing), but in the meantime the test fails frequently. So, introduced a Python positive_address() function in the test module, that takes a purported machine address and, if negative, converts it to a non-negative value "with the same bits". This should leave the test passing under all versions of Python. Belated thanks to Armin Rigo for teaching me the sick trick ;-) for determining the # of bits in a machine pointer via abuse of the struct module. Modified: python/trunk/Lib/ctypes/test/test_prototypes.py ============================================================================== --- python/trunk/Lib/ctypes/test/test_prototypes.py (original) +++ python/trunk/Lib/ctypes/test/test_prototypes.py Tue Apr 11 04:59:48 2006 @@ -24,6 +24,19 @@ import _ctypes_test testdll = cdll.load(_ctypes_test.__file__) +# Return machine address `a` as a (possibly long) non-negative integer. +# Starting with Python 2.5, id(anything) is always non-negative, and +# the ctypes addressof() inherits that via PyLong_FromVoidPtr(). +def positive_address(a): + if a >= 0: + return a + # View the bits in `a` as unsigned instead. + import struct + num_bits = struct.calcsize("P") * 8 # num bits in native machine address + a += 1L << num_bits + assert a >= 0 + return a + def c_wbuffer(init): n = len(init) + 1 return (c_wchar * n)(*init) @@ -43,7 +56,8 @@ ci = c_int(0) func.argtypes = POINTER(c_int), - self.failUnlessEqual(addressof(ci), func(byref(ci))) + self.failUnlessEqual(positive_address(addressof(ci)), + positive_address(func(byref(ci)))) func.argtypes = c_char_p, self.assertRaises(ArgumentError, func, byref(ci)) From buildbot at python.org Tue Apr 11 05:06:57 2006 From: buildbot at python.org (buildbot at python.org) Date: Tue, 11 Apr 2006 03:06:57 +0000 Subject: [Python-checkins] buildbot failure in x86 Ubuntu dapper (icc) trunk Message-ID: <20060411030657.6334C1E4007@bag.python.org> The Buildbot has detected a new failure of x86 Ubuntu dapper (icc) trunk. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520Ubuntu%2520dapper%2520%2528icc%2529%2520trunk/builds/76 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: tim.peters BUILD FAILED: failed failed slave lost sincerely, -The Buildbot From python-checkins at python.org Tue Apr 11 07:39:16 2006 From: python-checkins at python.org (anthony.baxter) Date: Tue, 11 Apr 2006 07:39:16 +0200 (CEST) Subject: [Python-checkins] r45257 - python/trunk/Parser/bitset.c python/trunk/Parser/firstsets.c python/trunk/Parser/grammar.c python/trunk/Parser/myreadline.c python/trunk/Parser/parser.c python/trunk/Parser/pgen.c python/trunk/Parser/tokenizer.c Message-ID: <20060411053916.100E01E4007@bag.python.org> Author: anthony.baxter Date: Tue Apr 11 07:39:14 2006 New Revision: 45257 Modified: python/trunk/Parser/bitset.c python/trunk/Parser/firstsets.c python/trunk/Parser/grammar.c python/trunk/Parser/myreadline.c python/trunk/Parser/parser.c python/trunk/Parser/pgen.c python/trunk/Parser/tokenizer.c Log: Fix the code in Parser/ to also compile with C++. This was mostly casts for malloc/realloc type functions, as well as renaming one variable called 'new' in tokensizer.c. Still lots more to be done, going to be checking in one chunk at a time or the patch will be massively huge. Still compiles ok with gcc. Modified: python/trunk/Parser/bitset.c ============================================================================== --- python/trunk/Parser/bitset.c (original) +++ python/trunk/Parser/bitset.c Tue Apr 11 07:39:14 2006 @@ -8,7 +8,7 @@ newbitset(int nbits) { int nbytes = NBYTES(nbits); - bitset ss = PyObject_MALLOC(sizeof(BYTE) * nbytes); + bitset ss = (char *)PyObject_MALLOC(sizeof(BYTE) * nbytes); if (ss == NULL) Py_FatalError("no mem for bitset"); Modified: python/trunk/Parser/firstsets.c ============================================================================== --- python/trunk/Parser/firstsets.c (original) +++ python/trunk/Parser/firstsets.c Tue Apr 11 07:39:14 2006 @@ -59,7 +59,7 @@ nbits = g->g_ll.ll_nlabels; result = newbitset(nbits); - sym = PyObject_MALLOC(sizeof(int)); + sym = (int *)PyObject_MALLOC(sizeof(int)); if (sym == NULL) Py_FatalError("no mem for new sym in calcfirstset"); nsyms = 1; @@ -73,7 +73,8 @@ break; } if (j >= nsyms) { /* New label */ - sym = PyObject_REALLOC(sym, sizeof(int) * (nsyms + 1)); + sym = (int *)PyObject_REALLOC(sym, + sizeof(int) * (nsyms + 1)); if (sym == NULL) Py_FatalError( "no mem to resize sym in calcfirstset"); Modified: python/trunk/Parser/grammar.c ============================================================================== --- python/trunk/Parser/grammar.c (original) +++ python/trunk/Parser/grammar.c Tue Apr 11 07:39:14 2006 @@ -20,7 +20,7 @@ { grammar *g; - g = PyObject_MALLOC(sizeof(grammar)); + g = (grammar *)PyObject_MALLOC(sizeof(grammar)); if (g == NULL) Py_FatalError("no mem for new grammar"); g->g_ndfas = 0; @@ -37,7 +37,8 @@ { dfa *d; - g->g_dfa = PyObject_REALLOC(g->g_dfa, sizeof(dfa) * (g->g_ndfas + 1)); + g->g_dfa = (dfa *)PyObject_REALLOC(g->g_dfa, + sizeof(dfa) * (g->g_ndfas + 1)); if (g->g_dfa == NULL) Py_FatalError("no mem to resize dfa in adddfa"); d = &g->g_dfa[g->g_ndfas++]; @@ -55,7 +56,7 @@ { state *s; - d->d_state = PyObject_REALLOC(d->d_state, + d->d_state = (state *)PyObject_REALLOC(d->d_state, sizeof(state) * (d->d_nstates + 1)); if (d->d_state == NULL) Py_FatalError("no mem to resize state in addstate"); @@ -79,7 +80,7 @@ assert(0 <= to && to < d->d_nstates); s = &d->d_state[from]; - s->s_arc = PyObject_REALLOC(s->s_arc, sizeof(arc) * (s->s_narcs + 1)); + s->s_arc = (arc *)PyObject_REALLOC(s->s_arc, sizeof(arc) * (s->s_narcs + 1)); if (s->s_arc == NULL) Py_FatalError("no mem to resize arc list in addarc"); a = &s->s_arc[s->s_narcs++]; @@ -98,7 +99,7 @@ strcmp(ll->ll_label[i].lb_str, str) == 0) return i; } - ll->ll_label = PyObject_REALLOC(ll->ll_label, + ll->ll_label = (label *)PyObject_REALLOC(ll->ll_label, sizeof(label) * (ll->ll_nlabels + 1)); if (ll->ll_label == NULL) Py_FatalError("no mem to resize labellist in addlabel"); @@ -197,7 +198,7 @@ name_len = p - src; else name_len = strlen(src); - dest = malloc(name_len + 1); + dest = (char *)malloc(name_len + 1); strncpy(dest, src, name_len); dest[name_len] = '\0'; free(lb->lb_str); Modified: python/trunk/Parser/myreadline.c ============================================================================== --- python/trunk/Parser/myreadline.c (original) +++ python/trunk/Parser/myreadline.c Tue Apr 11 07:39:14 2006 @@ -111,7 +111,7 @@ size_t n; char *p; n = 100; - if ((p = PyObject_MALLOC(n)) == NULL) + if ((p = (char *)PyObject_MALLOC(n)) == NULL) return NULL; fflush(sys_stdout); #ifndef RISCOS @@ -141,7 +141,7 @@ n = strlen(p); while (n > 0 && p[n-1] != '\n') { size_t incr = n+2; - p = PyObject_REALLOC(p, n + incr); + p = (char *)PyObject_REALLOC(p, n + incr); if (p == NULL) return NULL; if (incr > INT_MAX) { @@ -151,7 +151,7 @@ break; n += strlen(p+n); } - return PyObject_REALLOC(p, n+1); + return (char *)PyObject_REALLOC(p, n+1); } Modified: python/trunk/Parser/parser.c ============================================================================== --- python/trunk/Parser/parser.c (original) +++ python/trunk/Parser/parser.c Tue Apr 11 07:39:14 2006 @@ -75,7 +75,7 @@ if (!g->g_accel) PyGrammar_AddAccelerators(g); - ps = PyMem_MALLOC(sizeof(parser_state)); + ps = (parser_state *)PyMem_MALLOC(sizeof(parser_state)); if (ps == NULL) return NULL; ps->p_grammar = g; Modified: python/trunk/Parser/pgen.c ============================================================================== --- python/trunk/Parser/pgen.c (original) +++ python/trunk/Parser/pgen.c Tue Apr 11 07:39:14 2006 @@ -49,8 +49,8 @@ { nfastate *st; - nf->nf_state = PyObject_REALLOC(nf->nf_state, sizeof(nfastate) * - (nf->nf_nstates + 1)); + nf->nf_state = (nfastate *)PyObject_REALLOC(nf->nf_state, + sizeof(nfastate) * (nf->nf_nstates + 1)); if (nf->nf_state == NULL) Py_FatalError("out of mem"); st = &nf->nf_state[nf->nf_nstates++]; @@ -66,7 +66,7 @@ nfaarc *ar; st = &nf->nf_state[from]; - st->st_arc = PyObject_REALLOC(st->st_arc, + st->st_arc = (nfaarc *)PyObject_REALLOC(st->st_arc, sizeof(nfaarc) * (st->st_narcs + 1)); if (st->st_arc == NULL) Py_FatalError("out of mem"); @@ -81,7 +81,7 @@ nfa *nf; static int type = NT_OFFSET; /* All types will be disjunct */ - nf = PyObject_MALLOC(sizeof(nfa)); + nf = (nfa *)PyObject_MALLOC(sizeof(nfa)); if (nf == NULL) Py_FatalError("no mem for new nfa"); nf->nf_type = type++; @@ -106,7 +106,7 @@ { nfagrammar *gr; - gr = PyObject_MALLOC(sizeof(nfagrammar)); + gr = (nfagrammar *)PyObject_MALLOC(sizeof(nfagrammar)); if (gr == NULL) Py_FatalError("no mem for new nfa grammar"); gr->gr_nnfas = 0; @@ -123,7 +123,7 @@ nfa *nf; nf = newnfa(name); - gr->gr_nfa = PyObject_REALLOC(gr->gr_nfa, + gr->gr_nfa = (nfa **)PyObject_REALLOC(gr->gr_nfa, sizeof(nfa) * (gr->gr_nnfas + 1)); if (gr->gr_nfa == NULL) Py_FatalError("out of mem"); @@ -364,7 +364,7 @@ typedef struct _ss_state { bitset ss_ss; int ss_narcs; - ss_arc *ss_arc; + struct _ss_arc *ss_arc; int ss_deleted; int ss_finish; int ss_rename; @@ -395,7 +395,7 @@ ss = newbitset(nbits); addclosure(ss, nf, nf->nf_start); - xx_state = PyObject_MALLOC(sizeof(ss_state)); + xx_state = (ss_state *)PyObject_MALLOC(sizeof(ss_state)); if (xx_state == NULL) Py_FatalError("no mem for xx_state in makedfa"); xx_nstates = 1; @@ -435,8 +435,8 @@ } /* Add new arc for this state */ size = sizeof(ss_arc) * (yy->ss_narcs + 1); - yy->ss_arc = PyObject_REALLOC(yy->ss_arc, - size); + yy->ss_arc = (ss_arc *)PyObject_REALLOC( + yy->ss_arc, size); if (yy->ss_arc == NULL) Py_FatalError("out of mem"); zz = &yy->ss_arc[yy->ss_narcs++]; @@ -459,7 +459,8 @@ } } size = sizeof(ss_state) * (xx_nstates + 1); - xx_state = PyObject_REALLOC(xx_state, size); + xx_state = (ss_state *)PyObject_REALLOC(xx_state, + size); if (xx_state == NULL) Py_FatalError("out of mem"); zz->sa_arrow = xx_nstates; Modified: python/trunk/Parser/tokenizer.c ============================================================================== --- python/trunk/Parser/tokenizer.c (original) +++ python/trunk/Parser/tokenizer.c Tue Apr 11 07:39:14 2006 @@ -105,7 +105,8 @@ static struct tok_state * tok_new(void) { - struct tok_state *tok = PyMem_MALLOC(sizeof(struct tok_state)); + struct tok_state *tok = (struct tok_state *)PyMem_MALLOC( + sizeof(struct tok_state)); if (tok == NULL) return NULL; tok->buf = tok->cur = tok->end = tok->inp = tok->start = NULL; @@ -775,38 +776,38 @@ return Py_CHARMASK(*tok->cur++); } if (tok->prompt != NULL) { - char *new = PyOS_Readline(stdin, stdout, tok->prompt); + char *newtok = PyOS_Readline(stdin, stdout, tok->prompt); if (tok->nextprompt != NULL) tok->prompt = tok->nextprompt; - if (new == NULL) + if (newtok == NULL) tok->done = E_INTR; - else if (*new == '\0') { - PyObject_FREE(new); + else if (*newtok == '\0') { + PyObject_FREE(newtok); tok->done = E_EOF; } #if !defined(PGEN) && defined(Py_USING_UNICODE) - else if (tok_stdin_decode(tok, &new) != 0) - PyObject_FREE(new); + else if (tok_stdin_decode(tok, &newtok) != 0) + PyObject_FREE(newtok); #endif else if (tok->start != NULL) { size_t start = tok->start - tok->buf; size_t oldlen = tok->cur - tok->buf; - size_t newlen = oldlen + strlen(new); + size_t newlen = oldlen + strlen(newtok); char *buf = tok->buf; buf = (char *)PyObject_REALLOC(buf, newlen+1); tok->lineno++; if (buf == NULL) { PyObject_FREE(tok->buf); tok->buf = NULL; - PyObject_FREE(new); + PyObject_FREE(newtok); tok->done = E_NOMEM; return EOF; } tok->buf = buf; tok->cur = tok->buf + oldlen; tok->line_start = tok->cur; - strcpy(tok->buf + oldlen, new); - PyObject_FREE(new); + strcpy(tok->buf + oldlen, newtok); + PyObject_FREE(newtok); tok->inp = tok->buf + newlen; tok->end = tok->inp + 1; tok->start = tok->buf + start; @@ -815,7 +816,7 @@ tok->lineno++; if (tok->buf != NULL) PyObject_FREE(tok->buf); - tok->buf = new; + tok->buf = newtok; tok->line_start = tok->buf; tok->cur = tok->buf; tok->line_start = tok->buf; From python-checkins at python.org Tue Apr 11 08:27:13 2006 From: python-checkins at python.org (georg.brandl) Date: Tue, 11 Apr 2006 08:27:13 +0200 (CEST) Subject: [Python-checkins] r45258 - python/trunk/Doc/whatsnew/whatsnew25.tex Message-ID: <20060411062713.27F5D1E4007@bag.python.org> Author: georg.brandl Date: Tue Apr 11 08:27:12 2006 New Revision: 45258 Modified: python/trunk/Doc/whatsnew/whatsnew25.tex Log: Typo fix. Modified: python/trunk/Doc/whatsnew/whatsnew25.tex ============================================================================== --- python/trunk/Doc/whatsnew/whatsnew25.tex (original) +++ python/trunk/Doc/whatsnew/whatsnew25.tex Tue Apr 11 08:27:12 2006 @@ -1086,7 +1086,7 @@ A subset of Fredrik Lundh's ElementTree library for processing XML has been added to the standard library as \module{xmlcore.etree}. The -vailable modules are +available modules are \module{ElementTree}, \module{ElementPath}, and \module{ElementInclude} from ElementTree 1.2.6. The \module{cElementTree} accelerator module is also included. From python-checkins at python.org Tue Apr 11 08:47:45 2006 From: python-checkins at python.org (georg.brandl) Date: Tue, 11 Apr 2006 08:47:45 +0200 (CEST) Subject: [Python-checkins] r45259 - in python/trunk: Misc/NEWS Modules/posixmodule.c Message-ID: <20060411064745.B00B81E4007@bag.python.org> Author: georg.brandl Date: Tue Apr 11 08:47:43 2006 New Revision: 45259 Modified: python/trunk/Misc/NEWS python/trunk/Modules/posixmodule.c Log: Bug #1467952: os.listdir() now correctly raises an error if readdir() fails with an error condition. Modified: python/trunk/Misc/NEWS ============================================================================== --- python/trunk/Misc/NEWS (original) +++ python/trunk/Misc/NEWS Tue Apr 11 08:47:43 2006 @@ -41,6 +41,9 @@ Extension Modules ----------------- +- Bug #1467952: os.listdir() now correctly raises an error if readdir() + fails with an error condition. + - Fix bsddb.db.DBError derived exceptions so they can be unpickled. Library Modified: python/trunk/Modules/posixmodule.c ============================================================================== --- python/trunk/Modules/posixmodule.c (original) +++ python/trunk/Modules/posixmodule.c Tue Apr 11 08:47:43 2006 @@ -1901,6 +1901,12 @@ } Py_DECREF(v); } + if (errno != 0 && d != NULL) { + /* readdir() returned NULL and set errno */ + closedir(dirp); + Py_DECREF(d); + return posix_error_with_allocated_filename(name); + } closedir(dirp); PyMem_Free(name); From python-checkins at python.org Tue Apr 11 08:51:27 2006 From: python-checkins at python.org (georg.brandl) Date: Tue, 11 Apr 2006 08:51:27 +0200 (CEST) Subject: [Python-checkins] r45260 - in python/branches/release24-maint: Doc/lib/libos.tex Misc/NEWS Modules/posixmodule.c Message-ID: <20060411065127.256EF1E4007@bag.python.org> Author: georg.brandl Date: Tue Apr 11 08:51:25 2006 New Revision: 45260 Modified: python/branches/release24-maint/Doc/lib/libos.tex python/branches/release24-maint/Misc/NEWS python/branches/release24-maint/Modules/posixmodule.c Log: Bug #1467952: backport: make os.listdir() raise if readdir() fails Modified: python/branches/release24-maint/Doc/lib/libos.tex ============================================================================== --- python/branches/release24-maint/Doc/lib/libos.tex (original) +++ python/branches/release24-maint/Doc/lib/libos.tex Tue Apr 11 08:51:25 2006 @@ -540,7 +540,8 @@ This function is intended for low-level I/O. For normal usage, use the built-in function \function{open()}, which returns a ``file object'' with \method{read()} and \method{write()} methods (and many -more). +more). To wrap a file descriptor in a ``file object'', use +\function{fdopen()}. \end{notice} \end{funcdesc} Modified: python/branches/release24-maint/Misc/NEWS ============================================================================== --- python/branches/release24-maint/Misc/NEWS (original) +++ python/branches/release24-maint/Misc/NEWS Tue Apr 11 08:51:25 2006 @@ -15,6 +15,9 @@ Extension Modules ----------------- +- Bug #1467952: os.listdir() now correctly raises an error if readdir() + fails with an error condition. + - Fix bsddb.db.DBError derived exceptions so they can be unpickled. Library Modified: python/branches/release24-maint/Modules/posixmodule.c ============================================================================== --- python/branches/release24-maint/Modules/posixmodule.c (original) +++ python/branches/release24-maint/Modules/posixmodule.c Tue Apr 11 08:51:25 2006 @@ -1699,6 +1699,12 @@ } Py_DECREF(v); } + if (errno != 0 && d != NULL) { + /* readdir() returned NULL and set errno */ + closedir(dirp); + Py_DECREF(d); + return posix_error_with_allocated_filename(name); + } closedir(dirp); PyMem_Free(name); From buildbot at python.org Tue Apr 11 08:54:12 2006 From: buildbot at python.org (buildbot at python.org) Date: Tue, 11 Apr 2006 06:54:12 +0000 Subject: [Python-checkins] buildbot warnings in amd64 gentoo trunk Message-ID: <20060411065412.B51301E4009@bag.python.org> The Buildbot has detected a new failure of amd64 gentoo trunk. Full details are available at: http://www.python.org/dev/buildbot/all/amd64%2520gentoo%2520trunk/builds/397 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: georg.brandl Build Had Warnings: warnings test sincerely, -The Buildbot From python-checkins at python.org Tue Apr 11 08:54:32 2006 From: python-checkins at python.org (anthony.baxter) Date: Tue, 11 Apr 2006 08:54:32 +0200 (CEST) Subject: [Python-checkins] r45261 - python/trunk/Objects/bufferobject.c python/trunk/Objects/classobject.c python/trunk/Objects/enumobject.c python/trunk/Objects/fileobject.c python/trunk/Objects/floatobject.c python/trunk/Objects/intobject.c python/trunk/Objects/listobject.c python/trunk/Objects/longobject.c Message-ID: <20060411065432.4FEBC1E4007@bag.python.org> Author: anthony.baxter Date: Tue Apr 11 08:54:30 2006 New Revision: 45261 Modified: python/trunk/Objects/bufferobject.c python/trunk/Objects/classobject.c python/trunk/Objects/enumobject.c python/trunk/Objects/fileobject.c python/trunk/Objects/floatobject.c python/trunk/Objects/intobject.c python/trunk/Objects/listobject.c python/trunk/Objects/longobject.c Log: More C++-compliance. Note especially listobject.c - to get C++ to accept the PyTypeObject structures, I had to make prototypes for the functions, and move the structure definition ahead of the functions. I'd dearly like a better way to do this - to change this would make for a massive set of changes to the codebase. There's still some warnings - this is purely to get rid of errors first. Modified: python/trunk/Objects/bufferobject.c ============================================================================== --- python/trunk/Objects/bufferobject.c (original) +++ python/trunk/Objects/bufferobject.c Tue Apr 11 08:54:30 2006 @@ -169,7 +169,7 @@ } /* XXX: check for overflow in multiply */ /* Inline PyObject_New */ - o = PyObject_MALLOC(sizeof(*b) + size); + o = (PyObject *)PyObject_MALLOC(sizeof(*b) + size); if ( o == NULL ) return PyErr_NoMemory(); b = (PyBufferObject *) PyObject_INIT(o, &PyBuffer_Type); @@ -305,7 +305,7 @@ Py_ssize_t size; if (!get_buf(self, &ptr, &size)) return NULL; - return PyString_FromStringAndSize(ptr, size); + return PyString_FromStringAndSize((const char *)ptr, size); } /* Sequence methods */ Modified: python/trunk/Objects/classobject.c ============================================================================== --- python/trunk/Objects/classobject.c (original) +++ python/trunk/Objects/classobject.c Tue Apr 11 08:54:30 2006 @@ -208,7 +208,7 @@ { register PyObject *v; register char *sname = PyString_AsString(name); - PyClassObject *class; + PyClassObject *klass; descrgetfunc f; if (sname[0] == '_' && sname[1] == '_') { @@ -234,7 +234,7 @@ return v; } } - v = class_lookup(op, name, &class); + v = class_lookup(op, name, &klass); if (v == NULL) { PyErr_Format(PyExc_AttributeError, "class %.50s has no attribute '%.400s'", @@ -481,23 +481,23 @@ }; int -PyClass_IsSubclass(PyObject *class, PyObject *base) +PyClass_IsSubclass(PyObject *klass, PyObject *base) { Py_ssize_t i, n; PyClassObject *cp; - if (class == base) + if (klass == base) return 1; if (PyTuple_Check(base)) { n = PyTuple_GET_SIZE(base); for (i = 0; i < n; i++) { - if (PyClass_IsSubclass(class, PyTuple_GET_ITEM(base, i))) + if (PyClass_IsSubclass(klass, PyTuple_GET_ITEM(base, i))) return 1; } return 0; } - if (class == NULL || !PyClass_Check(class)) + if (klass == NULL || !PyClass_Check(klass)) return 0; - cp = (PyClassObject *)class; + cp = (PyClassObject *)klass; n = PyTuple_Size(cp->cl_bases); for (i = 0; i < n; i++) { if (PyClass_IsSubclass(PyTuple_GetItem(cp->cl_bases, i), base)) @@ -719,7 +719,7 @@ instance_getattr2(register PyInstanceObject *inst, PyObject *name) { register PyObject *v; - PyClassObject *class; + PyClassObject *klass; descrgetfunc f; v = PyDict_GetItem(inst->in_dict, name); @@ -727,7 +727,7 @@ Py_INCREF(v); return v; } - v = class_lookup(inst->in_class, name, &class); + v = class_lookup(inst->in_class, name, &klass); if (v != NULL) { Py_INCREF(v); f = TP_DESCR_GET(v->ob_type); @@ -767,7 +767,7 @@ _PyInstance_Lookup(PyObject *pinst, PyObject *name) { PyObject *v; - PyClassObject *class; + PyClassObject *klass; PyInstanceObject *inst; /* pinst cast to the right type */ assert(PyInstance_Check(pinst)); @@ -777,7 +777,7 @@ v = PyDict_GetItem(inst->in_dict, name); if (v == NULL) - v = class_lookup(inst->in_class, name, &class); + v = class_lookup(inst->in_class, name, &klass); return v; } @@ -2123,7 +2123,7 @@ static PyMethodObject *free_list; PyObject * -PyMethod_New(PyObject *func, PyObject *self, PyObject *class) +PyMethod_New(PyObject *func, PyObject *self, PyObject *klass) { register PyMethodObject *im; if (!PyCallable_Check(func)) { @@ -2145,8 +2145,8 @@ im->im_func = func; Py_XINCREF(self); im->im_self = self; - Py_XINCREF(class); - im->im_class = class; + Py_XINCREF(klass); + im->im_class = klass; _PyObject_GC_TRACK(im); return (PyObject *)im; } @@ -2368,15 +2368,15 @@ } static void -getclassname(PyObject *class, char *buf, int bufsize) +getclassname(PyObject *klass, char *buf, int bufsize) { PyObject *name; assert(bufsize > 1); strcpy(buf, "?"); /* Default outcome */ - if (class == NULL) + if (klass == NULL) return; - name = PyObject_GetAttrString(class, "__name__"); + name = PyObject_GetAttrString(klass, "__name__"); if (name == NULL) { /* This function cannot return an exception */ PyErr_Clear(); @@ -2392,7 +2392,7 @@ static void getinstclassname(PyObject *inst, char *buf, int bufsize) { - PyObject *class; + PyObject *klass; if (inst == NULL) { assert(bufsize > 0 && (size_t)bufsize > strlen("nothing")); @@ -2400,22 +2400,22 @@ return; } - class = PyObject_GetAttrString(inst, "__class__"); - if (class == NULL) { + klass = PyObject_GetAttrString(inst, "__class__"); + if (klass == NULL) { /* This function cannot return an exception */ PyErr_Clear(); - class = (PyObject *)(inst->ob_type); - Py_INCREF(class); + klass = (PyObject *)(inst->ob_type); + Py_INCREF(klass); } - getclassname(class, buf, bufsize); - Py_XDECREF(class); + getclassname(klass, buf, bufsize); + Py_XDECREF(klass); } static PyObject * instancemethod_call(PyObject *func, PyObject *arg, PyObject *kw) { PyObject *self = PyMethod_GET_SELF(func); - PyObject *class = PyMethod_GET_CLASS(func); + PyObject *klass = PyMethod_GET_CLASS(func); PyObject *result; func = PyMethod_GET_FUNCTION(func); @@ -2428,14 +2428,14 @@ if (self == NULL) ok = 0; else { - ok = PyObject_IsInstance(self, class); + ok = PyObject_IsInstance(self, klass); if (ok < 0) return NULL; } if (!ok) { char clsbuf[256]; char instbuf[256]; - getclassname(class, clsbuf, sizeof(clsbuf)); + getclassname(klass, clsbuf, sizeof(clsbuf)); getinstclassname(self, instbuf, sizeof(instbuf)); PyErr_Format(PyExc_TypeError, "unbound method %s%s must be called with " Modified: python/trunk/Objects/enumobject.c ============================================================================== --- python/trunk/Objects/enumobject.c (original) +++ python/trunk/Objects/enumobject.c Tue Apr 11 08:54:30 2006 @@ -9,8 +9,6 @@ PyObject* en_result; /* result tuple */ } enumobject; -PyTypeObject PyEnum_Type; - static PyObject * enum_new(PyTypeObject *type, PyObject *args, PyObject *kwds) { Modified: python/trunk/Objects/fileobject.c ============================================================================== --- python/trunk/Objects/fileobject.c (original) +++ python/trunk/Objects/fileobject.c Tue Apr 11 08:54:30 2006 @@ -313,7 +313,8 @@ PyMem_Free(file->f_setbuf); file->f_setbuf = NULL; } else { - file->f_setbuf = PyMem_Realloc(file->f_setbuf, bufsize); + file->f_setbuf = (char *)PyMem_Realloc(file->f_setbuf, + bufsize); } #ifdef HAVE_SETVBUF setvbuf(file->f_fp, file->f_setbuf, type, bufsize); @@ -1391,7 +1392,7 @@ goto cleanup; } totalread += nread; - p = memchr(buffer+nfilled, '\n', nread); + p = (char *)memchr(buffer+nfilled, '\n', nread); if (p == NULL) { /* Need a larger buffer to fit this line */ nfilled += nread; @@ -1431,7 +1432,7 @@ if (err != 0) goto error; q = p; - p = memchr(q, '\n', end-q); + p = (char *)memchr(q, '\n', end-q); } while (p != NULL); /* Move the remaining incomplete line to the start */ nfilled = end-q; @@ -1809,7 +1810,7 @@ else drop_readahead(f); } - if ((f->f_buf = PyMem_Malloc(bufsize)) == NULL) { + if ((f->f_buf = (char *)PyMem_Malloc(bufsize)) == NULL) { PyErr_NoMemory(); return -1; } @@ -1852,7 +1853,7 @@ if (len == 0) return (PyStringObject *) PyString_FromStringAndSize(NULL, skip); - bufptr = memchr(f->f_bufptr, '\n', len); + bufptr = (char *)memchr(f->f_bufptr, '\n', len); if (bufptr != NULL) { bufptr++; /* Count the '\n' */ len = bufptr - f->f_bufptr; Modified: python/trunk/Objects/floatobject.c ============================================================================== --- python/trunk/Objects/floatobject.c (original) +++ python/trunk/Objects/floatobject.c Tue Apr 11 08:54:30 2006 @@ -959,21 +959,21 @@ static PyObject * float_subtype_new(PyTypeObject *type, PyObject *args, PyObject *kwds) { - PyObject *tmp, *new; + PyObject *tmp, *newobj; assert(PyType_IsSubtype(type, &PyFloat_Type)); tmp = float_new(&PyFloat_Type, args, kwds); if (tmp == NULL) return NULL; assert(PyFloat_CheckExact(tmp)); - new = type->tp_alloc(type, 0); - if (new == NULL) { + newobj = type->tp_alloc(type, 0); + if (newobj == NULL) { Py_DECREF(tmp); return NULL; } - ((PyFloatObject *)new)->ob_fval = ((PyFloatObject *)tmp)->ob_fval; + ((PyFloatObject *)newobj)->ob_fval = ((PyFloatObject *)tmp)->ob_fval; Py_DECREF(tmp); - return new; + return newobj; } static PyObject * Modified: python/trunk/Objects/intobject.c ============================================================================== --- python/trunk/Objects/intobject.c (original) +++ python/trunk/Objects/intobject.c Tue Apr 11 08:54:30 2006 @@ -376,7 +376,7 @@ PyInt_FromUnicode(Py_UNICODE *s, Py_ssize_t length, int base) { PyObject *result; - char *buffer = PyMem_MALLOC(length+1); + char *buffer = (char *)PyMem_MALLOC(length+1); if (buffer == NULL) return NULL; @@ -982,7 +982,7 @@ static PyObject * int_subtype_new(PyTypeObject *type, PyObject *args, PyObject *kwds) { - PyObject *tmp, *new; + PyObject *tmp, *newobj; long ival; assert(PyType_IsSubtype(type, &PyInt_Type)); @@ -999,14 +999,14 @@ ival = ((PyIntObject *)tmp)->ob_ival; } - new = type->tp_alloc(type, 0); - if (new == NULL) { + newobj = type->tp_alloc(type, 0); + if (newobj == NULL) { Py_DECREF(tmp); return NULL; } - ((PyIntObject *)new)->ob_ival = ival; + ((PyIntObject *)newobj)->ob_ival = ival; Py_DECREF(tmp); - return new; + return newobj; } static PyObject * Modified: python/trunk/Objects/listobject.c ============================================================================== --- python/trunk/Objects/listobject.c (original) +++ python/trunk/Objects/listobject.c Tue Apr 11 08:54:30 2006 @@ -1805,28 +1805,11 @@ PyObject *value; } sortwrapperobject; -static PyTypeObject sortwrapper_type; - +PyDoc_STRVAR(sortwrapper_doc, "Object wrapper with a custom sort key."); static PyObject * -sortwrapper_richcompare(sortwrapperobject *a, sortwrapperobject *b, int op) -{ - if (!PyObject_TypeCheck(b, &sortwrapper_type)) { - PyErr_SetString(PyExc_TypeError, - "expected a sortwrapperobject"); - return NULL; - } - return PyObject_RichCompare(a->key, b->key, op); -} - +sortwrapper_richcompare(sortwrapperobject *, sortwrapperobject *, int); static void -sortwrapper_dealloc(sortwrapperobject *so) -{ - Py_XDECREF(so->key); - Py_XDECREF(so->value); - PyObject_Del(so); -} - -PyDoc_STRVAR(sortwrapper_doc, "Object wrapper with a custom sort key."); +sortwrapper_dealloc(sortwrapperobject *); static PyTypeObject sortwrapper_type = { PyObject_HEAD_INIT(&PyType_Type) @@ -1858,6 +1841,26 @@ (richcmpfunc)sortwrapper_richcompare, /* tp_richcompare */ }; + +static PyObject * +sortwrapper_richcompare(sortwrapperobject *a, sortwrapperobject *b, int op) +{ + if (!PyObject_TypeCheck(b, &sortwrapper_type)) { + PyErr_SetString(PyExc_TypeError, + "expected a sortwrapperobject"); + return NULL; + } + return PyObject_RichCompare(a->key, b->key, op); +} + +static void +sortwrapper_dealloc(sortwrapperobject *so) +{ + Py_XDECREF(so->key); + Py_XDECREF(so->value); + PyObject_Del(so); +} + /* Returns a new reference to a sortwrapper. Consumes the references to the two underlying objects. */ @@ -2698,7 +2701,53 @@ PyListObject *it_seq; /* Set to NULL when iterator is exhausted */ } listiterobject; -PyTypeObject PyListIter_Type; +static PyObject *list_iter(PyObject *); +static void listiter_dealloc(listiterobject *); +static int listiter_traverse(listiterobject *, visitproc, void *); +static PyObject *listiter_next(listiterobject *); +static PyObject *listiter_len(listiterobject *); + +PyDoc_STRVAR(length_hint_doc, "Private method returning an estimate of len(list(it))."); + +static PyMethodDef listiter_methods[] = { + {"__length_hint__", (PyCFunction)listiter_len, METH_NOARGS, length_hint_doc}, + {NULL, NULL} /* sentinel */ +}; + +PyTypeObject PyListIter_Type = { + PyObject_HEAD_INIT(&PyType_Type) + 0, /* ob_size */ + "listiterator", /* tp_name */ + sizeof(listiterobject), /* tp_basicsize */ + 0, /* tp_itemsize */ + /* methods */ + (destructor)listiter_dealloc, /* tp_dealloc */ + 0, /* tp_print */ + 0, /* tp_getattr */ + 0, /* tp_setattr */ + 0, /* tp_compare */ + 0, /* tp_repr */ + 0, /* tp_as_number */ + 0, /* tp_as_sequence */ + 0, /* tp_as_mapping */ + 0, /* tp_hash */ + 0, /* tp_call */ + 0, /* tp_str */ + PyObject_GenericGetAttr, /* tp_getattro */ + 0, /* tp_setattro */ + 0, /* tp_as_buffer */ + Py_TPFLAGS_DEFAULT | Py_TPFLAGS_HAVE_GC,/* tp_flags */ + 0, /* tp_doc */ + (traverseproc)listiter_traverse, /* tp_traverse */ + 0, /* tp_clear */ + 0, /* tp_richcompare */ + 0, /* tp_weaklistoffset */ + PyObject_SelfIter, /* tp_iter */ + (iternextfunc)listiter_next, /* tp_iternext */ + listiter_methods, /* tp_methods */ + 0, /* tp_members */ +}; + static PyObject * list_iter(PyObject *seq) @@ -2770,29 +2819,40 @@ } return PyInt_FromLong(0); } +/*********************** List Reverse Iterator **************************/ -PyDoc_STRVAR(length_hint_doc, "Private method returning an estimate of len(list(it))."); +typedef struct { + PyObject_HEAD + Py_ssize_t it_index; + PyListObject *it_seq; /* Set to NULL when iterator is exhausted */ +} listreviterobject; -static PyMethodDef listiter_methods[] = { - {"__length_hint__", (PyCFunction)listiter_len, METH_NOARGS, length_hint_doc}, - {NULL, NULL} /* sentinel */ +static PyObject *list_reversed(PyListObject *, PyObject *); +static void listreviter_dealloc(listreviterobject *); +static int listreviter_traverse(listreviterobject *, visitproc, void *); +static PyObject *listreviter_next(listreviterobject *); +static Py_ssize_t listreviter_len(listreviterobject *); + +static PySequenceMethods listreviter_as_sequence = { + (lenfunc)listreviter_len, /* sq_length */ + 0, /* sq_concat */ }; -PyTypeObject PyListIter_Type = { +PyTypeObject PyListRevIter_Type = { PyObject_HEAD_INIT(&PyType_Type) 0, /* ob_size */ - "listiterator", /* tp_name */ - sizeof(listiterobject), /* tp_basicsize */ + "listreverseiterator", /* tp_name */ + sizeof(listreviterobject), /* tp_basicsize */ 0, /* tp_itemsize */ /* methods */ - (destructor)listiter_dealloc, /* tp_dealloc */ + (destructor)listreviter_dealloc, /* tp_dealloc */ 0, /* tp_print */ 0, /* tp_getattr */ 0, /* tp_setattr */ 0, /* tp_compare */ 0, /* tp_repr */ 0, /* tp_as_number */ - 0, /* tp_as_sequence */ + &listreviter_as_sequence, /* tp_as_sequence */ 0, /* tp_as_mapping */ 0, /* tp_hash */ 0, /* tp_call */ @@ -2802,26 +2862,15 @@ 0, /* tp_as_buffer */ Py_TPFLAGS_DEFAULT | Py_TPFLAGS_HAVE_GC,/* tp_flags */ 0, /* tp_doc */ - (traverseproc)listiter_traverse, /* tp_traverse */ + (traverseproc)listreviter_traverse, /* tp_traverse */ 0, /* tp_clear */ 0, /* tp_richcompare */ 0, /* tp_weaklistoffset */ PyObject_SelfIter, /* tp_iter */ - (iternextfunc)listiter_next, /* tp_iternext */ - listiter_methods, /* tp_methods */ - 0, /* tp_members */ + (iternextfunc)listreviter_next, /* tp_iternext */ + 0, }; -/*********************** List Reverse Iterator **************************/ - -typedef struct { - PyObject_HEAD - Py_ssize_t it_index; - PyListObject *it_seq; /* Set to NULL when iterator is exhausted */ -} listreviterobject; - -PyTypeObject PyListRevIter_Type; - static PyObject * list_reversed(PyListObject *seq, PyObject *unused) { @@ -2884,40 +2933,3 @@ return len; } -static PySequenceMethods listreviter_as_sequence = { - (lenfunc)listreviter_len, /* sq_length */ - 0, /* sq_concat */ -}; - -PyTypeObject PyListRevIter_Type = { - PyObject_HEAD_INIT(&PyType_Type) - 0, /* ob_size */ - "listreverseiterator", /* tp_name */ - sizeof(listreviterobject), /* tp_basicsize */ - 0, /* tp_itemsize */ - /* methods */ - (destructor)listreviter_dealloc, /* tp_dealloc */ - 0, /* tp_print */ - 0, /* tp_getattr */ - 0, /* tp_setattr */ - 0, /* tp_compare */ - 0, /* tp_repr */ - 0, /* tp_as_number */ - &listreviter_as_sequence, /* tp_as_sequence */ - 0, /* tp_as_mapping */ - 0, /* tp_hash */ - 0, /* tp_call */ - 0, /* tp_str */ - PyObject_GenericGetAttr, /* tp_getattro */ - 0, /* tp_setattro */ - 0, /* tp_as_buffer */ - Py_TPFLAGS_DEFAULT | Py_TPFLAGS_HAVE_GC,/* tp_flags */ - 0, /* tp_doc */ - (traverseproc)listreviter_traverse, /* tp_traverse */ - 0, /* tp_clear */ - 0, /* tp_richcompare */ - 0, /* tp_weaklistoffset */ - PyObject_SelfIter, /* tp_iter */ - (iternextfunc)listreviter_next, /* tp_iternext */ - 0, -}; Modified: python/trunk/Objects/longobject.c ============================================================================== --- python/trunk/Objects/longobject.c (original) +++ python/trunk/Objects/longobject.c Tue Apr 11 08:54:30 2006 @@ -1476,7 +1476,7 @@ PyLong_FromUnicode(Py_UNICODE *u, Py_ssize_t length, int base) { PyObject *result; - char *buffer = PyMem_MALLOC(length+1); + char *buffer = (char *)PyMem_MALLOC(length+1); if (buffer == NULL) return NULL; @@ -3088,7 +3088,7 @@ static PyObject * long_subtype_new(PyTypeObject *type, PyObject *args, PyObject *kwds) { - PyLongObject *tmp, *new; + PyLongObject *tmp, *newobj; Py_ssize_t i, n; assert(PyType_IsSubtype(type, &PyLong_Type)); @@ -3099,17 +3099,17 @@ n = tmp->ob_size; if (n < 0) n = -n; - new = (PyLongObject *)type->tp_alloc(type, n); - if (new == NULL) { + newobj = (PyLongObject *)type->tp_alloc(type, n); + if (newobj == NULL) { Py_DECREF(tmp); return NULL; } - assert(PyLong_Check(new)); - new->ob_size = tmp->ob_size; + assert(PyLong_Check(newobj)); + newobj->ob_size = tmp->ob_size; for (i = 0; i < n; i++) - new->ob_digit[i] = tmp->ob_digit[i]; + newobj->ob_digit[i] = tmp->ob_digit[i]; Py_DECREF(tmp); - return (PyObject *)new; + return (PyObject *)newobj; } static PyObject * From buildbot at python.org Tue Apr 11 08:54:35 2006 From: buildbot at python.org (buildbot at python.org) Date: Tue, 11 Apr 2006 06:54:35 +0000 Subject: [Python-checkins] buildbot warnings in x86 gentoo trunk Message-ID: <20060411065435.C07891E4007@bag.python.org> The Buildbot has detected a new failure of x86 gentoo trunk. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520gentoo%2520trunk/builds/382 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: georg.brandl Build Had Warnings: warnings test sincerely, -The Buildbot From buildbot at python.org Tue Apr 11 08:56:28 2006 From: buildbot at python.org (buildbot at python.org) Date: Tue, 11 Apr 2006 06:56:28 +0000 Subject: [Python-checkins] buildbot warnings in x86 OpenBSD trunk Message-ID: <20060411065629.152F61E4007@bag.python.org> The Buildbot has detected a new failure of x86 OpenBSD trunk. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520OpenBSD%2520trunk/builds/229 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: georg.brandl Build Had Warnings: warnings test sincerely, -The Buildbot From buildbot at python.org Tue Apr 11 08:57:52 2006 From: buildbot at python.org (buildbot at python.org) Date: Tue, 11 Apr 2006 06:57:52 +0000 Subject: [Python-checkins] buildbot warnings in amd64 gentoo 2.4 Message-ID: <20060411065752.6BFA51E4007@bag.python.org> The Buildbot has detected a new failure of amd64 gentoo 2.4. Full details are available at: http://www.python.org/dev/buildbot/all/amd64%2520gentoo%25202.4/builds/83 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch branches/release24-maint] HEAD Blamelist: georg.brandl Build Had Warnings: warnings test sincerely, -The Buildbot From buildbot at python.org Tue Apr 11 08:58:09 2006 From: buildbot at python.org (buildbot at python.org) Date: Tue, 11 Apr 2006 06:58:09 +0000 Subject: [Python-checkins] buildbot warnings in x86 gentoo 2.4 Message-ID: <20060411065809.A2C0B1E4007@bag.python.org> The Buildbot has detected a new failure of x86 gentoo 2.4. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520gentoo%25202.4/builds/80 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch branches/release24-maint] HEAD Blamelist: georg.brandl Build Had Warnings: warnings test sincerely, -The Buildbot From buildbot at python.org Tue Apr 11 09:01:50 2006 From: buildbot at python.org (buildbot at python.org) Date: Tue, 11 Apr 2006 07:01:50 +0000 Subject: [Python-checkins] buildbot warnings in ppc Debian unstable 2.4 Message-ID: <20060411070150.8E6EA1E40CB@bag.python.org> The Buildbot has detected a new failure of ppc Debian unstable 2.4. Full details are available at: http://www.python.org/dev/buildbot/all/ppc%2520Debian%2520unstable%25202.4/builds/12 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch branches/release24-maint] HEAD Blamelist: georg.brandl Build Had Warnings: warnings test sincerely, -The Buildbot From buildbot at python.org Tue Apr 11 09:02:58 2006 From: buildbot at python.org (buildbot at python.org) Date: Tue, 11 Apr 2006 07:02:58 +0000 Subject: [Python-checkins] buildbot warnings in ppc Debian unstable trunk Message-ID: <20060411070258.968AD1E4007@bag.python.org> The Buildbot has detected a new failure of ppc Debian unstable trunk. Full details are available at: http://www.python.org/dev/buildbot/all/ppc%2520Debian%2520unstable%2520trunk/builds/69 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: anthony.baxter,georg.brandl Build Had Warnings: warnings test sincerely, -The Buildbot From python-checkins at python.org Tue Apr 11 09:04:07 2006 From: python-checkins at python.org (georg.brandl) Date: Tue, 11 Apr 2006 09:04:07 +0200 (CEST) Subject: [Python-checkins] r45262 - python/trunk/Modules/posixmodule.c Message-ID: <20060411070407.791751E4007@bag.python.org> Author: georg.brandl Date: Tue Apr 11 09:04:06 2006 New Revision: 45262 Modified: python/trunk/Modules/posixmodule.c Log: Clear errno before calling opendir() and readdir(). Modified: python/trunk/Modules/posixmodule.c ============================================================================== --- python/trunk/Modules/posixmodule.c (original) +++ python/trunk/Modules/posixmodule.c Tue Apr 11 09:04:06 2006 @@ -1845,6 +1845,7 @@ struct dirent *ep; int arg_is_unicode = 1; + errno = 0; if (!PyArg_ParseTuple(args, "U:listdir", &v)) { arg_is_unicode = 0; PyErr_Clear(); From buildbot at python.org Tue Apr 11 09:04:08 2006 From: buildbot at python.org (buildbot at python.org) Date: Tue, 11 Apr 2006 07:04:08 +0000 Subject: [Python-checkins] buildbot warnings in alpha Tru64 5.1 2.4 Message-ID: <20060411070408.61AB71E4007@bag.python.org> The Buildbot has detected a new failure of alpha Tru64 5.1 2.4. Full details are available at: http://www.python.org/dev/buildbot/all/alpha%2520Tru64%25205.1%25202.4/builds/36 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch branches/release24-maint] HEAD Blamelist: georg.brandl Build Had Warnings: warnings test sincerely, -The Buildbot From python-checkins at python.org Tue Apr 11 09:04:09 2006 From: python-checkins at python.org (georg.brandl) Date: Tue, 11 Apr 2006 09:04:09 +0200 (CEST) Subject: [Python-checkins] r45263 - python/branches/release24-maint/Modules/posixmodule.c Message-ID: <20060411070409.B14671E4007@bag.python.org> Author: georg.brandl Date: Tue Apr 11 09:04:09 2006 New Revision: 45263 Modified: python/branches/release24-maint/Modules/posixmodule.c Log: Clear errno before calling opendir() and readdir(). (backport from rev. 45262) Modified: python/branches/release24-maint/Modules/posixmodule.c ============================================================================== --- python/branches/release24-maint/Modules/posixmodule.c (original) +++ python/branches/release24-maint/Modules/posixmodule.c Tue Apr 11 09:04:09 2006 @@ -1648,6 +1648,7 @@ struct dirent *ep; int arg_is_unicode = 1; + errno = 0; if (!PyArg_ParseTuple(args, "U:listdir", &v)) { arg_is_unicode = 0; PyErr_Clear(); From buildbot at python.org Tue Apr 11 09:06:50 2006 From: buildbot at python.org (buildbot at python.org) Date: Tue, 11 Apr 2006 07:06:50 +0000 Subject: [Python-checkins] buildbot warnings in alpha Tru64 5.1 trunk Message-ID: <20060411070650.B7A5D1E4009@bag.python.org> The Buildbot has detected a new failure of alpha Tru64 5.1 trunk. Full details are available at: http://www.python.org/dev/buildbot/all/alpha%2520Tru64%25205.1%2520trunk/builds/164 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: anthony.baxter,georg.brandl Build Had Warnings: warnings test sincerely, -The Buildbot From buildbot at python.org Tue Apr 11 09:09:07 2006 From: buildbot at python.org (buildbot at python.org) Date: Tue, 11 Apr 2006 07:09:07 +0000 Subject: [Python-checkins] buildbot warnings in sparc solaris10 gcc 2.4 Message-ID: <20060411070907.621601E4010@bag.python.org> The Buildbot has detected a new failure of sparc solaris10 gcc 2.4. Full details are available at: http://www.python.org/dev/buildbot/all/sparc%2520solaris10%2520gcc%25202.4/builds/78 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch branches/release24-maint] HEAD Blamelist: georg.brandl Build Had Warnings: warnings test sincerely, -The Buildbot From buildbot at python.org Tue Apr 11 09:11:59 2006 From: buildbot at python.org (buildbot at python.org) Date: Tue, 11 Apr 2006 07:11:59 +0000 Subject: [Python-checkins] buildbot warnings in ia64 Debian unstable 2.4 Message-ID: <20060411071159.DBFBC1E4007@bag.python.org> The Buildbot has detected a new failure of ia64 Debian unstable 2.4. Full details are available at: http://www.python.org/dev/buildbot/all/ia64%2520Debian%2520unstable%25202.4/builds/12 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch branches/release24-maint] HEAD Blamelist: georg.brandl Build Had Warnings: warnings test sincerely, -The Buildbot From buildbot at python.org Tue Apr 11 09:12:34 2006 From: buildbot at python.org (buildbot at python.org) Date: Tue, 11 Apr 2006 07:12:34 +0000 Subject: [Python-checkins] buildbot warnings in sparc solaris10 gcc trunk Message-ID: <20060411071234.D62EB1E4007@bag.python.org> The Buildbot has detected a new failure of sparc solaris10 gcc trunk. Full details are available at: http://www.python.org/dev/buildbot/all/sparc%2520solaris10%2520gcc%2520trunk/builds/347 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: anthony.baxter,georg.brandl Build Had Warnings: warnings test sincerely, -The Buildbot From python-checkins at python.org Tue Apr 11 09:15:31 2006 From: python-checkins at python.org (martin.v.loewis) Date: Tue, 11 Apr 2006 09:15:31 +0200 (CEST) Subject: [Python-checkins] r45264 - python/trunk/configure python/trunk/configure.in Message-ID: <20060411071531.693D51E402B@bag.python.org> Author: martin.v.loewis Date: Tue Apr 11 09:15:30 2006 New Revision: 45264 Modified: python/trunk/configure python/trunk/configure.in Log: Correct test whether wchar_t is unsigned. Fixed crash in #1454485. Modified: python/trunk/configure ============================================================================== --- python/trunk/configure (original) +++ python/trunk/configure Tue Apr 11 09:15:30 2006 @@ -1,5 +1,5 @@ #! /bin/sh -# From configure.in Revision: 43615 . +# From configure.in Revision: 43748 . # Guess values for system-dependent variables and create Makefiles. # Generated by GNU Autoconf 2.59 for python 2.5. # @@ -20019,7 +20019,8 @@ #include int main() { - exit((((wchar_t) -1) < ((wchar_t) 0)) ? 1 : 0); + /* Success: exit code 0 */ + exit((((wchar_t) -1) < ((wchar_t) 0)) ? 0 : 1); } _ACEOF Modified: python/trunk/configure.in ============================================================================== --- python/trunk/configure.in (original) +++ python/trunk/configure.in Tue Apr 11 09:15:30 2006 @@ -2792,7 +2792,8 @@ #include int main() { - exit((((wchar_t) -1) < ((wchar_t) 0)) ? 1 : 0); + /* Success: exit code 0 */ + exit((((wchar_t) -1) < ((wchar_t) 0)) ? 0 : 1); } ], ac_cv_wchar_t_signed=yes, From python-checkins at python.org Tue Apr 11 09:17:10 2006 From: python-checkins at python.org (neal.norwitz) Date: Tue, 11 Apr 2006 09:17:10 +0200 (CEST) Subject: [Python-checkins] r45265 - python/trunk/Lib/inspect.py Message-ID: <20060411071710.EADE31E4007@bag.python.org> Author: neal.norwitz Date: Tue Apr 11 09:17:08 2006 New Revision: 45265 Modified: python/trunk/Lib/inspect.py Log: Add whitespace after comma Modified: python/trunk/Lib/inspect.py ============================================================================== --- python/trunk/Lib/inspect.py (original) +++ python/trunk/Lib/inspect.py Tue Apr 11 09:17:08 2006 @@ -353,7 +353,7 @@ if 'b' in mode and string.lower(filename[-len(suffix):]) == suffix: # Looks like a binary file. We want to only return a text file. return None - if os.path.exists(filename) or hasattr(getmodule(object),'__loader__'): + if os.path.exists(filename) or hasattr(getmodule(object), '__loader__'): return filename def getabsfile(object): From python-checkins at python.org Tue Apr 11 09:21:20 2006 From: python-checkins at python.org (neal.norwitz) Date: Tue, 11 Apr 2006 09:21:20 +0200 (CEST) Subject: [Python-checkins] r45266 - python/trunk/Lib/linecache.py python/trunk/Lib/site.py Message-ID: <20060411072120.CF6541E4009@bag.python.org> Author: neal.norwitz Date: Tue Apr 11 09:21:20 2006 New Revision: 45266 Modified: python/trunk/Lib/linecache.py python/trunk/Lib/site.py Log: Adjust whitespace. Modified: python/trunk/Lib/linecache.py ============================================================================== --- python/trunk/Lib/linecache.py (original) +++ python/trunk/Lib/linecache.py Tue Apr 11 09:21:20 2006 @@ -37,7 +37,7 @@ if filename in cache: return cache[filename][2] else: - return updatecache(filename,module_globals) + return updatecache(filename, module_globals) def checkcache(filename=None): @@ -85,13 +85,13 @@ if module_globals and '__loader__' in module_globals: name = module_globals.get('__name__') loader = module_globals['__loader__'] - get_source = getattr(loader, 'get_source' ,None) + get_source = getattr(loader, 'get_source', None) if name and get_source: if basename.startswith(name.split('.')[-1]+'.'): try: data = get_source(name) - except (ImportError,IOError): + except (ImportError, IOError): pass else: cache[filename] = ( Modified: python/trunk/Lib/site.py ============================================================================== --- python/trunk/Lib/site.py (original) +++ python/trunk/Lib/site.py Tue Apr 11 09:21:20 2006 @@ -69,7 +69,7 @@ def abs__file__(): """Set all module' __file__ attribute to an absolute path""" for m in sys.modules.values(): - if hasattr(m,'__loader__'): + if hasattr(m, '__loader__'): continue # don't mess with a PEP 302-supplied __file__ try: m.__file__ = os.path.abspath(m.__file__) From python-checkins at python.org Tue Apr 11 09:23:06 2006 From: python-checkins at python.org (anthony.baxter) Date: Tue, 11 Apr 2006 09:23:06 +0200 (CEST) Subject: [Python-checkins] r45267 - python/trunk/Include/asdl.h Message-ID: <20060411072306.393A01E4007@bag.python.org> Author: anthony.baxter Date: Tue Apr 11 09:23:05 2006 New Revision: 45267 Modified: python/trunk/Include/asdl.h Log: C++ already defines a perfectly good 'bool'. Use that. Modified: python/trunk/Include/asdl.h ============================================================================== --- python/trunk/Include/asdl.h (original) +++ python/trunk/Include/asdl.h Tue Apr 11 09:23:05 2006 @@ -5,7 +5,9 @@ typedef PyObject * string; typedef PyObject * object; +#ifndef __cplusplus typedef enum {false, true} bool; +#endif /* It would be nice if the code generated by asdl_c.py was completely independent of Python, but it is a goal the requires too much work From martin at v.loewis.de Tue Apr 11 09:23:35 2006 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 11 Apr 2006 09:23:35 +0200 Subject: [Python-checkins] r45259 - in python/trunk: Misc/NEWS Modules/posixmodule.c In-Reply-To: <20060411064745.B00B81E4007@bag.python.org> References: <20060411064745.B00B81E4007@bag.python.org> Message-ID: <443B5977.6020300@v.loewis.de> georg.brandl wrote: > Modified: python/trunk/Modules/posixmodule.c > ============================================================================== > --- python/trunk/Modules/posixmodule.c (original) > +++ python/trunk/Modules/posixmodule.c Tue Apr 11 08:47:43 2006 > @@ -1901,6 +1901,12 @@ > } > Py_DECREF(v); > } > + if (errno != 0 && d != NULL) { > + /* readdir() returned NULL and set errno */ > + closedir(dirp); > + Py_DECREF(d); > + return posix_error_with_allocated_filename(name); > + } > closedir(dirp); > PyMem_Free(name); I believe this patch is incomplete: if readdir succeeds, errno is not changed. So I think errno must be set to 0 before invoking readdir, or else you might get bogus exceptions. Regards, Martin From python-checkins at python.org Tue Apr 11 09:24:24 2006 From: python-checkins at python.org (neal.norwitz) Date: Tue, 11 Apr 2006 09:24:24 +0200 (CEST) Subject: [Python-checkins] r45268 - peps/trunk/pep-0356.txt Message-ID: <20060411072424.5F1C31E4007@bag.python.org> Author: neal.norwitz Date: Tue Apr 11 09:24:24 2006 New Revision: 45268 Modified: peps/trunk/pep-0356.txt Log: Add note about PJEs updates to support PEP 302 imports. Modified: peps/trunk/pep-0356.txt ============================================================================== --- peps/trunk/pep-0356.txt (original) +++ peps/trunk/pep-0356.txt Tue Apr 11 09:24:24 2006 @@ -90,6 +90,9 @@ - pysqlite was added to the standard library + - Add PEP 302 zipfile/__loader__ support to the following modules: + warnings, linecache, inspect, traceback, site, and doctest + Possible features for 2.5 From martin at v.loewis.de Tue Apr 11 09:29:01 2006 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 11 Apr 2006 09:29:01 +0200 Subject: [Python-checkins] r45261 - python/trunk/Objects/bufferobject.c python/trunk/Objects/classobject.c python/trunk/Objects/enumobject.c python/trunk/Objects/fileobject.c python/trunk/Objects/floatobject.c python/trunk/Objects/intobject.c python/trunk/Objects/listobject.c python/trunk/Objects/longobject.c In-Reply-To: <20060411065432.4FEBC1E4007@bag.python.org> References: <20060411065432.4FEBC1E4007@bag.python.org> Message-ID: <443B5ABD.4040203@v.loewis.de> anthony.baxter wrote: > More C++-compliance. Note especially listobject.c - to get C++ to accept the > PyTypeObject structures, I had to make prototypes for the functions, and > move the structure definition ahead of the functions. I'd dearly like a better > way to do this - to change this would make for a massive set of changes to > the codebase. You don't have to move the structures before all functions: just having it before the "new" function (e.g. list_iter) would be enough. So the order should be this: forward-declare new function define methods define type structures define new function Regards, Martin From buildbot at python.org Tue Apr 11 09:30:35 2006 From: buildbot at python.org (buildbot at python.org) Date: Tue, 11 Apr 2006 07:30:35 +0000 Subject: [Python-checkins] buildbot warnings in g4 osx.4 2.4 Message-ID: <20060411073036.2DE1B1E4007@bag.python.org> The Buildbot has detected a new failure of g4 osx.4 2.4. Full details are available at: http://www.python.org/dev/buildbot/all/g4%2520osx.4%25202.4/builds/76 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch branches/release24-maint] HEAD Blamelist: georg.brandl Build Had Warnings: warnings test sincerely, -The Buildbot From g.brandl at gmx.net Tue Apr 11 09:31:56 2006 From: g.brandl at gmx.net (Georg Brandl) Date: Tue, 11 Apr 2006 09:31:56 +0200 Subject: [Python-checkins] r45259 - in python/trunk: Misc/NEWS Modules/posixmodule.c In-Reply-To: <443B5977.6020300@v.loewis.de> References: <20060411064745.B00B81E4007@bag.python.org> <443B5977.6020300@v.loewis.de> Message-ID: Martin v. L?wis wrote: > georg.brandl wrote: >> Modified: python/trunk/Modules/posixmodule.c >> ============================================================================== >> --- python/trunk/Modules/posixmodule.c (original) >> +++ python/trunk/Modules/posixmodule.c Tue Apr 11 08:47:43 2006 >> @@ -1901,6 +1901,12 @@ >> } >> Py_DECREF(v); >> } >> + if (errno != 0 && d != NULL) { >> + /* readdir() returned NULL and set errno */ >> + closedir(dirp); >> + Py_DECREF(d); >> + return posix_error_with_allocated_filename(name); >> + } >> closedir(dirp); >> PyMem_Free(name); > > I believe this patch is incomplete: if readdir succeeds, errno is > not changed. So I think errno must be set to 0 before invoking readdir, > or else you might get bogus exceptions. Does r45262 suffice? It sets errno = 0 before invoking opendir(). Georg From martin at v.loewis.de Tue Apr 11 09:32:08 2006 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 11 Apr 2006 09:32:08 +0200 Subject: [Python-checkins] r45267 - python/trunk/Include/asdl.h In-Reply-To: <20060411072306.393A01E4007@bag.python.org> References: <20060411072306.393A01E4007@bag.python.org> Message-ID: <443B5B78.4070506@v.loewis.de> anthony.baxter wrote: > C++ already defines a perfectly good 'bool'. Use that. Actually, I think asdl.h should just drop the boolean type, and map ASDL bool to C int. Regards, Martin From martin at v.loewis.de Tue Apr 11 09:35:13 2006 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 11 Apr 2006 09:35:13 +0200 Subject: [Python-checkins] r45259 - in python/trunk: Misc/NEWS Modules/posixmodule.c In-Reply-To: References: <20060411064745.B00B81E4007@bag.python.org> <443B5977.6020300@v.loewis.de> Message-ID: <443B5C31.2060601@v.loewis.de> Georg Brandl wrote: > Does r45262 suffice? It sets errno = 0 before invoking opendir(). In the current code, yes. If there where any further C library calls that could set errno and not cause a Python exception, then errno would need to be cleared immediately before each readdir call. Regards, Martin From python-checkins at python.org Tue Apr 11 09:42:39 2006 From: python-checkins at python.org (anthony.baxter) Date: Tue, 11 Apr 2006 09:42:39 +0200 (CEST) Subject: [Python-checkins] r45269 - python/trunk/Objects/obmalloc.c python/trunk/Objects/stringobject.c python/trunk/Objects/tupleobject.c python/trunk/Objects/typeobject.c python/trunk/Objects/unicodeobject.c Message-ID: <20060411074239.F1BBF1E4007@bag.python.org> Author: anthony.baxter Date: Tue Apr 11 09:42:36 2006 New Revision: 45269 Modified: python/trunk/Objects/obmalloc.c python/trunk/Objects/stringobject.c python/trunk/Objects/tupleobject.c python/trunk/Objects/typeobject.c python/trunk/Objects/unicodeobject.c Log: More low-hanging fruit. Still need to re-arrange some code (or find a better solution) in the same way as listobject.c got changed. Hoping for a better solution. Modified: python/trunk/Objects/obmalloc.c ============================================================================== --- python/trunk/Objects/obmalloc.c (original) +++ python/trunk/Objects/obmalloc.c Tue Apr 11 09:42:36 2006 @@ -529,7 +529,7 @@ nbytes = numarenas * sizeof(*arenas); if (nbytes / sizeof(*arenas) != numarenas) return NULL; /* overflow */ - arenaobj = realloc(arenas, nbytes); + arenaobj = (arena_object *)realloc(arenas, nbytes); if (arenaobj == NULL) return NULL; arenas = arenaobj; Modified: python/trunk/Objects/stringobject.c ============================================================================== --- python/trunk/Objects/stringobject.c (original) +++ python/trunk/Objects/stringobject.c Tue Apr 11 09:42:36 2006 @@ -1051,7 +1051,7 @@ lastchar = sub[shortsub]; last = s + PyString_GET_SIZE(a) - len_sub + 1; while (s < last) { - s = memchr(s, firstchar, last-s); + s = (char *)memchr(s, firstchar, last-s); if (s == NULL) return 0; assert(s < last); @@ -1210,7 +1210,7 @@ } else { source_buf = PyString_AsString((PyObject*)self); - result_buf = PyMem_Malloc(slicelength); + result_buf = (char *)PyMem_Malloc(slicelength); if (result_buf == NULL) return PyErr_NoMemory(); @@ -2028,12 +2028,12 @@ { char *s = PyString_AS_STRING(self), *s_new; Py_ssize_t i, n = PyString_GET_SIZE(self); - PyObject *new; + PyObject *newobj; - new = PyString_FromStringAndSize(NULL, n); - if (new == NULL) + newobj = PyString_FromStringAndSize(NULL, n); + if (newobj == NULL) return NULL; - s_new = PyString_AsString(new); + s_new = PyString_AsString(newobj); for (i = 0; i < n; i++) { int c = Py_CHARMASK(*s++); if (isupper(c)) { @@ -2042,7 +2042,7 @@ *s_new = c; s_new++; } - return new; + return newobj; } @@ -2056,12 +2056,12 @@ { char *s = PyString_AS_STRING(self), *s_new; Py_ssize_t i, n = PyString_GET_SIZE(self); - PyObject *new; + PyObject *newobj; - new = PyString_FromStringAndSize(NULL, n); - if (new == NULL) + newobj = PyString_FromStringAndSize(NULL, n); + if (newobj == NULL) return NULL; - s_new = PyString_AsString(new); + s_new = PyString_AsString(newobj); for (i = 0; i < n; i++) { int c = Py_CHARMASK(*s++); if (islower(c)) { @@ -2070,7 +2070,7 @@ *s_new = c; s_new++; } - return new; + return newobj; } @@ -2086,12 +2086,12 @@ char *s = PyString_AS_STRING(self), *s_new; Py_ssize_t i, n = PyString_GET_SIZE(self); int previous_is_cased = 0; - PyObject *new; + PyObject *newobj; - new = PyString_FromStringAndSize(NULL, n); - if (new == NULL) + newobj = PyString_FromStringAndSize(NULL, n); + if (newobj == NULL) return NULL; - s_new = PyString_AsString(new); + s_new = PyString_AsString(newobj); for (i = 0; i < n; i++) { int c = Py_CHARMASK(*s++); if (islower(c)) { @@ -2106,7 +2106,7 @@ previous_is_cased = 0; *s_new++ = c; } - return new; + return newobj; } PyDoc_STRVAR(capitalize__doc__, @@ -2120,12 +2120,12 @@ { char *s = PyString_AS_STRING(self), *s_new; Py_ssize_t i, n = PyString_GET_SIZE(self); - PyObject *new; + PyObject *newobj; - new = PyString_FromStringAndSize(NULL, n); - if (new == NULL) + newobj = PyString_FromStringAndSize(NULL, n); + if (newobj == NULL) return NULL; - s_new = PyString_AsString(new); + s_new = PyString_AsString(newobj); if (0 < n) { int c = Py_CHARMASK(*s++); if (islower(c)) @@ -2142,7 +2142,7 @@ *s_new = c; s_new++; } - return new; + return newobj; } @@ -2199,7 +2199,7 @@ } if (i >= m) break; - t = memchr(s+i, sub[0], m-i); + t = (const char *)memchr(s+i, sub[0], m-i); if (t == NULL) break; i = t - s; @@ -2218,12 +2218,12 @@ { char *s = PyString_AS_STRING(self), *s_new; Py_ssize_t i, n = PyString_GET_SIZE(self); - PyObject *new; + PyObject *newobj; - new = PyString_FromStringAndSize(NULL, n); - if (new == NULL) + newobj = PyString_FromStringAndSize(NULL, n); + if (newobj == NULL) return NULL; - s_new = PyString_AsString(new); + s_new = PyString_AsString(newobj); for (i = 0; i < n; i++) { int c = Py_CHARMASK(*s++); if (islower(c)) { @@ -2236,7 +2236,7 @@ *s_new = c; s_new++; } - return new; + return newobj; } @@ -2524,7 +2524,7 @@ const Py_ssize_t len = PyString_GET_SIZE(self); Py_ssize_t sub_len, repl_len, out_len; int count = -1; - PyObject *new; + PyObject *newobj; PyObject *subobj, *replobj; if (!PyArg_ParseTuple(args, "OO|i:replace", @@ -2563,20 +2563,20 @@ if (out_len == -1) { if (PyString_CheckExact(self)) { /* we're returning another reference to self */ - new = (PyObject*)self; - Py_INCREF(new); + newobj = (PyObject*)self; + Py_INCREF(newobj); } else { - new = PyString_FromStringAndSize(str, len); - if (new == NULL) + newobj = PyString_FromStringAndSize(str, len); + if (newobj == NULL) return NULL; } } else { - new = PyString_FromStringAndSize(new_s, out_len); + newobj = PyString_FromStringAndSize(new_s, out_len); PyMem_FREE(new_s); } - return new; + return newobj; } Modified: python/trunk/Objects/tupleobject.c ============================================================================== --- python/trunk/Objects/tupleobject.c (original) +++ python/trunk/Objects/tupleobject.c Tue Apr 11 09:42:36 2006 @@ -547,7 +547,7 @@ static PyObject * tuple_subtype_new(PyTypeObject *type, PyObject *args, PyObject *kwds) { - PyObject *tmp, *new, *item; + PyObject *tmp, *newobj, *item; Py_ssize_t i, n; assert(PyType_IsSubtype(type, &PyTuple_Type)); @@ -555,16 +555,16 @@ if (tmp == NULL) return NULL; assert(PyTuple_Check(tmp)); - new = type->tp_alloc(type, n = PyTuple_GET_SIZE(tmp)); - if (new == NULL) + newobj = type->tp_alloc(type, n = PyTuple_GET_SIZE(tmp)); + if (newobj == NULL) return NULL; for (i = 0; i < n; i++) { item = PyTuple_GET_ITEM(tmp, i); Py_INCREF(item); - PyTuple_SET_ITEM(new, i, item); + PyTuple_SET_ITEM(newobj, i, item); } Py_DECREF(tmp); - return new; + return newobj; } PyDoc_STRVAR(tuple_doc, Modified: python/trunk/Objects/typeobject.c ============================================================================== --- python/trunk/Objects/typeobject.c (original) +++ python/trunk/Objects/typeobject.c Tue Apr 11 09:42:36 2006 @@ -453,7 +453,7 @@ if (PyType_IS_GC(type)) obj = _PyObject_GC_Malloc(size); else - obj = PyObject_MALLOC(size); + obj = (PyObject *)PyObject_MALLOC(size); if (obj == NULL) return PyErr_NoMemory(); @@ -1150,7 +1150,7 @@ remain[i] is the index of the next base in to_merge[i] that is not included in acc. */ - remain = PyMem_MALLOC(SIZEOF_INT*to_merge_size); + remain = (int *)PyMem_MALLOC(SIZEOF_INT*to_merge_size); if (remain == NULL) return -1; for (i = 0; i < to_merge_size; i++) @@ -1896,7 +1896,7 @@ PyObject *doc = PyDict_GetItemString(dict, "__doc__"); if (doc != NULL && PyString_Check(doc)) { const size_t n = (size_t)PyString_GET_SIZE(doc); - char *tp_doc = PyObject_MALLOC(n+1); + char *tp_doc = (char *)PyObject_MALLOC(n+1); if (tp_doc == NULL) { Py_DECREF(type); return NULL; @@ -2446,23 +2446,23 @@ } static int -compatible_for_assignment(PyTypeObject* old, PyTypeObject* new, char* attr) +compatible_for_assignment(PyTypeObject* oldto, PyTypeObject* newto, char* attr) { PyTypeObject *newbase, *oldbase; - if (new->tp_dealloc != old->tp_dealloc || - new->tp_free != old->tp_free) + if (newto->tp_dealloc != oldto->tp_dealloc || + newto->tp_free != oldto->tp_free) { PyErr_Format(PyExc_TypeError, "%s assignment: " "'%s' deallocator differs from '%s'", attr, - new->tp_name, - old->tp_name); + newto->tp_name, + oldto->tp_name); return 0; } - newbase = new; - oldbase = old; + newbase = newto; + oldbase = oldto; while (equiv_structs(newbase, newbase->tp_base)) newbase = newbase->tp_base; while (equiv_structs(oldbase, oldbase->tp_base)) @@ -2474,8 +2474,8 @@ "%s assignment: " "'%s' object layout differs from '%s'", attr, - new->tp_name, - old->tp_name); + newto->tp_name, + oldto->tp_name); return 0; } @@ -2485,8 +2485,8 @@ static int object_set_class(PyObject *self, PyObject *value, void *closure) { - PyTypeObject *old = self->ob_type; - PyTypeObject *new; + PyTypeObject *oldto = self->ob_type; + PyTypeObject *newto; if (value == NULL) { PyErr_SetString(PyExc_TypeError, @@ -2499,18 +2499,18 @@ value->ob_type->tp_name); return -1; } - new = (PyTypeObject *)value; - if (!(new->tp_flags & Py_TPFLAGS_HEAPTYPE) || - !(old->tp_flags & Py_TPFLAGS_HEAPTYPE)) + newto = (PyTypeObject *)value; + if (!(newto->tp_flags & Py_TPFLAGS_HEAPTYPE) || + !(oldto->tp_flags & Py_TPFLAGS_HEAPTYPE)) { PyErr_Format(PyExc_TypeError, "__class__ assignment: only for heap types"); return -1; } - if (compatible_for_assignment(new, old, "__class__")) { - Py_INCREF(new); - self->ob_type = new; - Py_DECREF(old); + if (compatible_for_assignment(newto, oldto, "__class__")) { + Py_INCREF(newto); + self->ob_type = newto; + Py_DECREF(oldto); return 0; } else { @@ -3332,7 +3332,7 @@ { Py_ssize_t i; int result; - PyObject *list, *ref, *new; + PyObject *list, *ref, *newobj; list = base->tp_subclasses; if (list == NULL) { @@ -3341,16 +3341,16 @@ return -1; } assert(PyList_Check(list)); - new = PyWeakref_NewRef((PyObject *)type, NULL); + newobj = PyWeakref_NewRef((PyObject *)type, NULL); i = PyList_GET_SIZE(list); while (--i >= 0) { ref = PyList_GET_ITEM(list, i); assert(PyWeakref_CheckRef(ref)); if (PyWeakref_GET_OBJECT(ref) == Py_None) - return PyList_SetItem(list, i, new); + return PyList_SetItem(list, i, newobj); } - result = PyList_Append(list, new); - Py_DECREF(new); + result = PyList_Append(list, newobj); + Py_DECREF(newobj); return result; } @@ -5746,7 +5746,7 @@ super_descr_get(PyObject *self, PyObject *obj, PyObject *type) { superobject *su = (superobject *)self; - superobject *new; + superobject *newobj; if (obj == NULL || obj == Py_None || su->obj != NULL) { /* Not binding to an object, or already bound */ @@ -5763,16 +5763,16 @@ PyTypeObject *obj_type = supercheck(su->type, obj); if (obj_type == NULL) return NULL; - new = (superobject *)PySuper_Type.tp_new(&PySuper_Type, + newobj = (superobject *)PySuper_Type.tp_new(&PySuper_Type, NULL, NULL); - if (new == NULL) + if (newobj == NULL) return NULL; Py_INCREF(su->type); Py_INCREF(obj); - new->type = su->type; - new->obj = obj; - new->obj_type = obj_type; - return (PyObject *)new; + newobj->type = su->type; + newobj->obj = obj; + newobj->obj_type = obj_type; + return (PyObject *)newobj; } } Modified: python/trunk/Objects/unicodeobject.c ============================================================================== --- python/trunk/Objects/unicodeobject.c (original) +++ python/trunk/Objects/unicodeobject.c Tue Apr 11 09:42:36 2006 @@ -149,7 +149,7 @@ oldstr = unicode->str; PyMem_RESIZE(unicode->str, Py_UNICODE, length + 1); if (!unicode->str) { - unicode->str = oldstr; + unicode->str = (Py_UNICODE *)oldstr; PyErr_NoMemory(); return -1; } @@ -1884,7 +1884,7 @@ Py_DECREF(m); if (api == NULL) goto ucnhashError; - ucnhash_CAPI = PyCObject_AsVoidPtr(api); + ucnhash_CAPI = (_PyUnicode_Name_CAPI *)PyCObject_AsVoidPtr(api); Py_DECREF(api); if (ucnhash_CAPI == NULL) goto ucnhashError; @@ -2499,8 +2499,8 @@ /* current output position */ Py_ssize_t respos = 0; Py_ssize_t ressize; - char *encoding = (limit == 256) ? "latin-1" : "ascii"; - char *reason = (limit == 256) ? "ordinal not in range(256)" : "ordinal not in range(128)"; + const char *encoding = (limit == 256) ? "latin-1" : "ascii"; + const char *reason = (limit == 256) ? "ordinal not in range(256)" : "ordinal not in range(128)"; PyObject *errorHandler = NULL; PyObject *exc = NULL; /* the following variable is used for caching string comparisons @@ -6488,7 +6488,8 @@ return PyUnicode_FromUnicode(NULL, 0); } else { source_buf = PyUnicode_AS_UNICODE((PyObject*)self); - result_buf = PyMem_MALLOC(slicelength*sizeof(Py_UNICODE)); + result_buf = (Py_UNICODE *)PyMem_MALLOC(slicelength* + sizeof(Py_UNICODE)); if (result_buf == NULL) return PyErr_NoMemory(); From python-checkins at python.org Tue Apr 11 09:43:48 2006 From: python-checkins at python.org (anthony.baxter) Date: Tue, 11 Apr 2006 09:43:48 +0200 (CEST) Subject: [Python-checkins] r45270 - python/trunk/Python/Python-ast.c python/trunk/Python/ast.c python/trunk/Python/ceval.c python/trunk/Python/exceptions.c Message-ID: <20060411074348.0214E1E4007@bag.python.org> Author: anthony.baxter Date: Tue Apr 11 09:43:46 2006 New Revision: 45270 Modified: python/trunk/Python/Python-ast.c python/trunk/Python/ast.c python/trunk/Python/ceval.c python/trunk/Python/exceptions.c Log: low-hanging fruit in Python/ - g++ still hates all the enum_kind declarations in Python/Python-ast.c. Not sure what to do about those. Modified: python/trunk/Python/Python-ast.c ============================================================================== --- python/trunk/Python/Python-ast.c (original) +++ python/trunk/Python/Python-ast.c Tue Apr 11 09:43:46 2006 @@ -2328,6 +2328,8 @@ result = PyType_GenericNew(Continue_type, NULL, NULL); if (!result) goto failed; break; + default: + ; } value = ast2obj_int(o->lineno); if (!value) goto failed; Modified: python/trunk/Python/ast.c ============================================================================== --- python/trunk/Python/ast.c (original) +++ python/trunk/Python/ast.c Tue Apr 11 09:43:46 2006 @@ -1365,7 +1365,7 @@ int i, nops; expr_ty expr1, expr2, result; - operator_ty operator; + operator_ty newoperator; expr1 = ast_for_expr(c, CHILD(n, 0)); if (!expr1) @@ -1375,11 +1375,11 @@ if (!expr2) return NULL; - operator = get_operator(CHILD(n, 1)); - if (!operator) + newoperator = get_operator(CHILD(n, 1)); + if (!newoperator) return NULL; - result = BinOp(expr1, operator, expr2, LINENO(n), n->n_col_offset, + result = BinOp(expr1, newoperator, expr2, LINENO(n), n->n_col_offset, c->c_arena); if (!result) return NULL; @@ -1389,15 +1389,15 @@ expr_ty tmp_result, tmp; const node* next_oper = CHILD(n, i * 2 + 1); - operator = get_operator(next_oper); - if (!operator) + newoperator = get_operator(next_oper); + if (!newoperator) return NULL; tmp = ast_for_expr(c, CHILD(n, i * 2 + 2)); if (!tmp) return NULL; - tmp_result = BinOp(result, operator, tmp, + tmp_result = BinOp(result, newoperator, tmp, LINENO(next_oper), next_oper->n_col_offset, c->c_arena); if (!tmp) @@ -1610,10 +1610,10 @@ } for (i = 1; i < NCH(n); i += 2) { /* XXX cmpop_ty is just an enum */ - cmpop_ty operator; + cmpop_ty newoperator; - operator = ast_for_comp_op(CHILD(n, i)); - if (!operator) { + newoperator = ast_for_comp_op(CHILD(n, i)); + if (!newoperator) { return NULL; } @@ -1622,7 +1622,7 @@ return NULL; } - asdl_seq_SET(ops, i / 2, (void *)(Py_uintptr_t)operator); + asdl_seq_SET(ops, i / 2, (void *)(Py_uintptr_t)newoperator); asdl_seq_SET(cmps, i / 2, expression); } expression = ast_for_expr(c, CHILD(n, 0)); @@ -1882,7 +1882,7 @@ } else if (TYPE(CHILD(n, 1)) == augassign) { expr_ty expr1, expr2; - operator_ty operator; + operator_ty newoperator; node *ch = CHILD(n, 0); if (TYPE(ch) == testlist) @@ -1924,11 +1924,11 @@ if (!expr2) return NULL; - operator = ast_for_augassign(CHILD(n, 1)); - if (!operator) + newoperator = ast_for_augassign(CHILD(n, 1)); + if (!newoperator) return NULL; - return AugAssign(expr1, operator, expr2, LINENO(n), n->n_col_offset, c->c_arena); + return AugAssign(expr1, newoperator, expr2, LINENO(n), n->n_col_offset, c->c_arena); } else { int i; @@ -2541,8 +2541,8 @@ int off = 5 + (n_elif - i - 1) * 4; expr_ty expression; asdl_seq *suite_seq; - asdl_seq *new = asdl_seq_new(1, c->c_arena); - if (!new) + asdl_seq *newobj = asdl_seq_new(1, c->c_arena); + if (!newobj) return NULL; expression = ast_for_expr(c, CHILD(n, off)); if (!expression) @@ -2551,10 +2551,10 @@ if (!suite_seq) return NULL; - asdl_seq_SET(new, 0, + asdl_seq_SET(newobj, 0, If(expression, suite_seq, orelse, LINENO(CHILD(n, off)), CHILD(n, off)->n_col_offset, c->c_arena)); - orelse = new; + orelse = newobj; } return If(ast_for_expr(c, CHILD(n, 1)), ast_for_suite(c, CHILD(n, 3)), Modified: python/trunk/Python/ceval.c ============================================================================== --- python/trunk/Python/ceval.c (original) +++ python/trunk/Python/ceval.c Tue Apr 11 09:43:46 2006 @@ -507,7 +507,7 @@ } PyObject * -PyEval_EvalFrameEx(PyFrameObject *f, int throw) +PyEval_EvalFrameEx(PyFrameObject *f, int throwflag) { #ifdef DXPAIRS int lastopcode = 0; @@ -756,7 +756,7 @@ x = Py_None; /* Not a reference, just anything non-NULL */ w = NULL; - if (throw) { /* support for generator.throw() */ + if (throwflag) { /* support for generator.throw() */ why = WHY_EXCEPTION; goto on_error; } Modified: python/trunk/Python/exceptions.c ============================================================================== --- python/trunk/Python/exceptions.c (original) +++ python/trunk/Python/exceptions.c Tue Apr 11 09:43:46 2006 @@ -893,7 +893,7 @@ if (have_filename) bufsize += PyString_GET_SIZE(filename); - buffer = PyMem_MALLOC(bufsize); + buffer = (char *)PyMem_MALLOC(bufsize); if (buffer != NULL) { if (have_filename && have_lineno) PyOS_snprintf(buffer, bufsize, "%s (%s, line %ld)", From buildbot at python.org Tue Apr 11 09:45:50 2006 From: buildbot at python.org (buildbot at python.org) Date: Tue, 11 Apr 2006 07:45:50 +0000 Subject: [Python-checkins] buildbot warnings in x86 Ubuntu dapper (icc) 2.4 Message-ID: <20060411074550.3F9731E4036@bag.python.org> The Buildbot has detected a new failure of x86 Ubuntu dapper (icc) 2.4. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520Ubuntu%2520dapper%2520%2528icc%2529%25202.4/builds/16 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch branches/release24-maint] HEAD Blamelist: georg.brandl Build Had Warnings: warnings test sincerely, -The Buildbot From buildbot at python.org Tue Apr 11 09:54:56 2006 From: buildbot at python.org (buildbot at python.org) Date: Tue, 11 Apr 2006 07:54:56 +0000 Subject: [Python-checkins] buildbot failure in amd64 gentoo trunk Message-ID: <20060411075456.DE17B1E4007@bag.python.org> The Buildbot has detected a new failure of amd64 gentoo trunk. Full details are available at: http://www.python.org/dev/buildbot/all/amd64%2520gentoo%2520trunk/builds/401 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: anthony.baxter BUILD FAILED: failed compile sincerely, -The Buildbot From buildbot at python.org Tue Apr 11 09:55:49 2006 From: buildbot at python.org (buildbot at python.org) Date: Tue, 11 Apr 2006 07:55:49 +0000 Subject: [Python-checkins] buildbot failure in x86 gentoo trunk Message-ID: <20060411075549.D00641E4007@bag.python.org> The Buildbot has detected a new failure of x86 gentoo trunk. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520gentoo%2520trunk/builds/386 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: anthony.baxter BUILD FAILED: failed compile sincerely, -The Buildbot From buildbot at python.org Tue Apr 11 09:56:35 2006 From: buildbot at python.org (buildbot at python.org) Date: Tue, 11 Apr 2006 07:56:35 +0000 Subject: [Python-checkins] buildbot failure in hppa Ubuntu dapper trunk Message-ID: <20060411075635.201391E4007@bag.python.org> The Buildbot has detected a new failure of hppa Ubuntu dapper trunk. Full details are available at: http://www.python.org/dev/buildbot/all/hppa%2520Ubuntu%2520dapper%2520trunk/builds/70 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: anthony.baxter BUILD FAILED: failed compile sincerely, -The Buildbot From python-checkins at python.org Tue Apr 11 09:58:55 2006 From: python-checkins at python.org (neal.norwitz) Date: Tue, 11 Apr 2006 09:58:55 +0200 (CEST) Subject: [Python-checkins] r45271 - python/trunk/Objects/obmalloc.c Message-ID: <20060411075855.E46F81E404D@bag.python.org> Author: neal.norwitz Date: Tue Apr 11 09:58:54 2006 New Revision: 45271 Modified: python/trunk/Objects/obmalloc.c Log: Get compiling again Modified: python/trunk/Objects/obmalloc.c ============================================================================== --- python/trunk/Objects/obmalloc.c (original) +++ python/trunk/Objects/obmalloc.c Tue Apr 11 09:58:54 2006 @@ -529,7 +529,7 @@ nbytes = numarenas * sizeof(*arenas); if (nbytes / sizeof(*arenas) != numarenas) return NULL; /* overflow */ - arenaobj = (arena_object *)realloc(arenas, nbytes); + arenaobj = (struct arena_object *)realloc(arenas, nbytes); if (arenaobj == NULL) return NULL; arenas = arenaobj; From buildbot at python.org Tue Apr 11 10:01:55 2006 From: buildbot at python.org (buildbot at python.org) Date: Tue, 11 Apr 2006 08:01:55 +0000 Subject: [Python-checkins] buildbot warnings in g4 osx.4 trunk Message-ID: <20060411080155.2A1531E4009@bag.python.org> The Buildbot has detected a new failure of g4 osx.4 trunk. Full details are available at: http://www.python.org/dev/buildbot/all/g4%2520osx.4%2520trunk/builds/341 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: anthony.baxter,georg.brandl Build Had Warnings: warnings test sincerely, -The Buildbot From python-checkins at python.org Tue Apr 11 10:06:50 2006 From: python-checkins at python.org (martin.v.loewis) Date: Tue, 11 Apr 2006 10:06:50 +0200 (CEST) Subject: [Python-checkins] r45272 - in python/trunk: Include/Python-ast.h Parser/asdl_c.py Python/Python-ast.c Message-ID: <20060411080650.CD2481E4007@bag.python.org> Author: martin.v.loewis Date: Tue Apr 11 10:06:50 2006 New Revision: 45272 Modified: python/trunk/Include/Python-ast.h python/trunk/Parser/asdl_c.py python/trunk/Python/Python-ast.c Log: Make _kind types global for C++ compilation. Explicitly cast void* to int to cmpop_ty. Modified: python/trunk/Include/Python-ast.h ============================================================================== --- python/trunk/Include/Python-ast.h (original) +++ python/trunk/Include/Python-ast.h Tue Apr 11 10:06:50 2006 @@ -35,9 +35,10 @@ typedef struct _alias *alias_ty; +enum _mod_kind {Module_kind=1, Interactive_kind=2, Expression_kind=3, + Suite_kind=4}; struct _mod { - enum { Module_kind=1, Interactive_kind=2, Expression_kind=3, - Suite_kind=4 } kind; + enum _mod_kind kind; union { struct { asdl_seq *body; @@ -58,14 +59,15 @@ } v; }; +enum _stmt_kind {FunctionDef_kind=1, ClassDef_kind=2, Return_kind=3, + Delete_kind=4, Assign_kind=5, AugAssign_kind=6, Print_kind=7, + For_kind=8, While_kind=9, If_kind=10, With_kind=11, + Raise_kind=12, TryExcept_kind=13, TryFinally_kind=14, + Assert_kind=15, Import_kind=16, ImportFrom_kind=17, + Exec_kind=18, Global_kind=19, Expr_kind=20, Pass_kind=21, + Break_kind=22, Continue_kind=23}; struct _stmt { - enum { FunctionDef_kind=1, ClassDef_kind=2, Return_kind=3, - Delete_kind=4, Assign_kind=5, AugAssign_kind=6, Print_kind=7, - For_kind=8, While_kind=9, If_kind=10, With_kind=11, - Raise_kind=12, TryExcept_kind=13, TryFinally_kind=14, - Assert_kind=15, Import_kind=16, ImportFrom_kind=17, - Exec_kind=18, Global_kind=19, Expr_kind=20, Pass_kind=21, - Break_kind=22, Continue_kind=23 } kind; + enum _stmt_kind kind; union { struct { identifier name; @@ -181,12 +183,14 @@ int col_offset; }; +enum _expr_kind {BoolOp_kind=1, BinOp_kind=2, UnaryOp_kind=3, Lambda_kind=4, + IfExp_kind=5, Dict_kind=6, ListComp_kind=7, + GeneratorExp_kind=8, Yield_kind=9, Compare_kind=10, + Call_kind=11, Repr_kind=12, Num_kind=13, Str_kind=14, + Attribute_kind=15, Subscript_kind=16, Name_kind=17, + List_kind=18, Tuple_kind=19}; struct _expr { - enum { BoolOp_kind=1, BinOp_kind=2, UnaryOp_kind=3, Lambda_kind=4, - IfExp_kind=5, Dict_kind=6, ListComp_kind=7, GeneratorExp_kind=8, - Yield_kind=9, Compare_kind=10, Call_kind=11, Repr_kind=12, - Num_kind=13, Str_kind=14, Attribute_kind=15, Subscript_kind=16, - Name_kind=17, List_kind=18, Tuple_kind=19 } kind; + enum _expr_kind kind; union { struct { boolop_ty op; @@ -292,9 +296,9 @@ int col_offset; }; +enum _slice_kind {Ellipsis_kind=1, Slice_kind=2, ExtSlice_kind=3, Index_kind=4}; struct _slice { - enum { Ellipsis_kind=1, Slice_kind=2, ExtSlice_kind=3, Index_kind=4 } - kind; + enum _slice_kind kind; union { struct { expr_ty lower; Modified: python/trunk/Parser/asdl_c.py ============================================================================== --- python/trunk/Parser/asdl_c.py (original) +++ python/trunk/Parser/asdl_c.py Tue Apr 11 10:06:50 2006 @@ -155,8 +155,10 @@ type = sum.types[i] enum.append("%s_kind=%d" % (type.name, i + 1)) + emit("enum _%(name)s_kind {" + ", ".join(enum) + "};") + emit("struct _%(name)s {") - emit("enum { " + ", ".join(enum) + " } kind;", depth + 1) + emit("enum _%(name)s_kind kind;", depth + 1) emit("union {", depth + 1) for t in sum.types: self.visit(t, depth + 2) @@ -679,8 +681,8 @@ self.emit("if (!value) goto failed;", depth+1) self.emit("for(i = 0; i < n; i++)", depth+1) # This cannot fail, so no need for error handling - self.emit("PyList_SET_ITEM(value, i, ast2obj_%s((%s_ty)asdl_seq_GET(%s, i)));" % - (field.type, field.type, value), depth+2, reflow=False) + self.emit("PyList_SET_ITEM(value, i, ast2obj_cmpop((cmpop_ty)(int)asdl_seq_GET(%s, i)));" % value, + depth+2, reflow=False) self.emit("}", depth) else: self.emit("value = ast2obj_list(%s, ast2obj_%s);" % (value, field.type), depth) Modified: python/trunk/Python/Python-ast.c ============================================================================== --- python/trunk/Python/Python-ast.c (original) +++ python/trunk/Python/Python-ast.c Tue Apr 11 10:06:50 2006 @@ -2505,7 +2505,7 @@ value = PyList_New(n); if (!value) goto failed; for(i = 0; i < n; i++) - PyList_SET_ITEM(value, i, ast2obj_cmpop((cmpop_ty)asdl_seq_GET(o->v.Compare.ops, i))); + PyList_SET_ITEM(value, i, ast2obj_cmpop((cmpop_ty)(int)asdl_seq_GET(o->v.Compare.ops, i))); } if (!value) goto failed; if (PyObject_SetAttrString(result, "ops", value) == -1) From python-checkins at python.org Tue Apr 11 10:19:15 2006 From: python-checkins at python.org (neal.norwitz) Date: Tue, 11 Apr 2006 10:19:15 +0200 (CEST) Subject: [Python-checkins] r45273 - python/trunk/Parser/myreadline.c python/trunk/Parser/pgenmain.c python/trunk/Parser/tokenizer.c Message-ID: <20060411081915.BE0FD1E4007@bag.python.org> Author: neal.norwitz Date: Tue Apr 11 10:19:15 2006 New Revision: 45273 Modified: python/trunk/Parser/myreadline.c python/trunk/Parser/pgenmain.c python/trunk/Parser/tokenizer.c Log: As discussed on python-dev, really fix the PyMem_*/PyObject_* memory API mismatches. At least I hope this fixes them all. This reverts part of my change from yesterday that converted everything in Parser/*.c to use PyObject_* API. The encoding doesn't really need to use PyMem_*, however, it uses new_string() which must return PyMem_* for handling the result of PyOS_Readline() which returns PyMem_* memory. If there were 2 versions of new_string() one that returned PyMem_* for tokens and one that return PyObject_* for encodings that could also fix this problem. I'm not sure which version would be clearer. This seems to fix both Guido's and Phillip's problems, so it's good enough for now. After this change, it would be good to review Parser/*.c for consistent use of the 2 memory APIs. Modified: python/trunk/Parser/myreadline.c ============================================================================== --- python/trunk/Parser/myreadline.c (original) +++ python/trunk/Parser/myreadline.c Tue Apr 11 10:19:15 2006 @@ -111,7 +111,7 @@ size_t n; char *p; n = 100; - if ((p = (char *)PyObject_MALLOC(n)) == NULL) + if ((p = (char *)PyMem_MALLOC(n)) == NULL) return NULL; fflush(sys_stdout); #ifndef RISCOS @@ -130,7 +130,7 @@ case 0: /* Normal case */ break; case 1: /* Interrupt */ - PyObject_FREE(p); + PyMem_FREE(p); return NULL; case -1: /* EOF */ case -2: /* Error */ @@ -141,7 +141,7 @@ n = strlen(p); while (n > 0 && p[n-1] != '\n') { size_t incr = n+2; - p = (char *)PyObject_REALLOC(p, n + incr); + p = (char *)PyMem_REALLOC(p, n + incr); if (p == NULL) return NULL; if (incr > INT_MAX) { @@ -151,14 +151,14 @@ break; n += strlen(p+n); } - return (char *)PyObject_REALLOC(p, n+1); + return (char *)PyMem_REALLOC(p, n+1); } /* By initializing this function pointer, systems embedding Python can override the readline function. - Note: Python expects in return a buffer allocated with PyObject_Malloc. */ + Note: Python expects in return a buffer allocated with PyMem_Malloc. */ char *(*PyOS_ReadlineFunctionPointer)(FILE *, FILE *, char *); Modified: python/trunk/Parser/pgenmain.c ============================================================================== --- python/trunk/Parser/pgenmain.c (original) +++ python/trunk/Parser/pgenmain.c Tue Apr 11 10:19:15 2006 @@ -136,7 +136,7 @@ PyOS_Readline(FILE *sys_stdin, FILE *sys_stdout, char *prompt) { size_t n = 1000; - char *p = PyObject_MALLOC(n); + char *p = PyMem_MALLOC(n); char *q; if (p == NULL) return NULL; @@ -149,7 +149,7 @@ n = strlen(p); if (n > 0 && p[n-1] != '\n') p[n-1] = '\n'; - return PyObject_REALLOC(p, n+1); + return PyMem_REALLOC(p, n+1); } /* No-nonsense fgets */ Modified: python/trunk/Parser/tokenizer.c ============================================================================== --- python/trunk/Parser/tokenizer.c (original) +++ python/trunk/Parser/tokenizer.c Tue Apr 11 10:19:15 2006 @@ -164,7 +164,7 @@ { tok->decoding_erred = 1; if (tok->fp != NULL && tok->buf != NULL) /* see PyTokenizer_Free */ - PyObject_FREE(tok->buf); + PyMem_FREE(tok->buf); tok->buf = NULL; return NULL; /* as if it were EOF */ } @@ -172,7 +172,7 @@ static char * new_string(const char *s, Py_ssize_t len) { - char* result = (char *)PyObject_MALLOC(len + 1); + char* result = (char *)PyMem_MALLOC(len + 1); if (result != NULL) { memcpy(result, s, len); result[len] = '\0'; @@ -237,7 +237,7 @@ char* r = new_string(begin, t - begin); char* q = get_normal_name(r); if (r != q) { - PyObject_FREE(r); + PyMem_FREE(r); r = new_string(q, strlen(q)); } return r; @@ -278,18 +278,18 @@ tok->decoding_state = -1; } else - PyObject_FREE(cs); + PyMem_FREE(cs); #else /* Without Unicode support, we cannot process the coding spec. Since there won't be any Unicode literals, that won't matter. */ - PyObject_FREE(cs); + PyMem_FREE(cs); #endif } } else { /* then, compare cs with BOM */ r = (strcmp(tok->encoding, cs) == 0); - PyObject_FREE(cs); + PyMem_FREE(cs); } } if (!r) { @@ -335,7 +335,7 @@ return 1; } if (tok->encoding != NULL) - PyObject_FREE(tok->encoding); + PyMem_FREE(tok->encoding); tok->encoding = new_string("utf-8", 5); /* resulting is in utf-8 */ return 1; NON_BOM: @@ -657,7 +657,7 @@ struct tok_state *tok = tok_new(); if (tok == NULL) return NULL; - if ((tok->buf = (char *)PyObject_MALLOC(BUFSIZ)) == NULL) { + if ((tok->buf = (char *)PyMem_MALLOC(BUFSIZ)) == NULL) { PyTokenizer_Free(tok); return NULL; } @@ -676,13 +676,13 @@ PyTokenizer_Free(struct tok_state *tok) { if (tok->encoding != NULL) - PyObject_FREE(tok->encoding); + PyMem_FREE(tok->encoding); #ifndef PGEN Py_XDECREF(tok->decoding_readline); Py_XDECREF(tok->decoding_buffer); #endif if (tok->fp != NULL && tok->buf != NULL) - PyObject_FREE(tok->buf); + PyMem_FREE(tok->buf); PyMem_FREE(tok); } @@ -722,10 +722,10 @@ if (converted == NULL) goto error_nomem; - PyObject_FREE(*inp); + PyMem_FREE(*inp); *inp = converted; if (tok->encoding != NULL) - PyObject_FREE(tok->encoding); + PyMem_FREE(tok->encoding); tok->encoding = new_string(encoding, strlen(encoding)); if (tok->encoding == NULL) goto error_nomem; @@ -782,24 +782,24 @@ if (newtok == NULL) tok->done = E_INTR; else if (*newtok == '\0') { - PyObject_FREE(newtok); + PyMem_FREE(newtok); tok->done = E_EOF; } #if !defined(PGEN) && defined(Py_USING_UNICODE) else if (tok_stdin_decode(tok, &newtok) != 0) - PyObject_FREE(newtok); + PyMem_FREE(newtok); #endif else if (tok->start != NULL) { size_t start = tok->start - tok->buf; size_t oldlen = tok->cur - tok->buf; size_t newlen = oldlen + strlen(newtok); char *buf = tok->buf; - buf = (char *)PyObject_REALLOC(buf, newlen+1); + buf = (char *)PyMem_REALLOC(buf, newlen+1); tok->lineno++; if (buf == NULL) { - PyObject_FREE(tok->buf); + PyMem_FREE(tok->buf); tok->buf = NULL; - PyObject_FREE(newtok); + PyMem_FREE(newtok); tok->done = E_NOMEM; return EOF; } @@ -807,7 +807,7 @@ tok->cur = tok->buf + oldlen; tok->line_start = tok->cur; strcpy(tok->buf + oldlen, newtok); - PyObject_FREE(newtok); + PyMem_FREE(newtok); tok->inp = tok->buf + newlen; tok->end = tok->inp + 1; tok->start = tok->buf + start; @@ -815,7 +815,7 @@ else { tok->lineno++; if (tok->buf != NULL) - PyObject_FREE(tok->buf); + PyMem_FREE(tok->buf); tok->buf = newtok; tok->line_start = tok->buf; tok->cur = tok->buf; @@ -831,7 +831,7 @@ if (tok->start == NULL) { if (tok->buf == NULL) { tok->buf = (char *) - PyObject_MALLOC(BUFSIZ); + PyMem_MALLOC(BUFSIZ); if (tok->buf == NULL) { tok->done = E_NOMEM; return EOF; @@ -866,8 +866,8 @@ Py_ssize_t curvalid = tok->inp - tok->buf; Py_ssize_t newsize = curvalid + BUFSIZ; char *newbuf = tok->buf; - newbuf = (char *)PyObject_REALLOC(newbuf, - newsize); + newbuf = (char *)PyMem_REALLOC(newbuf, + newsize); if (newbuf == NULL) { tok->done = E_NOMEM; tok->cur = tok->inp; From buildbot at python.org Tue Apr 11 10:21:24 2006 From: buildbot at python.org (buildbot at python.org) Date: Tue, 11 Apr 2006 08:21:24 +0000 Subject: [Python-checkins] buildbot warnings in hppa Ubuntu dapper trunk Message-ID: <20060411082124.ECAB31E4007@bag.python.org> The Buildbot has detected a new failure of hppa Ubuntu dapper trunk. Full details are available at: http://www.python.org/dev/buildbot/all/hppa%2520Ubuntu%2520dapper%2520trunk/builds/71 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: neal.norwitz Build Had Warnings: warnings test sincerely, -The Buildbot From buildbot at python.org Tue Apr 11 10:23:12 2006 From: buildbot at python.org (buildbot at python.org) Date: Tue, 11 Apr 2006 08:23:12 +0000 Subject: [Python-checkins] buildbot failure in x86 XP trunk Message-ID: <20060411082312.4E12B1E4007@bag.python.org> The Buildbot has detected a new failure of x86 XP trunk. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520XP%2520trunk/builds/358 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: anthony.baxter,georg.brandl,martin.v.loewis,neal.norwitz BUILD FAILED: failed compile sincerely, -The Buildbot From buildbot at python.org Tue Apr 11 10:25:31 2006 From: buildbot at python.org (buildbot at python.org) Date: Tue, 11 Apr 2006 08:25:31 +0000 Subject: [Python-checkins] buildbot failure in ia64 Debian unstable trunk Message-ID: <20060411082531.E5E4A1E4043@bag.python.org> The Buildbot has detected a new failure of ia64 Debian unstable trunk. Full details are available at: http://www.python.org/dev/buildbot/all/ia64%2520Debian%2520unstable%2520trunk/builds/73 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: anthony.baxter,martin.v.loewis,neal.norwitz BUILD FAILED: failed compile sincerely, -The Buildbot From buildbot at python.org Tue Apr 11 10:30:04 2006 From: buildbot at python.org (buildbot at python.org) Date: Tue, 11 Apr 2006 08:30:04 +0000 Subject: [Python-checkins] buildbot failure in g4 osx.4 trunk Message-ID: <20060411083004.D61641E4007@bag.python.org> The Buildbot has detected a new failure of g4 osx.4 trunk. Full details are available at: http://www.python.org/dev/buildbot/all/g4%2520osx.4%2520trunk/builds/342 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: anthony.baxter,georg.brandl,martin.v.loewis,neal.norwitz BUILD FAILED: failed compile sincerely, -The Buildbot From buildbot at python.org Tue Apr 11 10:32:35 2006 From: buildbot at python.org (buildbot at python.org) Date: Tue, 11 Apr 2006 08:32:35 +0000 Subject: [Python-checkins] buildbot failure in alpha Tru64 5.1 trunk Message-ID: <20060411083235.880F31E4007@bag.python.org> The Buildbot has detected a new failure of alpha Tru64 5.1 trunk. Full details are available at: http://www.python.org/dev/buildbot/all/alpha%2520Tru64%25205.1%2520trunk/builds/166 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: anthony.baxter,martin.v.loewis,neal.norwitz BUILD FAILED: failed compile sincerely, -The Buildbot From buildbot at python.org Tue Apr 11 11:02:56 2006 From: buildbot at python.org (buildbot at python.org) Date: Tue, 11 Apr 2006 09:02:56 +0000 Subject: [Python-checkins] buildbot warnings in g4 osx.4 trunk Message-ID: <20060411090256.8C8321E4105@bag.python.org> The Buildbot has detected a new failure of g4 osx.4 trunk. Full details are available at: http://www.python.org/dev/buildbot/all/g4%2520osx.4%2520trunk/builds/343 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: martin.v.loewis,neal.norwitz Build Had Warnings: warnings test sincerely, -The Buildbot From python-checkins at python.org Tue Apr 11 11:03:33 2006 From: python-checkins at python.org (martin.v.loewis) Date: Tue, 11 Apr 2006 11:03:33 +0200 (CEST) Subject: [Python-checkins] r45274 - python/trunk/Python/Python-ast.c Message-ID: <20060411090333.A8D4E1E422F@bag.python.org> Author: martin.v.loewis Date: Tue Apr 11 11:03:33 2006 New Revision: 45274 Modified: python/trunk/Python/Python-ast.c Log: Regenerate. Modified: python/trunk/Python/Python-ast.c ============================================================================== --- python/trunk/Python/Python-ast.c (original) +++ python/trunk/Python/Python-ast.c Tue Apr 11 11:03:33 2006 @@ -2328,8 +2328,6 @@ result = PyType_GenericNew(Continue_type, NULL, NULL); if (!result) goto failed; break; - default: - ; } value = ast2obj_int(o->lineno); if (!value) goto failed; From python-checkins at python.org Tue Apr 11 11:04:13 2006 From: python-checkins at python.org (martin.v.loewis) Date: Tue, 11 Apr 2006 11:04:13 +0200 (CEST) Subject: [Python-checkins] r45275 - python/trunk/Objects/rangeobject.c python/trunk/Objects/setobject.c python/trunk/Objects/tupleobject.c Message-ID: <20060411090413.ADA661E4092@bag.python.org> Author: martin.v.loewis Date: Tue Apr 11 11:04:12 2006 New Revision: 45275 Modified: python/trunk/Objects/rangeobject.c python/trunk/Objects/setobject.c python/trunk/Objects/tupleobject.c Log: Remove "static forward" declaration. Move constructors after the type objects. Modified: python/trunk/Objects/rangeobject.c ============================================================================== --- python/trunk/Objects/rangeobject.c (original) +++ python/trunk/Objects/rangeobject.c Tue Apr 11 11:04:12 2006 @@ -200,53 +200,6 @@ long len; } rangeiterobject; -static PyTypeObject Pyrangeiter_Type; - -static PyObject * -range_iter(PyObject *seq) -{ - rangeiterobject *it; - - if (!PyRange_Check(seq)) { - PyErr_BadInternalCall(); - return NULL; - } - it = PyObject_New(rangeiterobject, &Pyrangeiter_Type); - if (it == NULL) - return NULL; - it->index = 0; - it->start = ((rangeobject *)seq)->start; - it->step = ((rangeobject *)seq)->step; - it->len = ((rangeobject *)seq)->len; - return (PyObject *)it; -} - -static PyObject * -range_reverse(PyObject *seq) -{ - rangeiterobject *it; - long start, step, len; - - if (!PyRange_Check(seq)) { - PyErr_BadInternalCall(); - return NULL; - } - it = PyObject_New(rangeiterobject, &Pyrangeiter_Type); - if (it == NULL) - return NULL; - - start = ((rangeobject *)seq)->start; - step = ((rangeobject *)seq)->step; - len = ((rangeobject *)seq)->len; - - it->index = 0; - it->start = start + (len-1) * step; - it->step = -step; - it->len = len; - - return (PyObject *)it; -} - static PyObject * rangeiter_next(rangeiterobject *r) { @@ -301,3 +254,48 @@ rangeiter_methods, /* tp_methods */ 0, }; + +static PyObject * +range_iter(PyObject *seq) +{ + rangeiterobject *it; + + if (!PyRange_Check(seq)) { + PyErr_BadInternalCall(); + return NULL; + } + it = PyObject_New(rangeiterobject, &Pyrangeiter_Type); + if (it == NULL) + return NULL; + it->index = 0; + it->start = ((rangeobject *)seq)->start; + it->step = ((rangeobject *)seq)->step; + it->len = ((rangeobject *)seq)->len; + return (PyObject *)it; +} + +static PyObject * +range_reverse(PyObject *seq) +{ + rangeiterobject *it; + long start, step, len; + + if (!PyRange_Check(seq)) { + PyErr_BadInternalCall(); + return NULL; + } + it = PyObject_New(rangeiterobject, &Pyrangeiter_Type); + if (it == NULL) + return NULL; + + start = ((rangeobject *)seq)->start; + step = ((rangeobject *)seq)->step; + len = ((rangeobject *)seq)->len; + + it->index = 0; + it->start = start + (len-1) * step; + it->step = -step; + it->len = len; + + return (PyObject *)it; +} Modified: python/trunk/Objects/setobject.c ============================================================================== --- python/trunk/Objects/setobject.c (original) +++ python/trunk/Objects/setobject.c Tue Apr 11 11:04:12 2006 @@ -3,7 +3,7 @@ Written and maintained by Raymond D. Hettinger Derived from Lib/sets.py and Objects/dictobject.c. - Copyright (c) 2003-5 Python Software Foundation. + Copyright (c) 2003-6 Python Software Foundation. All rights reserved. */ @@ -719,8 +719,6 @@ /***** Set iterator type ***********************************************/ -static PyTypeObject PySetIter_Type; /* Forward */ - typedef struct { PyObject_HEAD PySetObject *si_set; /* Set to NULL when iterator is exhausted */ @@ -729,20 +727,6 @@ long len; } setiterobject; -static PyObject * -set_iter(PySetObject *so) -{ - setiterobject *si = PyObject_New(setiterobject, &PySetIter_Type); - if (si == NULL) - return NULL; - Py_INCREF(so); - si->si_set = so; - si->si_used = so->used; - si->si_pos = 0; - si->len = so->used; - return (PyObject *)si; -} - static void setiter_dealloc(setiterobject *si) { @@ -838,6 +822,20 @@ 0, }; +static PyObject * +set_iter(PySetObject *so) +{ + setiterobject *si = PyObject_New(setiterobject, &PySetIter_Type); + if (si == NULL) + return NULL; + Py_INCREF(so); + si->si_set = so; + si->si_used = so->used; + si->si_pos = 0; + si->len = so->used; + return (PyObject *)si; +} + static int set_update_internal(PySetObject *so, PyObject *other) { Modified: python/trunk/Objects/tupleobject.c ============================================================================== --- python/trunk/Objects/tupleobject.c (original) +++ python/trunk/Objects/tupleobject.c Tue Apr 11 11:04:12 2006 @@ -791,27 +791,6 @@ PyTupleObject *it_seq; /* Set to NULL when iterator is exhausted */ } tupleiterobject; -PyTypeObject PyTupleIter_Type; - -static PyObject * -tuple_iter(PyObject *seq) -{ - tupleiterobject *it; - - if (!PyTuple_Check(seq)) { - PyErr_BadInternalCall(); - return NULL; - } - it = PyObject_GC_New(tupleiterobject, &PyTupleIter_Type); - if (it == NULL) - return NULL; - it->it_index = 0; - Py_INCREF(seq); - it->it_seq = (PyTupleObject *)seq; - _PyObject_GC_TRACK(it); - return (PyObject *)it; -} - static void tupleiter_dealloc(tupleiterobject *it) { @@ -901,3 +880,22 @@ tupleiter_methods, /* tp_methods */ 0, }; + +static PyObject * +tuple_iter(PyObject *seq) +{ + tupleiterobject *it; + + if (!PyTuple_Check(seq)) { + PyErr_BadInternalCall(); + return NULL; + } + it = PyObject_GC_New(tupleiterobject, &PyTupleIter_Type); + if (it == NULL) + return NULL; + it->it_index = 0; + Py_INCREF(seq); + it->it_seq = (PyTupleObject *)seq; + _PyObject_GC_TRACK(it); + return (PyObject *)it; +} From buildbot at python.org Tue Apr 11 11:04:30 2006 From: buildbot at python.org (buildbot at python.org) Date: Tue, 11 Apr 2006 09:04:30 +0000 Subject: [Python-checkins] buildbot warnings in ia64 Debian unstable trunk Message-ID: <20060411090430.2689E1E4028@bag.python.org> The Buildbot has detected a new failure of ia64 Debian unstable trunk. Full details are available at: http://www.python.org/dev/buildbot/all/ia64%2520Debian%2520unstable%2520trunk/builds/74 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: martin.v.loewis,neal.norwitz Build Had Warnings: warnings test sincerely, -The Buildbot From python-checkins at python.org Tue Apr 11 11:08:03 2006 From: python-checkins at python.org (martin.v.loewis) Date: Tue, 11 Apr 2006 11:08:03 +0200 (CEST) Subject: [Python-checkins] r45276 - python/trunk/Objects/typeobject.c Message-ID: <20060411090803.5B81D1E4007@bag.python.org> Author: martin.v.loewis Date: Tue Apr 11 11:08:02 2006 New Revision: 45276 Modified: python/trunk/Objects/typeobject.c Log: Correct casts to char*. Modified: python/trunk/Objects/typeobject.c ============================================================================== --- python/trunk/Objects/typeobject.c (original) +++ python/trunk/Objects/typeobject.c Tue Apr 11 11:08:02 2006 @@ -5216,19 +5216,19 @@ assert(offset >= 0); assert(offset < offsetof(PyHeapTypeObject, as_buffer)); if (offset >= offsetof(PyHeapTypeObject, as_sequence)) { - ptr = (void *)type->tp_as_sequence; + ptr = (char *)type->tp_as_sequence; offset -= offsetof(PyHeapTypeObject, as_sequence); } else if (offset >= offsetof(PyHeapTypeObject, as_mapping)) { - ptr = (void *)type->tp_as_mapping; + ptr = (char *)type->tp_as_mapping; offset -= offsetof(PyHeapTypeObject, as_mapping); } else if (offset >= offsetof(PyHeapTypeObject, as_number)) { - ptr = (void *)type->tp_as_number; + ptr = (char *)type->tp_as_number; offset -= offsetof(PyHeapTypeObject, as_number); } else { - ptr = (void *)type; + ptr = (char *)type; } if (ptr != NULL) ptr += offset; From python-checkins at python.org Tue Apr 11 11:17:27 2006 From: python-checkins at python.org (martin.v.loewis) Date: Tue, 11 Apr 2006 11:17:27 +0200 (CEST) Subject: [Python-checkins] r45277 - python/trunk/Python/ast.c Message-ID: <20060411091727.9A5F01E4007@bag.python.org> Author: martin.v.loewis Date: Tue Apr 11 11:17:27 2006 New Revision: 45277 Modified: python/trunk/Python/ast.c Log: Convert 0 to their respective enum types. Convert void* to their respective _ty types. Fix signature of ast_for_exprlist. Modified: python/trunk/Python/ast.c ============================================================================== --- python/trunk/Python/ast.c (original) +++ python/trunk/Python/ast.c Tue Apr 11 11:17:27 2006 @@ -31,7 +31,7 @@ static expr_ty ast_for_expr(struct compiling *, const node *); static stmt_ty ast_for_stmt(struct compiling *, const node *); static asdl_seq *ast_for_suite(struct compiling *, const node *); -static asdl_seq *ast_for_exprlist(struct compiling *, const node *, int); +static asdl_seq *ast_for_exprlist(struct compiling *, const node *, expr_context_ty); static expr_ty ast_for_testlist(struct compiling *, const node *); static expr_ty ast_for_testlist_gexp(struct compiling *, const node *); @@ -316,7 +316,7 @@ case PERCENT: return Mod; default: - return 0; + return (operator_ty)0; } } @@ -424,7 +424,7 @@ int i; for (i = 0; i < asdl_seq_LEN(s); i++) { - if (!set_context(asdl_seq_GET(s, i), ctx, n)) + if (!set_context((expr_ty)asdl_seq_GET(s, i), ctx, n)) return 0; } } @@ -465,7 +465,7 @@ return Mult; default: PyErr_Format(PyExc_SystemError, "invalid augassign: %s", STR(n)); - return 0; + return (operator_ty)0; } } @@ -499,7 +499,7 @@ default: PyErr_Format(PyExc_SystemError, "invalid comp_op: %s", STR(n)); - return 0; + return (cmpop_ty)0; } } else if (NCH(n) == 2) { @@ -513,12 +513,12 @@ default: PyErr_Format(PyExc_SystemError, "invalid comp_op: %s %s", STR(CHILD(n, 0)), STR(CHILD(n, 1))); - return 0; + return (cmpop_ty)0; } } PyErr_Format(PyExc_SystemError, "invalid comp_op: has %d children", NCH(n)); - return 0; + return (cmpop_ty)0; } static asdl_seq * @@ -985,7 +985,7 @@ return NULL; if (asdl_seq_LEN(t) == 1) - lc = comprehension(asdl_seq_GET(t, 0), expression, NULL, + lc = comprehension((expr_ty)asdl_seq_GET(t, 0), expression, NULL, c->c_arena); else lc = comprehension(Tuple(t, Store, LINENO(ch), ch->n_col_offset, @@ -1131,7 +1131,7 @@ return NULL; if (asdl_seq_LEN(t) == 1) - ge = comprehension(asdl_seq_GET(t, 0), expression, + ge = comprehension((expr_ty)asdl_seq_GET(t, 0), expression, NULL, c->c_arena); else ge = comprehension(Tuple(t, Store, LINENO(ch), ch->n_col_offset, @@ -2002,7 +2002,7 @@ } static asdl_seq * -ast_for_exprlist(struct compiling *c, const node *n, int context) +ast_for_exprlist(struct compiling *c, const node *n, expr_context_ty context) { asdl_seq *seq; int i; @@ -2626,7 +2626,7 @@ if (!_target) return NULL; if (asdl_seq_LEN(_target) == 1) - target = asdl_seq_GET(_target, 0); + target = (expr_ty)asdl_seq_GET(_target, 0); else target = Tuple(_target, Store, LINENO(n), n->n_col_offset, c->c_arena); From python-checkins at python.org Tue Apr 11 13:12:48 2006 From: python-checkins at python.org (martin.v.loewis) Date: Tue, 11 Apr 2006 13:12:48 +0200 (CEST) Subject: [Python-checkins] r45278 - python/trunk/README python/trunk/configure python/trunk/configure.in python/trunk/setup.py Message-ID: <20060411111248.C88331E4009@bag.python.org> Author: martin.v.loewis Date: Tue Apr 11 13:12:43 2006 New Revision: 45278 Modified: python/trunk/README python/trunk/configure python/trunk/configure.in python/trunk/setup.py Log: Patch #1464444: Add --with-system-ffi. Modified: python/trunk/README ============================================================================== --- python/trunk/README (original) +++ python/trunk/README Tue Apr 11 13:12:43 2006 @@ -1068,6 +1068,9 @@ --with-tsc: Profile using the Pentium timestamping counter (TSC). +--with-system-ffi: Build the _ctypes extension module using an ffi + library installed on the system. + Building for multiple architectures (using the VPATH feature) ------------------------------------------------------------- Modified: python/trunk/configure ============================================================================== --- python/trunk/configure (original) +++ python/trunk/configure Tue Apr 11 13:12:43 2006 @@ -1,5 +1,5 @@ #! /bin/sh -# From configure.in Revision: 43748 . +# From configure.in Revision: 45264 . # Guess values for system-dependent variables and create Makefiles. # Generated by GNU Autoconf 2.59 for python 2.5. # @@ -863,6 +863,7 @@ --with-suffix=.exe set executable suffix --with-pydebug build with Py_DEBUG defined --with-libs='lib1 ...' link against additional libs + --with-system-ffi build _ctypes module using an installed ffi library --with-signal-module disable/enable signal module --with-dec-threads use DEC Alpha/OSF1 thread-safe libraries --with(out)-threads[=DIRECTORY] @@ -11780,6 +11781,22 @@ echo "${ECHO_T}no" >&6 fi; +# Check for use of the system libffi library +echo "$as_me:$LINENO: checking for --with-system-ffi" >&5 +echo $ECHO_N "checking for --with-system-ffi... $ECHO_C" >&6 + +# Check whether --with-system_ffi or --without-system_ffi was given. +if test "${with_system_ffi+set}" = set; then + withval="$with_system_ffi" + +fi; + +if test -z "$with_system_ffi" +then with_system_ffi="no" +fi +echo "$as_me:$LINENO: result: $with_system_ffi" >&5 +echo "${ECHO_T}$with_system_ffi" >&6 + # Determine if signalmodule should be used. Modified: python/trunk/configure.in ============================================================================== --- python/trunk/configure.in (original) +++ python/trunk/configure.in Tue Apr 11 13:12:43 2006 @@ -1604,6 +1604,16 @@ ], [AC_MSG_RESULT(no)]) +# Check for use of the system libffi library +AC_MSG_CHECKING(for --with-system-ffi) +AC_ARG_WITH(system_ffi, + AC_HELP_STRING(--with-system-ffi, build _ctypes module using an installed ffi library)) + +if test -z "$with_system_ffi" +then with_system_ffi="no" +fi +AC_MSG_RESULT($with_system_ffi) + # Determine if signalmodule should be used. AC_SUBST(USE_SIGNAL_MODULE) AC_SUBST(SIGNAL_OBJS) Modified: python/trunk/setup.py ============================================================================== --- python/trunk/setup.py (original) +++ python/trunk/setup.py Tue Apr 11 13:12:43 2006 @@ -973,7 +973,7 @@ exts.append( Extension('dl', ['dlmodule.c']) ) # Thomas Heller's _ctypes module - self.detect_ctypes() + self.detect_ctypes(inc_dirs, lib_dirs) # Platform-specific libraries if platform == 'linux2': @@ -1269,44 +1269,46 @@ # -lGL -lGLU -lXext -lXmu \ def configure_ctypes(self, ext): - (srcdir,) = sysconfig.get_config_vars('srcdir') - ffi_builddir = os.path.join(self.build_temp, 'libffi') - ffi_srcdir = os.path.abspath(os.path.join(srcdir, 'Modules', - '_ctypes', 'libffi')) - ffi_configfile = os.path.join(ffi_builddir, 'fficonfig.py') - - if self.force or not os.path.exists(ffi_configfile): - from distutils.dir_util import mkpath - mkpath(ffi_builddir) - config_args = [] - - # Pass empty CFLAGS because we'll just append the resulting CFLAGS - # to Python's; -g or -O2 is to be avoided. - cmd = "cd %s && env CFLAGS='' '%s/configure' %s" \ - % (ffi_builddir, ffi_srcdir, " ".join(config_args)) - - res = os.system(cmd) - if res or not os.path.exists(ffi_configfile): - print "Failed to configure _ctypes module" - return False - - fficonfig = {} - execfile(ffi_configfile, globals(), fficonfig) - ffi_srcdir = os.path.join(fficonfig['ffi_srcdir'], 'src') - - # Add .S (preprocessed assembly) to C compiler source extensions. - self.compiler.src_extensions.append('.S') - - include_dirs = [os.path.join(ffi_builddir, 'include'), - ffi_builddir, ffi_srcdir] - extra_compile_args = fficonfig['ffi_cflags'].split() - - ext.sources.extend(fficonfig['ffi_sources']) - ext.include_dirs.extend(include_dirs) - ext.extra_compile_args.extend(extra_compile_args) + if not self.use_system_libffi: + (srcdir,) = sysconfig.get_config_vars('srcdir') + ffi_builddir = os.path.join(self.build_temp, 'libffi') + ffi_srcdir = os.path.abspath(os.path.join(srcdir, 'Modules', + '_ctypes', 'libffi')) + ffi_configfile = os.path.join(ffi_builddir, 'fficonfig.py') + + if self.force or not os.path.exists(ffi_configfile): + from distutils.dir_util import mkpath + mkpath(ffi_builddir) + config_args = [] + + # Pass empty CFLAGS because we'll just append the resulting + # CFLAGS to Python's; -g or -O2 is to be avoided. + cmd = "cd %s && env CFLAGS='' '%s/configure' %s" \ + % (ffi_builddir, ffi_srcdir, " ".join(config_args)) + + res = os.system(cmd) + if res or not os.path.exists(ffi_configfile): + print "Failed to configure _ctypes module" + return False + + fficonfig = {} + execfile(ffi_configfile, globals(), fficonfig) + ffi_srcdir = os.path.join(fficonfig['ffi_srcdir'], 'src') + + # Add .S (preprocessed assembly) to C compiler source extensions. + self.compiler.src_extensions.append('.S') + + include_dirs = [os.path.join(ffi_builddir, 'include'), + ffi_builddir, ffi_srcdir] + extra_compile_args = fficonfig['ffi_cflags'].split() + + ext.sources.extend(fficonfig['ffi_sources']) + ext.include_dirs.extend(include_dirs) + ext.extra_compile_args.extend(extra_compile_args) return True - def detect_ctypes(self): + def detect_ctypes(self, inc_dirs, lib_dirs): + self.use_system_libffi = False include_dirs = [] extra_compile_args = [] sources = ['_ctypes/_ctypes.c', @@ -1326,12 +1328,40 @@ ext = Extension('_ctypes', include_dirs=include_dirs, extra_compile_args=extra_compile_args, + libraries=[], sources=sources, depends=depends) ext_test = Extension('_ctypes_test', sources=['_ctypes/_ctypes_test.c']) self.extensions.extend([ext, ext_test]) + if not '--with-system-ffi' in sysconfig.get_config_var("CONFIG_ARGS"): + return + + ffi_inc = find_file('ffi.h', [], inc_dirs) + if ffi_inc is not None: + ffi_h = ffi_inc[0] + '/ffi.h' + fp = open(ffi_h) + while 1: + line = fp.readline() + if not line: + ffi_inc = None + break + if line.startswith('#define LIBFFI_H'): + break + ffi_lib = None + if ffi_inc is not None: + for lib_name in ('ffi_convenience', 'ffi_pic', 'ffi'): + if (self.compiler.find_library_file(lib_dirs, lib_name)): + ffi_lib = lib_name + break + + if ffi_inc and ffi_lib: + ext.include_dirs.extend(ffi_inc) + ext.libraries.append(ffi_lib) + self.use_system_libffi = True + + class PyBuildInstall(install): # Suppress the warning about installation into the lib_dynload # directory, which is not in sys.path when running Python during From python-checkins at python.org Tue Apr 11 14:01:58 2006 From: python-checkins at python.org (anthony.baxter) Date: Tue, 11 Apr 2006 14:01:58 +0200 (CEST) Subject: [Python-checkins] r45279 - in python/trunk: Parser/pgenmain.c Python/compile.c Python/future.c Python/pyarena.c Python/pystate.c Message-ID: <20060411120158.281E41E4458@bag.python.org> Author: anthony.baxter Date: Tue Apr 11 14:01:56 2006 New Revision: 45279 Modified: python/trunk/Parser/pgenmain.c python/trunk/Python/compile.c python/trunk/Python/future.c python/trunk/Python/pyarena.c python/trunk/Python/pystate.c Log: more low-hanging fruit. Not entirely happy with the two new VISIT macros in compile.c, but I couldn't see a better approach. Modified: python/trunk/Parser/pgenmain.c ============================================================================== --- python/trunk/Parser/pgenmain.c (original) +++ python/trunk/Parser/pgenmain.c Tue Apr 11 14:01:56 2006 @@ -136,7 +136,7 @@ PyOS_Readline(FILE *sys_stdin, FILE *sys_stdout, char *prompt) { size_t n = 1000; - char *p = PyMem_MALLOC(n); + char *p = (char *)PyMem_MALLOC(n); char *q; if (p == NULL) return NULL; @@ -149,7 +149,7 @@ n = strlen(p); if (n > 0 && p[n-1] != '\n') p[n-1] = '\n'; - return PyMem_REALLOC(p, n+1); + return (char *)PyMem_REALLOC(p, n+1); } /* No-nonsense fgets */ Modified: python/trunk/Python/compile.c ============================================================================== --- python/trunk/Python/compile.c (original) +++ python/trunk/Python/compile.c Tue Apr 11 14:01:56 2006 @@ -197,19 +197,19 @@ static PyObject *__doc__; PyObject * -_Py_Mangle(PyObject *private, PyObject *ident) +_Py_Mangle(PyObject *privateobj, PyObject *ident) { /* Name mangling: __private becomes _classname__private. This is independent from how the name is used. */ const char *p, *name = PyString_AsString(ident); char *buffer; size_t nlen, plen; - if (private == NULL || name == NULL || name[0] != '_' || + if (privateobj == NULL || name == NULL || name[0] != '_' || name[1] != '_') { Py_INCREF(ident); return ident; } - p = PyString_AsString(private); + p = PyString_AsString(privateobj); nlen = strlen(name); if (name[nlen-1] == '_' && name[nlen-2] == '_') { Py_INCREF(ident); @@ -612,7 +612,7 @@ static unsigned int * markblocks(unsigned char *code, int len) { - unsigned int *blocks = PyMem_Malloc(len*sizeof(int)); + unsigned int *blocks = (unsigned int *)PyMem_Malloc(len*sizeof(int)); int i,j, opcode, blockcnt = 0; if (blocks == NULL) @@ -693,10 +693,11 @@ goto exitUnchanged; /* Make a modifiable copy of the code string */ - codestr = PyMem_Malloc(codelen); + codestr = (unsigned char *)PyMem_Malloc(codelen); if (codestr == NULL) goto exitUnchanged; - codestr = memcpy(codestr, PyString_AS_STRING(code), codelen); + codestr = (unsigned char *)memcpy(codestr, + PyString_AS_STRING(code), codelen); /* Verify that RETURN_VALUE terminates the codestring. This allows the various transformation patterns to look ahead several @@ -707,7 +708,7 @@ goto exitUnchanged; /* Mapping to new jump targets after NOPs are removed */ - addrmap = PyMem_Malloc(codelen * sizeof(int)); + addrmap = (int *)PyMem_Malloc(codelen * sizeof(int)); if (addrmap == NULL) goto exitUnchanged; @@ -1087,7 +1088,8 @@ { struct compiler_unit *u; - u = PyObject_Malloc(sizeof(struct compiler_unit)); + u = (struct compiler_unit *)PyObject_Malloc(sizeof( + struct compiler_unit)); if (!u) { PyErr_NoMemory(); return 0; @@ -1243,8 +1245,8 @@ { assert(b != NULL); if (b->b_instr == NULL) { - b->b_instr = PyObject_Malloc(sizeof(struct instr) * - DEFAULT_BLOCK_SIZE); + b->b_instr = (struct instr *)PyObject_Malloc( + sizeof(struct instr) * DEFAULT_BLOCK_SIZE); if (b->b_instr == NULL) { PyErr_NoMemory(); return -1; @@ -1262,7 +1264,8 @@ return -1; } b->b_ialloc <<= 1; - b->b_instr = PyObject_Realloc((void *)b->b_instr, newsize); + b->b_instr = (struct instr *)PyObject_Realloc( + (void *)b->b_instr, newsize); if (b->b_instr == NULL) return -1; memset((char *)b->b_instr + oldsize, 0, newsize - oldsize); @@ -1720,6 +1723,16 @@ } \ } +#define VISIT_SEQ_WITH_CAST(C, TYPE, SEQ, CAST) { \ + int _i; \ + asdl_seq *seq = (SEQ); /* avoid variable capture */ \ + for (_i = 0; _i < asdl_seq_LEN(seq); _i++) { \ + TYPE ## _ty elt = (CAST)asdl_seq_GET(seq, _i); \ + if (!compiler_visit_ ## TYPE((C), elt)) \ + return 0; \ + } \ +} + #define VISIT_SEQ_IN_SCOPE(C, TYPE, SEQ) { \ int _i; \ asdl_seq *seq = (SEQ); /* avoid variable capture */ \ @@ -1732,6 +1745,18 @@ } \ } +#define VISIT_SEQ_IN_SCOPE_WITH_CAST(C, TYPE, SEQ, CAST) { \ + int _i; \ + asdl_seq *seq = (SEQ); /* avoid variable capture */ \ + for (_i = 0; _i < asdl_seq_LEN(seq); _i++) { \ + TYPE ## _ty elt = (CAST)asdl_seq_GET(seq, _i); \ + if (!compiler_visit_ ## TYPE((C), elt)) { \ + compiler_exit_scope(c); \ + return 0; \ + } \ + } \ +} + static int compiler_isdocstring(stmt_ty s) { @@ -1750,7 +1775,7 @@ if (!asdl_seq_LEN(stmts)) return 1; - st = asdl_seq_GET(stmts, 0); + st = (stmt_ty)asdl_seq_GET(stmts, 0); if (compiler_isdocstring(st)) { i = 1; VISIT(c, expr, st->v.Expr.value); @@ -1758,7 +1783,7 @@ return 0; } for (; i < asdl_seq_LEN(stmts); i++) - VISIT(c, stmt, asdl_seq_GET(stmts, i)); + VISIT(c, stmt, (stmt_ty)asdl_seq_GET(stmts, i)); return 1; } @@ -1784,7 +1809,8 @@ break; case Interactive_kind: c->c_interactive = 1; - VISIT_SEQ_IN_SCOPE(c, stmt, mod->v.Interactive.body); + VISIT_SEQ_IN_SCOPE_WITH_CAST(c, stmt, + mod->v.Interactive.body, stmt_ty); break; case Expression_kind: VISIT_IN_SCOPE(c, expr, mod->v.Expression.body); @@ -1901,7 +1927,7 @@ return 1; for (i = 0; i < asdl_seq_LEN(decos); i++) { - VISIT(c, expr, asdl_seq_GET(decos, i)); + VISIT(c, expr, (expr_ty)asdl_seq_GET(decos, i)); } return 1; } @@ -1913,7 +1939,7 @@ int n = asdl_seq_LEN(args->args); /* Correctly handle nested argument lists */ for (i = 0; i < n; i++) { - expr_ty arg = asdl_seq_GET(args->args, i); + expr_ty arg = (expr_ty)asdl_seq_GET(args->args, i); if (arg->kind == Tuple_kind) { PyObject *id = PyString_FromFormat(".%d", i); if (id == NULL) { @@ -1945,12 +1971,12 @@ if (!compiler_decorators(c, decos)) return 0; if (args->defaults) - VISIT_SEQ(c, expr, args->defaults); + VISIT_SEQ_WITH_CAST(c, expr, args->defaults, expr_ty); if (!compiler_enter_scope(c, s->v.FunctionDef.name, (void *)s, s->lineno)) return 0; - st = asdl_seq_GET(s->v.FunctionDef.body, 0); + st = (stmt_ty)asdl_seq_GET(s->v.FunctionDef.body, 0); docstring = compiler_isdocstring(st); if (docstring) first_const = st->v.Expr.value->v.Str.s; @@ -1966,7 +1992,7 @@ n = asdl_seq_LEN(s->v.FunctionDef.body); /* if there was a docstring, we need to skip the first statement */ for (i = docstring; i < n; i++) { - stmt_ty s2 = asdl_seq_GET(s->v.FunctionDef.body, i); + stmt_ty s2 = (stmt_ty)asdl_seq_GET(s->v.FunctionDef.body, i); if (i == 0 && s2->kind == Expr_kind && s2->v.Expr.value->kind == Str_kind) continue; @@ -1998,7 +2024,7 @@ /* push the tuple of base classes on the stack */ n = asdl_seq_LEN(s->v.ClassDef.bases); if (n > 0) - VISIT_SEQ(c, expr, s->v.ClassDef.bases); + VISIT_SEQ_WITH_CAST(c, expr, s->v.ClassDef.bases, expr_ty); ADDOP_I(c, BUILD_TUPLE, n); if (!compiler_enter_scope(c, s->v.ClassDef.name, (void *)s, s->lineno)) @@ -2082,7 +2108,7 @@ } if (args->defaults) - VISIT_SEQ(c, expr, args->defaults); + VISIT_SEQ_WITH_CAST(c, expr, args->defaults, expr_ty); if (!compiler_enter_scope(c, name, (void *)e, e->lineno)) return 0; @@ -2155,12 +2181,12 @@ VISIT(c, expr, s->v.If.test); ADDOP_JREL(c, JUMP_IF_FALSE, next); ADDOP(c, POP_TOP); - VISIT_SEQ(c, stmt, s->v.If.body); + VISIT_SEQ_WITH_CAST(c, stmt, s->v.If.body, stmt_ty); ADDOP_JREL(c, JUMP_FORWARD, end); compiler_use_next_block(c, next); ADDOP(c, POP_TOP); if (s->v.If.orelse) - VISIT_SEQ(c, stmt, s->v.If.orelse); + VISIT_SEQ_WITH_CAST(c, stmt, s->v.If.orelse, stmt_ty); compiler_use_next_block(c, end); return 1; } @@ -2183,12 +2209,12 @@ compiler_use_next_block(c, start); ADDOP_JREL(c, FOR_ITER, cleanup); VISIT(c, expr, s->v.For.target); - VISIT_SEQ(c, stmt, s->v.For.body); + VISIT_SEQ_WITH_CAST(c, stmt, s->v.For.body, stmt_ty); ADDOP_JABS(c, JUMP_ABSOLUTE, start); compiler_use_next_block(c, cleanup); ADDOP(c, POP_BLOCK); compiler_pop_fblock(c, LOOP, start); - VISIT_SEQ(c, stmt, s->v.For.orelse); + VISIT_SEQ_WITH_CAST(c, stmt, s->v.For.orelse, stmt_ty); compiler_use_next_block(c, end); return 1; } @@ -2227,7 +2253,7 @@ ADDOP_JREL(c, JUMP_IF_FALSE, anchor); ADDOP(c, POP_TOP); } - VISIT_SEQ(c, stmt, s->v.While.body); + VISIT_SEQ_WITH_CAST(c, stmt, s->v.While.body, stmt_ty); ADDOP_JABS(c, JUMP_ABSOLUTE, loop); /* XXX should the two POP instructions be in a separate block @@ -2241,7 +2267,7 @@ } compiler_pop_fblock(c, LOOP, loop); if (orelse != NULL) /* what if orelse is just pass? */ - VISIT_SEQ(c, stmt, s->v.While.orelse); + VISIT_SEQ_WITH_CAST(c, stmt, s->v.While.orelse, stmt_ty); compiler_use_next_block(c, end); return 1; @@ -2322,7 +2348,7 @@ compiler_use_next_block(c, body); if (!compiler_push_fblock(c, FINALLY_TRY, body)) return 0; - VISIT_SEQ(c, stmt, s->v.TryFinally.body); + VISIT_SEQ_WITH_CAST(c, stmt, s->v.TryFinally.body, stmt_ty); ADDOP(c, POP_BLOCK); compiler_pop_fblock(c, FINALLY_TRY, body); @@ -2330,7 +2356,7 @@ compiler_use_next_block(c, end); if (!compiler_push_fblock(c, FINALLY_END, end)) return 0; - VISIT_SEQ(c, stmt, s->v.TryFinally.finalbody); + VISIT_SEQ_WITH_CAST(c, stmt, s->v.TryFinally.finalbody, stmt_ty); ADDOP(c, END_FINALLY); compiler_pop_fblock(c, FINALLY_END, end); @@ -2387,14 +2413,14 @@ compiler_use_next_block(c, body); if (!compiler_push_fblock(c, EXCEPT, body)) return 0; - VISIT_SEQ(c, stmt, s->v.TryExcept.body); + VISIT_SEQ_WITH_CAST(c, stmt, s->v.TryExcept.body, stmt_ty); ADDOP(c, POP_BLOCK); compiler_pop_fblock(c, EXCEPT, body); ADDOP_JREL(c, JUMP_FORWARD, orelse); n = asdl_seq_LEN(s->v.TryExcept.handlers); compiler_use_next_block(c, except); for (i = 0; i < n; i++) { - excepthandler_ty handler = asdl_seq_GET( + excepthandler_ty handler = (excepthandler_ty)asdl_seq_GET( s->v.TryExcept.handlers, i); if (!handler->type && i < n-1) return compiler_error(c, "default 'except:' must be last"); @@ -2418,7 +2444,7 @@ ADDOP(c, POP_TOP); } ADDOP(c, POP_TOP); - VISIT_SEQ(c, stmt, handler->body); + VISIT_SEQ_WITH_CAST(c, stmt, handler->body, stmt_ty); ADDOP_JREL(c, JUMP_FORWARD, end); compiler_use_next_block(c, except); if (handler->type) @@ -2426,7 +2452,7 @@ } ADDOP(c, END_FINALLY); compiler_use_next_block(c, orelse); - VISIT_SEQ(c, stmt, s->v.TryExcept.orelse); + VISIT_SEQ_WITH_CAST(c, stmt, s->v.TryExcept.orelse, stmt_ty); compiler_use_next_block(c, end); return 1; } @@ -2474,7 +2500,7 @@ int i, n = asdl_seq_LEN(s->v.Import.names); for (i = 0; i < n; i++) { - alias_ty alias = asdl_seq_GET(s->v.Import.names, i); + alias_ty alias = (alias_ty)asdl_seq_GET(s->v.Import.names, i); int r; PyObject *level; @@ -2538,7 +2564,7 @@ /* build up the names */ for (i = 0; i < n; i++) { - alias_ty alias = asdl_seq_GET(s->v.ImportFrom.names, i); + alias_ty alias = (alias_ty)asdl_seq_GET(s->v.ImportFrom.names, i); Py_INCREF(alias->name); PyTuple_SET_ITEM(names, i, alias->name); } @@ -2561,7 +2587,7 @@ Py_DECREF(names); ADDOP_NAME(c, IMPORT_NAME, s->v.ImportFrom.module, names); for (i = 0; i < n; i++) { - alias_ty alias = asdl_seq_GET(s->v.ImportFrom.names, i); + alias_ty alias = (alias_ty)asdl_seq_GET(s->v.ImportFrom.names, i); identifier store_name; if (i == 0 && *PyString_AS_STRING(alias->name) == '*') { @@ -2646,7 +2672,7 @@ ADDOP(c, RETURN_VALUE); break; case Delete_kind: - VISIT_SEQ(c, expr, s->v.Delete.targets) + VISIT_SEQ_WITH_CAST(c, expr, s->v.Delete.targets, expr_ty) break; case Assign_kind: n = asdl_seq_LEN(s->v.Assign.targets); @@ -3000,11 +3026,11 @@ s = e->v.BoolOp.values; n = asdl_seq_LEN(s) - 1; for (i = 0; i < n; ++i) { - VISIT(c, expr, asdl_seq_GET(s, i)); + VISIT(c, expr, (expr_ty)asdl_seq_GET(s, i)); ADDOP_JREL(c, jumpi, end); ADDOP(c, POP_TOP) } - VISIT(c, expr, asdl_seq_GET(s, n)); + VISIT(c, expr, (expr_ty)asdl_seq_GET(s, n)); compiler_use_next_block(c, end); return 1; } @@ -3016,7 +3042,7 @@ if (e->v.List.ctx == Store) { ADDOP_I(c, UNPACK_SEQUENCE, n); } - VISIT_SEQ(c, expr, e->v.List.elts); + VISIT_SEQ_WITH_CAST(c, expr, e->v.List.elts, expr_ty); if (e->v.List.ctx == Load) { ADDOP_I(c, BUILD_LIST, n); } @@ -3030,7 +3056,7 @@ if (e->v.Tuple.ctx == Store) { ADDOP_I(c, UNPACK_SEQUENCE, n); } - VISIT_SEQ(c, expr, e->v.Tuple.elts); + VISIT_SEQ_WITH_CAST(c, expr, e->v.Tuple.elts, expr_ty); if (e->v.Tuple.ctx == Load) { ADDOP_I(c, BUILD_TUPLE, n); } @@ -3051,7 +3077,8 @@ cleanup = compiler_new_block(c); if (cleanup == NULL) return 0; - VISIT(c, expr, asdl_seq_GET(e->v.Compare.comparators, 0)); + VISIT(c, expr, + (expr_ty)asdl_seq_GET(e->v.Compare.comparators, 0)); } for (i = 1; i < n; i++) { ADDOP(c, DUP_TOP); @@ -3063,9 +3090,10 @@ NEXT_BLOCK(c); ADDOP(c, POP_TOP); if (i < (n - 1)) - VISIT(c, expr, asdl_seq_GET(e->v.Compare.comparators, i)); + VISIT(c, expr, + (expr_ty)asdl_seq_GET(e->v.Compare.comparators, i)); } - VISIT(c, expr, asdl_seq_GET(e->v.Compare.comparators, n - 1)); + VISIT(c, expr, (expr_ty)asdl_seq_GET(e->v.Compare.comparators, n - 1)); ADDOP_I(c, COMPARE_OP, /* XXX We're casting a void* to cmpop_ty in the next stmt. */ cmpop((cmpop_ty)asdl_seq_GET(e->v.Compare.ops, n - 1))); @@ -3089,9 +3117,9 @@ VISIT(c, expr, e->v.Call.func); n = asdl_seq_LEN(e->v.Call.args); - VISIT_SEQ(c, expr, e->v.Call.args); + VISIT_SEQ_WITH_CAST(c, expr, e->v.Call.args, expr_ty); if (e->v.Call.keywords) { - VISIT_SEQ(c, keyword, e->v.Call.keywords); + VISIT_SEQ_WITH_CAST(c, keyword, e->v.Call.keywords, keyword_ty); n |= asdl_seq_LEN(e->v.Call.keywords) << 8; } if (e->v.Call.starargs) { @@ -3140,7 +3168,7 @@ anchor == NULL) return 0; - l = asdl_seq_GET(generators, gen_index); + l = (comprehension_ty)asdl_seq_GET(generators, gen_index); VISIT(c, expr, l->iter); ADDOP(c, GET_ITER); compiler_use_next_block(c, start); @@ -3151,7 +3179,7 @@ /* XXX this needs to be cleaned up...a lot! */ n = asdl_seq_LEN(l->ifs); for (i = 0; i < n; i++) { - expr_ty e = asdl_seq_GET(l->ifs, i); + expr_ty e = (expr_ty)asdl_seq_GET(l->ifs, i); VISIT(c, expr, e); ADDOP_JREL(c, JUMP_IF_FALSE, if_cleanup); NEXT_BLOCK(c); @@ -3236,7 +3264,7 @@ anchor == NULL || end == NULL) return 0; - ge = asdl_seq_GET(generators, gen_index); + ge = (comprehension_ty)asdl_seq_GET(generators, gen_index); ADDOP_JREL(c, SETUP_LOOP, end); if (!compiler_push_fblock(c, LOOP, start)) return 0; @@ -3259,7 +3287,7 @@ /* XXX this needs to be cleaned up...a lot! */ n = asdl_seq_LEN(ge->ifs); for (i = 0; i < n; i++) { - expr_ty e = asdl_seq_GET(ge->ifs, i); + expr_ty e = (expr_ty)asdl_seq_GET(ge->ifs, i); VISIT(c, expr, e); ADDOP_JREL(c, JUMP_IF_FALSE, if_cleanup); NEXT_BLOCK(c); @@ -3472,7 +3500,7 @@ } /* BLOCK code */ - VISIT_SEQ(c, stmt, s->v.With.body); + VISIT_SEQ_WITH_CAST(c, stmt, s->v.With.body, stmt_ty); /* End of try block; start the finally block */ ADDOP(c, POP_BLOCK); @@ -3531,9 +3559,11 @@ It wants the stack to look like (value) (dict) (key) */ for (i = 0; i < n; i++) { ADDOP(c, DUP_TOP); - VISIT(c, expr, asdl_seq_GET(e->v.Dict.values, i)); + VISIT(c, expr, + (expr_ty)asdl_seq_GET(e->v.Dict.values, i)); ADDOP(c, ROT_TWO); - VISIT(c, expr, asdl_seq_GET(e->v.Dict.keys, i)); + VISIT(c, expr, + (expr_ty)asdl_seq_GET(e->v.Dict.keys, i)); ADDOP(c, STORE_SUBSCR); } break; @@ -3900,7 +3930,8 @@ if (ctx != AugStore) { int i, n = asdl_seq_LEN(s->v.ExtSlice.dims); for (i = 0; i < n; i++) { - slice_ty sub = asdl_seq_GET(s->v.ExtSlice.dims, i); + slice_ty sub = (slice_ty)asdl_seq_GET( + s->v.ExtSlice.dims, i); if (!compiler_visit_nested_slice(c, sub, ctx)) return 0; } Modified: python/trunk/Python/future.c ============================================================================== --- python/trunk/Python/future.c (original) +++ python/trunk/Python/future.c Tue Apr 11 14:01:56 2006 @@ -19,7 +19,7 @@ names = s->v.ImportFrom.names; for (i = 0; i < asdl_seq_LEN(names); i++) { - alias_ty name = asdl_seq_GET(names, i); + alias_ty name = (alias_ty)asdl_seq_GET(names, i); const char *feature = PyString_AsString(name->name); if (!feature) return 0; @@ -73,7 +73,7 @@ for (i = 0; i < asdl_seq_LEN(mod->v.Module.body); i++) { - stmt_ty s = asdl_seq_GET(mod->v.Module.body, i); + stmt_ty s = (stmt_ty)asdl_seq_GET(mod->v.Module.body, i); if (done && s->lineno > prev_line) return 1; Modified: python/trunk/Python/pyarena.c ============================================================================== --- python/trunk/Python/pyarena.c (original) +++ python/trunk/Python/pyarena.c Tue Apr 11 14:01:56 2006 @@ -105,14 +105,14 @@ the default block, allocate a one-off block that is exactly the right size. */ /* TODO(jhylton): Think about space waste at end of block */ - block *new = block_new( + block *newbl = block_new( size < DEFAULT_BLOCK_SIZE ? DEFAULT_BLOCK_SIZE : size); - if (!new) + if (!newbl) return NULL; assert(!b->ab_next); - b->ab_next = new; - b = new; + b->ab_next = newbl; + b = newbl; } assert(b->ab_offset + size <= b->ab_size); Modified: python/trunk/Python/pystate.c ============================================================================== --- python/trunk/Python/pystate.c (original) +++ python/trunk/Python/pystate.c Tue Apr 11 14:01:56 2006 @@ -297,23 +297,23 @@ PyThreadState * -PyThreadState_Swap(PyThreadState *new) +PyThreadState_Swap(PyThreadState *newts) { - PyThreadState *old = _PyThreadState_Current; + PyThreadState *oldts = _PyThreadState_Current; - _PyThreadState_Current = new; + _PyThreadState_Current = newts; /* It should not be possible for more than one thread state to be used for a thread. Check this the best we can in debug builds. */ #if defined(Py_DEBUG) && defined(WITH_THREAD) - if (new) { + if (newts) { PyThreadState *check = PyGILState_GetThisThreadState(); - if (check && check->interp == new->interp && check != new) + if (check && check->interp == newts->interp && check != newts) Py_FatalError("Invalid thread state for this thread"); } #endif - return old; + return oldts; } /* An extension mechanism to store arbitrary additional per-thread state. @@ -491,7 +491,7 @@ called Py_Initialize() and usually PyEval_InitThreads(). */ assert(autoInterpreterState); /* Py_Initialize() hasn't been called! */ - tcur = PyThread_get_key_value(autoTLSkey); + tcur = (PyThreadState *)PyThread_get_key_value(autoTLSkey); if (tcur == NULL) { /* Create a new thread state for this thread */ tcur = PyThreadState_New(autoInterpreterState); @@ -518,7 +518,8 @@ void PyGILState_Release(PyGILState_STATE oldstate) { - PyThreadState *tcur = PyThread_get_key_value(autoTLSkey); + PyThreadState *tcur = (PyThreadState *)PyThread_get_key_value( + autoTLSkey); if (tcur == NULL) Py_FatalError("auto-releasing thread-state, " "but no thread-state for this thread"); From python-checkins at python.org Tue Apr 11 14:14:11 2006 From: python-checkins at python.org (anthony.baxter) Date: Tue, 11 Apr 2006 14:14:11 +0200 (CEST) Subject: [Python-checkins] r45280 - in python/trunk: Modules/gcmodule.c Modules/getpath.c Modules/main.c Modules/posixmodule.c Python/pystrtod.c Message-ID: <20060411121411.5FDBA1E4009@bag.python.org> Author: anthony.baxter Date: Tue Apr 11 14:14:09 2006 New Revision: 45280 Modified: python/trunk/Modules/gcmodule.c python/trunk/Modules/getpath.c python/trunk/Modules/main.c python/trunk/Modules/posixmodule.c python/trunk/Python/pystrtod.c Log: some more fruit. Modified: python/trunk/Modules/gcmodule.c ============================================================================== --- python/trunk/Modules/gcmodule.c (original) +++ python/trunk/Modules/gcmodule.c Tue Apr 11 14:14:09 2006 @@ -1281,7 +1281,8 @@ _PyObject_GC_Malloc(size_t basicsize) { PyObject *op; - PyGC_Head *g = PyObject_MALLOC(sizeof(PyGC_Head) + basicsize); + PyGC_Head *g = (PyGC_Head *)PyObject_MALLOC( + sizeof(PyGC_Head) + basicsize); if (g == NULL) return PyErr_NoMemory(); g->gc.gc_refs = GC_UNTRACKED; @@ -1323,7 +1324,7 @@ { const size_t basicsize = _PyObject_VAR_SIZE(op->ob_type, nitems); PyGC_Head *g = AS_GC(op); - g = PyObject_REALLOC(g, sizeof(PyGC_Head) + basicsize); + g = (PyGC_Head *)PyObject_REALLOC(g, sizeof(PyGC_Head) + basicsize); if (g == NULL) return (PyVarObject *)PyErr_NoMemory(); op = (PyVarObject *) FROM_GC(g); Modified: python/trunk/Modules/getpath.c ============================================================================== --- python/trunk/Modules/getpath.c (original) +++ python/trunk/Modules/getpath.c Tue Apr 11 14:14:09 2006 @@ -566,7 +566,7 @@ bufsz += strlen(exec_prefix) + 1; /* This is the only malloc call in this file */ - buf = PyMem_Malloc(bufsz); + buf = (char *)PyMem_Malloc(bufsz); if (buf == NULL) { /* We can't exit, so print a warning and limp along */ Modified: python/trunk/Modules/main.c ============================================================================== --- python/trunk/Modules/main.c (original) +++ python/trunk/Modules/main.c Tue Apr 11 14:14:09 2006 @@ -208,7 +208,7 @@ /* -c is the last option; following arguments that look like options are left for the command to interpret. */ - command = malloc(strlen(_PyOS_optarg) + 2); + command = (char *)malloc(strlen(_PyOS_optarg) + 2); if (command == NULL) Py_FatalError( "not enough memory to copy -c argument"); @@ -221,7 +221,7 @@ /* -m is the last option; following arguments that look like options are left for the module to interpret. */ - module = malloc(strlen(_PyOS_optarg) + 2); + module = (char *)malloc(strlen(_PyOS_optarg) + 2); if (module == NULL) Py_FatalError( "not enough memory to copy -m argument"); Modified: python/trunk/Modules/posixmodule.c ============================================================================== --- python/trunk/Modules/posixmodule.c (original) +++ python/trunk/Modules/posixmodule.c Tue Apr 11 14:14:09 2006 @@ -6009,7 +6009,7 @@ posix_putenv(PyObject *self, PyObject *args) { char *s1, *s2; - char *new; + char *newenv; PyObject *newstr; size_t len; @@ -6040,9 +6040,9 @@ newstr = PyString_FromStringAndSize(NULL, (int)len - 1); if (newstr == NULL) return PyErr_NoMemory(); - new = PyString_AS_STRING(newstr); - PyOS_snprintf(new, len, "%s=%s", s1, s2); - if (putenv(new)) { + newenv = PyString_AS_STRING(newstr); + PyOS_snprintf(newenv, len, "%s=%s", s1, s2); + if (putenv(newenv)) { Py_DECREF(newstr); posix_error(); return NULL; Modified: python/trunk/Python/pystrtod.c ============================================================================== --- python/trunk/Python/pystrtod.c (original) +++ python/trunk/Python/pystrtod.c Tue Apr 11 14:14:09 2006 @@ -101,7 +101,7 @@ char *copy, *c; /* We need to convert the '.' to the locale specific decimal point */ - copy = malloc(end - nptr + 1 + decimal_point_len); + copy = (char *)malloc(end - nptr + 1 + decimal_point_len); c = copy; memcpy(c, nptr, decimal_point_pos - nptr); From jeremy at alum.mit.edu Tue Apr 11 14:21:41 2006 From: jeremy at alum.mit.edu (Jeremy Hylton) Date: Tue, 11 Apr 2006 08:21:41 -0400 Subject: [Python-checkins] r45279 - in python/trunk: Parser/pgenmain.c Python/compile.c Python/future.c Python/pyarena.c Python/pystate.c In-Reply-To: <20060411120158.281E41E4458@bag.python.org> References: <20060411120158.281E41E4458@bag.python.org> Message-ID: On 4/11/06, anthony.baxter wrote: > Author: anthony.baxter > Date: Tue Apr 11 14:01:56 2006 > New Revision: 45279 > > Modified: > python/trunk/Parser/pgenmain.c > python/trunk/Python/compile.c > python/trunk/Python/future.c > python/trunk/Python/pyarena.c > python/trunk/Python/pystate.c > Log: > more low-hanging fruit. Not entirely happy with the two new VISIT macros in > compile.c, but I couldn't see a better approach. Could you provide more descriptive log messages? I can't fathom what kind of low-hanging fruit you are talking about. I'm guessing that you are dealing with warnings on some compiler, but I don't know which one or why we didn't see them before. As for the new visit macros, the second argument to VISIT is the type of the sequence elements that are being iterated over. You should be able to generate a cast from that without passing a separate cast token. I can't think of a case where TYPE did not imply CAST. - VISIT_SEQ(c, stmt, s->v.TryFinally.finalbody); + VISIT_SEQ_WITH_CAST(c, stmt, s->v.TryFinally.finalbody, stmt_ty); +#define VISIT_SEQ_WITH_CAST(C, TYPE, SEQ, CAST) { \ + int _i; \ + asdl_seq *seq = (SEQ); /* avoid variable capture */ \ + for (_i = 0; _i < asdl_seq_LEN(seq); _i++) { \ + TYPE ## _ty elt = (CAST)asdl_seq_GET(seq, _i); \ (##_ty) instead of CAST? + if (!compiler_visit_ ## TYPE((C), elt)) \ + return 0; \ + } \ +} + Jeremy From anthony at interlink.com.au Tue Apr 11 14:26:16 2006 From: anthony at interlink.com.au (Anthony Baxter) Date: Tue, 11 Apr 2006 22:26:16 +1000 Subject: [Python-checkins] r45279 - in python/trunk: Parser/pgenmain.c Python/compile.c Python/future.c Python/pyarena.c Python/pystate.c In-Reply-To: References: <20060411120158.281E41E4458@bag.python.org> Message-ID: <200604112226.17900.anthony@interlink.com.au> On Tuesday 11 April 2006 22:21, Jeremy Hylton wrote: > > more low-hanging fruit. Not entirely happy with the two new VISIT > > macros in compile.c, but I couldn't see a better approach. > > Could you provide more descriptive log messages? I can't fathom > what kind of low-hanging fruit you are talking about. I'm guessing > that you are dealing with warnings on some compiler, but I don't > know which one or why we didn't see them before. Ok. In this case, it's making the code compile with C++ compilers. I'll edit the log messages. > As for the new visit macros, the second argument to VISIT is the > type of the sequence elements that are being iterated over. You > should be able to generate a cast from that without passing a > separate cast token. I can't think of a case where TYPE did not > imply CAST. > > - VISIT_SEQ(c, stmt, s->v.TryFinally.finalbody); > + VISIT_SEQ_WITH_CAST(c, stmt, s->v.TryFinally.finalbody, > stmt_ty); > > +#define VISIT_SEQ_WITH_CAST(C, TYPE, SEQ, CAST) { \ > + int _i; \ > + asdl_seq *seq = (SEQ); /* avoid variable capture */ \ > + for (_i = 0; _i < asdl_seq_LEN(seq); _i++) { \ > + TYPE ## _ty elt = (CAST)asdl_seq_GET(seq, _i); \ > > (##_ty) instead of > CAST? Hm. That seems like a workable approach. Will give it a go. -- Anthony Baxter It's never too late to have a happy childhood. From python-checkins at python.org Tue Apr 11 14:38:06 2006 From: python-checkins at python.org (anthony.baxter) Date: Tue, 11 Apr 2006 14:38:06 +0200 (CEST) Subject: [Python-checkins] r45280 - svn:log Message-ID: <20060411123806.86B861E4009@bag.python.org> Author: anthony.baxter Revision: 45280 Property Name: svn:log New Property Value: Some more changes to make code compile under a C++ compiler. From python-checkins at python.org Tue Apr 11 14:38:54 2006 From: python-checkins at python.org (anthony.baxter) Date: Tue, 11 Apr 2006 14:38:54 +0200 (CEST) Subject: [Python-checkins] r45279 - svn:log Message-ID: <20060411123854.7EF381E4009@bag.python.org> Author: anthony.baxter Revision: 45279 Property Name: svn:log New Property Value: more low-hanging fruit to make code compile under a C++ compiler. Not entirely happy with the two new VISIT macros in compile.c, but I couldn't see a better approach. From jeremy at alum.mit.edu Tue Apr 11 15:04:36 2006 From: jeremy at alum.mit.edu (Jeremy Hylton) Date: Tue, 11 Apr 2006 09:04:36 -0400 Subject: [Python-checkins] r45279 - in python/trunk: Parser/pgenmain.c Python/compile.c Python/future.c Python/pyarena.c Python/pystate.c In-Reply-To: <200604112226.17900.anthony@interlink.com.au> References: <20060411120158.281E41E4458@bag.python.org> <200604112226.17900.anthony@interlink.com.au> Message-ID: On 4/11/06, Anthony Baxter wrote: > On Tuesday 11 April 2006 22:21, Jeremy Hylton wrote: > > > more low-hanging fruit. Not entirely happy with the two new VISIT > > > macros in compile.c, but I couldn't see a better approach. > > > > Could you provide more descriptive log messages? I can't fathom > > what kind of low-hanging fruit you are talking about. I'm guessing > > that you are dealing with warnings on some compiler, but I don't > > know which one or why we didn't see them before. > > Ok. In this case, it's making the code compile with C++ compilers. > > I'll edit the log messages. Thanks. I found your other log messages after I sent the messages, but in six months you'll be the only one who remembers that for this change "low-hanging fruit" meant "C++ compiler" :-). > > > As for the new visit macros, the second argument to VISIT is the > > type of the sequence elements that are being iterated over. You > > should be able to generate a cast from that without passing a > > separate cast token. I can't think of a case where TYPE did not > > imply CAST. > > > > - VISIT_SEQ(c, stmt, s->v.TryFinally.finalbody); > > + VISIT_SEQ_WITH_CAST(c, stmt, s->v.TryFinally.finalbody, > > stmt_ty); > > > > +#define VISIT_SEQ_WITH_CAST(C, TYPE, SEQ, CAST) { \ > > + int _i; \ > > + asdl_seq *seq = (SEQ); /* avoid variable capture */ \ > > + for (_i = 0; _i < asdl_seq_LEN(seq); _i++) { \ > > + TYPE ## _ty elt = (CAST)asdl_seq_GET(seq, _i); \ > > > > (##_ty) instead of > > CAST? > > Hm. That seems like a workable approach. Will give it a go. Cool. Jeremy From buildbot at python.org Tue Apr 11 15:04:44 2006 From: buildbot at python.org (buildbot at python.org) Date: Tue, 11 Apr 2006 13:04:44 +0000 Subject: [Python-checkins] buildbot warnings in x86 OpenBSD trunk Message-ID: <20060411130445.13F5D1E4009@bag.python.org> The Buildbot has detected a new failure of x86 OpenBSD trunk. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520OpenBSD%2520trunk/builds/238 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: anthony.baxter Build Had Warnings: warnings test sincerely, -The Buildbot From python-checkins at python.org Tue Apr 11 15:14:56 2006 From: python-checkins at python.org (andrew.kuchling) Date: Tue, 11 Apr 2006 15:14:56 +0200 (CEST) Subject: [Python-checkins] r45281 - python/trunk/Doc/whatsnew/whatsnew25.tex Message-ID: <20060411131456.96C331E4009@bag.python.org> Author: andrew.kuchling Date: Tue Apr 11 15:14:56 2006 New Revision: 45281 Modified: python/trunk/Doc/whatsnew/whatsnew25.tex Log: Complete the ElementTree section Modified: python/trunk/Doc/whatsnew/whatsnew25.tex ============================================================================== --- python/trunk/Doc/whatsnew/whatsnew25.tex (original) +++ python/trunk/Doc/whatsnew/whatsnew25.tex Tue Apr 11 15:14:56 2006 @@ -5,6 +5,9 @@ % Fix XXX comments % Distutils upload (PEP 243) % The easy_install stuff +% Access to ASTs with compile() flag +% Stateful codec changes +% ASCII is now default encoding for modules \title{What's New in Python 2.5} \release{0.1} @@ -1132,10 +1135,34 @@ \end{verbatim} Each XML element supports some dictionary-like and some list-like -access methods. Dictionary-like methods are used to access attribute -values, and list-like methods are used to access child nodes. +access methods. Dictionary-like operations are used to access attribute +values, and list-like operations are used to access child nodes. -% XXX finish this +\begin{tableii}{c|l}{code}{Operation}{Result} + \lineii{elem[n]}{Returns n'th child element.} + \lineii{elem[m:n]}{Returns list of m'th through n'th child elements.} + \lineii{len(elem)}{Returns number of child elements.} + \lineii{elem.getchildren()}{Returns list of child elements.} + \lineii{elem.append(elem2)}{Adds \var{elem2} as a child.} + \lineii{elem.insert(index, elem2)}{Inserts \var{elem2} at the specified location.} + \lineii{del elem[n]}{Deletes n'th child element.} + \lineii{elem.keys()}{Returns list of attribute names.} + \lineii{elem.get(name)}{Returns value of attribute \var{name}.} + \lineii{elem.set(name, value)}{Sets new value for attribute \var{name}.} + \lineii{elem.attrib}{Retrieves the dictionary containing attributes.} + \lineii{del elem.attrib[name]}{Deletes attribute \var{name}.} +\end{tableii} + +Comments and processing instructions are also represented as +\class{Element} nodes. To check if a node is a comment or processing +instructions: + +\begin{verbatim} +if elem.tag is ET.Comment: + ... +elif elem.tag is ET.ProcessingInstruction: + ... +\end{verbatim} To generate XML output, you should call the \method{ElementTree.write()} method. Like \function{parse()}, @@ -1156,8 +1183,8 @@ specify a different encoding such as UTF-8 that can handle any Unicode character.) - -% XXX write introduction +This section is only a partial description of the ElementTree interfaces. +Please read the package's official documentation for more details. \begin{seealso} From buildbot at python.org Tue Apr 11 15:48:52 2006 From: buildbot at python.org (buildbot at python.org) Date: Tue, 11 Apr 2006 13:48:52 +0000 Subject: [Python-checkins] buildbot warnings in x86 XP trunk Message-ID: <20060411134853.15E841E4009@bag.python.org> The Buildbot has detected a new failure of x86 XP trunk. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520XP%2520trunk/builds/366 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: andrew.kuchling Build Had Warnings: warnings test sincerely, -The Buildbot From python-checkins at python.org Tue Apr 11 21:12:34 2006 From: python-checkins at python.org (tim.peters) Date: Tue, 11 Apr 2006 21:12:34 +0200 (CEST) Subject: [Python-checkins] r45282 - python/trunk/Objects/object.c Message-ID: <20060411191234.7A11B1E4009@bag.python.org> Author: tim.peters Date: Tue Apr 11 21:12:33 2006 New Revision: 45282 Modified: python/trunk/Objects/object.c Log: _Py_PrintReferenceAddresses,_Py_PrintReferences: interpolate PY_FORMAT_SIZE_T for refcount display instead of casting refcounts to long. I understand that gcc on some boxes delivers nuisance warnings about this, but if any new ones appear because of this they'll get fixed by magic when the others get fixed. Modified: python/trunk/Objects/object.c ============================================================================== --- python/trunk/Objects/object.c (original) +++ python/trunk/Objects/object.c Tue Apr 11 21:12:33 2006 @@ -1900,9 +1900,7 @@ PyObject *op; fprintf(fp, "Remaining objects:\n"); for (op = refchain._ob_next; op != &refchain; op = op->_ob_next) { - /* XXX(twouters) cast refcount to long until %zd is - universally available */ - fprintf(fp, "%p [%ld] ", op, (long)op->ob_refcnt); + fprintf(fp, "%p [%" PY_FORMAT_SIZE_T "d] ", op, op->ob_refcnt); if (PyObject_Print(op, fp, 0) != 0) PyErr_Clear(); putc('\n', fp); @@ -1918,10 +1916,8 @@ PyObject *op; fprintf(fp, "Remaining object addresses:\n"); for (op = refchain._ob_next; op != &refchain; op = op->_ob_next) - /* XXX(twouters) cast refcount to long until %zd is - universally available */ - fprintf(fp, "%p [%ld] %s\n", op, (long)op->ob_refcnt, - op->ob_type->tp_name); + fprintf(fp, "%p [%" PY_FORMAT_SIZE_T "d] %s\n", op, + op->ob_refcnt, op->ob_type->tp_name); } PyObject * From python-checkins at python.org Tue Apr 11 23:42:01 2006 From: python-checkins at python.org (vinay.sajip) Date: Tue, 11 Apr 2006 23:42:01 +0200 (CEST) Subject: [Python-checkins] r45283 - python/trunk/Lib/logging/__init__.py Message-ID: <20060411214201.8B5401E4009@bag.python.org> Author: vinay.sajip Date: Tue Apr 11 23:42:00 2006 New Revision: 45283 Modified: python/trunk/Lib/logging/__init__.py Log: StreamHandler now checks explicitly for None before using sys.stderr as the stream (see SF bug #1463840). Modified: python/trunk/Lib/logging/__init__.py ============================================================================== --- python/trunk/Lib/logging/__init__.py (original) +++ python/trunk/Lib/logging/__init__.py Tue Apr 11 23:42:00 2006 @@ -719,7 +719,7 @@ If strm is not specified, sys.stderr is used. """ Handler.__init__(self) - if not strm: + if strm is None: strm = sys.stderr self.stream = strm self.formatter = None From python-checkins at python.org Wed Apr 12 01:50:38 2006 From: python-checkins at python.org (thomas.wouters) Date: Wed, 12 Apr 2006 01:50:38 +0200 (CEST) Subject: [Python-checkins] r45284 - python/trunk/Objects/intobject.c python/trunk/Objects/longobject.c Message-ID: <20060411235038.8016C1E4003@bag.python.org> Author: thomas.wouters Date: Wed Apr 12 01:50:33 2006 New Revision: 45284 Modified: python/trunk/Objects/intobject.c python/trunk/Objects/longobject.c Log: Fix int() and long() to repr() their argument when formatting the exception, to avoid confusing situations like: >>> int("") ValueError: invalid literal for int(): >>> int("2\n\n2") ValueError: invalid literal for int(): 2 2 Also report the base used, to avoid: ValueError: invalid literal for int(): 2 They now report: >>> int("") ValueError: invalid literal for int() with base 10: '' >>> int("2\n\n2") ValueError: invalid literal for int() with base 10: '2\n\n2' >>> int("2", 2) ValueError: invalid literal for int() with base 2: '2' (Reporting the base could be avoided when base is 10, which is the default, but hrm.) Another effect of these changes is that the errormessage can be longer; before, it was cut off at about 250 characters. Now, it can be up to four times as long, as the unrepr'ed string is cut off at 200 characters, instead. No tests were added or changed, since testing for exact errormsgs is (pardon the pun) somewhat errorprone, and I consider not testing the exact text preferable. The actually changed code is tested frequent enough in the test_builtin test as it is (120 runs for each of ints and longs.) Modified: python/trunk/Objects/intobject.c ============================================================================== --- python/trunk/Objects/intobject.c (original) +++ python/trunk/Objects/intobject.c Wed Apr 12 01:50:33 2006 @@ -335,7 +335,8 @@ { char *end; long x; - char buffer[256]; /* For errors */ + Py_ssize_t slen; + PyObject *sobj, *srepr; if ((base != 0 && base < 2) || base > 36) { PyErr_SetString(PyExc_ValueError, @@ -359,9 +360,18 @@ end++; if (*end != '\0') { bad: - PyOS_snprintf(buffer, sizeof(buffer), - "invalid literal for int(): %.200s", s); - PyErr_SetString(PyExc_ValueError, buffer); + slen = strlen(s) < 200 ? strlen(s) : 200; + sobj = PyString_FromStringAndSize(s, slen); + if (sobj == NULL) + return NULL; + srepr = PyObject_Repr(sobj); + Py_DECREF(sobj); + if (srepr == NULL) + return NULL; + PyErr_Format(PyExc_ValueError, + "invalid literal for int() with base %d: %s", + base, PyString_AS_STRING(srepr)); + Py_DECREF(srepr); return NULL; } else if (errno != 0) Modified: python/trunk/Objects/longobject.c ============================================================================== --- python/trunk/Objects/longobject.c (original) +++ python/trunk/Objects/longobject.c Wed Apr 12 01:50:33 2006 @@ -1400,6 +1400,8 @@ int sign = 1; char *start, *orig_str = str; PyLongObject *z; + PyObject *strobj, *strrepr; + Py_ssize_t slen; if ((base != 0 && base < 2) || base > 36) { PyErr_SetString(PyExc_ValueError, @@ -1465,9 +1467,19 @@ return (PyObject *) z; onError: - PyErr_Format(PyExc_ValueError, - "invalid literal for long(): %.200s", orig_str); Py_XDECREF(z); + slen = strlen(orig_str) < 200 ? strlen(orig_str) : 200; + strobj = PyString_FromStringAndSize(orig_str, slen); + if (strobj == NULL) + return NULL; + strrepr = PyObject_Repr(strobj); + Py_DECREF(strobj); + if (strrepr == NULL) + return NULL; + PyErr_Format(PyExc_ValueError, + "invalid literal for long() with base %d: %s", + base, PyString_AS_STRING(strrepr)); + Py_DECREF(strrepr); return NULL; } From python-checkins at python.org Wed Apr 12 02:06:35 2006 From: python-checkins at python.org (thomas.wouters) Date: Wed, 12 Apr 2006 02:06:35 +0200 (CEST) Subject: [Python-checkins] r45285 - python/trunk/Grammar/Grammar Message-ID: <20060412000635.6A7061E4003@bag.python.org> Author: thomas.wouters Date: Wed Apr 12 02:06:34 2006 New Revision: 45285 Modified: python/trunk/Grammar/Grammar Log: Fix SF bug #1466641: multiple adjacent 'if's in listcomps and genexps, as in [x for x in it if x if x], were broken for no good reason by the PEP 308 patch. Modified: python/trunk/Grammar/Grammar ============================================================================== --- python/trunk/Grammar/Grammar (original) +++ python/trunk/Grammar/Grammar Wed Apr 12 02:06:34 2006 @@ -131,11 +131,11 @@ list_iter: list_for | list_if list_for: 'for' exprlist 'in' testlist_safe [list_iter] -list_if: 'if' test [list_iter] +list_if: 'if' old_test [list_iter] gen_iter: gen_for | gen_if gen_for: 'for' exprlist 'in' or_test [gen_iter] -gen_if: 'if' test [gen_iter] +gen_if: 'if' old_test [gen_iter] testlist1: test (',' test)* From python-checkins at python.org Wed Apr 12 02:07:59 2006 From: python-checkins at python.org (thomas.wouters) Date: Wed, 12 Apr 2006 02:07:59 +0200 (CEST) Subject: [Python-checkins] r45286 - in python/trunk: Lib/test/test_grammar.py Python/graminit.c Message-ID: <20060412000759.E3C0D1E4003@bag.python.org> Author: thomas.wouters Date: Wed Apr 12 02:07:59 2006 New Revision: 45286 Modified: python/trunk/Lib/test/test_grammar.py python/trunk/Python/graminit.c Log: Part two of the fix for SF bug #1466641: Regenerate graminit.c and add test for the bogus failure. Modified: python/trunk/Lib/test/test_grammar.py ============================================================================== --- python/trunk/Lib/test/test_grammar.py (original) +++ python/trunk/Lib/test/test_grammar.py Wed Apr 12 02:07:59 2006 @@ -815,6 +815,11 @@ x = 5; t = True; verify([(i,j) for i in range(10) for j in range(5)] == list(g)) +# Grammar allows multiple adjacent 'if's in listcomps and genexps, +# even though it's silly. Make sure it works (ifelse broke this.) +verify([ x for x in range(10) if x % 2 if x % 3 ], [1, 5, 7]) +verify((x for x in range(10) if x % 2 if x % 3), [1, 5, 7]) + # Test ifelse expressions in various cases def _checkeval(msg, ret): "helper to check that evaluation of expressions is done correctly" Modified: python/trunk/Python/graminit.c ============================================================================== --- python/trunk/Python/graminit.c (original) +++ python/trunk/Python/graminit.c Wed Apr 12 02:07:59 2006 @@ -1678,7 +1678,7 @@ {91, 1}, }; static arc arcs_77_1[1] = { - {26, 2}, + {105, 2}, }; static arc arcs_77_2[2] = { {162, 3}, @@ -1735,7 +1735,7 @@ {91, 1}, }; static arc arcs_80_1[1] = { - {26, 2}, + {105, 2}, }; static arc arcs_80_2[2] = { {164, 3}, From buildbot at python.org Wed Apr 12 02:29:15 2006 From: buildbot at python.org (buildbot at python.org) Date: Wed, 12 Apr 2006 00:29:15 +0000 Subject: [Python-checkins] buildbot warnings in amd64 gentoo trunk Message-ID: <20060412002915.5E2C71E4003@bag.python.org> The Buildbot has detected a new failure of amd64 gentoo trunk. Full details are available at: http://www.python.org/dev/buildbot/all/amd64%2520gentoo%2520trunk/builds/414 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: thomas.wouters Build Had Warnings: warnings test sincerely, -The Buildbot From buildbot at python.org Wed Apr 12 02:29:29 2006 From: buildbot at python.org (buildbot at python.org) Date: Wed, 12 Apr 2006 00:29:29 +0000 Subject: [Python-checkins] buildbot warnings in x86 gentoo trunk Message-ID: <20060412002929.2CBB21E4003@bag.python.org> The Buildbot has detected a new failure of x86 gentoo trunk. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520gentoo%2520trunk/builds/399 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: thomas.wouters Build Had Warnings: warnings test sincerely, -The Buildbot From buildbot at python.org Wed Apr 12 02:41:27 2006 From: buildbot at python.org (buildbot at python.org) Date: Wed, 12 Apr 2006 00:41:27 +0000 Subject: [Python-checkins] buildbot warnings in x86 W2k trunk Message-ID: <20060412004127.399D71E4003@bag.python.org> The Buildbot has detected a new failure of x86 W2k trunk. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520W2k%2520trunk/builds/394 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: thomas.wouters Build Had Warnings: warnings test sincerely, -The Buildbot From buildbot at python.org Wed Apr 12 02:42:31 2006 From: buildbot at python.org (buildbot at python.org) Date: Wed, 12 Apr 2006 00:42:31 +0000 Subject: [Python-checkins] buildbot warnings in x86 OpenBSD trunk Message-ID: <20060412004232.1D8E71E4003@bag.python.org> The Buildbot has detected a new failure of x86 OpenBSD trunk. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520OpenBSD%2520trunk/builds/244 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: thomas.wouters Build Had Warnings: warnings test sincerely, -The Buildbot From python-checkins at python.org Wed Apr 12 02:43:10 2006 From: python-checkins at python.org (anthony.baxter) Date: Wed, 12 Apr 2006 02:43:10 +0200 (CEST) Subject: [Python-checkins] r45287 - python/trunk/Python/compile.c Message-ID: <20060412004310.B05BA1E4003@bag.python.org> Author: anthony.baxter Date: Wed Apr 12 02:43:09 2006 New Revision: 45287 Modified: python/trunk/Python/compile.c Log: per Jeremy's email, remove the _WITH_CAST versions of macros. g++ still has errors from the casts of asdl_seq_GET to cmpop_ty, but otherwise it's C++ clean. Modified: python/trunk/Python/compile.c ============================================================================== --- python/trunk/Python/compile.c (original) +++ python/trunk/Python/compile.c Wed Apr 12 02:43:09 2006 @@ -1717,17 +1717,7 @@ int _i; \ asdl_seq *seq = (SEQ); /* avoid variable capture */ \ for (_i = 0; _i < asdl_seq_LEN(seq); _i++) { \ - TYPE ## _ty elt = asdl_seq_GET(seq, _i); \ - if (!compiler_visit_ ## TYPE((C), elt)) \ - return 0; \ - } \ -} - -#define VISIT_SEQ_WITH_CAST(C, TYPE, SEQ, CAST) { \ - int _i; \ - asdl_seq *seq = (SEQ); /* avoid variable capture */ \ - for (_i = 0; _i < asdl_seq_LEN(seq); _i++) { \ - TYPE ## _ty elt = (CAST)asdl_seq_GET(seq, _i); \ + TYPE ## _ty elt = (TYPE ## _ty)asdl_seq_GET(seq, _i); \ if (!compiler_visit_ ## TYPE((C), elt)) \ return 0; \ } \ @@ -1737,19 +1727,7 @@ int _i; \ asdl_seq *seq = (SEQ); /* avoid variable capture */ \ for (_i = 0; _i < asdl_seq_LEN(seq); _i++) { \ - TYPE ## _ty elt = asdl_seq_GET(seq, _i); \ - if (!compiler_visit_ ## TYPE((C), elt)) { \ - compiler_exit_scope(c); \ - return 0; \ - } \ - } \ -} - -#define VISIT_SEQ_IN_SCOPE_WITH_CAST(C, TYPE, SEQ, CAST) { \ - int _i; \ - asdl_seq *seq = (SEQ); /* avoid variable capture */ \ - for (_i = 0; _i < asdl_seq_LEN(seq); _i++) { \ - TYPE ## _ty elt = (CAST)asdl_seq_GET(seq, _i); \ + TYPE ## _ty elt = (TYPE ## _ty)asdl_seq_GET(seq, _i); \ if (!compiler_visit_ ## TYPE((C), elt)) { \ compiler_exit_scope(c); \ return 0; \ @@ -1809,8 +1787,8 @@ break; case Interactive_kind: c->c_interactive = 1; - VISIT_SEQ_IN_SCOPE_WITH_CAST(c, stmt, - mod->v.Interactive.body, stmt_ty); + VISIT_SEQ_IN_SCOPE(c, stmt, + mod->v.Interactive.body); break; case Expression_kind: VISIT_IN_SCOPE(c, expr, mod->v.Expression.body); @@ -1971,7 +1949,7 @@ if (!compiler_decorators(c, decos)) return 0; if (args->defaults) - VISIT_SEQ_WITH_CAST(c, expr, args->defaults, expr_ty); + VISIT_SEQ(c, expr, args->defaults); if (!compiler_enter_scope(c, s->v.FunctionDef.name, (void *)s, s->lineno)) return 0; @@ -2024,7 +2002,7 @@ /* push the tuple of base classes on the stack */ n = asdl_seq_LEN(s->v.ClassDef.bases); if (n > 0) - VISIT_SEQ_WITH_CAST(c, expr, s->v.ClassDef.bases, expr_ty); + VISIT_SEQ(c, expr, s->v.ClassDef.bases); ADDOP_I(c, BUILD_TUPLE, n); if (!compiler_enter_scope(c, s->v.ClassDef.name, (void *)s, s->lineno)) @@ -2108,7 +2086,7 @@ } if (args->defaults) - VISIT_SEQ_WITH_CAST(c, expr, args->defaults, expr_ty); + VISIT_SEQ(c, expr, args->defaults); if (!compiler_enter_scope(c, name, (void *)e, e->lineno)) return 0; @@ -2181,12 +2159,12 @@ VISIT(c, expr, s->v.If.test); ADDOP_JREL(c, JUMP_IF_FALSE, next); ADDOP(c, POP_TOP); - VISIT_SEQ_WITH_CAST(c, stmt, s->v.If.body, stmt_ty); + VISIT_SEQ(c, stmt, s->v.If.body); ADDOP_JREL(c, JUMP_FORWARD, end); compiler_use_next_block(c, next); ADDOP(c, POP_TOP); if (s->v.If.orelse) - VISIT_SEQ_WITH_CAST(c, stmt, s->v.If.orelse, stmt_ty); + VISIT_SEQ(c, stmt, s->v.If.orelse); compiler_use_next_block(c, end); return 1; } @@ -2209,12 +2187,12 @@ compiler_use_next_block(c, start); ADDOP_JREL(c, FOR_ITER, cleanup); VISIT(c, expr, s->v.For.target); - VISIT_SEQ_WITH_CAST(c, stmt, s->v.For.body, stmt_ty); + VISIT_SEQ(c, stmt, s->v.For.body); ADDOP_JABS(c, JUMP_ABSOLUTE, start); compiler_use_next_block(c, cleanup); ADDOP(c, POP_BLOCK); compiler_pop_fblock(c, LOOP, start); - VISIT_SEQ_WITH_CAST(c, stmt, s->v.For.orelse, stmt_ty); + VISIT_SEQ(c, stmt, s->v.For.orelse); compiler_use_next_block(c, end); return 1; } @@ -2253,7 +2231,7 @@ ADDOP_JREL(c, JUMP_IF_FALSE, anchor); ADDOP(c, POP_TOP); } - VISIT_SEQ_WITH_CAST(c, stmt, s->v.While.body, stmt_ty); + VISIT_SEQ(c, stmt, s->v.While.body); ADDOP_JABS(c, JUMP_ABSOLUTE, loop); /* XXX should the two POP instructions be in a separate block @@ -2267,7 +2245,7 @@ } compiler_pop_fblock(c, LOOP, loop); if (orelse != NULL) /* what if orelse is just pass? */ - VISIT_SEQ_WITH_CAST(c, stmt, s->v.While.orelse, stmt_ty); + VISIT_SEQ(c, stmt, s->v.While.orelse); compiler_use_next_block(c, end); return 1; @@ -2348,7 +2326,7 @@ compiler_use_next_block(c, body); if (!compiler_push_fblock(c, FINALLY_TRY, body)) return 0; - VISIT_SEQ_WITH_CAST(c, stmt, s->v.TryFinally.body, stmt_ty); + VISIT_SEQ(c, stmt, s->v.TryFinally.body); ADDOP(c, POP_BLOCK); compiler_pop_fblock(c, FINALLY_TRY, body); @@ -2356,7 +2334,7 @@ compiler_use_next_block(c, end); if (!compiler_push_fblock(c, FINALLY_END, end)) return 0; - VISIT_SEQ_WITH_CAST(c, stmt, s->v.TryFinally.finalbody, stmt_ty); + VISIT_SEQ(c, stmt, s->v.TryFinally.finalbody); ADDOP(c, END_FINALLY); compiler_pop_fblock(c, FINALLY_END, end); @@ -2413,7 +2391,7 @@ compiler_use_next_block(c, body); if (!compiler_push_fblock(c, EXCEPT, body)) return 0; - VISIT_SEQ_WITH_CAST(c, stmt, s->v.TryExcept.body, stmt_ty); + VISIT_SEQ(c, stmt, s->v.TryExcept.body); ADDOP(c, POP_BLOCK); compiler_pop_fblock(c, EXCEPT, body); ADDOP_JREL(c, JUMP_FORWARD, orelse); @@ -2444,7 +2422,7 @@ ADDOP(c, POP_TOP); } ADDOP(c, POP_TOP); - VISIT_SEQ_WITH_CAST(c, stmt, handler->body, stmt_ty); + VISIT_SEQ(c, stmt, handler->body); ADDOP_JREL(c, JUMP_FORWARD, end); compiler_use_next_block(c, except); if (handler->type) @@ -2452,7 +2430,7 @@ } ADDOP(c, END_FINALLY); compiler_use_next_block(c, orelse); - VISIT_SEQ_WITH_CAST(c, stmt, s->v.TryExcept.orelse, stmt_ty); + VISIT_SEQ(c, stmt, s->v.TryExcept.orelse); compiler_use_next_block(c, end); return 1; } @@ -2672,7 +2650,7 @@ ADDOP(c, RETURN_VALUE); break; case Delete_kind: - VISIT_SEQ_WITH_CAST(c, expr, s->v.Delete.targets, expr_ty) + VISIT_SEQ(c, expr, s->v.Delete.targets) break; case Assign_kind: n = asdl_seq_LEN(s->v.Assign.targets); @@ -3042,7 +3020,7 @@ if (e->v.List.ctx == Store) { ADDOP_I(c, UNPACK_SEQUENCE, n); } - VISIT_SEQ_WITH_CAST(c, expr, e->v.List.elts, expr_ty); + VISIT_SEQ(c, expr, e->v.List.elts); if (e->v.List.ctx == Load) { ADDOP_I(c, BUILD_LIST, n); } @@ -3056,7 +3034,7 @@ if (e->v.Tuple.ctx == Store) { ADDOP_I(c, UNPACK_SEQUENCE, n); } - VISIT_SEQ_WITH_CAST(c, expr, e->v.Tuple.elts, expr_ty); + VISIT_SEQ(c, expr, e->v.Tuple.elts); if (e->v.Tuple.ctx == Load) { ADDOP_I(c, BUILD_TUPLE, n); } @@ -3117,9 +3095,9 @@ VISIT(c, expr, e->v.Call.func); n = asdl_seq_LEN(e->v.Call.args); - VISIT_SEQ_WITH_CAST(c, expr, e->v.Call.args, expr_ty); + VISIT_SEQ(c, expr, e->v.Call.args); if (e->v.Call.keywords) { - VISIT_SEQ_WITH_CAST(c, keyword, e->v.Call.keywords, keyword_ty); + VISIT_SEQ(c, keyword, e->v.Call.keywords); n |= asdl_seq_LEN(e->v.Call.keywords) << 8; } if (e->v.Call.starargs) { @@ -3500,7 +3478,7 @@ } /* BLOCK code */ - VISIT_SEQ_WITH_CAST(c, stmt, s->v.With.body, stmt_ty); + VISIT_SEQ(c, stmt, s->v.With.body); /* End of try block; start the finally block */ ADDOP(c, POP_BLOCK); From buildbot at python.org Wed Apr 12 02:53:27 2006 From: buildbot at python.org (buildbot at python.org) Date: Wed, 12 Apr 2006 00:53:27 +0000 Subject: [Python-checkins] buildbot warnings in x86 XP trunk Message-ID: <20060412005327.671461E4003@bag.python.org> The Buildbot has detected a new failure of x86 XP trunk. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520XP%2520trunk/builds/370 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: thomas.wouters Build Had Warnings: warnings test sincerely, -The Buildbot From buildbot at python.org Wed Apr 12 03:01:46 2006 From: buildbot at python.org (buildbot at python.org) Date: Wed, 12 Apr 2006 01:01:46 +0000 Subject: [Python-checkins] buildbot warnings in sparc solaris10 gcc trunk Message-ID: <20060412010146.A11441E4009@bag.python.org> The Buildbot has detected a new failure of sparc solaris10 gcc trunk. Full details are available at: http://www.python.org/dev/buildbot/all/sparc%2520solaris10%2520gcc%2520trunk/builds/360 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: thomas.wouters Build Had Warnings: warnings test sincerely, -The Buildbot From buildbot at python.org Wed Apr 12 03:14:04 2006 From: buildbot at python.org (buildbot at python.org) Date: Wed, 12 Apr 2006 01:14:04 +0000 Subject: [Python-checkins] buildbot warnings in ppc Debian unstable trunk Message-ID: <20060412011404.9BC951E4003@bag.python.org> The Buildbot has detected a new failure of ppc Debian unstable trunk. Full details are available at: http://www.python.org/dev/buildbot/all/ppc%2520Debian%2520unstable%2520trunk/builds/82 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: thomas.wouters Build Had Warnings: warnings test sincerely, -The Buildbot From buildbot at python.org Wed Apr 12 03:29:23 2006 From: buildbot at python.org (buildbot at python.org) Date: Wed, 12 Apr 2006 01:29:23 +0000 Subject: [Python-checkins] buildbot warnings in alpha Tru64 5.1 trunk Message-ID: <20060412012923.C8E071E4003@bag.python.org> The Buildbot has detected a new failure of alpha Tru64 5.1 trunk. Full details are available at: http://www.python.org/dev/buildbot/all/alpha%2520Tru64%25205.1%2520trunk/builds/177 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: thomas.wouters Build Had Warnings: warnings test sincerely, -The Buildbot From buildbot at python.org Wed Apr 12 03:45:06 2006 From: buildbot at python.org (buildbot at python.org) Date: Wed, 12 Apr 2006 01:45:06 +0000 Subject: [Python-checkins] buildbot warnings in x86 Ubuntu dapper (icc) trunk Message-ID: <20060412014506.581EE1E4017@bag.python.org> The Buildbot has detected a new failure of x86 Ubuntu dapper (icc) trunk. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520Ubuntu%2520dapper%2520%2528icc%2529%2520trunk/builds/90 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: anthony.baxter,thomas.wouters Build Had Warnings: warnings test sincerely, -The Buildbot From buildbot at python.org Wed Apr 12 05:00:40 2006 From: buildbot at python.org (buildbot at python.org) Date: Wed, 12 Apr 2006 03:00:40 +0000 Subject: [Python-checkins] buildbot warnings in x86 XP-2 trunk Message-ID: <20060412030040.51C491E4003@bag.python.org> The Buildbot has detected a new failure of x86 XP-2 trunk. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520XP-2%2520trunk/builds/234 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: anthony.baxter,thomas.wouters Build Had Warnings: warnings test sincerely, -The Buildbot From python-checkins at python.org Wed Apr 12 06:00:50 2006 From: python-checkins at python.org (anthony.baxter) Date: Wed, 12 Apr 2006 06:00:50 +0200 (CEST) Subject: [Python-checkins] r45288 - python/trunk/Python/symtable.c Message-ID: <20060412040050.D19381E4003@bag.python.org> Author: anthony.baxter Date: Wed Apr 12 06:00:50 2006 New Revision: 45288 Modified: python/trunk/Python/symtable.c Log: Make symtable.c safe for C++ compilers. Changed macros in the same way as compile.c to add a cast. Modified: python/trunk/Python/symtable.c ============================================================================== --- python/trunk/Python/symtable.c (original) +++ python/trunk/Python/symtable.c Wed Apr 12 06:00:50 2006 @@ -227,7 +227,8 @@ case Module_kind: seq = mod->v.Module.body; for (i = 0; i < asdl_seq_LEN(seq); i++) - if (!symtable_visit_stmt(st, asdl_seq_GET(seq, i))) + if (!symtable_visit_stmt(st, + (stmt_ty)asdl_seq_GET(seq, i))) goto error; break; case Expression_kind: @@ -237,7 +238,8 @@ case Interactive_kind: seq = mod->v.Interactive.body; for (i = 0; i < asdl_seq_LEN(seq); i++) - if (!symtable_visit_stmt(st, asdl_seq_GET(seq, i))) + if (!symtable_visit_stmt(st, + (stmt_ty)asdl_seq_GET(seq, i))) goto error; break; case Suite_kind: @@ -506,7 +508,7 @@ */ static int update_symbols(PyObject *symbols, PyObject *scope, - PyObject *bound, PyObject *free, int class) + PyObject *bound, PyObject *free, int classflag) { PyObject *name, *v, *u, *w, *free_value = NULL; Py_ssize_t pos = 0; @@ -541,7 +543,7 @@ the class that has the same name as a local or global in the class scope. */ - if (class && + if (classflag && PyInt_AS_LONG(o) & (DEF_BOUND | DEF_GLOBAL)) { long i = PyInt_AS_LONG(o) | DEF_FREE_CLASS; o = PyInt_FromLong(i); @@ -851,7 +853,7 @@ int i; \ asdl_seq *seq = (SEQ); /* avoid variable capture */ \ for (i = 0; i < asdl_seq_LEN(seq); i++) { \ - TYPE ## _ty elt = asdl_seq_GET(seq, i); \ + TYPE ## _ty elt = (TYPE ## _ty)asdl_seq_GET(seq, i); \ if (!symtable_visit_ ## TYPE((ST), elt)) \ return 0; \ } \ @@ -861,7 +863,7 @@ int i; \ asdl_seq *seq = (SEQ); /* avoid variable capture */ \ for (i = 0; i < asdl_seq_LEN(seq); i++) { \ - TYPE ## _ty elt = asdl_seq_GET(seq, i); \ + TYPE ## _ty elt = (TYPE ## _ty)asdl_seq_GET(seq, i); \ if (!symtable_visit_ ## TYPE((ST), elt)) { \ symtable_exit_block((ST), (S)); \ return 0; \ @@ -873,7 +875,7 @@ int i; \ asdl_seq *seq = (SEQ); /* avoid variable capture */ \ for (i = (START); i < asdl_seq_LEN(seq); i++) { \ - TYPE ## _ty elt = asdl_seq_GET(seq, i); \ + TYPE ## _ty elt = (TYPE ## _ty)asdl_seq_GET(seq, i); \ if (!symtable_visit_ ## TYPE((ST), elt)) \ return 0; \ } \ @@ -883,7 +885,7 @@ int i; \ asdl_seq *seq = (SEQ); /* avoid variable capture */ \ for (i = (START); i < asdl_seq_LEN(seq); i++) { \ - TYPE ## _ty elt = asdl_seq_GET(seq, i); \ + TYPE ## _ty elt = (TYPE ## _ty)asdl_seq_GET(seq, i); \ if (!symtable_visit_ ## TYPE((ST), elt)) { \ symtable_exit_block((ST), (S)); \ return 0; \ @@ -1036,7 +1038,7 @@ int i; asdl_seq *seq = s->v.Global.names; for (i = 0; i < asdl_seq_LEN(seq); i++) { - identifier name = asdl_seq_GET(seq, i); + identifier name = (identifier)asdl_seq_GET(seq, i); char *c_name = PyString_AS_STRING(name); long cur = symtable_lookup(st, name); if (cur < 0) @@ -1200,7 +1202,7 @@ /* go through all the toplevel arguments first */ for (i = 0; i < asdl_seq_LEN(args); i++) { - expr_ty arg = asdl_seq_GET(args, i); + expr_ty arg = (expr_ty)asdl_seq_GET(args, i); if (arg->kind == Name_kind) { assert(arg->v.Name.ctx == Param || (arg->v.Name.ctx == Store && !toplevel)); @@ -1236,7 +1238,7 @@ { int i; for (i = 0; i < asdl_seq_LEN(args); i++) { - expr_ty arg = asdl_seq_GET(args, i); + expr_ty arg = (expr_ty)asdl_seq_GET(args, i); if (arg->kind == Tuple_kind && !symtable_visit_params(st, arg->v.Tuple.elts, 0)) return 0; From python-checkins at python.org Wed Apr 12 06:08:46 2006 From: python-checkins at python.org (anthony.baxter) Date: Wed, 12 Apr 2006 06:08:46 +0200 (CEST) Subject: [Python-checkins] r45289 - python/trunk/Modules/threadmodule.c Message-ID: <20060412040846.842AE1E4003@bag.python.org> Author: anthony.baxter Date: Wed Apr 12 06:08:46 2006 New Revision: 45289 Modified: python/trunk/Modules/threadmodule.c Log: remove forward declarations, move constructor functions. makes code C++ safe. Modified: python/trunk/Modules/threadmodule.c ============================================================================== --- python/trunk/Modules/threadmodule.c (original) +++ python/trunk/Modules/threadmodule.c Wed Apr 12 06:08:46 2006 @@ -22,24 +22,6 @@ PyThread_type_lock lock_lock; } lockobject; -static PyTypeObject Locktype; - -static lockobject * -newlockobject(void) -{ - lockobject *self; - self = PyObject_New(lockobject, &Locktype); - if (self == NULL) - return NULL; - self->lock_lock = PyThread_allocate_lock(); - if (self->lock_lock == NULL) { - PyObject_Del(self); - self = NULL; - PyErr_SetString(ThreadError, "can't allocate lock"); - } - return self; -} - static void lock_dealloc(lockobject *self) { @@ -166,6 +148,22 @@ 0, /*tp_repr*/ }; +static lockobject * +newlockobject(void) +{ + lockobject *self; + self = PyObject_New(lockobject, &Locktype); + if (self == NULL) + return NULL; + self->lock_lock = PyThread_allocate_lock(); + if (self->lock_lock == NULL) { + PyObject_Del(self); + self = NULL; + PyErr_SetString(ThreadError, "can't allocate lock"); + } + return self; +} + /* Thread-local objects */ #include "structmember.h" @@ -178,8 +176,6 @@ PyObject *dict; } localobject; -static PyTypeObject localtype; - static PyObject * local_new(PyTypeObject *type, PyObject *args, PyObject *kw) { @@ -315,29 +311,6 @@ return ldict; } -static PyObject * -local_getattro(localobject *self, PyObject *name) -{ - PyObject *ldict, *value; - - ldict = _ldict(self); - if (ldict == NULL) - return NULL; - - if (self->ob_type != &localtype) - /* use generic lookup for subtypes */ - return PyObject_GenericGetAttr((PyObject *)self, name); - - /* Optimization: just look in dict ourselves */ - value = PyDict_GetItem(ldict, name); - if (value == NULL) - /* Fall back on generic to get __class__ and __dict__ */ - return PyObject_GenericGetAttr((PyObject *)self, name); - - Py_INCREF(value); - return value; -} - static int local_setattro(localobject *self, PyObject *name, PyObject *v) { @@ -368,6 +341,8 @@ {NULL} /* Sentinel */ }; +static PyObject *local_getattro(localobject *, PyObject *); + static PyTypeObject localtype = { PyObject_HEAD_INIT(NULL) /* ob_size */ 0, @@ -412,6 +387,28 @@ /* tp_is_gc */ 0, /* For PyObject_IS_GC */ }; +static PyObject * +local_getattro(localobject *self, PyObject *name) +{ + PyObject *ldict, *value; + + ldict = _ldict(self); + if (ldict == NULL) + return NULL; + + if (self->ob_type != &localtype) + /* use generic lookup for subtypes */ + return PyObject_GenericGetAttr((PyObject *)self, name); + + /* Optimization: just look in dict ourselves */ + value = PyDict_GetItem(ldict, name); + if (value == NULL) + /* Fall back on generic to get __class__ and __dict__ */ + return PyObject_GenericGetAttr((PyObject *)self, name); + + Py_INCREF(value); + return value; +} /* Module functions */ @@ -560,6 +557,8 @@ } #endif +static lockobject *newlockobject(void); + static PyObject * thread_PyThread_allocate_lock(PyObject *self) { From python-checkins at python.org Wed Apr 12 06:26:16 2006 From: python-checkins at python.org (anthony.baxter) Date: Wed, 12 Apr 2006 06:26:16 +0200 (CEST) Subject: [Python-checkins] r45290 - python/trunk/Modules/_sre.c Message-ID: <20060412042616.6BE851E4003@bag.python.org> Author: anthony.baxter Date: Wed Apr 12 06:26:11 2006 New Revision: 45290 Modified: python/trunk/Modules/_sre.c Log: Move constructors, add some casts to make C++ compiler happy. Still a problem with the getstring() results in pattern_subx. Will come back to that. Modified: python/trunk/Modules/_sre.c ============================================================================== --- python/trunk/Modules/_sre.c (original) +++ python/trunk/Modules/_sre.c Wed Apr 12 06:26:11 2006 @@ -275,7 +275,7 @@ data_stack_dealloc(state); return SRE_ERROR_MEMORY; } - state->data_stack = stack; + state->data_stack = (char *)stack; state->data_stack_size = cursize; } return 0; @@ -335,7 +335,7 @@ { /* check if pointer is at given position */ - int this, that; + int thisp, thatp; switch (at) { @@ -362,57 +362,57 @@ case SRE_AT_BOUNDARY: if (state->beginning == state->end) return 0; - that = ((void*) ptr > state->beginning) ? + thatp = ((void*) ptr > state->beginning) ? SRE_IS_WORD((int) ptr[-1]) : 0; - this = ((void*) ptr < state->end) ? + thisp = ((void*) ptr < state->end) ? SRE_IS_WORD((int) ptr[0]) : 0; - return this != that; + return thisp != thatp; case SRE_AT_NON_BOUNDARY: if (state->beginning == state->end) return 0; - that = ((void*) ptr > state->beginning) ? + thatp = ((void*) ptr > state->beginning) ? SRE_IS_WORD((int) ptr[-1]) : 0; - this = ((void*) ptr < state->end) ? + thisp = ((void*) ptr < state->end) ? SRE_IS_WORD((int) ptr[0]) : 0; - return this == that; + return thisp == thatp; case SRE_AT_LOC_BOUNDARY: if (state->beginning == state->end) return 0; - that = ((void*) ptr > state->beginning) ? + thatp = ((void*) ptr > state->beginning) ? SRE_LOC_IS_WORD((int) ptr[-1]) : 0; - this = ((void*) ptr < state->end) ? + thisp = ((void*) ptr < state->end) ? SRE_LOC_IS_WORD((int) ptr[0]) : 0; - return this != that; + return thisp != thatp; case SRE_AT_LOC_NON_BOUNDARY: if (state->beginning == state->end) return 0; - that = ((void*) ptr > state->beginning) ? + thatp = ((void*) ptr > state->beginning) ? SRE_LOC_IS_WORD((int) ptr[-1]) : 0; - this = ((void*) ptr < state->end) ? + thisp = ((void*) ptr < state->end) ? SRE_LOC_IS_WORD((int) ptr[0]) : 0; - return this == that; + return thisp == thatp; #if defined(HAVE_UNICODE) case SRE_AT_UNI_BOUNDARY: if (state->beginning == state->end) return 0; - that = ((void*) ptr > state->beginning) ? + thatp = ((void*) ptr > state->beginning) ? SRE_UNI_IS_WORD((int) ptr[-1]) : 0; - this = ((void*) ptr < state->end) ? + thisp = ((void*) ptr < state->end) ? SRE_UNI_IS_WORD((int) ptr[0]) : 0; - return this != that; + return thisp != thatp; case SRE_AT_UNI_NON_BOUNDARY: if (state->beginning == state->end) return 0; - that = ((void*) ptr > state->beginning) ? + thatp = ((void*) ptr > state->beginning) ? SRE_UNI_IS_WORD((int) ptr[-1]) : 0; - this = ((void*) ptr < state->end) ? + thisp = ((void*) ptr < state->end) ? SRE_UNI_IS_WORD((int) ptr[0]) : 0; - return this == that; + return thisp == thatp; #endif } @@ -516,8 +516,8 @@ SRE_COUNT(SRE_STATE* state, SRE_CODE* pattern, int maxcount) { SRE_CODE chr; - SRE_CHAR* ptr = state->ptr; - SRE_CHAR* end = state->end; + SRE_CHAR* ptr = (SRE_CHAR *)state->ptr; + SRE_CHAR* end = (SRE_CHAR *)state->end; int i; /* adjust end */ @@ -803,7 +803,7 @@ LOCAL(int) SRE_MATCH(SRE_STATE* state, SRE_CODE* pattern) { - SRE_CHAR* end = state->end; + SRE_CHAR* end = (SRE_CHAR *)state->end; int alloc_pos, ctx_pos = -1; int i, ret = 0; int jump; @@ -821,7 +821,7 @@ entrance: - ctx->ptr = state->ptr; + ctx->ptr = (SRE_CHAR *)state->ptr; if (ctx->pattern[0] == SRE_OP_INFO) { /* optimization info block */ @@ -1477,8 +1477,8 @@ LOCAL(int) SRE_SEARCH(SRE_STATE* state, SRE_CODE* pattern) { - SRE_CHAR* ptr = state->start; - SRE_CHAR* end = state->end; + SRE_CHAR* ptr = (SRE_CHAR *)state->start; + SRE_CHAR* end = (SRE_CHAR *)state->end; int status = 0; int prefix_len = 0; int prefix_skip = 0; @@ -1524,7 +1524,7 @@ /* pattern starts with a known prefix. use the overlap table to skip forward as fast as we possibly can */ int i = 0; - end = state->end; + end = (SRE_CHAR *)state->end; while (ptr < end) { for (;;) { if ((SRE_CODE) ptr[0] != prefix[i]) { @@ -1559,7 +1559,7 @@ /* pattern starts with a literal character. this is used for short prefixes, and if fast search is disabled */ SRE_CODE chr = pattern[1]; - end = state->end; + end = (SRE_CHAR *)state->end; for (;;) { while (ptr < end && (SRE_CODE) ptr[0] != chr) ptr++; @@ -1576,7 +1576,7 @@ } } else if (charset) { /* pattern starts with a character from a known set */ - end = state->end; + end = (SRE_CHAR *)state->end; for (;;) { while (ptr < end && !SRE_CHARSET(charset, ptr[0])) ptr++; @@ -1619,72 +1619,8 @@ /* factories and destructors */ /* see sre.h for object declarations */ - -static PyTypeObject Pattern_Type; -static PyTypeObject Match_Type; -static PyTypeObject Scanner_Type; - -static PyObject * -_compile(PyObject* self_, PyObject* args) -{ - /* "compile" pattern descriptor to pattern object */ - - PatternObject* self; - int i, n; - - PyObject* pattern; - int flags = 0; - PyObject* code; - int groups = 0; - PyObject* groupindex = NULL; - PyObject* indexgroup = NULL; - if (!PyArg_ParseTuple(args, "OiO!|iOO", &pattern, &flags, - &PyList_Type, &code, &groups, - &groupindex, &indexgroup)) - return NULL; - - n = PyList_GET_SIZE(code); - - self = PyObject_NEW_VAR(PatternObject, &Pattern_Type, n); - if (!self) - return NULL; - - self->codesize = n; - - for (i = 0; i < n; i++) { - PyObject *o = PyList_GET_ITEM(code, i); - unsigned long value = PyInt_Check(o) ? (unsigned long)PyInt_AsLong(o) - : PyLong_AsUnsignedLong(o); - self->code[i] = (SRE_CODE) value; - if ((unsigned long) self->code[i] != value) { - PyErr_SetString(PyExc_OverflowError, - "regular expression code size limit exceeded"); - break; - } - } - - if (PyErr_Occurred()) { - PyObject_DEL(self); - return NULL; - } - - Py_INCREF(pattern); - self->pattern = pattern; - - self->flags = flags; - - self->groups = groups; - - Py_XINCREF(groupindex); - self->groupindex = groupindex; - - Py_XINCREF(indexgroup); - self->indexgroup = indexgroup; - - self->weakreflist = NULL; - - return (PyObject*) self; -} +static PyObject*pattern_new_match(PatternObject*, SRE_STATE*, int); +static PyObject*pattern_scanner(PatternObject*, PyObject*); static PyObject * sre_codesize(PyObject* self, PyObject* args) @@ -1900,98 +1836,6 @@ } } -static PyObject* -pattern_new_match(PatternObject* pattern, SRE_STATE* state, int status) -{ - /* create match object (from state object) */ - - MatchObject* match; - int i, j; - char* base; - int n; - - if (status > 0) { - - /* create match object (with room for extra group marks) */ - match = PyObject_NEW_VAR(MatchObject, &Match_Type, - 2*(pattern->groups+1)); - if (!match) - return NULL; - - Py_INCREF(pattern); - match->pattern = pattern; - - Py_INCREF(state->string); - match->string = state->string; - - match->regs = NULL; - match->groups = pattern->groups+1; - - /* fill in group slices */ - - base = (char*) state->beginning; - n = state->charsize; - - match->mark[0] = ((char*) state->start - base) / n; - match->mark[1] = ((char*) state->ptr - base) / n; - - for (i = j = 0; i < pattern->groups; i++, j+=2) - if (j+1 <= state->lastmark && state->mark[j] && state->mark[j+1]) { - match->mark[j+2] = ((char*) state->mark[j] - base) / n; - match->mark[j+3] = ((char*) state->mark[j+1] - base) / n; - } else - match->mark[j+2] = match->mark[j+3] = -1; /* undefined */ - - match->pos = state->pos; - match->endpos = state->endpos; - - match->lastindex = state->lastindex; - - return (PyObject*) match; - - } else if (status == 0) { - - /* no match */ - Py_INCREF(Py_None); - return Py_None; - - } - - /* internal error */ - pattern_error(status); - return NULL; -} - -static PyObject* -pattern_scanner(PatternObject* pattern, PyObject* args) -{ - /* create search state object */ - - ScannerObject* self; - - PyObject* string; - int start = 0; - int end = INT_MAX; - if (!PyArg_ParseTuple(args, "O|ii:scanner", &string, &start, &end)) - return NULL; - - /* create scanner object */ - self = PyObject_NEW(ScannerObject, &Scanner_Type); - if (!self) - return NULL; - - string = state_init(&self->state, pattern, string, start, end); - if (!string) { - PyObject_DEL(self); - return NULL; - } - - Py_INCREF(pattern); - self->pattern = (PyObject*) pattern; - - return (PyObject*) self; -} - static void pattern_dealloc(PatternObject* self) { @@ -2414,7 +2258,7 @@ } static PyObject* -pattern_subx(PatternObject* self, PyObject* template, PyObject* string, +pattern_subx(PatternObject* self, PyObject* ptemplate, PyObject* string, int count, int subn) { SRE_STATE state; @@ -2429,15 +2273,15 @@ int i, b, e; int filter_is_callable; - if (PyCallable_Check(template)) { + if (PyCallable_Check(ptemplate)) { /* sub/subn takes either a function or a template */ - filter = template; + filter = ptemplate; Py_INCREF(filter); filter_is_callable = 1; } else { /* if not callable, check if it's a literal string */ int literal; - ptr = getstring(template, &n, &b); + ptr = getstring(ptemplate, &n, &b); if (ptr) { if (b == 1) { literal = sre_literal_template(ptr, n); @@ -2451,14 +2295,14 @@ literal = 0; } if (literal) { - filter = template; + filter = ptemplate; Py_INCREF(filter); filter_is_callable = 0; } else { /* not a literal; hand it over to the template compiler */ filter = call( SRE_PY_MODULE, "_subx", - PyTuple_Pack(2, self, template) + PyTuple_Pack(2, self, ptemplate) ); if (!filter) return NULL; @@ -2597,29 +2441,29 @@ static PyObject* pattern_sub(PatternObject* self, PyObject* args, PyObject* kw) { - PyObject* template; + PyObject* ptemplate; PyObject* string; int count = 0; static char* kwlist[] = { "repl", "string", "count", NULL }; if (!PyArg_ParseTupleAndKeywords(args, kw, "OO|i:sub", kwlist, - &template, &string, &count)) + &ptemplate, &string, &count)) return NULL; - return pattern_subx(self, template, string, count, 0); + return pattern_subx(self, ptemplate, string, count, 0); } static PyObject* pattern_subn(PatternObject* self, PyObject* args, PyObject* kw) { - PyObject* template; + PyObject* ptemplate; PyObject* string; int count = 0; static char* kwlist[] = { "repl", "string", "count", NULL }; if (!PyArg_ParseTupleAndKeywords(args, kw, "OO|i:subn", kwlist, - &template, &string, &count)) + &ptemplate, &string, &count)) return NULL; - return pattern_subx(self, template, string, count, 1); + return pattern_subx(self, ptemplate, string, count, 1); } static PyObject* @@ -2799,6 +2643,68 @@ offsetof(PatternObject, weakreflist), /* tp_weaklistoffset */ }; +static PyObject * +_compile(PyObject* self_, PyObject* args) +{ + /* "compile" pattern descriptor to pattern object */ + + PatternObject* self; + int i, n; + + PyObject* pattern; + int flags = 0; + PyObject* code; + int groups = 0; + PyObject* groupindex = NULL; + PyObject* indexgroup = NULL; + if (!PyArg_ParseTuple(args, "OiO!|iOO", &pattern, &flags, + &PyList_Type, &code, &groups, + &groupindex, &indexgroup)) + return NULL; + + n = PyList_GET_SIZE(code); + + self = PyObject_NEW_VAR(PatternObject, &Pattern_Type, n); + if (!self) + return NULL; + + self->codesize = n; + + for (i = 0; i < n; i++) { + PyObject *o = PyList_GET_ITEM(code, i); + unsigned long value = PyInt_Check(o) ? (unsigned long)PyInt_AsLong(o) + : PyLong_AsUnsignedLong(o); + self->code[i] = (SRE_CODE) value; + if ((unsigned long) self->code[i] != value) { + PyErr_SetString(PyExc_OverflowError, + "regular expression code size limit exceeded"); + break; + } + } + + if (PyErr_Occurred()) { + PyObject_DEL(self); + return NULL; + } + + Py_INCREF(pattern); + self->pattern = pattern; + + self->flags = flags; + + self->groups = groups; + + Py_XINCREF(groupindex); + self->groupindex = groupindex; + + Py_XINCREF(indexgroup); + self->indexgroup = indexgroup; + + self->weakreflist = NULL; + + return (PyObject*) self; +} + /* -------------------------------------------------------------------- */ /* match methods */ @@ -2868,14 +2774,14 @@ static PyObject* match_expand(MatchObject* self, PyObject* args) { - PyObject* template; - if (!PyArg_ParseTuple(args, "O:expand", &template)) + PyObject* ptemplate; + if (!PyArg_ParseTuple(args, "O:expand", &ptemplate)) return NULL; /* delegate to Python code */ return call( SRE_PY_MODULE, "_expand", - PyTuple_Pack(3, self->pattern, self, template) + PyTuple_Pack(3, self->pattern, self, ptemplate) ); } @@ -3262,6 +3168,69 @@ (getattrfunc)match_getattr /*tp_getattr*/ }; +static PyObject* +pattern_new_match(PatternObject* pattern, SRE_STATE* state, int status) +{ + /* create match object (from state object) */ + + MatchObject* match; + int i, j; + char* base; + int n; + + if (status > 0) { + + /* create match object (with room for extra group marks) */ + match = PyObject_NEW_VAR(MatchObject, &Match_Type, + 2*(pattern->groups+1)); + if (!match) + return NULL; + + Py_INCREF(pattern); + match->pattern = pattern; + + Py_INCREF(state->string); + match->string = state->string; + + match->regs = NULL; + match->groups = pattern->groups+1; + + /* fill in group slices */ + + base = (char*) state->beginning; + n = state->charsize; + + match->mark[0] = ((char*) state->start - base) / n; + match->mark[1] = ((char*) state->ptr - base) / n; + + for (i = j = 0; i < pattern->groups; i++, j+=2) + if (j+1 <= state->lastmark && state->mark[j] && state->mark[j+1]) { + match->mark[j+2] = ((char*) state->mark[j] - base) / n; + match->mark[j+3] = ((char*) state->mark[j+1] - base) / n; + } else + match->mark[j+2] = match->mark[j+3] = -1; /* undefined */ + + match->pos = state->pos; + match->endpos = state->endpos; + + match->lastindex = state->lastindex; + + return (PyObject*) match; + + } else if (status == 0) { + + /* no match */ + Py_INCREF(Py_None); + return Py_None; + + } + + /* internal error */ + pattern_error(status); + return NULL; +} + + /* -------------------------------------------------------------------- */ /* scanner methods (experimental) */ @@ -3372,6 +3341,36 @@ (getattrfunc)scanner_getattr, /*tp_getattr*/ }; +static PyObject* +pattern_scanner(PatternObject* pattern, PyObject* args) +{ + /* create search state object */ + + ScannerObject* self; + + PyObject* string; + int start = 0; + int end = INT_MAX; + if (!PyArg_ParseTuple(args, "O|ii:scanner", &string, &start, &end)) + return NULL; + + /* create scanner object */ + self = PyObject_NEW(ScannerObject, &Scanner_Type); + if (!self) + return NULL; + + string = state_init(&self->state, pattern, string, start, end); + if (!string) { + PyObject_DEL(self); + return NULL; + } + + Py_INCREF(pattern); + self->pattern = (PyObject*) pattern; + + return (PyObject*) self; +} + static PyMethodDef _functions[] = { {"compile", _compile, METH_VARARGS}, {"getcodesize", sre_codesize, METH_VARARGS}, From python-checkins at python.org Wed Apr 12 06:29:02 2006 From: python-checkins at python.org (anthony.baxter) Date: Wed, 12 Apr 2006 06:29:02 +0200 (CEST) Subject: [Python-checkins] r45291 - python/trunk/Modules/xxsubtype.c python/trunk/Modules/zipimport.c Message-ID: <20060412042902.0E77C1E4003@bag.python.org> Author: anthony.baxter Date: Wed Apr 12 06:29:01 2006 New Revision: 45291 Modified: python/trunk/Modules/xxsubtype.c python/trunk/Modules/zipimport.c Log: remove forward declarations. No constructors to move for these files. Makes code work with C++ compilers. Modified: python/trunk/Modules/xxsubtype.c ============================================================================== --- python/trunk/Modules/xxsubtype.c (original) +++ python/trunk/Modules/xxsubtype.c Wed Apr 12 06:29:01 2006 @@ -79,8 +79,6 @@ {NULL, NULL}, }; -static PyTypeObject spamlist_type; - static int spamlist_init(spamlistobject *self, PyObject *args, PyObject *kwds) { @@ -179,8 +177,6 @@ {NULL, NULL}, }; -static PyTypeObject spamdict_type; - static int spamdict_init(spamdictobject *self, PyObject *args, PyObject *kwds) { Modified: python/trunk/Modules/zipimport.c ============================================================================== --- python/trunk/Modules/zipimport.c (original) +++ python/trunk/Modules/zipimport.c Wed Apr 12 06:29:01 2006 @@ -40,7 +40,6 @@ PyObject *files; /* dict with file info {path: toc_entry} */ }; -static PyTypeObject ZipImporter_Type; static PyObject *ZipImportError; static PyObject *zip_directory_cache = NULL; @@ -958,7 +957,7 @@ PyObject *fixed_source; /* one char extra for trailing \n and one for terminating \0 */ - buf = PyMem_Malloc(PyString_Size(source) + 2); + buf = (char *)PyMem_Malloc(PyString_Size(source) + 2); if (buf == NULL) { PyErr_SetString(PyExc_MemoryError, "zipimport: no memory to allocate " From python-checkins at python.org Wed Apr 12 06:38:55 2006 From: python-checkins at python.org (anthony.baxter) Date: Wed, 12 Apr 2006 06:38:55 +0200 (CEST) Subject: [Python-checkins] r45292 - python/trunk/Python/getargs.c Message-ID: <20060412043855.415641E4003@bag.python.org> Author: anthony.baxter Date: Wed Apr 12 06:38:54 2006 New Revision: 45292 Modified: python/trunk/Python/getargs.c Log: avoid C++ name mangling for the _Py.*SizeT functions Modified: python/trunk/Python/getargs.c ============================================================================== --- python/trunk/Python/getargs.c (original) +++ python/trunk/Python/getargs.c Wed Apr 12 06:38:54 2006 @@ -6,6 +6,9 @@ #include +#ifdef __cplusplus +extern "C" { +#endif int PyArg_Parse(PyObject *, const char *, ...); int PyArg_ParseTuple(PyObject *, const char *, ...); int PyArg_VaParse(PyObject *, const char *, va_list); @@ -1742,3 +1745,6 @@ funcname); return 0; } +#ifdef __cplusplus +}; +#endif From python-checkins at python.org Wed Apr 12 06:56:36 2006 From: python-checkins at python.org (neal.norwitz) Date: Wed, 12 Apr 2006 06:56:36 +0200 (CEST) Subject: [Python-checkins] r45293 - in python/trunk/Mac/Modules: carbonevt/_CarbonEvtmodule.c qd/_Qdmodule.c Message-ID: <20060412045636.134FC1E4003@bag.python.org> Author: neal.norwitz Date: Wed Apr 12 06:56:35 2006 New Revision: 45293 Modified: python/trunk/Mac/Modules/carbonevt/_CarbonEvtmodule.c python/trunk/Mac/Modules/qd/_Qdmodule.c Log: Get rid of some warnings on Mac Modified: python/trunk/Mac/Modules/carbonevt/_CarbonEvtmodule.c ============================================================================== --- python/trunk/Mac/Modules/carbonevt/_CarbonEvtmodule.c (original) +++ python/trunk/Mac/Modules/carbonevt/_CarbonEvtmodule.c Wed Apr 12 06:56:35 2006 @@ -40,7 +40,7 @@ PyMac_GetOSType, &(out->eventClass), &(out->eventKind))) return 1; - return NULL; + return 0; } /********** end EventTypeSpec *******/ @@ -78,7 +78,7 @@ { if (PyArg_ParseTuple(v, "ll", &out->signature, &out->id)) return 1; - return NULL; + return 0; } /********** end EventHotKeyID *******/ Modified: python/trunk/Mac/Modules/qd/_Qdmodule.c ============================================================================== --- python/trunk/Mac/Modules/qd/_Qdmodule.c (original) +++ python/trunk/Mac/Modules/qd/_Qdmodule.c Wed Apr 12 06:56:35 2006 @@ -5824,7 +5824,6 @@ { PyObject *_res = NULL; char *textBuf__in__; - int textBuf__len__; int textBuf__in_len__; short firstByte; short byteCount; @@ -5885,7 +5884,6 @@ PyObject *_res = NULL; short _rv; char *textBuf__in__; - int textBuf__len__; int textBuf__in_len__; short firstByte; short byteCount; @@ -6471,7 +6469,6 @@ { PyObject *_res = NULL; char *textBuf__in__; - int textBuf__len__; int textBuf__in_len__; short firstByte; short byteCount; @@ -6534,7 +6531,7 @@ return NULL; if ( PyString_Size(source) != sizeof(BitMap) && PyString_Size(source) != sizeof(PixMap) ) { PyErr_Format(PyExc_TypeError, - "Argument size was %d, should be %d (sizeof BitMap) or %d (sizeof PixMap)", + "Argument size was %ld, should be %lu (sizeof BitMap) or %lu (sizeof PixMap)", PyString_Size(source), sizeof(BitMap), sizeof(PixMap)); return NULL; } From python-checkins at python.org Wed Apr 12 07:16:31 2006 From: python-checkins at python.org (anthony.baxter) Date: Wed, 12 Apr 2006 07:16:31 +0200 (CEST) Subject: [Python-checkins] r45294 - python/trunk/Grammar/Grammar Message-ID: <20060412051631.783961E4040@bag.python.org> Author: anthony.baxter Date: Wed Apr 12 07:16:30 2006 New Revision: 45294 Modified: python/trunk/Grammar/Grammar Log: put in a reference to PEP 306 in a comment at the top Modified: python/trunk/Grammar/Grammar ============================================================================== --- python/trunk/Grammar/Grammar (original) +++ python/trunk/Grammar/Grammar Wed Apr 12 07:16:30 2006 @@ -7,6 +7,9 @@ # with someone who can; ask around on python-dev for help. Fred # Drake will probably be listening there. +# NOTE WELL: You should also follow all the steps listed in PEP 306, +# "How to Change Python's Grammar" + # Commands for Kees Blom's railroad program #diagram:token NAME #diagram:token NUMBER From python-checkins at python.org Wed Apr 12 07:16:37 2006 From: python-checkins at python.org (neal.norwitz) Date: Wed, 12 Apr 2006 07:16:37 +0200 (CEST) Subject: [Python-checkins] r45295 - peps/trunk/pep-0356.txt Message-ID: <20060412051637.159061E4003@bag.python.org> Author: neal.norwitz Date: Wed Apr 12 07:16:36 2006 New Revision: 45295 Modified: peps/trunk/pep-0356.txt Log: Update from pje Modified: peps/trunk/pep-0356.txt ============================================================================== --- peps/trunk/pep-0356.txt (original) +++ peps/trunk/pep-0356.txt Wed Apr 12 07:16:36 2006 @@ -119,11 +119,8 @@ (Owner: ???) http://mail.python.org/pipermail/python-dev/2006-February/060926.html - - bdist_egg in distutils package - (Owner: ???) - - setuptools to the standard library - (Owner: ???) + (Owner: Phillip Eby) - wsgiref to the standard library (Owner: Phillip Eby) @@ -150,7 +147,8 @@ Deferred until 2.6: - - None + - bdist_egg in distutils package + (though bsdist_egg is supported in setuptools) Open issues From python-checkins at python.org Wed Apr 12 07:24:43 2006 From: python-checkins at python.org (neal.norwitz) Date: Wed, 12 Apr 2006 07:24:43 +0200 (CEST) Subject: [Python-checkins] r45296 - python/trunk/Modules/parsermodule.c Message-ID: <20060412052443.A848E1E4009@bag.python.org> Author: neal.norwitz Date: Wed Apr 12 07:24:39 2006 New Revision: 45296 Modified: python/trunk/Modules/parsermodule.c Log: Update for new grammar Modified: python/trunk/Modules/parsermodule.c ============================================================================== --- python/trunk/Modules/parsermodule.c (original) +++ python/trunk/Modules/parsermodule.c Wed Apr 12 07:24:39 2006 @@ -1326,7 +1326,7 @@ return res; } -/* list_if: 'if' test [list_iter] +/* list_if: 'if' old_test [list_iter] */ static int validate_list_if(node *tree) @@ -1341,12 +1341,12 @@ if (res) res = (validate_name(CHILD(tree, 0), "if") - && validate_test(CHILD(tree, 1))); + && validate_old_test(CHILD(tree, 1))); return res; } -/* gen_if: 'if' test [gen_iter] +/* gen_if: 'if' old_test [gen_iter] */ static int validate_gen_if(node *tree) @@ -1361,7 +1361,7 @@ if (res) res = (validate_name(CHILD(tree, 0), "if") - && validate_test(CHILD(tree, 1))); + && validate_old_test(CHILD(tree, 1))); return res; } From python-checkins at python.org Wed Apr 12 07:27:46 2006 From: python-checkins at python.org (neal.norwitz) Date: Wed, 12 Apr 2006 07:27:46 +0200 (CEST) Subject: [Python-checkins] r45297 - python/trunk/Lib/test/test_parser.py Message-ID: <20060412052746.BA4451E4003@bag.python.org> Author: neal.norwitz Date: Wed Apr 12 07:27:46 2006 New Revision: 45297 Modified: python/trunk/Lib/test/test_parser.py Log: Add another little test to make sure we roundtrip multiple list comp ifs ok. Add tests for generator expressions too. Modified: python/trunk/Lib/test/test_parser.py ============================================================================== --- python/trunk/Lib/test/test_parser.py (original) +++ python/trunk/Lib/test/test_parser.py Wed Apr 12 07:27:46 2006 @@ -51,6 +51,10 @@ self.check_expr("[1, 2, 3]") self.check_expr("[x**3 for x in range(20)]") self.check_expr("[x**3 for x in range(20) if x % 3]") + self.check_expr("[x**3 for x in range(20) if x % 2 if x % 3]") + self.check_expr("list(x**3 for x in range(20))") + self.check_expr("list(x**3 for x in range(20) if x % 3)") + self.check_expr("list(x**3 for x in range(20) if x % 2 if x % 3)") self.check_expr("foo(*args)") self.check_expr("foo(*args, **kw)") self.check_expr("foo(**kw)") From nnorwitz at gmail.com Wed Apr 12 07:33:38 2006 From: nnorwitz at gmail.com (Neal Norwitz) Date: Tue, 11 Apr 2006 22:33:38 -0700 Subject: [Python-checkins] r45283 - python/trunk/Lib/logging/__init__.py In-Reply-To: <20060411214201.8B5401E4009@bag.python.org> References: <20060411214201.8B5401E4009@bag.python.org> Message-ID: Vinay, Should this be backported? n -- On 4/11/06, vinay.sajip wrote: > Author: vinay.sajip > Date: Tue Apr 11 23:42:00 2006 > New Revision: 45283 > > Modified: > python/trunk/Lib/logging/__init__.py > Log: > StreamHandler now checks explicitly for None before using sys.stderr as the stream (see SF bug #1463840). > > Modified: python/trunk/Lib/logging/__init__.py > ============================================================================== > --- python/trunk/Lib/logging/__init__.py (original) > +++ python/trunk/Lib/logging/__init__.py Tue Apr 11 23:42:00 2006 > @@ -719,7 +719,7 @@ > If strm is not specified, sys.stderr is used. > """ > Handler.__init__(self) > - if not strm: > + if strm is None: > strm = sys.stderr > self.stream = strm > self.formatter = None > _______________________________________________ > Python-checkins mailing list > Python-checkins at python.org > http://mail.python.org/mailman/listinfo/python-checkins > From python-checkins at python.org Wed Apr 12 07:56:01 2006 From: python-checkins at python.org (neal.norwitz) Date: Wed, 12 Apr 2006 07:56:01 +0200 (CEST) Subject: [Python-checkins] r45298 - python/trunk/Misc/build.sh Message-ID: <20060412055601.38D101E4003@bag.python.org> Author: neal.norwitz Date: Wed Apr 12 07:56:00 2006 New Revision: 45298 Modified: python/trunk/Misc/build.sh Log: Update comments and the skip list, maybe some of these tests don't report failures, we'll see. Skip certain hopeless tests: compiler and logging. compiler will likely always show varying leaks since it doesn't work on a defined set of modules unless -u compiler is specified. But that takes forever (we only run with -u network currently). logging causes hangs when running with -R. Modified: python/trunk/Misc/build.sh ============================================================================== --- python/trunk/Misc/build.sh (original) +++ python/trunk/Misc/build.sh Wed Apr 12 07:56:00 2006 @@ -55,12 +55,19 @@ RSYNC_OPTS="-aC -e ssh" REFLOG="build/reflog.txt.out" -# These tests are not stable and often falsely report leaks. +# These tests are not stable and falsely report leaks sometimes. # The entire leak report will be mailed if any test not in this list leaks. # Note: test_XXX (none currently) really leak, but are disabled # so we don't send spam. Any test which really leaks should only # be listed here if there are also test cases under Lib/test/leakers. -LEAKY_TESTS="test_(capi|cfgparser|charmapcodec|cmd_line|compiler|ctypes|filecmp|quopri|socket|threaded_import|threadedtempfile|threading|threading_local)" +LEAKY_TESTS="test_(cmd_line|ctypes|filecmp|socket|threadedtempfile|threading|threading_local|urllib2)" + +# Skip these tests altogether when looking for leaks. These tests +# do not need to be stored above in LEAKY_TESTS too. +# test_compiler almost never finishes with the same number of refs +# since it depends on other modules, skip it. +# test_logging causes hangs, skip it. +LEAKY_SKIPS="-x test_compiler test_logging" # Change this flag to "yes" for old releases to only update/build the docs. BUILD_DISABLED="no" @@ -160,7 +167,7 @@ ## run the tests looking for leaks F=make-test-refleak.out start=`current_time` - ./python ./Lib/test/regrtest.py -R 4:3:$REFLOG -u network >& build/$F + ./python ./Lib/test/regrtest.py -R 4:3:$REFLOG -u network $LEAKY_SKIPS >& build/$F NUM_FAILURES=`egrep -vc "$LEAKY_TESTS" $REFLOG` update_status "Testing refleaks ($NUM_FAILURES failures)" "$F" $start mail_on_failure "refleak" $REFLOG From python-checkins at python.org Wed Apr 12 08:44:37 2006 From: python-checkins at python.org (tim.peters) Date: Wed, 12 Apr 2006 08:44:37 +0200 (CEST) Subject: [Python-checkins] r45299 - python/trunk/Objects/genobject.c Message-ID: <20060412064437.2863F1E4003@bag.python.org> Author: tim.peters Date: Wed Apr 12 08:44:36 2006 New Revision: 45299 Modified: python/trunk/Objects/genobject.c Log: gen_throw(): The caller doesn't own PyArg_ParseTuple() "O" arguments, so must not decref them. This accounts for why running test_contextlib.test_main() in a loop eventually tried to deallocate Py_None. Modified: python/trunk/Objects/genobject.c ============================================================================== --- python/trunk/Objects/genobject.c (original) +++ python/trunk/Objects/genobject.c Wed Apr 12 08:44:36 2006 @@ -217,10 +217,8 @@ /* First, check the traceback argument, replacing None with NULL. */ - if (tb == Py_None) { - Py_DECREF(tb); + if (tb == Py_None) tb = NULL; - } else if (tb != NULL && !PyTraceBack_Check(tb)) { PyErr_SetString(PyExc_TypeError, "throw() third argument must be a traceback object"); From python-checkins at python.org Wed Apr 12 08:56:56 2006 From: python-checkins at python.org (neal.norwitz) Date: Wed, 12 Apr 2006 08:56:56 +0200 (CEST) Subject: [Python-checkins] r45300 - python/trunk/Objects/genobject.c Message-ID: <20060412065656.90BFB1E4003@bag.python.org> Author: neal.norwitz Date: Wed Apr 12 08:56:56 2006 New Revision: 45300 Modified: python/trunk/Objects/genobject.c Log: wrap docstrings so they are less than 80 columns. add spaces after commas. Modified: python/trunk/Objects/genobject.c ============================================================================== --- python/trunk/Objects/genobject.c (original) +++ python/trunk/Objects/genobject.c Wed Apr 12 08:56:56 2006 @@ -106,7 +106,8 @@ } PyDoc_STRVAR(send_doc, -"send(arg) -> send 'arg' into generator, return next yielded value or raise StopIteration."); +"send(arg) -> send 'arg' into generator,\n\ +return next yielded value or raise StopIteration."); static PyObject * gen_send(PyGenObject *gen, PyObject *arg) @@ -203,7 +204,8 @@ PyDoc_STRVAR(throw_doc, -"throw(typ[,val[,tb]]) -> raise exception in generator, return next yielded value or raise StopIteration."); +"throw(typ[,val[,tb]]) -> raise exception in generator,\n\ +return next yielded value or raise StopIteration."); static PyObject * gen_throw(PyGenObject *gen, PyObject *args) @@ -259,7 +261,7 @@ goto failed_throw; } - PyErr_Restore(typ,val,tb); + PyErr_Restore(typ, val, tb); return gen_send_ex(gen, Py_None, 1); failed_throw: From neal at metaslash.com Wed Apr 12 11:04:21 2006 From: neal at metaslash.com (Neal Norwitz) Date: Wed, 12 Apr 2006 05:04:21 -0400 Subject: [Python-checkins] Python Regression Test Failures refleak (1) Message-ID: <20060412090421.GA6096@python.psfb.org> test_cmd_line leaked [0, -17, 0] references test_generators leaked [1, 1, 1] references test_socket leaked [-217, 0, 0] references test_threadedtempfile leaked [-82, 0, 2] references test_threading_local leaked [42, -58, -59] references test_urllib2 leaked [-121, 88, 99] references From python-checkins at python.org Wed Apr 12 12:02:01 2006 From: python-checkins at python.org (vinay.sajip) Date: Wed, 12 Apr 2006 12:02:01 +0200 (CEST) Subject: [Python-checkins] r45301 - python/branches/release24-maint/Lib/logging/__init__.py Message-ID: <20060412100201.723141E4003@bag.python.org> Author: vinay.sajip Date: Wed Apr 12 12:02:00 2006 New Revision: 45301 Modified: python/branches/release24-maint/Lib/logging/__init__.py Log: StreamHandler now checks explicitly for None before using sys.stderr as the stream (see SF bug #1463840). Modified: python/branches/release24-maint/Lib/logging/__init__.py ============================================================================== --- python/branches/release24-maint/Lib/logging/__init__.py (original) +++ python/branches/release24-maint/Lib/logging/__init__.py Wed Apr 12 12:02:00 2006 @@ -706,7 +706,7 @@ If strm is not specified, sys.stderr is used. """ Handler.__init__(self) - if not strm: + if strm is None: strm = sys.stderr self.stream = strm self.formatter = None From python-checkins at python.org Wed Apr 12 12:09:16 2006 From: python-checkins at python.org (walter.doerwald) Date: Wed, 12 Apr 2006 12:09:16 +0200 (CEST) Subject: [Python-checkins] r45302 - in python/trunk/Lib: calendar.py test/test_calendar.py Message-ID: <20060412100916.C45601E4003@bag.python.org> Author: walter.doerwald Date: Wed Apr 12 12:09:16 2006 New Revision: 45302 Modified: python/trunk/Lib/calendar.py python/trunk/Lib/test/test_calendar.py Log: Patch #1463288: use a context manager to temporarily switch locales. Add tests for the output of the TextCalendar and HTMLCalendar classes. Modified: python/trunk/Lib/calendar.py ============================================================================== --- python/trunk/Lib/calendar.py (original) +++ python/trunk/Lib/calendar.py Wed Apr 12 12:09:16 2006 @@ -5,6 +5,7 @@ Sunday as the last (the European convention). Use setfirstweekday() to set the first day of the week (0=Monday, 6=Sunday).""" +from __future__ import with_statement import sys, datetime, locale __all__ = ["IllegalMonthError", "IllegalWeekdayError", "setfirstweekday", @@ -479,6 +480,21 @@ return ''.join(v).encode(encoding, "xmlcharrefreplace") +class TimeEncoding: + def __init__(self, locale): + self.locale = locale + + def __context__(self): + return self + + def __enter__(self): + self.oldlocale = locale.setlocale(locale.LC_TIME, self.locale) + return locale.getlocale(locale.LC_TIME)[1] + + def __exit__(self, *args): + locale.setlocale(locale.LC_TIME, self.oldlocale) + + class LocaleTextCalendar(TextCalendar): """ This class can be passed a locale name in the constructor and will return @@ -494,9 +510,7 @@ self.locale = locale def formatweekday(self, day, width): - oldlocale = locale.setlocale(locale.LC_TIME, self.locale) - try: - encoding = locale.getlocale(locale.LC_TIME)[1] + with TimeEncoding(self.locale) as encoding: if width >= 9: names = day_name else: @@ -504,24 +518,16 @@ name = names[day] if encoding is not None: name = name.decode(encoding) - result = name[:width].center(width) - finally: - locale.setlocale(locale.LC_TIME, oldlocale) - return result + return name[:width].center(width) def formatmonthname(self, theyear, themonth, width, withyear=True): - oldlocale = locale.setlocale(locale.LC_TIME, self.locale) - try: - encoding = locale.getlocale(locale.LC_TIME)[1] + with TimeEncoding(self.locale) as encoding: s = month_name[themonth] if encoding is not None: s = s.decode(encoding) if withyear: s = "%s %r" % (s, theyear) - result = s.center(width) - finally: - locale.setlocale(locale.LC_TIME, oldlocale) - return result + return s.center(width) class LocaleHTMLCalendar(HTMLCalendar): @@ -538,30 +544,20 @@ self.locale = locale def formatweekday(self, day): - oldlocale = locale.setlocale(locale.LC_TIME, self.locale) - try: - encoding = locale.getlocale(locale.LC_TIME)[1] + with TimeEncoding(self.locale) as encoding: s = day_abbr[day] if encoding is not None: s = s.decode(encoding) - result = '' % (self.cssclasses[day], s) - finally: - locale.setlocale(locale.LC_TIME, oldlocale) - return result + return '' % (self.cssclasses[day], s) def formatmonthname(self, theyear, themonth, withyear=True): - oldlocale = locale.setlocale(locale.LC_TIME, self.locale) - try: - encoding = locale.getlocale(locale.LC_TIME)[1] + with TimeEncoding(self.locale) as encoding: s = month_name[themonth] if encoding is not None: s = s.decode(encoding) if withyear: s = '%s %s' % (s, theyear) - result = '' % s - finally: - locale.setlocale(locale.LC_TIME, oldlocale) - return result + return '' % s # Support for old module level interface Modified: python/trunk/Lib/test/test_calendar.py ============================================================================== --- python/trunk/Lib/test/test_calendar.py (original) +++ python/trunk/Lib/test/test_calendar.py Wed Apr 12 12:09:16 2006 @@ -4,7 +4,7 @@ from test import test_support -result_2004 = """ +result_2004_text = """ 2004 January February March @@ -42,9 +42,135 @@ 25 26 27 28 29 30 31 29 30 27 28 29 30 31 """ +result_2004_html = """ + + + + + + +Calendar for 2004 + +
%s%s
%s
%s
+
2004
+ + + + + + + +
January
MonTueWedThuFriSatSun
   1234
567891011
12131415161718
19202122232425
262728293031 
+		 + + + + + + + +
February
MonTueWedThuFriSatSun
      1
2345678
9101112131415
16171819202122
23242526272829
+		 + + + + + + + +
March
MonTueWedThuFriSatSun
1234567
891011121314
15161718192021
22232425262728
293031    
+
+ + + + + + + +
April
MonTueWedThuFriSatSun
   1234
567891011
12131415161718
19202122232425
2627282930  
+		 + + + + + + + + +
May
MonTueWedThuFriSatSun
     12
3456789
10111213141516
17181920212223
24252627282930
31      
+		 + + + + + + + +
June
MonTueWedThuFriSatSun
 123456
78910111213
14151617181920
21222324252627
282930    
+
+ + + + + + + +
July
MonTueWedThuFriSatSun
   1234
567891011
12131415161718
19202122232425
262728293031 
+		 + + + + + + + + +
August
MonTueWedThuFriSatSun
      1
2345678
9101112131415
16171819202122
23242526272829
3031     
+		 + + + + + + + +
September
MonTueWedThuFriSatSun
  12345
6789101112
13141516171819
20212223242526
27282930   
+
+ + + + + + + +
October
MonTueWedThuFriSatSun
    123
45678910
11121314151617
18192021222324
25262728293031
+		 + + + + + + + +
November
MonTueWedThuFriSatSun
1234567
891011121314
15161718192021
22232425262728
2930     
+		 + + + + + + + +
December
MonTueWedThuFriSatSun
  12345
6789101112
13141516171819
20212223242526
2728293031  
+
+ +""" + class OutputTestCase(unittest.TestCase): def normalize_calendar(self, s): + # Filters out locale dependant strings def neitherspacenordigit(c): return not c.isspace() and not c.isdigit() @@ -58,7 +184,19 @@ def test_output(self): self.assertEqual( self.normalize_calendar(calendar.calendar(2004)), - self.normalize_calendar(result_2004) + self.normalize_calendar(result_2004_text) + ) + + def test_output_textcalendar(self): + self.assertEqual( + calendar.TextCalendar().formatyear(2004).strip(), + result_2004_text.strip() + ) + + def test_output_htmlcalendar(self): + self.assertEqual( + calendar.HTMLCalendar().formatyearpage(2004).strip(), + result_2004_html.strip() ) From buildbot at python.org Wed Apr 12 12:23:32 2006 From: buildbot at python.org (buildbot at python.org) Date: Wed, 12 Apr 2006 10:23:32 +0000 Subject: [Python-checkins] buildbot warnings in x86 OpenBSD 2.4 Message-ID: <20060412102332.729E01E4003@bag.python.org> The Buildbot has detected a new failure of x86 OpenBSD 2.4. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520OpenBSD%25202.4/builds/49 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch branches/release24-maint] HEAD Blamelist: vinay.sajip Build Had Warnings: warnings test sincerely, -The Buildbot From python-checkins at python.org Wed Apr 12 13:59:28 2006 From: python-checkins at python.org (armin.rigo) Date: Wed, 12 Apr 2006 13:59:28 +0200 (CEST) Subject: [Python-checkins] r45303 - python/trunk/Modules/socketmodule.c Message-ID: <20060412115928.7A6A11E4003@bag.python.org> Author: armin.rigo Date: Wed Apr 12 13:59:26 2006 New Revision: 45303 Modified: python/trunk/Modules/socketmodule.c Log: Off-by-one buffer overflow error. Modified: python/trunk/Modules/socketmodule.c ============================================================================== --- python/trunk/Modules/socketmodule.c (original) +++ python/trunk/Modules/socketmodule.c Wed Apr 12 13:59:26 2006 @@ -1098,7 +1098,7 @@ addr = (struct sockaddr_un*)&(s->sock_addr).un; if (!PyArg_Parse(args, "t#", &path, &len)) return 0; - if (len > sizeof addr->sun_path) { + if (len >= sizeof addr->sun_path) { PyErr_SetString(socket_error, "AF_UNIX path too long"); return 0; From python-checkins at python.org Wed Apr 12 14:04:11 2006 From: python-checkins at python.org (armin.rigo) Date: Wed, 12 Apr 2006 14:04:11 +0200 (CEST) Subject: [Python-checkins] r45304 - python/branches/release24-maint/Modules/socketmodule.c Message-ID: <20060412120411.E30981E4003@bag.python.org> Author: armin.rigo Date: Wed Apr 12 14:04:10 2006 New Revision: 45304 Modified: python/branches/release24-maint/Modules/socketmodule.c Log: Backport of r45303: buffer overflow. Modified: python/branches/release24-maint/Modules/socketmodule.c ============================================================================== --- python/branches/release24-maint/Modules/socketmodule.c (original) +++ python/branches/release24-maint/Modules/socketmodule.c Wed Apr 12 14:04:10 2006 @@ -1083,7 +1083,7 @@ addr = (struct sockaddr_un*)&(s->sock_addr).un; if (!PyArg_Parse(args, "t#", &path, &len)) return 0; - if (len > sizeof addr->sun_path) { + if (len >= sizeof addr->sun_path) { PyErr_SetString(socket_error, "AF_UNIX path too long"); return 0; From python-checkins at python.org Wed Apr 12 14:07:32 2006 From: python-checkins at python.org (anthony.baxter) Date: Wed, 12 Apr 2006 14:07:32 +0200 (CEST) Subject: [Python-checkins] r45305 - in python/trunk: Lib/SimpleXMLRPCServer.py Misc/NEWS Message-ID: <20060412120732.9CD6E1E4003@bag.python.org> Author: anthony.baxter Date: Wed Apr 12 14:07:31 2006 New Revision: 45305 Modified: python/trunk/Lib/SimpleXMLRPCServer.py python/trunk/Misc/NEWS Log: Bug #1469163: SimpleXMLRPCServer unconditionally attempted to import fcntl. Wrapped in a try/except. Modified: python/trunk/Lib/SimpleXMLRPCServer.py ============================================================================== --- python/trunk/Lib/SimpleXMLRPCServer.py (original) +++ python/trunk/Lib/SimpleXMLRPCServer.py Wed Apr 12 14:07:31 2006 @@ -104,7 +104,11 @@ import SocketServer import BaseHTTPServer import sys -import os, fcntl +import os +try: + import fcntl +except ImportError: + fcntl = None def resolve_dotted_attribute(obj, attr, allow_dotted_names=True): """resolve_dotted_attribute(a, 'b.c.d') => a.b.c.d @@ -493,7 +497,7 @@ # [Bug #1222790] If possible, set close-on-exec flag; if a # method spawns a subprocess, the subprocess shouldn't have # the listening socket open. - if hasattr(fcntl, 'FD_CLOEXEC'): + if fcntl is not None and hasattr(fcntl, 'FD_CLOEXEC'): flags = fcntl.fcntl(self.fileno(), fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC fcntl.fcntl(self.fileno(), fcntl.F_SETFD, flags) Modified: python/trunk/Misc/NEWS ============================================================================== --- python/trunk/Misc/NEWS (original) +++ python/trunk/Misc/NEWS Wed Apr 12 14:07:31 2006 @@ -49,6 +49,9 @@ Library ------- +- SimpleXMLRPCServer relied on the fcntl module, which is unavailable on + Windows. Bug #1469163. + - The warnings, linecache, inspect, traceback, site, and doctest modules were updated to work correctly with modules imported from zipfiles or via other PEP 302 __loader__ objects. From buildbot at python.org Wed Apr 12 14:09:23 2006 From: buildbot at python.org (buildbot at python.org) Date: Wed, 12 Apr 2006 12:09:23 +0000 Subject: [Python-checkins] buildbot failure in x86 cygwin trunk Message-ID: <20060412120926.27C7E1E4009@bag.python.org> The Buildbot has detected a new failure of x86 cygwin trunk. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520cygwin%2520trunk/builds/106 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: armin.rigo BUILD FAILED: failed failed slave lost sincerely, -The Buildbot From buildbot at python.org Wed Apr 12 14:09:24 2006 From: buildbot at python.org (buildbot at python.org) Date: Wed, 12 Apr 2006 12:09:24 +0000 Subject: [Python-checkins] buildbot failure in x86 Ubuntu dapper (icc) trunk Message-ID: <20060412120926.26BD91E4003@bag.python.org> The Buildbot has detected a new failure of x86 Ubuntu dapper (icc) trunk. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520Ubuntu%2520dapper%2520%2528icc%2529%2520trunk/builds/97 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: armin.rigo BUILD FAILED: failed failed slave lost sincerely, -The Buildbot From python-checkins at python.org Wed Apr 12 14:16:32 2006 From: python-checkins at python.org (andrew.kuchling) Date: Wed, 12 Apr 2006 14:16:32 +0200 (CEST) Subject: [Python-checkins] r45306 - python/trunk/Doc/whatsnew/whatsnew25.tex Message-ID: <20060412121632.178601E4003@bag.python.org> Author: andrew.kuchling Date: Wed Apr 12 14:16:31 2006 New Revision: 45306 Modified: python/trunk/Doc/whatsnew/whatsnew25.tex Log: Mention access to ASTs Modified: python/trunk/Doc/whatsnew/whatsnew25.tex ============================================================================== --- python/trunk/Doc/whatsnew/whatsnew25.tex (original) +++ python/trunk/Doc/whatsnew/whatsnew25.tex Wed Apr 12 14:16:31 2006 @@ -5,7 +5,6 @@ % Fix XXX comments % Distutils upload (PEP 243) % The easy_install stuff -% Access to ASTs with compile() flag % Stateful codec changes % ASCII is now default encoding for modules @@ -1380,6 +1379,20 @@ the parse tree is converted to an abstract syntax tree (or AST), and it is the abstract syntax tree that's traversed to produce the bytecode. +It's possible for Python code to obtain AST objects by using the +\function{compile()} built-in and specifying 0x400 as the value of the +\var{flags} parameter: + +\begin{verbatim} +ast = compile("""a=0 +for i in range(10): + a += i +""", "", 'exec', 0x0400) + +assignment = ast.body[0] +for_loop = ast.body[1] +\end{verbatim} + No documentation has been written for the AST code yet. To start learning about it, read the definition of the various AST nodes in \file{Parser/Python.asdl}. A Python script reads this file and From python-checkins at python.org Wed Apr 12 14:27:50 2006 From: python-checkins at python.org (andrew.kuchling) Date: Wed, 12 Apr 2006 14:27:50 +0200 (CEST) Subject: [Python-checkins] r45307 - python/trunk/Doc/whatsnew/whatsnew25.tex Message-ID: <20060412122750.8EF471E4003@bag.python.org> Author: andrew.kuchling Date: Wed Apr 12 14:27:50 2006 New Revision: 45307 Modified: python/trunk/Doc/whatsnew/whatsnew25.tex Log: Note C API incompatibilities Modified: python/trunk/Doc/whatsnew/whatsnew25.tex ============================================================================== --- python/trunk/Doc/whatsnew/whatsnew25.tex (original) +++ python/trunk/Doc/whatsnew/whatsnew25.tex Wed Apr 12 14:27:50 2006 @@ -1453,6 +1453,23 @@ the operating system. (Implemented by Evan Jones, and reworked by Tim Peters.) +Note that this change means extension modules need to be more careful +with how they allocate memory. Python's API has a number of different +functions for allocating memory that are grouped into families. For +example, \cfunction{PyMem_Malloc()}, \cfunction{PyMem_Realloc()}, and +\cfunction{PyMem_Free()} are one family that allocates raw memory, +while \cfunction{PyObject_Malloc()}, \cfunction{PyObject_Realloc()}, +and \cfunction{PyObject_Free()} are another family that's supposed to +be used for creating Python objects. + +Previously these different families all reduced to the platform's +\cfunction{malloc()} and \cfunction{free()} functions. This meant +it didn't matter if you got things wrong and allocated memory with the +\cfunction{PyMem} function but freed it with the \cfunction{PyObject} +function. With the obmalloc change, these families now do different +things, and mismatches will probably result in a segfault. You should +carefully test your C extension modules with Python 2.5. + \item Coverity, a company that markets a source code analysis tool called Prevent, provided the results of their examination of the Python source code. The analysis found a number of refcounting bugs, often @@ -1472,6 +1489,21 @@ \item The \module{pickle} module no longer uses the deprecated \var{bin} parameter. +\item C API: Many functions now use \ctype{Py_ssize_t} +instead of \ctype{int} to allow processing more data +on 64-bit machines. Extension code may need to make +the same change to avoid warnings and to support 64-bit machines. +See the earlier +section~ref{section-353} for a discussion of this change. + +\item C API: +The obmalloc changes mean that +you must be careful to not mix usage +of the \cfunction{PyMem_*()} and \cfunction{PyObject_*()} +families of functions. Memory allocated with +one family's \cfunction{*_Malloc()} must be +freed with the corresponding family's \cfunction{*_Free()} function. + \end{itemize} From python-checkins at python.org Wed Apr 12 14:44:37 2006 From: python-checkins at python.org (georg.brandl) Date: Wed, 12 Apr 2006 14:44:37 +0200 (CEST) Subject: [Python-checkins] r45308 - python/trunk/Lib/test/test_sundry.py Message-ID: <20060412124437.56AD91E4003@bag.python.org> Author: georg.brandl Date: Wed Apr 12 14:44:36 2006 New Revision: 45308 Modified: python/trunk/Lib/test/test_sundry.py Log: Update test_sundry. Many modules have now tests, but e.g. SimpleXMLRPCServer wasn't in here yet. Modified: python/trunk/Lib/test/test_sundry.py ============================================================================== --- python/trunk/Lib/test/test_sundry.py (original) +++ python/trunk/Lib/test/test_sundry.py Wed Apr 12 14:44:36 2006 @@ -12,74 +12,50 @@ from test.test_support import verbose import BaseHTTPServer +import DocXMLRPCServer import CGIHTTPServer -import Queue import SimpleHTTPServer -import SocketServer +import SimpleXMLRPCServer import aifc -import anydbm import audiodev import bdb +import cgitb import cmd import code -import codeop -import colorsys -import commands import compileall -try: - import curses # not available on Windows -except ImportError: - if verbose: - print "skipping curses" -import dircache -import dis -import distutils -import doctest -import dumbdbm import encodings -import fnmatch import formatter -import fpformat import ftplib import getpass -import glob import gopherlib import htmlentitydefs -import htmllib -import httplib -import imaplib +import ihooks import imghdr import imputil import keyword -import macpath +import linecache import macurl2path import mailcap -import mhlib -import mimetypes import mimify -import multifile import mutex import nntplib import nturl2path +import opcode +import os2emxpath import pdb import pipes #import poplib import posixfile -import profile import pstats import py_compile -import repr +import pydoc +import rexec try: import rlcompleter # not available on Windows except ImportError: if verbose: print "skipping rlcompleter" -import robotparser import sched -import sgmllib -import shelve -import shlex -import shutil import smtplib import sndhdr import statvfs @@ -89,12 +65,12 @@ import symbol import tabnanny import telnetlib -import test +import timeit import toaiff -import urllib2 +import token +import tty # Can't test the "user" module -- if the user has a ~/.pythonrc.py, it # can screw up all sorts of things (esp. if it prints!). #import user import webbrowser -import whichdb import xml From python-checkins at python.org Wed Apr 12 15:03:35 2006 From: python-checkins at python.org (andrew.kuchling) Date: Wed, 12 Apr 2006 15:03:35 +0200 (CEST) Subject: [Python-checkins] r45309 - python/trunk/Doc/whatsnew/whatsnew25.tex Message-ID: <20060412130335.C700F1E4003@bag.python.org> Author: andrew.kuchling Date: Wed Apr 12 15:03:35 2006 New Revision: 45309 Modified: python/trunk/Doc/whatsnew/whatsnew25.tex Log: Add PEP 243 section Modified: python/trunk/Doc/whatsnew/whatsnew25.tex ============================================================================== --- python/trunk/Doc/whatsnew/whatsnew25.tex (original) +++ python/trunk/Doc/whatsnew/whatsnew25.tex Wed Apr 12 15:03:35 2006 @@ -3,7 +3,6 @@ % $Id$ % Fix XXX comments -% Distutils upload (PEP 243) % The easy_install stuff % Stateful codec changes % ASCII is now default encoding for modules @@ -36,6 +35,32 @@ %====================================================================== +\section{PEP 243: Uploading Modules to PyPI} + +PEP 243 describes an HTTP-based protocol for submitting software +packages to a central archive. The Python package index at +\url{http://cheeseshop.python.org} now supports package uploads, and +the new \command{upload} Distutils command will upload a package to the +repository. + +Before a package can be uploaded, you must be able to build a +distribution using the \command{sdist} Distutils command. Once that +works, you can run \code{python setup.py upload} to add your package +to the PyPI archive. Optionally you can GPG-sign the package by +supplying the \programopt{--sign} and +\programopt{--identity} options. + +\begin{seealso} + +\seepep{243}{Module Repository Upload Mechanism}{PEP written by +Sean Reifschneider; implemented by Martin von L\"owis +and Richard Jones. Note that the PEP doesn't exactly +describe what's implemented in PyPI.} + +\end{seealso} + + +%====================================================================== \section{PEP 308: Conditional Expressions} For a long time, people have been requesting a way to write From buildbot at python.org Wed Apr 12 15:20:20 2006 From: buildbot at python.org (buildbot at python.org) Date: Wed, 12 Apr 2006 13:20:20 +0000 Subject: [Python-checkins] buildbot warnings in x86 cygwin trunk Message-ID: <20060412132020.670D81E4043@bag.python.org> The Buildbot has detected a new failure of x86 cygwin trunk. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520cygwin%2520trunk/builds/107 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: anthony.baxter Build Had Warnings: warnings test sincerely, -The Buildbot From python-checkins at python.org Wed Apr 12 17:28:49 2006 From: python-checkins at python.org (georg.brandl) Date: Wed, 12 Apr 2006 17:28:49 +0200 (CEST) Subject: [Python-checkins] r45310 - python/trunk/Lib/lib-tk/tkFont.py Message-ID: <20060412152849.AC29B1E4003@bag.python.org> Author: georg.brandl Date: Wed Apr 12 17:28:49 2006 New Revision: 45310 Modified: python/trunk/Lib/lib-tk/tkFont.py Log: Patch #1468808: don't complain if Tkinter is already deleted at the time Font.__del__ is run. Modified: python/trunk/Lib/lib-tk/tkFont.py ============================================================================== --- python/trunk/Lib/lib-tk/tkFont.py (original) +++ python/trunk/Lib/lib-tk/tkFont.py Wed Apr 12 17:28:49 2006 @@ -108,7 +108,9 @@ try: if self.delete_font: self._call("font", "delete", self.name) - except (AttributeError, Tkinter.TclError): + except (KeyboardInterrupt, SystemExit): + raise + except Exception: pass def copy(self): From python-checkins at python.org Wed Apr 12 17:28:53 2006 From: python-checkins at python.org (georg.brandl) Date: Wed, 12 Apr 2006 17:28:53 +0200 (CEST) Subject: [Python-checkins] r45311 - python/branches/release24-maint/Lib/lib-tk/tkFont.py Message-ID: <20060412152853.1092D1E4025@bag.python.org> Author: georg.brandl Date: Wed Apr 12 17:28:52 2006 New Revision: 45311 Modified: python/branches/release24-maint/Lib/lib-tk/tkFont.py Log: Patch #1468808: don't complain if Tkinter is already deleted at the time Font.__del__ is run. (backport from rev. 45310) Modified: python/branches/release24-maint/Lib/lib-tk/tkFont.py ============================================================================== --- python/branches/release24-maint/Lib/lib-tk/tkFont.py (original) +++ python/branches/release24-maint/Lib/lib-tk/tkFont.py Wed Apr 12 17:28:52 2006 @@ -117,7 +117,9 @@ try: if self.delete_font: self._call("font", "delete", self.name) - except (AttributeError, Tkinter.TclError): + except (KeyboardInterrupt, SystemExit): + raise + except Exception: pass def copy(self): From buildbot at python.org Wed Apr 12 18:39:12 2006 From: buildbot at python.org (buildbot at python.org) Date: Wed, 12 Apr 2006 16:39:12 +0000 Subject: [Python-checkins] buildbot warnings in ia64 Debian unstable 2.4 Message-ID: <20060412163912.737961E4010@bag.python.org> The Buildbot has detected a new failure of ia64 Debian unstable 2.4. Full details are available at: http://www.python.org/dev/buildbot/all/ia64%2520Debian%2520unstable%25202.4/builds/16 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch branches/release24-maint] HEAD Blamelist: georg.brandl Build Had Warnings: warnings failed slave lost sincerely, -The Buildbot From martin at v.loewis.de Wed Apr 12 18:53:04 2006 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 12 Apr 2006 18:53:04 +0200 Subject: [Python-checkins] r45306 - python/trunk/Doc/whatsnew/whatsnew25.tex In-Reply-To: <20060412121632.178601E4003@bag.python.org> References: <20060412121632.178601E4003@bag.python.org> Message-ID: <443D3070.3000402@v.loewis.de> andrew.kuchling wrote: > Mention access to ASTs Instead of 0x400, it is better to use _ast.PyCF_ONLY_AST. Regards, Martin From python-checkins at python.org Wed Apr 12 19:06:06 2006 From: python-checkins at python.org (armin.rigo) Date: Wed, 12 Apr 2006 19:06:06 +0200 (CEST) Subject: [Python-checkins] r45312 - in python/trunk: Include/object.h Objects/dictobject.c Objects/object.c Objects/setobject.c Python/pythonrun.c Python/sysmodule.c Message-ID: <20060412170606.492AB1E4003@bag.python.org> Author: armin.rigo Date: Wed Apr 12 19:06:05 2006 New Revision: 45312 Modified: python/trunk/Include/object.h python/trunk/Objects/dictobject.c python/trunk/Objects/object.c python/trunk/Objects/setobject.c python/trunk/Python/pythonrun.c python/trunk/Python/sysmodule.c Log: Ignore the references to the dummy objects used as deleted keys in dicts and sets when computing the total number of references. Modified: python/trunk/Include/object.h ============================================================================== --- python/trunk/Include/object.h (original) +++ python/trunk/Include/object.h Wed Apr 12 19:06:05 2006 @@ -578,6 +578,9 @@ PyAPI_DATA(Py_ssize_t) _Py_RefTotal; PyAPI_FUNC(void) _Py_NegativeRefcount(const char *fname, int lineno, PyObject *op); +PyAPI_FUNC(PyObject *) _PyDict_Dummy(void); +PyAPI_FUNC(PyObject *) _PySet_Dummy(void); +PyAPI_FUNC(Py_ssize_t) _Py_GetRefTotal(void); #define _Py_INC_REFTOTAL _Py_RefTotal++ #define _Py_DEC_REFTOTAL _Py_RefTotal-- #define _Py_REF_DEBUG_COMMA , Modified: python/trunk/Objects/dictobject.c ============================================================================== --- python/trunk/Objects/dictobject.c (original) +++ python/trunk/Objects/dictobject.c Wed Apr 12 19:06:05 2006 @@ -115,6 +115,14 @@ /* Object used as dummy key to fill deleted entries */ static PyObject *dummy = NULL; /* Initialized by first call to newdictobject() */ +#ifdef Py_REF_DEBUG +PyObject * +_PyDict_Dummy(void) +{ + return dummy; +} +#endif + /* forward declarations */ static dictentry * lookdict_string(dictobject *mp, PyObject *key, long hash); Modified: python/trunk/Objects/object.c ============================================================================== --- python/trunk/Objects/object.c (original) +++ python/trunk/Objects/object.c Wed Apr 12 19:06:05 2006 @@ -5,7 +5,24 @@ #ifdef Py_REF_DEBUG Py_ssize_t _Py_RefTotal; -#endif + +Py_ssize_t +_Py_GetRefTotal(void) +{ + PyObject *o; + Py_ssize_t total = _Py_RefTotal; + /* ignore the references to the dummy object of the dicts and sets + because they are not reliable and not useful (now that the + hash table code is well-tested) */ + o = _PyDict_Dummy(); + if (o != NULL) + total -= o->ob_refcnt; + o = _PySet_Dummy(); + if (o != NULL) + total -= o->ob_refcnt; + return total; +} +#endif /* Py_REF_DEBUG */ int Py_DivisionWarningFlag; Modified: python/trunk/Objects/setobject.c ============================================================================== --- python/trunk/Objects/setobject.c (original) +++ python/trunk/Objects/setobject.c Wed Apr 12 19:06:05 2006 @@ -16,6 +16,14 @@ /* Object used as dummy key to fill deleted entries */ static PyObject *dummy = NULL; /* Initialized by first call to make_new_set() */ +#ifdef Py_REF_DEBUG +PyObject * +_PySet_Dummy(void) +{ + return dummy; +} +#endif + #define INIT_NONZERO_SET_SLOTS(so) do { \ (so)->table = (so)->smalltable; \ (so)->mask = PySet_MINSIZE - 1; \ Modified: python/trunk/Python/pythonrun.c ============================================================================== --- python/trunk/Python/pythonrun.c (original) +++ python/trunk/Python/pythonrun.c Wed Apr 12 19:06:05 2006 @@ -34,7 +34,7 @@ #else /* Py_REF_DEBUG */ #define PRINT_TOTAL_REFS() fprintf(stderr, \ "[%" PY_FORMAT_SIZE_T "d refs]\n", \ - _Py_RefTotal) + _Py_GetRefTotal()) #endif extern char *Py_GetPath(void); Modified: python/trunk/Python/sysmodule.c ============================================================================== --- python/trunk/Python/sysmodule.c (original) +++ python/trunk/Python/sysmodule.c Wed Apr 12 19:06:05 2006 @@ -604,10 +604,9 @@ static PyObject * sys_gettotalrefcount(PyObject *self) { - return PyInt_FromSsize_t(_Py_RefTotal); + return PyInt_FromSsize_t(_Py_GetRefTotal()); } - -#endif /* Py_TRACE_REFS */ +#endif /* Py_REF_DEBUG */ PyDoc_STRVAR(getrefcount_doc, "getrefcount(object) -> integer\n\ From python-checkins at python.org Wed Apr 12 19:07:00 2006 From: python-checkins at python.org (armin.rigo) Date: Wed, 12 Apr 2006 19:07:00 +0200 (CEST) Subject: [Python-checkins] r45313 - in python/branches/release24-maint: Include/object.h Objects/dictobject.c Objects/object.c Python/pythonrun.c Python/sysmodule.c Message-ID: <20060412170700.EC0D51E4003@bag.python.org> Author: armin.rigo Date: Wed Apr 12 19:06:58 2006 New Revision: 45313 Modified: python/branches/release24-maint/Include/object.h python/branches/release24-maint/Objects/dictobject.c python/branches/release24-maint/Objects/object.c python/branches/release24-maint/Python/pythonrun.c python/branches/release24-maint/Python/sysmodule.c Log: Ignore the references to the dummy objects used as deleted keys in dicts and sets when computing the total number of references. Modified: python/branches/release24-maint/Include/object.h ============================================================================== --- python/branches/release24-maint/Include/object.h (original) +++ python/branches/release24-maint/Include/object.h Wed Apr 12 19:06:58 2006 @@ -556,6 +556,8 @@ PyAPI_DATA(long) _Py_RefTotal; PyAPI_FUNC(void) _Py_NegativeRefcount(const char *fname, int lineno, PyObject *op); +PyAPI_FUNC(PyObject *) _PyDict_Dummy(void); +PyAPI_FUNC(long) _Py_GetRefTotal(void); #define _Py_INC_REFTOTAL _Py_RefTotal++ #define _Py_DEC_REFTOTAL _Py_RefTotal-- #define _Py_REF_DEBUG_COMMA , Modified: python/branches/release24-maint/Objects/dictobject.c ============================================================================== --- python/branches/release24-maint/Objects/dictobject.c (original) +++ python/branches/release24-maint/Objects/dictobject.c Wed Apr 12 19:06:58 2006 @@ -115,6 +115,14 @@ /* Object used as dummy key to fill deleted entries */ static PyObject *dummy; /* Initialized by first call to newdictobject() */ +#ifdef Py_REF_DEBUG +PyObject * +_PyDict_Dummy(void) +{ + return dummy; +} +#endif + /* forward declarations */ static dictentry * lookdict_string(dictobject *mp, PyObject *key, long hash); Modified: python/branches/release24-maint/Objects/object.c ============================================================================== --- python/branches/release24-maint/Objects/object.c (original) +++ python/branches/release24-maint/Objects/object.c Wed Apr 12 19:06:58 2006 @@ -5,7 +5,20 @@ #ifdef Py_REF_DEBUG long _Py_RefTotal; -#endif +long +_Py_GetRefTotal(void) +{ + PyObject *o; + long total = _Py_RefTotal; + /* ignore the references to the dummy object of the dicts + because they are not reliable and not useful (now that the + hash table code is well-tested) */ + o = _PyDict_Dummy(); + if (o != NULL) + total -= o->ob_refcnt; + return total; +} +#endif /* Py_REF_DEBUG */ int Py_DivisionWarningFlag; Modified: python/branches/release24-maint/Python/pythonrun.c ============================================================================== --- python/branches/release24-maint/Python/pythonrun.c (original) +++ python/branches/release24-maint/Python/pythonrun.c Wed Apr 12 19:06:58 2006 @@ -379,7 +379,7 @@ #endif #ifdef Py_REF_DEBUG - fprintf(stderr, "[%ld refs]\n", _Py_RefTotal); + fprintf(stderr, "[%ld refs]\n", _Py_GetRefTotal()); #endif #ifdef Py_TRACE_REFS @@ -694,7 +694,7 @@ for (;;) { ret = PyRun_InteractiveOneFlags(fp, filename, flags); #ifdef Py_REF_DEBUG - fprintf(stderr, "[%ld refs]\n", _Py_RefTotal); + fprintf(stderr, "[%ld refs]\n", _Py_GetRefTotal()); #endif if (ret == E_EOF) return 0; Modified: python/branches/release24-maint/Python/sysmodule.c ============================================================================== --- python/branches/release24-maint/Python/sysmodule.c (original) +++ python/branches/release24-maint/Python/sysmodule.c Wed Apr 12 19:06:58 2006 @@ -604,10 +604,9 @@ static PyObject * sys_gettotalrefcount(PyObject *self) { - return PyInt_FromLong(_Py_RefTotal); + return PyInt_FromLong(_Py_GetRefTotal()); } - -#endif /* Py_TRACE_REFS */ +#endif /* Py_REF_DEBUG */ PyDoc_STRVAR(getrefcount_doc, "getrefcount(object) -> integer\n\ From buildbot at python.org Wed Apr 12 19:27:22 2006 From: buildbot at python.org (buildbot at python.org) Date: Wed, 12 Apr 2006 17:27:22 +0000 Subject: [Python-checkins] buildbot warnings in x86 gentoo trunk Message-ID: <20060412172722.82C741E4056@bag.python.org> The Buildbot has detected a new failure of x86 gentoo trunk. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520gentoo%2520trunk/builds/418 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: armin.rigo Build Had Warnings: warnings test sincerely, -The Buildbot From python-checkins at python.org Wed Apr 12 20:52:11 2006 From: python-checkins at python.org (andrew.kuchling) Date: Wed, 12 Apr 2006 20:52:11 +0200 (CEST) Subject: [Python-checkins] r45314 - python/trunk/Doc/whatsnew/whatsnew21.tex Message-ID: <20060412185211.E70A71E4003@bag.python.org> Author: andrew.kuchling Date: Wed Apr 12 20:52:09 2006 New Revision: 45314 Modified: python/trunk/Doc/whatsnew/whatsnew21.tex Log: Bump version number; rearrange introduction a bit Modified: python/trunk/Doc/whatsnew/whatsnew21.tex ============================================================================== --- python/trunk/Doc/whatsnew/whatsnew21.tex (original) +++ python/trunk/Doc/whatsnew/whatsnew21.tex Wed Apr 12 20:52:09 2006 @@ -5,7 +5,7 @@ % $Id$ \title{What's New in Python 2.1} -\release{1.00} +\release{1.01} \author{A.M. Kuchling} \authoraddress{ \strong{Python Software Foundation}\\ @@ -16,14 +16,7 @@ \section{Introduction} -It's that time again... time for a new Python release, Python 2.1. -One recent goal of the Python development team has been to accelerate -the pace of new releases, with a new release coming every 6 to 9 -months. 2.1 is the first release to come out at this faster pace, with -the first alpha appearing in January, 3 months after the final version -of 2.0 was released. - -This article explains the new features in 2.1. While there aren't as +This article explains the new features in Python 2.1. While there aren't as many changes in 2.1 as there were in Python 2.0, there are still some pleasant surprises in store. 2.1 is the first release to be steered through the use of Python Enhancement Proposals, or PEPs, so most of @@ -34,6 +27,12 @@ Refer to the Python 2.1 documentation, or to the specific PEP, for more details about any new feature that particularly interests you. +One recent goal of the Python development team has been to accelerate +the pace of new releases, with a new release coming every 6 to 9 +months. 2.1 is the first release to come out at this faster pace, with +the first alpha appearing in January, 3 months after the final version +of 2.0 was released. + The final release of Python 2.1 was made on April 17, 2001. %====================================================================== From python-checkins at python.org Wed Apr 12 20:54:01 2006 From: python-checkins at python.org (andrew.kuchling) Date: Wed, 12 Apr 2006 20:54:01 +0200 (CEST) Subject: [Python-checkins] r45315 - python/trunk/Doc/whatsnew/whatsnew25.tex Message-ID: <20060412185401.22C331E4003@bag.python.org> Author: andrew.kuchling Date: Wed Apr 12 20:54:00 2006 New Revision: 45315 Modified: python/trunk/Doc/whatsnew/whatsnew25.tex Log: Mention ASCII as default encoding; update TODO list; use PyCF_ONLY_AST by MvL's suggestion; typographical tidying of MvL's name Modified: python/trunk/Doc/whatsnew/whatsnew25.tex ============================================================================== --- python/trunk/Doc/whatsnew/whatsnew25.tex (original) +++ python/trunk/Doc/whatsnew/whatsnew25.tex Wed Apr 12 20:54:00 2006 @@ -5,7 +5,8 @@ % Fix XXX comments % The easy_install stuff % Stateful codec changes -% ASCII is now default encoding for modules +% Write ctypes examples +% Count up the patches and bugs \title{What's New in Python 2.5} \release{0.1} @@ -53,7 +54,7 @@ \begin{seealso} \seepep{243}{Module Repository Upload Mechanism}{PEP written by -Sean Reifschneider; implemented by Martin von L\"owis +Sean Reifschneider; implemented by Martin von~L\"owis and Richard Jones. Note that the PEP doesn't exactly describe what's implemented in PyPI.} @@ -783,7 +784,7 @@ \begin{seealso} -\seepep{353}{Using ssize_t as the index type}{PEP written and implemented by Martin von L\"owis.} +\seepep{353}{Using ssize_t as the index type}{PEP written and implemented by Martin von~L\"owis.} \end{seealso} @@ -868,6 +869,17 @@ all of the values returned by the iterator evaluate as being true. (Suggested by GvR, and implemented by Raymond Hettinger.) +\item ASCII is now the default encoding for modules. It's now +a syntax error if a module contains string literals with 8-bit +characters but doesn't have an encoding declaration. In Python 2.4 +this triggered a warning, not a syntax error. See \pep{263} +for how to declare a module's encoding; for example, you might add +a line like this near the top of the source file: + +\begin{verbatim} +# -*- coding: latin1 -*- +\end{verbatim} + \item The list of base classes in a class definition can now be empty. As an example, this is now legal: @@ -1056,7 +1068,7 @@ family, type, and protocol values for the socket. \item New module: \module{spwd} provides functions for accessing the -shadow password database on systems that support it. +shadow password database on systems that support it. % XXX give example % XXX patch #1382163: sys.subversion, Py_GetBuildNumber() @@ -1095,8 +1107,6 @@ %====================================================================== % whole new modules get described in subsections here -% XXX new distutils features: upload - \subsection{The ctypes package} The \module{ctypes} package, written by Thomas Heller, has been added @@ -1405,14 +1415,16 @@ the abstract syntax tree that's traversed to produce the bytecode. It's possible for Python code to obtain AST objects by using the -\function{compile()} built-in and specifying 0x400 as the value of the +\function{compile()} built-in and specifying \code{_ast.PyCF_ONLY_AST} +as the value of the \var{flags} parameter: \begin{verbatim} +from _ast import PyCF_ONLY_AST ast = compile("""a=0 for i in range(10): a += i -""", "", 'exec', 0x0400) +""", "", 'exec', PyCF_ONLY_AST) assignment = ast.body[0] for_loop = ast.body[1] @@ -1512,6 +1524,11 @@ \begin{itemize} +\item ASCII is now the default encoding for modules. It's now +a syntax error if a module contains string literals with 8-bit +characters but doesn't have an encoding declaration. In Python 2.4 +this triggered a warning, not a syntax error. + \item The \module{pickle} module no longer uses the deprecated \var{bin} parameter. \item C API: Many functions now use \ctype{Py_ssize_t} @@ -1537,6 +1554,6 @@ The author would like to thank the following people for offering suggestions, corrections and assistance with various drafts of this -article: Mike Rovner, Thomas Wouters. +article: Martin von~L\"owis, Mike Rovner, Thomas Wouters. \end{document} From buildbot at python.org Wed Apr 12 20:57:08 2006 From: buildbot at python.org (buildbot at python.org) Date: Wed, 12 Apr 2006 18:57:08 +0000 Subject: [Python-checkins] buildbot warnings in x86 Ubuntu dapper (icc) 2.4 Message-ID: <20060412185708.487AD1E4009@bag.python.org> The Buildbot has detected a new failure of x86 Ubuntu dapper (icc) 2.4. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520Ubuntu%2520dapper%2520%2528icc%2529%25202.4/builds/21 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch branches/release24-maint] HEAD Blamelist: armin.rigo Build Had Warnings: warnings test sincerely, -The Buildbot From python-checkins at python.org Wed Apr 12 21:07:15 2006 From: python-checkins at python.org (phillip.eby) Date: Wed, 12 Apr 2006 21:07:15 +0200 (CEST) Subject: [Python-checkins] r45316 - in python/trunk: Include/genobject.h Objects/genobject.c Message-ID: <20060412190715.BF0121E4003@bag.python.org> Author: phillip.eby Date: Wed Apr 12 21:07:15 2006 New Revision: 45316 Modified: python/trunk/Include/genobject.h python/trunk/Objects/genobject.c Log: Don't set gi_frame to Py_None, use NULL instead, eliminating some insane pointer dereferences. Modified: python/trunk/Include/genobject.h ============================================================================== --- python/trunk/Include/genobject.h (original) +++ python/trunk/Include/genobject.h Wed Apr 12 21:07:15 2006 @@ -13,6 +13,7 @@ PyObject_HEAD /* The gi_ prefix is intended to remind of generator-iterator. */ + /* Note: gi_frame can be NULL if the generator is "finished" */ struct _frame *gi_frame; /* True if generator is being executed. */ Modified: python/trunk/Objects/genobject.c ============================================================================== --- python/trunk/Objects/genobject.c (original) +++ python/trunk/Objects/genobject.c Wed Apr 12 21:07:15 2006 @@ -10,7 +10,8 @@ static int gen_traverse(PyGenObject *gen, visitproc visit, void *arg) { - return visit((PyObject *)gen->gi_frame, arg); + Py_VISIT(gen->gi_frame); + return 0; } static void @@ -26,7 +27,7 @@ _PyObject_GC_TRACK(self); - if (gen->gi_frame->f_stacktop!=NULL) { + if (gen->gi_frame!=NULL && gen->gi_frame->f_stacktop!=NULL) { /* Generator is paused, so we need to close */ gen->ob_type->tp_del(self); if (self->ob_refcnt > 0) @@ -51,7 +52,7 @@ "generator already executing"); return NULL; } - if ((PyObject *)f == Py_None || f->f_stacktop == NULL) { + if (f==NULL || f->f_stacktop == NULL) { /* Only set exception if called from send() */ if (arg && !exc) PyErr_SetNone(PyExc_StopIteration); return NULL; @@ -98,8 +99,7 @@ if (!result || f->f_stacktop == NULL) { /* generator can't be rerun, so release the frame */ Py_DECREF(f); - gen->gi_frame = (PyFrameObject *)Py_None; - Py_INCREF(Py_None); + gen->gi_frame = NULL; } return result; @@ -147,7 +147,7 @@ PyObject *error_type, *error_value, *error_traceback; PyGenObject *gen = (PyGenObject *)self; - if ((PyObject *)gen->gi_frame == Py_None || gen->gi_frame->f_stacktop==NULL) + if (!gen->gi_frame || gen->gi_frame->f_stacktop==NULL) /* Generator isn't paused, so no need to close */ return; @@ -366,7 +366,7 @@ int i; PyFrameObject *f = gen->gi_frame; - if ((PyObject *)f == Py_None || f->f_stacktop==NULL || f->f_iblock<=0) + if (f == NULL || f->f_stacktop==NULL || f->f_iblock<=0) return 0; /* no frame or no blockstack == no finalization */ for (i=f->f_iblock; i>=0; i--) { From python-checkins at python.org Wed Apr 12 21:07:36 2006 From: python-checkins at python.org (thomas.heller) Date: Wed, 12 Apr 2006 21:07:36 +0200 (CEST) Subject: [Python-checkins] r45317 - in python/trunk: Lib/ctypes/test/test_pointers.py Modules/_ctypes/_ctypes.c Message-ID: <20060412190736.F198D1E4003@bag.python.org> Author: thomas.heller Date: Wed Apr 12 21:07:36 2006 New Revision: 45317 Modified: python/trunk/Lib/ctypes/test/test_pointers.py python/trunk/Modules/_ctypes/_ctypes.c Log: Fix for a bug found by Armin Rigo, plus test. https://sourceforge.net/tracker/?func=detail&atid=532154&aid=1467852&group_id=71702 Modified: python/trunk/Lib/ctypes/test/test_pointers.py ============================================================================== --- python/trunk/Lib/ctypes/test/test_pointers.py (original) +++ python/trunk/Lib/ctypes/test/test_pointers.py Wed Apr 12 21:07:36 2006 @@ -166,6 +166,18 @@ result = func( byref(argc), argv ) assert result == 'world', result + def test_bug_1467852(self): + # http://sourceforge.net/tracker/?func=detail&atid=532154&aid=1467852&group_id=71702 + x = c_int(5) + dummy = [] + for i in range(32000): + dummy.append(c_int(i)) + y = c_int(6) + p = pointer(x) + pp = pointer(p) + q = pointer(y) + pp[0] = q # <== + self.failUnlessEqual(p[0], 6) if __name__ == '__main__': unittest.main() Modified: python/trunk/Modules/_ctypes/_ctypes.c ============================================================================== --- python/trunk/Modules/_ctypes/_ctypes.c (original) +++ python/trunk/Modules/_ctypes/_ctypes.c Wed Apr 12 21:07:36 2006 @@ -548,7 +548,7 @@ return NULL; stgdict->size = sizeof(void *); stgdict->align = getentry("P")->pffi_type->alignment; - stgdict->length = 2; + stgdict->length = 1; stgdict->ffi_type = ffi_type_pointer; proto = PyDict_GetItemString(typedict, "_type_"); /* Borrowed ref */ From buildbot at python.org Wed Apr 12 21:59:53 2006 From: buildbot at python.org (buildbot at python.org) Date: Wed, 12 Apr 2006 19:59:53 +0000 Subject: [Python-checkins] buildbot warnings in x86 Ubuntu dapper (icc) trunk Message-ID: <20060412195953.87E111E4003@bag.python.org> The Buildbot has detected a new failure of x86 Ubuntu dapper (icc) trunk. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520Ubuntu%2520dapper%2520%2528icc%2529%2520trunk/builds/102 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: andrew.kuchling Build Had Warnings: warnings test sincerely, -The Buildbot From python-checkins at python.org Wed Apr 12 22:16:57 2006 From: python-checkins at python.org (gregory.p.smith) Date: Wed, 12 Apr 2006 22:16:57 +0200 (CEST) Subject: [Python-checkins] r45318 - in python/trunk/Lib: bsddb/__init__.py test/test_bsddb.py Message-ID: <20060412201657.698C81E4003@bag.python.org> Author: gregory.p.smith Date: Wed Apr 12 22:16:56 2006 New Revision: 45318 Modified: python/trunk/Lib/bsddb/__init__.py python/trunk/Lib/test/test_bsddb.py Log: Closes bug #1149413 Using None for a filename with the 'n' flag when calling bsddb.btopen would cause an error while checking if the file None existed. error not likely to be seen as anyone using None for a filename would likely use the 'c' flag in the first place. Modified: python/trunk/Lib/bsddb/__init__.py ============================================================================== --- python/trunk/Lib/bsddb/__init__.py (original) +++ python/trunk/Lib/bsddb/__init__.py Wed Apr 12 22:16:56 2006 @@ -358,7 +358,7 @@ #flags = db.DB_CREATE | db.DB_TRUNCATE # we used db.DB_TRUNCATE flag for this before but BerkeleyDB # 4.2.52 changed to disallowed truncate with txn environments. - if os.path.isfile(file): + if file is not None and os.path.isfile(file): os.unlink(file) else: raise error, "flags should be one of 'r', 'w', 'c' or 'n'" Modified: python/trunk/Lib/test/test_bsddb.py ============================================================================== --- python/trunk/Lib/test/test_bsddb.py (original) +++ python/trunk/Lib/test/test_bsddb.py Wed Apr 12 22:16:56 2006 @@ -11,9 +11,10 @@ from sets import Set class TestBSDDB(unittest.TestCase): + openflag = 'c' def setUp(self): - self.f = self.openmethod[0](self.fname, 'c') + self.f = self.openmethod[0](self.fname, self.openflag) self.d = dict(q='Guido', w='van', e='Rossum', r='invented', t='Python', y='') for k, v in self.d.iteritems(): self.f[k] = v @@ -267,6 +268,11 @@ fname = None openmethod = [bsddb.btopen] +class TestBTree_InMemory_Truncate(TestBSDDB): + fname = None + openflag = 'n' + openmethod = [bsddb.btopen] + class TestHashTable(TestBSDDB): fname = test_support.TESTFN openmethod = [bsddb.hashopen] @@ -285,6 +291,7 @@ TestHashTable, TestBTree_InMemory, TestHashTable_InMemory, + TestBTree_InMemory_Truncate, ) if __name__ == "__main__": From python-checkins at python.org Wed Apr 12 22:35:02 2006 From: python-checkins at python.org (gregory.p.smith) Date: Wed, 12 Apr 2006 22:35:02 +0200 (CEST) Subject: [Python-checkins] r45319 - in python/trunk/Lib: bsddb/__init__.py test/test_bsddb.py Message-ID: <20060412203502.A194B1E4009@bag.python.org> Author: gregory.p.smith Date: Wed Apr 12 22:35:02 2006 New Revision: 45319 Modified: python/trunk/Lib/bsddb/__init__.py python/trunk/Lib/test/test_bsddb.py Log: Fixes bug #1117761 bsddb.*open() methods cachesize parameter wouldn't work (raised an internal bsddb.db exception when it was given). The set_cachesize call needed to be moved from the DB object to the DBEnv since the env was introduced to allow for threading. (will backport to 2.4) Modified: python/trunk/Lib/bsddb/__init__.py ============================================================================== --- python/trunk/Lib/bsddb/__init__.py (original) +++ python/trunk/Lib/bsddb/__init__.py Wed Apr 12 22:35:02 2006 @@ -287,10 +287,9 @@ cachesize=None, lorder=None, hflags=0): flags = _checkflag(flag, file) - e = _openDBEnv() + e = _openDBEnv(cachesize) d = db.DB(e) d.set_flags(hflags) - if cachesize is not None: d.set_cachesize(0, cachesize) if pgsize is not None: d.set_pagesize(pgsize) if lorder is not None: d.set_lorder(lorder) if ffactor is not None: d.set_h_ffactor(ffactor) @@ -305,9 +304,8 @@ pgsize=None, lorder=None): flags = _checkflag(flag, file) - e = _openDBEnv() + e = _openDBEnv(cachesize) d = db.DB(e) - if cachesize is not None: d.set_cachesize(0, cachesize) if pgsize is not None: d.set_pagesize(pgsize) if lorder is not None: d.set_lorder(lorder) d.set_flags(btflags) @@ -324,9 +322,8 @@ rlen=None, delim=None, source=None, pad=None): flags = _checkflag(flag, file) - e = _openDBEnv() + e = _openDBEnv(cachesize) d = db.DB(e) - if cachesize is not None: d.set_cachesize(0, cachesize) if pgsize is not None: d.set_pagesize(pgsize) if lorder is not None: d.set_lorder(lorder) d.set_flags(rnflags) @@ -339,8 +336,13 @@ #---------------------------------------------------------------------- -def _openDBEnv(): +def _openDBEnv(cachesize): e = db.DBEnv() + if cachesize is not None: + if cachesize >= 20480: + e.set_cachesize(0, cachesize) + else: + raise error, "cachesize must be >= 20480" e.open('.', db.DB_PRIVATE | db.DB_CREATE | db.DB_THREAD | db.DB_INIT_LOCK | db.DB_INIT_MPOOL) return e Modified: python/trunk/Lib/test/test_bsddb.py ============================================================================== --- python/trunk/Lib/test/test_bsddb.py (original) +++ python/trunk/Lib/test/test_bsddb.py Wed Apr 12 22:35:02 2006 @@ -14,7 +14,7 @@ openflag = 'c' def setUp(self): - self.f = self.openmethod[0](self.fname, self.openflag) + self.f = self.openmethod[0](self.fname, self.openflag, cachesize=32768) self.d = dict(q='Guido', w='van', e='Rossum', r='invented', t='Python', y='') for k, v in self.d.iteritems(): self.f[k] = v From python-checkins at python.org Wed Apr 12 22:37:59 2006 From: python-checkins at python.org (gregory.p.smith) Date: Wed, 12 Apr 2006 22:37:59 +0200 (CEST) Subject: [Python-checkins] r45320 - in python/branches/release24-maint/Lib: bsddb/__init__.py test/test_bsddb.py Message-ID: <20060412203759.3D09C1E400B@bag.python.org> Author: gregory.p.smith Date: Wed Apr 12 22:37:58 2006 New Revision: 45320 Modified: python/branches/release24-maint/Lib/bsddb/__init__.py python/branches/release24-maint/Lib/test/test_bsddb.py Log: backport 43519. fixes SF bug 1117761. bsddb.*open() methods were failing when the cachesize parameter was supplied. Modified: python/branches/release24-maint/Lib/bsddb/__init__.py ============================================================================== --- python/branches/release24-maint/Lib/bsddb/__init__.py (original) +++ python/branches/release24-maint/Lib/bsddb/__init__.py Wed Apr 12 22:37:58 2006 @@ -288,10 +288,9 @@ cachesize=None, lorder=None, hflags=0): flags = _checkflag(flag, file) - e = _openDBEnv() + e = _openDBEnv(cachesize) d = db.DB(e) d.set_flags(hflags) - if cachesize is not None: d.set_cachesize(0, cachesize) if pgsize is not None: d.set_pagesize(pgsize) if lorder is not None: d.set_lorder(lorder) if ffactor is not None: d.set_h_ffactor(ffactor) @@ -306,9 +305,8 @@ pgsize=None, lorder=None): flags = _checkflag(flag, file) - e = _openDBEnv() + e = _openDBEnv(cachesize) d = db.DB(e) - if cachesize is not None: d.set_cachesize(0, cachesize) if pgsize is not None: d.set_pagesize(pgsize) if lorder is not None: d.set_lorder(lorder) d.set_flags(btflags) @@ -325,9 +323,8 @@ rlen=None, delim=None, source=None, pad=None): flags = _checkflag(flag, file) - e = _openDBEnv() + e = _openDBEnv(cachesize) d = db.DB(e) - if cachesize is not None: d.set_cachesize(0, cachesize) if pgsize is not None: d.set_pagesize(pgsize) if lorder is not None: d.set_lorder(lorder) d.set_flags(rnflags) @@ -340,8 +337,13 @@ #---------------------------------------------------------------------- -def _openDBEnv(): +def _openDBEnv(cachesize): e = db.DBEnv() + if cachesize is not None: + if cachesize >= 20480: + e.set_cachesize(0, cachesize) + else: + raise error, "cachesize must be >= 20480" e.open('.', db.DB_PRIVATE | db.DB_CREATE | db.DB_THREAD | db.DB_INIT_LOCK | db.DB_INIT_MPOOL) return e Modified: python/branches/release24-maint/Lib/test/test_bsddb.py ============================================================================== --- python/branches/release24-maint/Lib/test/test_bsddb.py (original) +++ python/branches/release24-maint/Lib/test/test_bsddb.py Wed Apr 12 22:37:58 2006 @@ -13,7 +13,7 @@ class TestBSDDB(unittest.TestCase): def setUp(self): - self.f = self.openmethod[0](self.fname, 'c') + self.f = self.openmethod[0](self.fname, 'c', cachesize=32768) self.d = dict(q='Guido', w='van', e='Rossum', r='invented', t='Python', y='') for k, v in self.d.iteritems(): self.f[k] = v From python-checkins at python.org Wed Apr 12 23:14:09 2006 From: python-checkins at python.org (georg.brandl) Date: Wed, 12 Apr 2006 23:14:09 +0200 (CEST) Subject: [Python-checkins] r45321 - in python/trunk: Lib/test/test_traceback.py Lib/traceback.py Misc/NEWS Message-ID: <20060412211409.000EF1E4003@bag.python.org> Author: georg.brandl Date: Wed Apr 12 23:14:09 2006 New Revision: 45321 Modified: python/trunk/Lib/test/test_traceback.py python/trunk/Lib/traceback.py python/trunk/Misc/NEWS Log: Patch #860326: traceback.format_exception_only() now prepends the exception's module name to non-builtin exceptions, like the interpreter itself does. Modified: python/trunk/Lib/test/test_traceback.py ============================================================================== --- python/trunk/Lib/test/test_traceback.py (original) +++ python/trunk/Lib/test/test_traceback.py Wed Apr 12 23:14:09 2006 @@ -5,6 +5,9 @@ import traceback +class TbError(Exception): + pass + class TracebackCases(unittest.TestCase): # For now, a very minimal set of tests. I want to be sure that # formatting of SyntaxErrors works based on changes for 2.1. @@ -103,6 +106,24 @@ import sys sys.exc_traceback.__members__ + def raise_tberror(self): + raise TbError + + def raise_typeerror(self): + raise TypeError + + def test_modulename(self): + # Bug 860326: format_exception_only should prepend module name + # to exceptions not in "exceptions", like PyErr_Print does. + err = self.get_exception_format(self.raise_tberror, TbError) + self.assertEquals(len(err), 1) + self.assert_(err[0] == '__main__.TbError\n' or + err[0] == 'test.test_traceback.TbError\n') + + err = self.get_exception_format(self.raise_typeerror, TypeError) + self.assertEquals(err[0], 'TypeError\n') + + def test_main(): run_unittest(TracebackCases) Modified: python/trunk/Lib/traceback.py ============================================================================== --- python/trunk/Lib/traceback.py (original) +++ python/trunk/Lib/traceback.py Wed Apr 12 23:14:09 2006 @@ -158,8 +158,12 @@ """ list = [] if (type(etype) == types.ClassType - or (isinstance(etype, type) and issubclass(etype, Exception))): + or (isinstance(etype, type) and issubclass(etype, BaseException))): stype = etype.__name__ + if not hasattr(etype, '__module__'): + stype = '.' + stype + elif etype.__module__ != 'exceptions': + stype = etype.__module__ + '.' + stype else: stype = etype if value is None: Modified: python/trunk/Misc/NEWS ============================================================================== --- python/trunk/Misc/NEWS (original) +++ python/trunk/Misc/NEWS Wed Apr 12 23:14:09 2006 @@ -49,6 +49,10 @@ Library ------- +- Patch #860326: traceback.format_exception_only() now prepends the + exception's module name to non-builtin exceptions, like the interpreter + itself does. + - SimpleXMLRPCServer relied on the fcntl module, which is unavailable on Windows. Bug #1469163. From python-checkins at python.org Wed Apr 12 23:14:13 2006 From: python-checkins at python.org (georg.brandl) Date: Wed, 12 Apr 2006 23:14:13 +0200 (CEST) Subject: [Python-checkins] r45322 - in python/branches/release24-maint: Lib/test/test_traceback.py Lib/traceback.py Misc/NEWS Message-ID: <20060412211413.E2E121E4003@bag.python.org> Author: georg.brandl Date: Wed Apr 12 23:14:12 2006 New Revision: 45322 Modified: python/branches/release24-maint/Lib/test/test_traceback.py python/branches/release24-maint/Lib/traceback.py python/branches/release24-maint/Misc/NEWS Log: Patch #860326: traceback.format_exception_only() now prepends the exception's module name to non-builtin exceptions, like the interpreter itself does. (backport from rev. 45321) Modified: python/branches/release24-maint/Lib/test/test_traceback.py ============================================================================== --- python/branches/release24-maint/Lib/test/test_traceback.py (original) +++ python/branches/release24-maint/Lib/test/test_traceback.py Wed Apr 12 23:14:12 2006 @@ -5,6 +5,9 @@ import traceback +class TbError(Exception): + pass + class TracebackCases(unittest.TestCase): # For now, a very minimal set of tests. I want to be sure that # formatting of SyntaxErrors works based on changes for 2.1. @@ -85,6 +88,24 @@ os.unlink(os.path.join(testdir, f)) os.rmdir(testdir) + def raise_tberror(self): + raise TbError + + def raise_typeerror(self): + raise TypeError + + def test_modulename(self): + # Bug 860326: format_exception_only should prepend module name + # to exceptions not in "exceptions", like PyErr_Print does. + err = self.get_exception_format(self.raise_tberror, TbError) + self.assertEquals(len(err), 1) + self.assert_(err[0] == '__main__.TbError\n' or + err[0] == 'test.test_traceback.TbError\n') + + err = self.get_exception_format(self.raise_typeerror, TypeError) + self.assertEquals(err[0], 'TypeError\n') + + def test_main(): run_unittest(TracebackCases) Modified: python/branches/release24-maint/Lib/traceback.py ============================================================================== --- python/branches/release24-maint/Lib/traceback.py (original) +++ python/branches/release24-maint/Lib/traceback.py Wed Apr 12 23:14:12 2006 @@ -159,6 +159,10 @@ list = [] if type(etype) == types.ClassType: stype = etype.__name__ + if not hasattr(etype, '__module__'): + stype = '.' + stype + elif etype.__module__ != 'exceptions': + stype = etype.__module__ + '.' + stype else: stype = etype if value is None: Modified: python/branches/release24-maint/Misc/NEWS ============================================================================== --- python/branches/release24-maint/Misc/NEWS (original) +++ python/branches/release24-maint/Misc/NEWS Wed Apr 12 23:14:12 2006 @@ -23,6 +23,10 @@ Library ------- +- Patch #860326: traceback.format_exception_only() now prepends the + exception's module name to non-builtin exceptions, like the interpreter + itself does. + - The email module's parsedate_tz function now sets the daylight savings flag to -1 (unknown) since it can't tell from the date whether it should be set. From neal at metaslash.com Wed Apr 12 23:23:09 2006 From: neal at metaslash.com (Neal Norwitz) Date: Wed, 12 Apr 2006 17:23:09 -0400 Subject: [Python-checkins] Python Regression Test Failures refleak (1) Message-ID: <20060412212309.GA26856@python.psfb.org> test_cmd_line leaked [0, -34, 17] references test_generators leaked [1, 1, 1] references test_threadedtempfile leaked [-85, 0, 0] references test_threading_local leaked [42, 32, 24] references test_urllib2 leaked [-121, 88, 99] references From buildbot at python.org Wed Apr 12 23:36:06 2006 From: buildbot at python.org (buildbot at python.org) Date: Wed, 12 Apr 2006 21:36:06 +0000 Subject: [Python-checkins] buildbot warnings in amd64 gentoo trunk Message-ID: <20060412213606.D73D51E400B@bag.python.org> The Buildbot has detected a new failure of amd64 gentoo trunk. Full details are available at: http://www.python.org/dev/buildbot/all/amd64%2520gentoo%2520trunk/builds/437 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: georg.brandl Build Had Warnings: warnings test sincerely, -The Buildbot From buildbot at python.org Wed Apr 12 23:38:48 2006 From: buildbot at python.org (buildbot at python.org) Date: Wed, 12 Apr 2006 21:38:48 +0000 Subject: [Python-checkins] buildbot warnings in x86 XP-2 trunk Message-ID: <20060412213848.00A101E4003@bag.python.org> The Buildbot has detected a new failure of x86 XP-2 trunk. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520XP-2%2520trunk/builds/250 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: georg.brandl Build Had Warnings: warnings test sincerely, -The Buildbot From buildbot at python.org Wed Apr 12 23:41:13 2006 From: buildbot at python.org (buildbot at python.org) Date: Wed, 12 Apr 2006 21:41:13 +0000 Subject: [Python-checkins] buildbot warnings in x86 W2k trunk Message-ID: <20060412214114.21A2F1E402F@bag.python.org> The Buildbot has detected a new failure of x86 W2k trunk. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520W2k%2520trunk/builds/415 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: georg.brandl Build Had Warnings: warnings test sincerely, -The Buildbot From buildbot at python.org Wed Apr 12 23:46:19 2006 From: buildbot at python.org (buildbot at python.org) Date: Wed, 12 Apr 2006 21:46:19 +0000 Subject: [Python-checkins] buildbot warnings in x86 gentoo 2.4 Message-ID: <20060412214619.489701E4003@bag.python.org> The Buildbot has detected a new failure of x86 gentoo 2.4. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520gentoo%25202.4/builds/87 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch branches/release24-maint] HEAD Blamelist: georg.brandl Build Had Warnings: warnings test sincerely, -The Buildbot From buildbot at python.org Wed Apr 12 23:46:30 2006 From: buildbot at python.org (buildbot at python.org) Date: Wed, 12 Apr 2006 21:46:30 +0000 Subject: [Python-checkins] buildbot warnings in x86 XP trunk Message-ID: <20060412214630.ED38C1E4003@bag.python.org> The Buildbot has detected a new failure of x86 XP trunk. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520XP%2520trunk/builds/388 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: gregory.p.smith Build Had Warnings: warnings test sincerely, -The Buildbot From buildbot at python.org Wed Apr 12 23:49:50 2006 From: buildbot at python.org (buildbot at python.org) Date: Wed, 12 Apr 2006 21:49:50 +0000 Subject: [Python-checkins] buildbot warnings in amd64 gentoo 2.4 Message-ID: <20060412214950.715361E4003@bag.python.org> The Buildbot has detected a new failure of amd64 gentoo 2.4. Full details are available at: http://www.python.org/dev/buildbot/all/amd64%2520gentoo%25202.4/builds/90 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch branches/release24-maint] HEAD Blamelist: georg.brandl Build Had Warnings: warnings test sincerely, -The Buildbot From buildbot at python.org Wed Apr 12 23:55:56 2006 From: buildbot at python.org (buildbot at python.org) Date: Wed, 12 Apr 2006 21:55:56 +0000 Subject: [Python-checkins] buildbot warnings in x86 XP-2 2.4 Message-ID: <20060412215556.7D97B1E4003@bag.python.org> The Buildbot has detected a new failure of x86 XP-2 2.4. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520XP-2%25202.4/builds/69 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch branches/release24-maint] HEAD Blamelist: georg.brandl Build Had Warnings: warnings test sincerely, -The Buildbot From buildbot at python.org Wed Apr 12 23:57:52 2006 From: buildbot at python.org (buildbot at python.org) Date: Wed, 12 Apr 2006 21:57:52 +0000 Subject: [Python-checkins] buildbot warnings in ppc Debian unstable 2.4 Message-ID: <20060412215752.CE6CA1E4003@bag.python.org> The Buildbot has detected a new failure of ppc Debian unstable 2.4. Full details are available at: http://www.python.org/dev/buildbot/all/ppc%2520Debian%2520unstable%25202.4/builds/19 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch branches/release24-maint] HEAD Blamelist: georg.brandl Build Had Warnings: warnings test sincerely, -The Buildbot From buildbot at python.org Wed Apr 12 23:59:31 2006 From: buildbot at python.org (buildbot at python.org) Date: Wed, 12 Apr 2006 21:59:31 +0000 Subject: [Python-checkins] buildbot warnings in x86 OpenBSD 2.4 Message-ID: <20060412215931.CACF11E4003@bag.python.org> The Buildbot has detected a new failure of x86 OpenBSD 2.4. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520OpenBSD%25202.4/builds/54 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch branches/release24-maint] HEAD Blamelist: georg.brandl Build Had Warnings: warnings test sincerely, -The Buildbot From buildbot at python.org Wed Apr 12 23:59:59 2006 From: buildbot at python.org (buildbot at python.org) Date: Wed, 12 Apr 2006 21:59:59 +0000 Subject: [Python-checkins] buildbot warnings in x86 W2k 2.4 Message-ID: <20060412215959.EA0431E4003@bag.python.org> The Buildbot has detected a new failure of x86 W2k 2.4. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520W2k%25202.4/builds/85 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch branches/release24-maint] HEAD Blamelist: georg.brandl Build Had Warnings: warnings test sincerely, -The Buildbot From buildbot at python.org Thu Apr 13 00:05:27 2006 From: buildbot at python.org (buildbot at python.org) Date: Wed, 12 Apr 2006 22:05:27 +0000 Subject: [Python-checkins] buildbot warnings in x86 gentoo trunk Message-ID: <20060412220527.BF8471E4009@bag.python.org> The Buildbot has detected a new failure of x86 gentoo trunk. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520gentoo%2520trunk/builds/423 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: georg.brandl Build Had Warnings: warnings test sincerely, -The Buildbot From buildbot at python.org Thu Apr 13 00:12:55 2006 From: buildbot at python.org (buildbot at python.org) Date: Wed, 12 Apr 2006 22:12:55 +0000 Subject: [Python-checkins] buildbot warnings in x86 XP 2.4 Message-ID: <20060412221255.347CC1E4003@bag.python.org> The Buildbot has detected a new failure of x86 XP 2.4. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520XP%25202.4/builds/81 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch branches/release24-maint] HEAD Blamelist: georg.brandl Build Had Warnings: warnings test sincerely, -The Buildbot From buildbot at python.org Thu Apr 13 00:25:22 2006 From: buildbot at python.org (buildbot at python.org) Date: Wed, 12 Apr 2006 22:25:22 +0000 Subject: [Python-checkins] buildbot warnings in sparc solaris10 gcc 2.4 Message-ID: <20060412222522.8A21F1E4018@bag.python.org> The Buildbot has detected a new failure of sparc solaris10 gcc 2.4. Full details are available at: http://www.python.org/dev/buildbot/all/sparc%2520solaris10%2520gcc%25202.4/builds/85 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch branches/release24-maint] HEAD Blamelist: georg.brandl Build Had Warnings: warnings test sincerely, -The Buildbot From buildbot at python.org Thu Apr 13 00:25:43 2006 From: buildbot at python.org (buildbot at python.org) Date: Wed, 12 Apr 2006 22:25:43 +0000 Subject: [Python-checkins] buildbot warnings in ppc Debian unstable trunk Message-ID: <20060412222543.980E11E402C@bag.python.org> The Buildbot has detected a new failure of ppc Debian unstable trunk. Full details are available at: http://www.python.org/dev/buildbot/all/ppc%2520Debian%2520unstable%2520trunk/builds/100 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: georg.brandl Build Had Warnings: warnings test sincerely, -The Buildbot From buildbot at python.org Thu Apr 13 00:29:17 2006 From: buildbot at python.org (buildbot at python.org) Date: Wed, 12 Apr 2006 22:29:17 +0000 Subject: [Python-checkins] buildbot warnings in x86 OpenBSD trunk Message-ID: <20060412222917.3B1821E4003@bag.python.org> The Buildbot has detected a new failure of x86 OpenBSD trunk. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520OpenBSD%2520trunk/builds/264 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: georg.brandl Build Had Warnings: warnings test sincerely, -The Buildbot From buildbot at python.org Thu Apr 13 00:46:42 2006 From: buildbot at python.org (buildbot at python.org) Date: Wed, 12 Apr 2006 22:46:42 +0000 Subject: [Python-checkins] buildbot warnings in ia64 Debian unstable 2.4 Message-ID: <20060412224642.7F9C21E4003@bag.python.org> The Buildbot has detected a new failure of ia64 Debian unstable 2.4. Full details are available at: http://www.python.org/dev/buildbot/all/ia64%2520Debian%2520unstable%25202.4/builds/19 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch branches/release24-maint] HEAD Blamelist: georg.brandl Build Had Warnings: warnings test sincerely, -The Buildbot From buildbot at python.org Thu Apr 13 00:57:27 2006 From: buildbot at python.org (buildbot at python.org) Date: Wed, 12 Apr 2006 22:57:27 +0000 Subject: [Python-checkins] buildbot warnings in g4 osx.4 trunk Message-ID: <20060412225727.9F5B21E4003@bag.python.org> The Buildbot has detected a new failure of g4 osx.4 trunk. Full details are available at: http://www.python.org/dev/buildbot/all/g4%2520osx.4%2520trunk/builds/373 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: georg.brandl Build Had Warnings: warnings test sincerely, -The Buildbot From buildbot at python.org Thu Apr 13 01:01:01 2006 From: buildbot at python.org (buildbot at python.org) Date: Wed, 12 Apr 2006 23:01:01 +0000 Subject: [Python-checkins] buildbot warnings in sparc solaris10 gcc trunk Message-ID: <20060412230101.ABE521E4003@bag.python.org> The Buildbot has detected a new failure of sparc solaris10 gcc trunk. Full details are available at: http://www.python.org/dev/buildbot/all/sparc%2520solaris10%2520gcc%2520trunk/builds/379 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: georg.brandl Build Had Warnings: warnings test sincerely, -The Buildbot From buildbot at python.org Thu Apr 13 01:25:32 2006 From: buildbot at python.org (buildbot at python.org) Date: Wed, 12 Apr 2006 23:25:32 +0000 Subject: [Python-checkins] buildbot warnings in x86 Ubuntu dapper (icc) 2.4 Message-ID: <20060412232532.67D2A1E4003@bag.python.org> The Buildbot has detected a new failure of x86 Ubuntu dapper (icc) 2.4. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520Ubuntu%2520dapper%2520%2528icc%2529%25202.4/builds/23 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch branches/release24-maint] HEAD Blamelist: georg.brandl Build Had Warnings: warnings test sincerely, -The Buildbot From buildbot at python.org Thu Apr 13 01:40:25 2006 From: buildbot at python.org (buildbot at python.org) Date: Wed, 12 Apr 2006 23:40:25 +0000 Subject: [Python-checkins] buildbot warnings in alpha Tru64 5.1 2.4 Message-ID: <20060412234025.A56E91E4003@bag.python.org> The Buildbot has detected a new failure of alpha Tru64 5.1 2.4. Full details are available at: http://www.python.org/dev/buildbot/all/alpha%2520Tru64%25205.1%25202.4/builds/43 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch branches/release24-maint] HEAD Blamelist: georg.brandl Build Had Warnings: warnings test sincerely, -The Buildbot From buildbot at python.org Thu Apr 13 02:21:59 2006 From: buildbot at python.org (buildbot at python.org) Date: Thu, 13 Apr 2006 00:21:59 +0000 Subject: [Python-checkins] buildbot warnings in x86 Ubuntu dapper (icc) trunk Message-ID: <20060413002159.EE6D51E4003@bag.python.org> The Buildbot has detected a new failure of x86 Ubuntu dapper (icc) trunk. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520Ubuntu%2520dapper%2520%2528icc%2529%2520trunk/builds/105 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: georg.brandl Build Had Warnings: warnings test sincerely, -The Buildbot From python-checkins at python.org Thu Apr 13 02:23:55 2006 From: python-checkins at python.org (anthony.baxter) Date: Thu, 13 Apr 2006 02:23:55 +0200 (CEST) Subject: [Python-checkins] r45323 - peps/trunk/pep-0356.txt Message-ID: <20060413002355.AD1471E4003@bag.python.org> Author: anthony.baxter Date: Thu Apr 13 02:23:55 2006 New Revision: 45323 Modified: peps/trunk/pep-0356.txt Log: add note about 1465834 Modified: peps/trunk/pep-0356.txt ============================================================================== --- peps/trunk/pep-0356.txt (original) +++ peps/trunk/pep-0356.txt Thu Apr 13 02:23:55 2006 @@ -153,6 +153,9 @@ Open issues + - 1465834: bdist_wininst preinstall scripts broken by lack of + PyRun_SimpleString (and others?) as symbols instead of macros. + - AST compiler problems (Owner: Jeremy Hylton) * eval(str(-sys.maxint - 1)) should produce an int, not long. From buildbot at python.org Thu Apr 13 02:28:06 2006 From: buildbot at python.org (buildbot at python.org) Date: Thu, 13 Apr 2006 00:28:06 +0000 Subject: [Python-checkins] buildbot warnings in alpha Tru64 5.1 trunk Message-ID: <20060413002806.D8D4C1E4003@bag.python.org> The Buildbot has detected a new failure of alpha Tru64 5.1 trunk. Full details are available at: http://www.python.org/dev/buildbot/all/alpha%2520Tru64%25205.1%2520trunk/builds/193 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: georg.brandl Build Had Warnings: warnings test sincerely, -The Buildbot From python-checkins at python.org Thu Apr 13 03:07:29 2006 From: python-checkins at python.org (anthony.baxter) Date: Thu, 13 Apr 2006 03:07:29 +0200 (CEST) Subject: [Python-checkins] r45324 - python/trunk/Objects/genobject.c Message-ID: <20060413010729.1D9BE1E4009@bag.python.org> Author: anthony.baxter Date: Thu Apr 13 03:07:27 2006 New Revision: 45324 Modified: python/trunk/Objects/genobject.c Log: Add a cast to make code compile with a C++ compiler. Modified: python/trunk/Objects/genobject.c ============================================================================== --- python/trunk/Objects/genobject.c (original) +++ python/trunk/Objects/genobject.c Thu Apr 13 03:07:27 2006 @@ -10,7 +10,7 @@ static int gen_traverse(PyGenObject *gen, visitproc visit, void *arg) { - Py_VISIT(gen->gi_frame); + Py_VISIT((PyObject *)gen->gi_frame); return 0; } From python-checkins at python.org Thu Apr 13 03:23:28 2006 From: python-checkins at python.org (anthony.baxter) Date: Thu, 13 Apr 2006 03:23:28 +0200 (CEST) Subject: [Python-checkins] r45325 - python/trunk/Python/compile.c Message-ID: <20060413012328.B65F51E4009@bag.python.org> Author: anthony.baxter Date: Thu Apr 13 03:23:28 2006 New Revision: 45325 Modified: python/trunk/Python/compile.c Log: casting nastiness to make C++ compiler happy Modified: python/trunk/Python/compile.c ============================================================================== --- python/trunk/Python/compile.c (original) +++ python/trunk/Python/compile.c Thu Apr 13 03:23:28 2006 @@ -3058,12 +3058,18 @@ VISIT(c, expr, (expr_ty)asdl_seq_GET(e->v.Compare.comparators, 0)); } +#ifdef __cplusplus +#define CMPCAST (intptr_t) +#else +#define CMPCAST +#endif for (i = 1; i < n; i++) { ADDOP(c, DUP_TOP); ADDOP(c, ROT_THREE); /* XXX We're casting a void* to cmpop_ty in the next stmt. */ ADDOP_I(c, COMPARE_OP, - cmpop((cmpop_ty)asdl_seq_GET(e->v.Compare.ops, i - 1))); + cmpop((cmpop_ty)( CMPCAST asdl_seq_GET( + e->v.Compare.ops, i - 1)))); ADDOP_JREL(c, JUMP_IF_FALSE, cleanup); NEXT_BLOCK(c); ADDOP(c, POP_TOP); @@ -3074,7 +3080,8 @@ VISIT(c, expr, (expr_ty)asdl_seq_GET(e->v.Compare.comparators, n - 1)); ADDOP_I(c, COMPARE_OP, /* XXX We're casting a void* to cmpop_ty in the next stmt. */ - cmpop((cmpop_ty)asdl_seq_GET(e->v.Compare.ops, n - 1))); + cmpop((cmpop_ty)( CMPCAST asdl_seq_GET(e->v.Compare.ops, + n - 1)))); if (n > 1) { basicblock *end = compiler_new_block(c); if (end == NULL) @@ -3087,6 +3094,7 @@ } return 1; } +#undef CMPCAST static int compiler_call(struct compiler *c, expr_ty e) From python-checkins at python.org Thu Apr 13 03:29:12 2006 From: python-checkins at python.org (anthony.baxter) Date: Thu, 13 Apr 2006 03:29:12 +0200 (CEST) Subject: [Python-checkins] r45326 - in python/branches/release24-maint: Lib/test/test_traceback.py Lib/traceback.py Misc/NEWS Message-ID: <20060413012912.736511E400B@bag.python.org> Author: anthony.baxter Date: Thu Apr 13 03:29:10 2006 New Revision: 45326 Modified: python/branches/release24-maint/Lib/test/test_traceback.py python/branches/release24-maint/Lib/traceback.py python/branches/release24-maint/Misc/NEWS Log: reverting r45322 - "traceback.format_exception_only() now prepends the exception's module name to non-builtin exceptions, like the interpreter itself does." Reverting because: A - broke tests B - changed behaviour of 2.4. Modified: python/branches/release24-maint/Lib/test/test_traceback.py ============================================================================== --- python/branches/release24-maint/Lib/test/test_traceback.py (original) +++ python/branches/release24-maint/Lib/test/test_traceback.py Thu Apr 13 03:29:10 2006 @@ -5,9 +5,6 @@ import traceback -class TbError(Exception): - pass - class TracebackCases(unittest.TestCase): # For now, a very minimal set of tests. I want to be sure that # formatting of SyntaxErrors works based on changes for 2.1. @@ -88,24 +85,6 @@ os.unlink(os.path.join(testdir, f)) os.rmdir(testdir) - def raise_tberror(self): - raise TbError - - def raise_typeerror(self): - raise TypeError - - def test_modulename(self): - # Bug 860326: format_exception_only should prepend module name - # to exceptions not in "exceptions", like PyErr_Print does. - err = self.get_exception_format(self.raise_tberror, TbError) - self.assertEquals(len(err), 1) - self.assert_(err[0] == '__main__.TbError\n' or - err[0] == 'test.test_traceback.TbError\n') - - err = self.get_exception_format(self.raise_typeerror, TypeError) - self.assertEquals(err[0], 'TypeError\n') - - def test_main(): run_unittest(TracebackCases) Modified: python/branches/release24-maint/Lib/traceback.py ============================================================================== --- python/branches/release24-maint/Lib/traceback.py (original) +++ python/branches/release24-maint/Lib/traceback.py Thu Apr 13 03:29:10 2006 @@ -159,10 +159,6 @@ list = [] if type(etype) == types.ClassType: stype = etype.__name__ - if not hasattr(etype, '__module__'): - stype = '.' + stype - elif etype.__module__ != 'exceptions': - stype = etype.__module__ + '.' + stype else: stype = etype if value is None: Modified: python/branches/release24-maint/Misc/NEWS ============================================================================== --- python/branches/release24-maint/Misc/NEWS (original) +++ python/branches/release24-maint/Misc/NEWS Thu Apr 13 03:29:10 2006 @@ -23,10 +23,6 @@ Library ------- -- Patch #860326: traceback.format_exception_only() now prepends the - exception's module name to non-builtin exceptions, like the interpreter - itself does. - - The email module's parsedate_tz function now sets the daylight savings flag to -1 (unknown) since it can't tell from the date whether it should be set. From python-checkins at python.org Thu Apr 13 03:34:35 2006 From: python-checkins at python.org (anthony.baxter) Date: Thu, 13 Apr 2006 03:34:35 +0200 (CEST) Subject: [Python-checkins] r45327 - in python/trunk: Lib/test/test_traceback.py Lib/traceback.py Misc/NEWS Message-ID: <20060413013435.DDA501E4009@bag.python.org> Author: anthony.baxter Date: Thu Apr 13 03:34:33 2006 New Revision: 45327 Modified: python/trunk/Lib/test/test_traceback.py python/trunk/Lib/traceback.py python/trunk/Misc/NEWS Log: reverting r45321: Patch #860326: traceback.format_exception_only() now prepends the exception's module name to non-builtin exceptions, like the interpreter itself does. broke a number of doctests. should be discussed before checking in (see discussion on python-dev). Modified: python/trunk/Lib/test/test_traceback.py ============================================================================== --- python/trunk/Lib/test/test_traceback.py (original) +++ python/trunk/Lib/test/test_traceback.py Thu Apr 13 03:34:33 2006 @@ -5,9 +5,6 @@ import traceback -class TbError(Exception): - pass - class TracebackCases(unittest.TestCase): # For now, a very minimal set of tests. I want to be sure that # formatting of SyntaxErrors works based on changes for 2.1. @@ -106,24 +103,6 @@ import sys sys.exc_traceback.__members__ - def raise_tberror(self): - raise TbError - - def raise_typeerror(self): - raise TypeError - - def test_modulename(self): - # Bug 860326: format_exception_only should prepend module name - # to exceptions not in "exceptions", like PyErr_Print does. - err = self.get_exception_format(self.raise_tberror, TbError) - self.assertEquals(len(err), 1) - self.assert_(err[0] == '__main__.TbError\n' or - err[0] == 'test.test_traceback.TbError\n') - - err = self.get_exception_format(self.raise_typeerror, TypeError) - self.assertEquals(err[0], 'TypeError\n') - - def test_main(): run_unittest(TracebackCases) Modified: python/trunk/Lib/traceback.py ============================================================================== --- python/trunk/Lib/traceback.py (original) +++ python/trunk/Lib/traceback.py Thu Apr 13 03:34:33 2006 @@ -158,12 +158,8 @@ """ list = [] if (type(etype) == types.ClassType - or (isinstance(etype, type) and issubclass(etype, BaseException))): + or (isinstance(etype, type) and issubclass(etype, Exception))): stype = etype.__name__ - if not hasattr(etype, '__module__'): - stype = '.' + stype - elif etype.__module__ != 'exceptions': - stype = etype.__module__ + '.' + stype else: stype = etype if value is None: Modified: python/trunk/Misc/NEWS ============================================================================== --- python/trunk/Misc/NEWS (original) +++ python/trunk/Misc/NEWS Thu Apr 13 03:34:33 2006 @@ -49,10 +49,6 @@ Library ------- -- Patch #860326: traceback.format_exception_only() now prepends the - exception's module name to non-builtin exceptions, like the interpreter - itself does. - - SimpleXMLRPCServer relied on the fcntl module, which is unavailable on Windows. Bug #1469163. From python-checkins at python.org Thu Apr 13 04:00:58 2006 From: python-checkins at python.org (skip.montanaro) Date: Thu, 13 Apr 2006 04:00:58 +0200 (CEST) Subject: [Python-checkins] r45328 - python/trunk/configure python/trunk/configure.in Message-ID: <20060413020058.7EA021E401D@bag.python.org> Author: skip.montanaro Date: Thu Apr 13 04:00:56 2006 New Revision: 45328 Modified: python/trunk/configure python/trunk/configure.in Log: If compiling with g++ don't use -Wstrict-prototpes. Modified: python/trunk/configure ============================================================================== --- python/trunk/configure (original) +++ python/trunk/configure Thu Apr 13 04:00:56 2006 @@ -1,5 +1,5 @@ #! /bin/sh -# From configure.in Revision: 45264 . +# From configure.in Revision: 45278 . # Guess values for system-dependent variables and create Makefiles. # Generated by GNU Autoconf 2.59 for python 2.5. # @@ -3792,18 +3792,21 @@ then case $GCC in yes) + if test "$CC" != 'g++' ; then + STRICT_PROTO="-Wstrict-prototypes" + fi case $ac_cv_prog_cc_g in yes) if test "$Py_DEBUG" = 'true' ; then # Optimization messes up debuggers, so turn it off for # debug builds. - OPT="-g -Wall -Wstrict-prototypes" + OPT="-g -Wall $STRICT_PROTO" else - OPT="-g -O3 -Wall -Wstrict-prototypes" + OPT="-g -O3 -Wall $STRICT_PROTO" fi ;; *) - OPT="-O3 -Wall -Wstrict-prototypes" + OPT="-O3 -Wall $STRICT_PROTO" ;; esac case $ac_sys_system in Modified: python/trunk/configure.in ============================================================================== --- python/trunk/configure.in (original) +++ python/trunk/configure.in Thu Apr 13 04:00:56 2006 @@ -685,18 +685,21 @@ then case $GCC in yes) + if test "$CC" != 'g++' ; then + STRICT_PROTO="-Wstrict-prototypes" + fi case $ac_cv_prog_cc_g in yes) if test "$Py_DEBUG" = 'true' ; then # Optimization messes up debuggers, so turn it off for # debug builds. - OPT="-g -Wall -Wstrict-prototypes" + OPT="-g -Wall $STRICT_PROTO" else - OPT="-g -O3 -Wall -Wstrict-prototypes" + OPT="-g -O3 -Wall $STRICT_PROTO" fi ;; *) - OPT="-O3 -Wall -Wstrict-prototypes" + OPT="-O3 -Wall $STRICT_PROTO" ;; esac case $ac_sys_system in From python-checkins at python.org Thu Apr 13 04:04:45 2006 From: python-checkins at python.org (andrew.kuchling) Date: Thu, 13 Apr 2006 04:04:45 +0200 (CEST) Subject: [Python-checkins] r45329 - python/trunk/Doc/whatsnew/whatsnew25.tex Message-ID: <20060413020445.29D061E4009@bag.python.org> Author: andrew.kuchling Date: Thu Apr 13 04:04:42 2006 New Revision: 45329 Modified: python/trunk/Doc/whatsnew/whatsnew25.tex Log: Write some ctypes examples Modified: python/trunk/Doc/whatsnew/whatsnew25.tex ============================================================================== --- python/trunk/Doc/whatsnew/whatsnew25.tex (original) +++ python/trunk/Doc/whatsnew/whatsnew25.tex Thu Apr 13 04:04:42 2006 @@ -5,7 +5,6 @@ % Fix XXX comments % The easy_install stuff % Stateful codec changes -% Write ctypes examples % Count up the patches and bugs \title{What's New in Python 2.5} @@ -1111,13 +1110,83 @@ The \module{ctypes} package, written by Thomas Heller, has been added to the standard library. \module{ctypes} lets you call arbitrary functions -in shared libraries or DLLs. +in shared libraries or DLLs. Long-time users may remember the \module{dl} module, which +provides functions for loading shared libraries and calling functions in them. The \module{ctypes} package is much fancier. -In subsequent alpha releases of Python 2.5, I'll add a brief -introduction that shows some basic usage of the module. +To load a shared library or DLL, you must create an instance of the +\class{CDLL} class and provide the name or path of the shared library +or DLL. Once that's done, you can call arbitrary functions +by accessing them as attributes of the \class{CDLL} object. + +\begin{verbatim} +import ctypes + +libc = ctypes.CDLL('libc.so.6') +result = libc.printf("Line of output\n") +\end{verbatim} + +Type constructors for the various C types are provided: \function{c_int}, +\function{c_float}, \function{c_double}, \function{c_char_p} (equivalent to \ctype{char *}), and so forth. Unlike Python's types, the C versions are all mutable; you can assign to their \member{value} attribute +to change the wrapped value. Python integers and strings will be automatically +converted to the corresponding C types, but for other types you +must call the correct type constructor. (And I mean \emph{must}; +getting it wrong will often result in the interpreter crashing +with a segmentation fault.) + +You shouldn't use \function{c_char_p} with a Python string when the C function will be modifying the memory area, because Python strings are +supposed to be immutable; breaking this rule will cause puzzling bugs. When you need a modifiable memory area, +use \function{create_string_buffer(): + +\begin{verbatim} +s = "this is a string" +buf = ctypes.create_string_buffer(s) +libc.strfry(buf) +\end{verbatim} + +C functions are assumed to return integers, but you can set +the \member{restype} attribute of the function object to +change this: + +\begin{verbatim} +>>> libc.atof('2.71828') +-1783957616 +>>> libc.atof.restype = ctypes.c_double +>>> libc.atof('2.71828') +2.71828 +\end{verbatim} + +\module{ctypes} also provides a wrapper for Python's C API +as the \code{ctypes.pythonapi} object. This object does \emph{not} +release the global interpreter lock before calling a function, because the lock must be held when calling into the interpreter's code. +There's a \class{py_object()} type constructor that will create a +\ctype{PyObject *} pointer. A simple usage: + +\begin{verbatim} +import ctypes + +d = {} +ctypes.pythonapi.PyObject_SetItem(ctypes.py_object(d), + ctypes.py_object("abc"), ctypes.py_object(1)) +# d is now {'abc', 1}. +\end{verbatim} + +Don't forget to use \class{py_object()}; if it's omitted you end +up with a segmentation fault. + +\module{ctypes} has been around for a while, but people still write +and distribution hand-coded extension modules because you can't rely on \module{ctypes} being present. +Perhaps developers will begin to write +Python wrappers atop a library accessed through \module{ctypes} instead +of extension modules, now that \module{ctypes} is included with core Python. % XXX write introduction +\begin{seealso} + +\seeurl{http://starship.python.net/crew/theller/ctypes/} +{The ctypes web page, with a tutorial, reference, and FAQ.} + +\end{seealso} \subsection{The ElementTree package} From python-checkins at python.org Thu Apr 13 04:06:13 2006 From: python-checkins at python.org (anthony.baxter) Date: Thu, 13 Apr 2006 04:06:13 +0200 (CEST) Subject: [Python-checkins] r45330 - in python/trunk: Modules/getpath.c Modules/main.c Modules/posixmodule.c Objects/fileobject.c Objects/object.c Objects/unicodeobject.c Python/errors.c Python/getmtime.c Python/getopt.c Python/import.c Python/pystate.c Python/pythonrun.c Message-ID: <20060413020613.4037D1E4009@bag.python.org> Author: anthony.baxter Date: Thu Apr 13 04:06:09 2006 New Revision: 45330 Modified: python/trunk/Modules/getpath.c python/trunk/Modules/main.c python/trunk/Modules/posixmodule.c python/trunk/Objects/fileobject.c python/trunk/Objects/object.c python/trunk/Objects/unicodeobject.c python/trunk/Python/errors.c python/trunk/Python/getmtime.c python/trunk/Python/getopt.c python/trunk/Python/import.c python/trunk/Python/pystate.c python/trunk/Python/pythonrun.c Log: spread the extern "C" { } magic pixie dust around. Python itself builds now using a C++ compiler. Still lots and lots of errors in the modules built by setup.py, and a bunch of warnings from g++ in the core. Modified: python/trunk/Modules/getpath.c ============================================================================== --- python/trunk/Modules/getpath.c (original) +++ python/trunk/Modules/getpath.c Thu Apr 13 04:06:09 2006 @@ -91,6 +91,11 @@ * process to find the installed Python tree. */ +#ifdef __cplusplus + extern "C" { +#endif + + #ifndef VERSION #if defined(__VMS) #define VERSION "2_1" @@ -681,3 +686,9 @@ calculate_path(); return progpath; } + + +#ifdef __cplusplus +} +#endif + Modified: python/trunk/Modules/main.c ============================================================================== --- python/trunk/Modules/main.c (original) +++ python/trunk/Modules/main.c Thu Apr 13 04:06:09 2006 @@ -29,6 +29,10 @@ "Type \"help\", \"copyright\", \"credits\" or \"license\" " \ "for more information." +#ifdef __cplusplus +extern "C" { +#endif + /* For Py_GetArgcArgv(); set by main() */ static char **orig_argv; static int orig_argc; @@ -540,3 +544,8 @@ *argc = orig_argc; *argv = orig_argv; } + +#ifdef __cplusplus +} +#endif + Modified: python/trunk/Modules/posixmodule.c ============================================================================== --- python/trunk/Modules/posixmodule.c (original) +++ python/trunk/Modules/posixmodule.c Thu Apr 13 04:06:09 2006 @@ -22,6 +22,10 @@ # include #endif /* defined(__VMS) */ +#ifdef __cplusplus +extern "C" { +#endif + PyDoc_STRVAR(posix__doc__, "This module provides access to operating system functionality that is\n\ standardized by the C Standard and the POSIX standard (a thinly\n\ @@ -8253,3 +8257,8 @@ PyModule_AddObject(m, "statvfs_result", (PyObject*) &StatVFSResultType); } + +#ifdef __cplusplus +} +#endif + Modified: python/trunk/Objects/fileobject.c ============================================================================== --- python/trunk/Objects/fileobject.c (original) +++ python/trunk/Objects/fileobject.c Thu Apr 13 04:06:09 2006 @@ -48,6 +48,10 @@ #define NEWLINE_LF 2 /* \n newline seen */ #define NEWLINE_CRLF 4 /* \r\n newline seen */ +#ifdef __cplusplus +extern "C" { +#endif + FILE * PyFile_AsFile(PyObject *f) { @@ -2441,3 +2445,8 @@ f->f_skipnextlf = skipnextlf; return dst - buf; } + +#ifdef __cplusplus +} +#endif + Modified: python/trunk/Objects/object.c ============================================================================== --- python/trunk/Objects/object.c (original) +++ python/trunk/Objects/object.c Thu Apr 13 04:06:09 2006 @@ -3,6 +3,10 @@ #include "Python.h" +#ifdef __cplusplus +extern "C" { +#endif + #ifdef Py_REF_DEBUG Py_ssize_t _Py_RefTotal; @@ -2112,3 +2116,8 @@ --_PyTrash_delete_nesting; } } + +#ifdef __cplusplus +} +#endif + Modified: python/trunk/Objects/unicodeobject.c ============================================================================== --- python/trunk/Objects/unicodeobject.c (original) +++ python/trunk/Objects/unicodeobject.c Thu Apr 13 04:06:09 2006 @@ -83,6 +83,11 @@ */ + +#ifdef __cplusplus +extern "C" { +#endif + /* Free list for Unicode objects */ static PyUnicodeObject *unicode_freelist; static int unicode_freelist_size; @@ -7418,6 +7423,11 @@ unicode_freelist_size = 0; } +#ifdef __cplusplus +} +#endif + + /* Local variables: c-basic-offset: 4 Modified: python/trunk/Python/errors.c ============================================================================== --- python/trunk/Python/errors.c (original) +++ python/trunk/Python/errors.c Thu Apr 13 04:06:09 2006 @@ -16,6 +16,11 @@ #include +#ifdef __cplusplus +extern "C" { +#endif + + void PyErr_Restore(PyObject *type, PyObject *value, PyObject *traceback) { @@ -786,3 +791,8 @@ } return NULL; } + +#ifdef __cplusplus +} +#endif + Modified: python/trunk/Python/getmtime.c ============================================================================== --- python/trunk/Python/getmtime.c (original) +++ python/trunk/Python/getmtime.c Thu Apr 13 04:06:09 2006 @@ -6,6 +6,10 @@ #include "Python.h" #include "pyconfig.h" +#ifdef __cplusplus +extern "C" { +#endif + time_t PyOS_GetLastModificationTime(char *path, FILE *fp) { @@ -15,3 +19,8 @@ else return st.st_mtime; } + +#ifdef __cplusplus +} +#endif + Modified: python/trunk/Python/getopt.c ============================================================================== --- python/trunk/Python/getopt.c (original) +++ python/trunk/Python/getopt.c Thu Apr 13 04:06:09 2006 @@ -27,6 +27,10 @@ #include #include +#ifdef __cplusplus +extern "C" { +#endif + int _PyOS_opterr = 1; /* generate error messages */ int _PyOS_optind = 1; /* index into argv array */ char *_PyOS_optarg = NULL; /* optional argument */ @@ -81,3 +85,8 @@ return option; } + +#ifdef __cplusplus +} +#endif + Modified: python/trunk/Python/import.c ============================================================================== --- python/trunk/Python/import.c (original) +++ python/trunk/Python/import.c Thu Apr 13 04:06:09 2006 @@ -17,6 +17,9 @@ #ifdef HAVE_FCNTL_H #include #endif +#ifdef __cplusplus +extern "C" { +#endif extern time_t PyOS_GetLastModificationTime(char *, FILE *); /* In getmtime.c */ @@ -2947,3 +2950,7 @@ return PyImport_ExtendInittab(newtab); } + +#ifdef __cplusplus +} +#endif Modified: python/trunk/Python/pystate.c ============================================================================== --- python/trunk/Python/pystate.c (original) +++ python/trunk/Python/pystate.c Thu Apr 13 04:06:09 2006 @@ -37,6 +37,10 @@ #define HEAD_LOCK() PyThread_acquire_lock(head_mutex, WAIT_LOCK) #define HEAD_UNLOCK() PyThread_release_lock(head_mutex) +#ifdef __cplusplus +extern "C" { +#endif + /* The single PyInterpreterState used by this process' GILState implementation */ @@ -552,4 +556,11 @@ else if (oldstate == PyGILState_UNLOCKED) PyEval_SaveThread(); } + +#ifdef __cplusplus +} +#endif + #endif /* WITH_THREAD */ + + Modified: python/trunk/Python/pythonrun.c ============================================================================== --- python/trunk/Python/pythonrun.c (original) +++ python/trunk/Python/pythonrun.c Thu Apr 13 04:06:09 2006 @@ -37,6 +37,10 @@ _Py_GetRefTotal()) #endif +#ifdef __cplusplus +extern "C" { +#endif + extern char *Py_GetPath(void); extern grammar _PyParser_Grammar; /* From graminit.c */ @@ -1692,3 +1696,8 @@ { return PyParser_SimpleParseStringFlags(str, start, 0); } + +#ifdef __cplusplus +} +#endif + From python-checkins at python.org Thu Apr 13 04:10:19 2006 From: python-checkins at python.org (andrew.kuchling) Date: Thu, 13 Apr 2006 04:10:19 +0200 (CEST) Subject: [Python-checkins] r45331 - python/trunk/Doc/whatsnew/whatsnew20.tex python/trunk/Doc/whatsnew/whatsnew24.tex Message-ID: <20060413021019.49D0C1E4009@bag.python.org> Author: andrew.kuchling Date: Thu Apr 13 04:10:16 2006 New Revision: 45331 Modified: python/trunk/Doc/whatsnew/whatsnew20.tex python/trunk/Doc/whatsnew/whatsnew24.tex Log: Fix typography of Martin's name Modified: python/trunk/Doc/whatsnew/whatsnew20.tex ============================================================================== --- python/trunk/Doc/whatsnew/whatsnew20.tex (original) +++ python/trunk/Doc/whatsnew/whatsnew20.tex Thu Apr 13 04:10:16 2006 @@ -1214,8 +1214,8 @@ \item{\module{gettext}:} This module provides internationalization (I18N) and localization (L10N) support for Python programs by providing an interface to the GNU gettext message catalog library. -(Integrated by Barry Warsaw, from separate contributions by Martin von -Loewis, Peter Funk, and James Henstridge.) +(Integrated by Barry Warsaw, from separate contributions by Martin +von~L\"owis, Peter Funk, and James Henstridge.) \item{\module{linuxaudiodev}:} Support for the \file{/dev/audio} device on Linux, a twin to the existing \module{sunaudiodev} module. Modified: python/trunk/Doc/whatsnew/whatsnew24.tex ============================================================================== --- python/trunk/Doc/whatsnew/whatsnew24.tex (original) +++ python/trunk/Doc/whatsnew/whatsnew24.tex Thu Apr 13 04:10:16 2006 @@ -1336,7 +1336,7 @@ \item The \module{marshal} module now shares interned strings on unpacking a data structure. This may shrink the size of certain pickle strings, but the primary effect is to make \file{.pyc} files significantly smaller. -(Contributed by Martin von Loewis.) +(Contributed by Martin von~L\"owis.) \item The \module{nntplib} module's \class{NNTP} class gained \method{description()} and \method{descriptions()} methods to retrieve @@ -1688,7 +1688,7 @@ \begin{itemize} \item The Windows port now builds under MSVC++ 7.1 as well as version 6. - (Contributed by Martin von Loewis.) + (Contributed by Martin von~L\"owis.) \end{itemize} From skip at pobox.com Thu Apr 13 04:14:30 2006 From: skip at pobox.com (skip at pobox.com) Date: Wed, 12 Apr 2006 21:14:30 -0500 Subject: [Python-checkins] r45325 - python/trunk/Python/compile.c In-Reply-To: <20060413012328.B65F51E4009@bag.python.org> References: <20060413012328.B65F51E4009@bag.python.org> Message-ID: <17469.46086.889340.985248@montanaro.dyndns.org> anthony> Modified: anthony> python/trunk/Python/compile.c anthony> Log: anthony> casting nastiness to make C++ compiler happy I posted an alternative patch to SF for this. I don't know if a union is any better than a simple cast though: http://python.org/sf/1469594 If so, it probably needs a _SET_ENUM macro for symmetry. Skip From buildbot at python.org Thu Apr 13 04:34:31 2006 From: buildbot at python.org (buildbot at python.org) Date: Thu, 13 Apr 2006 02:34:31 +0000 Subject: [Python-checkins] buildbot failure in x86 XP trunk Message-ID: <20060413023431.5597E1E4009@bag.python.org> The Buildbot has detected a new failure of x86 XP trunk. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520XP%2520trunk/builds/392 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: andrew.kuchling,anthony.baxter,skip.montanaro BUILD FAILED: failed compile sincerely, -The Buildbot From python-checkins at python.org Thu Apr 13 05:09:42 2006 From: python-checkins at python.org (tim.peters) Date: Thu, 13 Apr 2006 05:09:42 +0200 (CEST) Subject: [Python-checkins] r45332 - python/trunk/Lib/test/test_sundry.py Message-ID: <20060413030942.E8BB31E4009@bag.python.org> Author: tim.peters Date: Thu Apr 13 05:09:40 2006 New Revision: 45332 Modified: python/trunk/Lib/test/test_sundry.py Log: tty isn't supported on all boxes. Modified: python/trunk/Lib/test/test_sundry.py ============================================================================== --- python/trunk/Lib/test/test_sundry.py (original) +++ python/trunk/Lib/test/test_sundry.py Thu Apr 13 05:09:40 2006 @@ -68,7 +68,12 @@ import timeit import toaiff import token -import tty +try: + import tty # not available on Windows +except ImportError: + if verbose: + print "skipping tty" + # Can't test the "user" module -- if the user has a ~/.pythonrc.py, it # can screw up all sorts of things (esp. if it prints!). #import user From dynkin at gmail.com Thu Apr 13 05:05:37 2006 From: dynkin at gmail.com (George Yoshida) Date: Thu, 13 Apr 2006 12:05:37 +0900 Subject: [Python-checkins] r45329 - python/trunk/Doc/whatsnew/whatsnew25.tex In-Reply-To: <20060413020445.29D061E4009@bag.python.org> References: <20060413020445.29D061E4009@bag.python.org> Message-ID: <2f188ee80604122005n37b11481w1d2f9850a4a337f3@mail.gmail.com> On 4/13/06, andrew.kuchling wrote: > Author: andrew.kuchling > Date: Thu Apr 13 04:04:42 2006 > New Revision: 45329 > > Modified: > python/trunk/Doc/whatsnew/whatsnew25.tex > Log: > Write some ctypes examples > > Modified: python/trunk/Doc/whatsnew/whatsnew25.tex This commit results in compile error. '}' is missing for \function. Please see below. Index: Doc/whatsnew/whatsnew25.tex =================================================================== --- Doc/whatsnew/whatsnew25.tex (revision 45331) +++ Doc/whatsnew/whatsnew25.tex (working copy) @@ -1135,7 +1135,7 @@ You shouldn't use \function{c_char_p} with a Python string when the C function will be modifying the memory area, because Python strings are supposed to be immutable; breaking this rule will cause puzzling bugs. When you need a modifiable memory area, -use \function{create_string_buffer(): +use \function{create_string_buffer()}: \begin{verbatim} s = "this is a string" -- george From python-checkins at python.org Thu Apr 13 05:41:04 2006 From: python-checkins at python.org (neal.norwitz) Date: Thu, 13 Apr 2006 05:41:04 +0200 (CEST) Subject: [Python-checkins] r45333 - python/trunk/Doc/whatsnew/whatsnew25.tex Message-ID: <20060413034104.C43F71E4009@bag.python.org> Author: neal.norwitz Date: Thu Apr 13 05:41:04 2006 New Revision: 45333 Modified: python/trunk/Doc/whatsnew/whatsnew25.tex Log: Fix {} mismatch spotted by George Yoshida. Modified: python/trunk/Doc/whatsnew/whatsnew25.tex ============================================================================== --- python/trunk/Doc/whatsnew/whatsnew25.tex (original) +++ python/trunk/Doc/whatsnew/whatsnew25.tex Thu Apr 13 05:41:04 2006 @@ -1135,7 +1135,7 @@ You shouldn't use \function{c_char_p} with a Python string when the C function will be modifying the memory area, because Python strings are supposed to be immutable; breaking this rule will cause puzzling bugs. When you need a modifiable memory area, -use \function{create_string_buffer(): +use \function{create_string_buffer()}: \begin{verbatim} s = "this is a string" From python-checkins at python.org Thu Apr 13 06:35:40 2006 From: python-checkins at python.org (neal.norwitz) Date: Thu, 13 Apr 2006 06:35:40 +0200 (CEST) Subject: [Python-checkins] r45334 - python/trunk/Lib/test/leakers/test_gen1.py python/trunk/Lib/test/leakers/test_generator_cycle.py python/trunk/Lib/test/leakers/test_tee.py Message-ID: <20060413043540.51D6F1E404D@bag.python.org> Author: neal.norwitz Date: Thu Apr 13 06:35:36 2006 New Revision: 45334 Added: python/trunk/Lib/test/leakers/test_gen1.py (contents, props changed) Removed: python/trunk/Lib/test/leakers/test_generator_cycle.py python/trunk/Lib/test/leakers/test_tee.py Log: Remove tests that no longer leak. There is still one leaking generator test Added: python/trunk/Lib/test/leakers/test_gen1.py ============================================================================== --- (empty file) +++ python/trunk/Lib/test/leakers/test_gen1.py Thu Apr 13 06:35:36 2006 @@ -0,0 +1,19 @@ +import gc + +# Taken from test_generators + +def f(): + try: + yield + except GeneratorExit: + yield "foo!" + +def inner_leak(): + g = f() + g.next() + +def leak(): + inner_leak() + gc.collect() + gc.collect() + gc.collect() Deleted: /python/trunk/Lib/test/leakers/test_generator_cycle.py ============================================================================== --- /python/trunk/Lib/test/leakers/test_generator_cycle.py Thu Apr 13 06:35:36 2006 +++ (empty file) @@ -1,10 +0,0 @@ - -# This leaks since the introduction of yield-expr and the use of generators -# as coroutines, trunk revision 39239. The cycle-GC doesn't seem to pick up -# the cycle, or decides it can't clean it up. - -def leak(): - def gen(): - while True: - yield g - g = gen() Deleted: /python/trunk/Lib/test/leakers/test_tee.py ============================================================================== --- /python/trunk/Lib/test/leakers/test_tee.py Thu Apr 13 06:35:36 2006 +++ (empty file) @@ -1,20 +0,0 @@ - -# Test case taken from test_itertools -# See http://mail.python.org/pipermail/python-dev/2005-November/058339.html -# When this is fixed remember to remove from LEAKY_TESTS in Misc/build.sh. - -from itertools import tee - -def leak(): - def fib(): - def yield_identity_forever(g): - while 1: - yield g - def _fib(): - for i in yield_identity_forever(head): - yield i - head, tail, result = tee(_fib(), 3) - return result - - x = fib() - x.next() From python-checkins at python.org Thu Apr 13 06:49:25 2006 From: python-checkins at python.org (anthony.baxter) Date: Thu, 13 Apr 2006 06:49:25 +0200 (CEST) Subject: [Python-checkins] r45335 - python/trunk/Modules/config.c.in Message-ID: <20060413044925.B85E91E4009@bag.python.org> Author: anthony.baxter Date: Thu Apr 13 06:49:25 2006 New Revision: 45335 Modified: python/trunk/Modules/config.c.in Log: whoops. missed one in an auto-generated file. another extern "C" {} for C++ compiler compatibility Modified: python/trunk/Modules/config.c.in ============================================================================== --- python/trunk/Modules/config.c.in (original) +++ python/trunk/Modules/config.c.in Thu Apr 13 06:49:25 2006 @@ -17,6 +17,10 @@ #include "Python.h" +#ifdef __cplusplus +extern "C" { +#endif + /* -- ADDMODULE MARKER 1 -- */ @@ -50,3 +54,9 @@ /* Sentinel */ {0, 0} }; + + +#ifdef __cplusplus +} +#endif + From buildbot at python.org Thu Apr 13 07:02:14 2006 From: buildbot at python.org (buildbot at python.org) Date: Thu, 13 Apr 2006 05:02:14 +0000 Subject: [Python-checkins] buildbot warnings in alpha Tru64 5.1 trunk Message-ID: <20060413050214.704711E4009@bag.python.org> The Buildbot has detected a new failure of alpha Tru64 5.1 trunk. Full details are available at: http://www.python.org/dev/buildbot/all/alpha%2520Tru64%25205.1%2520trunk/builds/196 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: tim.peters Build Had Warnings: warnings test sincerely, -The Buildbot From nnorwitz at gmail.com Thu Apr 13 07:26:06 2006 From: nnorwitz at gmail.com (Neal Norwitz) Date: Wed, 12 Apr 2006 22:26:06 -0700 Subject: [Python-checkins] r43299 - python/trunk/Lib/traceback.py In-Reply-To: References: <20060324221055.5853B1E4021@bag.python.org> Message-ID: ping On 3/24/06, Neal Norwitz wrote: > How about a test case? > > Is this just for strings? If so, should you just check for basestring instead of type? > > n > -- > > On 3/24/06, phillip.eby wrote: > > Author: phillip.eby > > Date: Fri Mar 24 23:10:54 2006 > > New Revision: 43299 > > > > Modified: > > python/trunk/Lib/traceback.py > > Log: > > Revert r42719, because the isinstance() check wasn't redundant; formatting a > > string exception was causing a TypeError. > > > > > > Modified: python/trunk/Lib/traceback.py > > ============================================================================== > > --- python/trunk/Lib/traceback.py (original) > > +++ python/trunk/Lib/traceback.py Fri Mar 24 23:10:54 2006 > > @@ -158,7 +158,7 @@ > > """ > > list = [] > > if (type(etype) == types.ClassType > > - or issubclass(etype, Exception)): > > + or (isinstance(etype, type) and issubclass(etype, Exception))): > > stype = etype.__name__ > > else: > > stype = etype > > _______________________________________________ > > Python-checkins mailing list > > Python-checkins at python.org > > http://mail.python.org/mailman/listinfo/python-checkins > > > From python-checkins at python.org Thu Apr 13 08:06:09 2006 From: python-checkins at python.org (martin.v.loewis) Date: Thu, 13 Apr 2006 08:06:09 +0200 (CEST) Subject: [Python-checkins] r45336 - python/trunk/Objects/unicodeobject.c Message-ID: <20060413060609.B74C21E400B@bag.python.org> Author: martin.v.loewis Date: Thu Apr 13 08:06:08 2006 New Revision: 45336 Modified: python/trunk/Objects/unicodeobject.c Log: Revert 34153: Py_UNICODE should not be signed. Modified: python/trunk/Objects/unicodeobject.c ============================================================================== --- python/trunk/Objects/unicodeobject.c (original) +++ python/trunk/Objects/unicodeobject.c Thu Apr 13 08:06:08 2006 @@ -135,14 +135,9 @@ /* Resizing shared object (unicode_empty or single character objects) in-place is not allowed. Use PyUnicode_Resize() instead ! */ - if (unicode == unicode_empty || - (unicode->length == 1 && - /* MvL said unicode->str[] may be signed. Python generally assumes - * an int contains at least 32 bits, and we don't use more than - * 32 bits even in a UCS4 build, so casting to unsigned int should - * be correct. - */ - (unsigned int)unicode->str[0] < 256U && + if (unicode == unicode_empty || + (unicode->length == 1 && + unicode->str[0] < 256U && unicode_latin1[unicode->str[0]] == unicode)) { PyErr_SetString(PyExc_SystemError, "can't resize shared unicode objects"); From python-checkins at python.org Thu Apr 13 08:34:32 2006 From: python-checkins at python.org (martin.v.loewis) Date: Thu, 13 Apr 2006 08:34:32 +0200 (CEST) Subject: [Python-checkins] r45337 - python/trunk/Objects/unicodeobject.c Message-ID: <20060413063432.C87AD1E4010@bag.python.org> Author: martin.v.loewis Date: Thu Apr 13 08:34:32 2006 New Revision: 45337 Modified: python/trunk/Objects/unicodeobject.c Log: Change more ints to Py_ssize_t. Modified: python/trunk/Objects/unicodeobject.c ============================================================================== --- python/trunk/Objects/unicodeobject.c (original) +++ python/trunk/Objects/unicodeobject.c Thu Apr 13 08:34:32 2006 @@ -154,8 +154,7 @@ return -1; } unicode->str[length] = 0; - assert(length < INT_MAX); - unicode->length = (int)length; + unicode->length = length; reset: /* Reset the object caches */ @@ -368,7 +367,7 @@ #else { register Py_UNICODE *u; - register int i; + register Py_ssize_t i; u = PyUnicode_AS_UNICODE(unicode); for (i = size; i > 0; i--) *u++ = *w++; @@ -396,7 +395,7 @@ #else { register Py_UNICODE *u; - register int i; + register Py_ssize_t i; u = PyUnicode_AS_UNICODE(unicode); for (i = size; i > 0; i--) *w++ = *u++; @@ -1358,7 +1357,7 @@ PyObject *v; /* result string object */ char *p; /* next free byte in output buffer */ Py_ssize_t nallocated; /* number of result bytes allocated */ - int nneeded; /* number of result bytes needed */ + Py_ssize_t nneeded; /* number of result bytes needed */ char stackbuf[MAX_SHORT_UNICHARS * 4]; assert(s != NULL); @@ -1427,13 +1426,13 @@ if (v == NULL) { /* This was stack allocated. */ - nneeded = Py_SAFE_DOWNCAST(p - stackbuf, long, int); + nneeded = p - stackbuf; assert(nneeded <= nallocated); v = PyString_FromStringAndSize(stackbuf, nneeded); } else { /* Cut back to size actually needed. */ - nneeded = Py_SAFE_DOWNCAST(p - PyString_AS_STRING(v), long, int); + nneeded = p - PyString_AS_STRING(v); assert(nneeded <= nallocated); _PyString_Resize(&v, nneeded); } @@ -1934,7 +1933,7 @@ nextByte: ; } - if (_PyUnicode_Resize(&v, (int)(p - PyUnicode_AS_UNICODE(v))) < 0) + if (_PyUnicode_Resize(&v, p - PyUnicode_AS_UNICODE(v)) < 0) goto onError; Py_XDECREF(errorHandler); Py_XDECREF(exc); @@ -2003,7 +2002,7 @@ #ifdef Py_UNICODE_WIDE /* Map 21-bit characters to '\U00xxxxxx' */ else if (ch >= 0x10000) { - int offset = p - PyString_AS_STRING(repr); + Py_ssize_t offset = p - PyString_AS_STRING(repr); /* Resize the string if necessary */ if (offset + 12 > PyString_GET_SIZE(repr)) { @@ -2205,7 +2204,7 @@ nextByte: ; } - if (_PyUnicode_Resize(&v, (int)(p - PyUnicode_AS_UNICODE(v))) < 0) + if (_PyUnicode_Resize(&v, p - PyUnicode_AS_UNICODE(v)) < 0) goto onError; Py_XDECREF(errorHandler); Py_XDECREF(exc); @@ -2348,7 +2347,7 @@ } } - if (_PyUnicode_Resize(&v, (int)(p - PyUnicode_AS_UNICODE(v))) < 0) + if (_PyUnicode_Resize(&v, p - PyUnicode_AS_UNICODE(v)) < 0) goto onError; Py_XDECREF(errorHandler); Py_XDECREF(exc); @@ -2723,7 +2722,7 @@ } } if (p - PyUnicode_AS_UNICODE(v) < PyString_GET_SIZE(v)) - if (_PyUnicode_Resize(&v, (int)(p - PyUnicode_AS_UNICODE(v))) < 0) + if (_PyUnicode_Resize(&v, p - PyUnicode_AS_UNICODE(v)) < 0) goto onError; Py_XDECREF(errorHandler); Py_XDECREF(exc); @@ -2982,7 +2981,7 @@ } } if (p - PyUnicode_AS_UNICODE(v) < PyUnicode_GET_SIZE(v)) - if (_PyUnicode_Resize(&v, (int)(p - PyUnicode_AS_UNICODE(v))) < 0) + if (_PyUnicode_Resize(&v, p - PyUnicode_AS_UNICODE(v)) < 0) goto onError; Py_XDECREF(errorHandler); Py_XDECREF(exc); @@ -3336,9 +3335,9 @@ Py_ssize_t startpos, Py_ssize_t endpos, Py_ssize_t *newpos) { - static char *argparse = "O!i;translating error handler must return (unicode, int) tuple"; + static char *argparse = "O!n;translating error handler must return (unicode, int) tuple"; - int i_newpos; + Py_ssize_t i_newpos; PyObject *restuple; PyObject *resunicode; @@ -3798,7 +3797,7 @@ Py_ssize_t end, PyUnicodeObject *substring) { - int count = 0; + Py_ssize_t count = 0; if (start < 0) start += self->length; @@ -4157,7 +4156,7 @@ PyObject *fseq; /* PySequence_Fast(seq) */ Py_ssize_t seqlen; /* len(fseq) -- number of items in sequence */ PyObject *item; - int i; + Py_ssize_t i; fseq = PySequence_Fast(seq, ""); if (fseq == NULL) { @@ -4206,7 +4205,7 @@ } /* Get space. */ - res = _PyUnicode_New((int)res_alloc); + res = _PyUnicode_New(res_alloc); if (res == NULL) goto onError; res_p = PyUnicode_AS_UNICODE(res); @@ -4236,11 +4235,11 @@ /* Make sure we have enough space for the separator and the item. */ itemlen = PyUnicode_GET_SIZE(item); new_res_used = res_used + itemlen; - if (new_res_used < res_used || new_res_used > INT_MAX) + if (new_res_used < res_used || new_res_used > PY_SSIZE_T_MAX) goto Overflow; if (i < seqlen - 1) { new_res_used += seplen; - if (new_res_used < res_used || new_res_used > INT_MAX) + if (new_res_used < res_used || new_res_used > PY_SSIZE_T_MAX) goto Overflow; } if (new_res_used > res_alloc) { @@ -4248,10 +4247,10 @@ do { size_t oldsize = res_alloc; res_alloc += res_alloc; - if (res_alloc < oldsize || res_alloc > INT_MAX) + if (res_alloc < oldsize || res_alloc > PY_SSIZE_T_MAX) goto Overflow; } while (new_res_used > res_alloc); - if (_PyUnicode_Resize(&res, (int)res_alloc) < 0) { + if (_PyUnicode_Resize(&res, res_alloc) < 0) { Py_DECREF(item); goto onError; } @@ -4259,10 +4258,10 @@ } /* Copy item, and maybe the separator. */ - Py_UNICODE_COPY(res_p, PyUnicode_AS_UNICODE(item), (int)itemlen); + Py_UNICODE_COPY(res_p, PyUnicode_AS_UNICODE(item), itemlen); res_p += itemlen; if (i < seqlen - 1) { - Py_UNICODE_COPY(res_p, sep, (int)seplen); + Py_UNICODE_COPY(res_p, sep, seplen); res_p += seplen; } Py_DECREF(item); @@ -4272,7 +4271,7 @@ /* Shrink res to match the used area; this probably can't fail, * but it's cheap to check. */ - if (_PyUnicode_Resize(&res, (int)res_used) < 0) + if (_PyUnicode_Resize(&res, res_used) < 0) goto onError; Done: @@ -4605,7 +4604,7 @@ PyObject *list; if (maxcount < 0) - maxcount = INT_MAX; + maxcount = PY_SSIZE_T_MAX; list = PyList_New(0); if (!list) @@ -4634,7 +4633,7 @@ PyObject *list; if (maxcount < 0) - maxcount = INT_MAX; + maxcount = PY_SSIZE_T_MAX; list = PyList_New(0); if (!list) @@ -4664,10 +4663,10 @@ PyUnicodeObject *u; if (maxcount < 0) - maxcount = INT_MAX; + maxcount = PY_SSIZE_T_MAX; if (str1->length == 1 && str2->length == 1) { - int i; + Py_ssize_t i; /* replace characters */ if (!findchar(self->str, self->length, str1->str[0]) && @@ -5088,7 +5087,7 @@ { PyUnicodeObject *substring; Py_ssize_t start = 0; - Py_ssize_t end = INT_MAX; + Py_ssize_t end = PY_SSIZE_T_MAX; PyObject *result; if (!PyArg_ParseTuple(args, "O|O&O&:count", &substring, @@ -5265,7 +5264,7 @@ { PyUnicodeObject *substring; Py_ssize_t start = 0; - Py_ssize_t end = INT_MAX; + Py_ssize_t end = PY_SSIZE_T_MAX; PyObject *result; if (!PyArg_ParseTuple(args, "O|O&O&:find", &substring, @@ -5331,7 +5330,7 @@ Py_ssize_t result; PyUnicodeObject *substring; Py_ssize_t start = 0; - Py_ssize_t end = INT_MAX; + Py_ssize_t end = PY_SSIZE_T_MAX; if (!PyArg_ParseTuple(args, "O|O&O&:index", &substring, _PyEval_SliceIndex, &start, _PyEval_SliceIndex, &end)) @@ -5669,10 +5668,10 @@ static PyObject * unicode_ljust(PyUnicodeObject *self, PyObject *args) { - int width; + Py_ssize_t width; Py_UNICODE fillchar = ' '; - if (!PyArg_ParseTuple(args, "i|O&:ljust", &width, convert_uc, &fillchar)) + if (!PyArg_ParseTuple(args, "n|O&:ljust", &width, convert_uc, &fillchar)) return NULL; if (self->length >= width && PyUnicode_CheckExact(self)) { @@ -5996,7 +5995,7 @@ { PyUnicodeObject *substring; Py_ssize_t start = 0; - Py_ssize_t end = INT_MAX; + Py_ssize_t end = PY_SSIZE_T_MAX; PyObject *result; if (!PyArg_ParseTuple(args, "O|O&O&:rfind", &substring, @@ -6024,7 +6023,7 @@ Py_ssize_t result; PyUnicodeObject *substring; Py_ssize_t start = 0; - Py_ssize_t end = INT_MAX; + Py_ssize_t end = PY_SSIZE_T_MAX; if (!PyArg_ParseTuple(args, "O|O&O&:rindex", &substring, _PyEval_SliceIndex, &start, _PyEval_SliceIndex, &end)) @@ -6053,10 +6052,10 @@ static PyObject * unicode_rjust(PyUnicodeObject *self, PyObject *args) { - int width; + Py_ssize_t width; Py_UNICODE fillchar = ' '; - if (!PyArg_ParseTuple(args, "i|O&:rjust", &width, convert_uc, &fillchar)) + if (!PyArg_ParseTuple(args, "n|O&:rjust", &width, convert_uc, &fillchar)) return NULL; if (self->length >= width && PyUnicode_CheckExact(self)) { @@ -6318,7 +6317,7 @@ { PyUnicodeObject *substring; Py_ssize_t start = 0; - Py_ssize_t end = INT_MAX; + Py_ssize_t end = PY_SSIZE_T_MAX; PyObject *result; if (!PyArg_ParseTuple(args, "O|O&O&:startswith", &substring, @@ -6349,7 +6348,7 @@ { PyUnicodeObject *substring; Py_ssize_t start = 0; - Py_ssize_t end = INT_MAX; + Py_ssize_t end = PY_SSIZE_T_MAX; PyObject *result; if (!PyArg_ParseTuple(args, "O|O&O&:endswith", &substring, From python-checkins at python.org Thu Apr 13 08:34:59 2006 From: python-checkins at python.org (neal.norwitz) Date: Thu, 13 Apr 2006 08:34:59 +0200 (CEST) Subject: [Python-checkins] r45338 - python/trunk/Lib/test/test_compiler.py Message-ID: <20060413063459.E65BA1E4009@bag.python.org> Author: neal.norwitz Date: Thu Apr 13 08:34:59 2006 New Revision: 45338 Modified: python/trunk/Lib/test/test_compiler.py Log: test_compile can be really long if we are using -u compiler. This may be causing the debian sparc buildbot to fail. Print a little message to let the user ^w buildbot know it's still thinking. We may want to adjust the time period which is currently 5 minutes. Will backport. Modified: python/trunk/Lib/test/test_compiler.py ============================================================================== --- python/trunk/Lib/test/test_compiler.py (original) +++ python/trunk/Lib/test/test_compiler.py Thu Apr 13 08:34:59 2006 @@ -1,10 +1,12 @@ import compiler from compiler.ast import flatten -import os +import os, sys, time, unittest import test.test_support -import unittest from random import random +# How much time in seconds can pass before we print a 'Still working' message. +_PRINT_WORKING_MSG_INTERVAL = 5 * 60 + class CompilerTest(unittest.TestCase): def testCompileLibrary(self): @@ -13,11 +15,18 @@ # that any of the code is correct, merely the compiler is able # to generate some kind of code for it. + next_time = time.time() + _PRINT_WORKING_MSG_INTERVAL libdir = os.path.dirname(unittest.__file__) testdir = os.path.dirname(test.test_support.__file__) for dir in [libdir, testdir]: for basename in os.listdir(dir): + # Print still working message since this test can be really slow + if next_time <= time.time(): + next_time = time.time() + _PRINT_WORKING_MSG_INTERVAL + print >>sys.__stdout__, \ + ' testCompileLibrary still working, be patient...' + if not basename.endswith(".py"): continue if not TEST_ALL and random() < 0.98: From python-checkins at python.org Thu Apr 13 08:36:31 2006 From: python-checkins at python.org (martin.v.loewis) Date: Thu, 13 Apr 2006 08:36:31 +0200 (CEST) Subject: [Python-checkins] r45339 - python/trunk/Modules/unicodedata.c Message-ID: <20060413063631.906F01E4009@bag.python.org> Author: martin.v.loewis Date: Thu Apr 13 08:36:31 2006 New Revision: 45339 Modified: python/trunk/Modules/unicodedata.c Log: Support NFD of very long strings. Modified: python/trunk/Modules/unicodedata.c ============================================================================== --- python/trunk/Modules/unicodedata.c (original) +++ python/trunk/Modules/unicodedata.c Thu Apr 13 08:36:31 2006 @@ -486,8 +486,8 @@ Py_UNICODE *i, *end, *o; /* Longest decomposition in Unicode 3.2: U+FDFA */ Py_UNICODE stack[20]; - int space, stackptr, isize; - int index, prefix, count; + Py_ssize_t space, isize; + int index, prefix, count, stackptr; unsigned char prev, cur; stackptr = 0; @@ -508,7 +508,7 @@ /* Hangul Decomposition adds three characters in a single step, so we need atleast that much room. */ if (space < 3) { - int newsize = PyString_GET_SIZE(result) + 10; + Py_ssize_t newsize = PyString_GET_SIZE(result) + 10; space += 10; if (PyUnicode_Resize(&result, newsize) == -1) return NULL; From python-checkins at python.org Thu Apr 13 09:15:31 2006 From: python-checkins at python.org (neal.norwitz) Date: Thu, 13 Apr 2006 09:15:31 +0200 (CEST) Subject: [Python-checkins] r45340 - python/branches/release24-maint/Lib/test/test_compiler.py Message-ID: <20060413071531.6CDAB1E4068@bag.python.org> Author: neal.norwitz Date: Thu Apr 13 09:15:30 2006 New Revision: 45340 Modified: python/branches/release24-maint/Lib/test/test_compiler.py Log: Backport 45338: test_compile can be really long if we are using -u compiler. This may be causing the debian sparc buildbot to fail. Print a little message to let the user ^w buildbot know it's still thinking. We may want to adjust the time period which is currently 5 minutes. Modified: python/branches/release24-maint/Lib/test/test_compiler.py ============================================================================== --- python/branches/release24-maint/Lib/test/test_compiler.py (original) +++ python/branches/release24-maint/Lib/test/test_compiler.py Thu Apr 13 09:15:30 2006 @@ -1,9 +1,11 @@ import compiler -import os +import os, sys, time, unittest import test.test_support -import unittest from random import random +# How much time in seconds can pass before we print a 'Still working' message. +_PRINT_WORKING_MSG_INTERVAL = 5 * 60 + class CompilerTest(unittest.TestCase): def testCompileLibrary(self): @@ -12,11 +14,18 @@ # that any of the code is correct, merely the compiler is able # to generate some kind of code for it. + next_time = time.time() + _PRINT_WORKING_MSG_INTERVAL libdir = os.path.dirname(unittest.__file__) testdir = os.path.dirname(test.test_support.__file__) for dir in [libdir, testdir]: for basename in os.listdir(dir): + # Print still working message since this test can be really slow + if next_time <= time.time(): + next_time = time.time() + _PRINT_WORKING_MSG_INTERVAL + print >>sys.__stdout__, \ + ' testCompileLibrary still working, be patient...' + if not basename.endswith(".py"): continue if not TEST_ALL and random() < 0.98: From python-checkins at python.org Thu Apr 13 09:19:02 2006 From: python-checkins at python.org (anthony.baxter) Date: Thu, 13 Apr 2006 09:19:02 +0200 (CEST) Subject: [Python-checkins] r45341 - python/trunk/Modules/_hotshot.c python/trunk/Modules/stropmodule.c python/trunk/Modules/timemodule.c Message-ID: <20060413071902.959281E4009@bag.python.org> Author: anthony.baxter Date: Thu Apr 13 09:19:01 2006 New Revision: 45341 Modified: python/trunk/Modules/_hotshot.c python/trunk/Modules/stropmodule.c python/trunk/Modules/timemodule.c Log: C++ compiler changes. casts, rename variables with reserved names. Modified: python/trunk/Modules/_hotshot.c ============================================================================== --- python/trunk/Modules/_hotshot.c (original) +++ python/trunk/Modules/_hotshot.c Thu Apr 13 09:19:01 2006 @@ -308,7 +308,7 @@ if ((err = unpack_packed_int(self, &len, 0))) return err; - buf = malloc(len); + buf = (char *)malloc(len); for (i=0; i < len; i++) { ch = fgetc(self->logfp); buf[i] = ch; @@ -1403,7 +1403,7 @@ ++rev; while (rev[i] != ' ' && rev[i] != '\0') ++i; - buffer = malloc(i + 1); + buffer = (char *)malloc(i + 1); if (buffer != NULL) { memmove(buffer, rev, i); buffer[i] = '\0'; Modified: python/trunk/Modules/stropmodule.c ============================================================================== --- python/trunk/Modules/stropmodule.c (original) +++ python/trunk/Modules/stropmodule.c Thu Apr 13 09:19:01 2006 @@ -446,16 +446,16 @@ { char *s, *s_new; Py_ssize_t i, n; - PyObject *new; + PyObject *newstr; int changed; WARN; if (PyString_AsStringAndSize(args, &s, &n)) return NULL; - new = PyString_FromStringAndSize(NULL, n); - if (new == NULL) + newstr = PyString_FromStringAndSize(NULL, n); + if (newstr == NULL) return NULL; - s_new = PyString_AsString(new); + s_new = PyString_AsString(newstr); changed = 0; for (i = 0; i < n; i++) { int c = Py_CHARMASK(*s++); @@ -467,11 +467,11 @@ s_new++; } if (!changed) { - Py_DECREF(new); + Py_DECREF(newstr); Py_INCREF(args); return args; } - return new; + return newstr; } @@ -485,16 +485,16 @@ { char *s, *s_new; Py_ssize_t i, n; - PyObject *new; + PyObject *newstr; int changed; WARN; if (PyString_AsStringAndSize(args, &s, &n)) return NULL; - new = PyString_FromStringAndSize(NULL, n); - if (new == NULL) + newstr = PyString_FromStringAndSize(NULL, n); + if (newstr == NULL) return NULL; - s_new = PyString_AsString(new); + s_new = PyString_AsString(newstr); changed = 0; for (i = 0; i < n; i++) { int c = Py_CHARMASK(*s++); @@ -506,11 +506,11 @@ s_new++; } if (!changed) { - Py_DECREF(new); + Py_DECREF(newstr); Py_INCREF(args); return args; } - return new; + return newstr; } @@ -525,16 +525,16 @@ { char *s, *s_new; Py_ssize_t i, n; - PyObject *new; + PyObject *newstr; int changed; WARN; if (PyString_AsStringAndSize(args, &s, &n)) return NULL; - new = PyString_FromStringAndSize(NULL, n); - if (new == NULL) + newstr = PyString_FromStringAndSize(NULL, n); + if (newstr == NULL) return NULL; - s_new = PyString_AsString(new); + s_new = PyString_AsString(newstr); changed = 0; if (0 < n) { int c = Py_CHARMASK(*s++); @@ -555,11 +555,11 @@ s_new++; } if (!changed) { - Py_DECREF(new); + Py_DECREF(newstr); Py_INCREF(args); return args; } - return new; + return newstr; } @@ -691,16 +691,16 @@ { char *s, *s_new; Py_ssize_t i, n; - PyObject *new; + PyObject *newstr; int changed; WARN; if (PyString_AsStringAndSize(args, &s, &n)) return NULL; - new = PyString_FromStringAndSize(NULL, n); - if (new == NULL) + newstr = PyString_FromStringAndSize(NULL, n); + if (newstr == NULL) return NULL; - s_new = PyString_AsString(new); + s_new = PyString_AsString(newstr); changed = 0; for (i = 0; i < n; i++) { int c = Py_CHARMASK(*s++); @@ -717,11 +717,11 @@ s_new++; } if (!changed) { - Py_DECREF(new); + Py_DECREF(newstr); Py_INCREF(args); return args; } - return new; + return newstr; } @@ -1141,7 +1141,7 @@ char *str, *pat,*sub,*new_s; Py_ssize_t len,pat_len,sub_len,out_len; Py_ssize_t count = -1; - PyObject *new; + PyObject *newstr; WARN; if (!PyArg_ParseTuple(args, "t#t#t#|n:replace", @@ -1165,14 +1165,14 @@ } if (out_len == -1) { /* we're returning another reference to the input string */ - new = PyTuple_GetItem(args, 0); - Py_XINCREF(new); + newstr = PyTuple_GetItem(args, 0); + Py_XINCREF(newstr); } else { - new = PyString_FromStringAndSize(new_s, out_len); + newstr = PyString_FromStringAndSize(new_s, out_len); PyMem_FREE(new_s); } - return new; + return newstr; } Modified: python/trunk/Modules/timemodule.c ============================================================================== --- python/trunk/Modules/timemodule.c (original) +++ python/trunk/Modules/timemodule.c Thu Apr 13 09:19:01 2006 @@ -443,7 +443,7 @@ * will be ahead of time... */ for (i = 1024; ; i += i) { - outbuf = malloc(i); + outbuf = (char *)malloc(i); if (outbuf == NULL) { return PyErr_NoMemory(); } From python-checkins at python.org Thu Apr 13 09:22:52 2006 From: python-checkins at python.org (martin.v.loewis) Date: Thu, 13 Apr 2006 09:22:52 +0200 (CEST) Subject: [Python-checkins] r45342 - python/trunk/Objects/stringobject.c Message-ID: <20060413072252.6E7641E4009@bag.python.org> Author: martin.v.loewis Date: Thu Apr 13 09:22:51 2006 New Revision: 45342 Modified: python/trunk/Objects/stringobject.c Log: Replace most INT_MAX with PY_SSIZE_T_MAX. Modified: python/trunk/Objects/stringobject.c ============================================================================== --- python/trunk/Objects/stringobject.c (original) +++ python/trunk/Objects/stringobject.c Thu Apr 13 09:22:51 2006 @@ -105,7 +105,7 @@ assert(str != NULL); size = strlen(str); - if (size > INT_MAX) { + if (size > PY_SSIZE_T_MAX) { PyErr_SetString(PyExc_OverflowError, "string is too long for a Python string"); return NULL; @@ -814,7 +814,7 @@ register PyStringObject* op = (PyStringObject*) obj; size_t newsize = 2 + 4 * op->ob_size; PyObject *v; - if (newsize > INT_MAX) { + if (newsize > PY_SSIZE_T_MAX) { PyErr_SetString(PyExc_OverflowError, "string is too large to make repr"); } @@ -1414,7 +1414,7 @@ if (!PyArg_ParseTuple(args, "|Oi:split", &subobj, &maxsplit)) return NULL; if (maxsplit < 0) - maxsplit = INT_MAX; + maxsplit = PY_SSIZE_T_MAX; if (subobj == Py_None) return split_whitespace(s, len, maxsplit); if (PyString_Check(subobj)) { @@ -1555,7 +1555,7 @@ if (!PyArg_ParseTuple(args, "|Oi:rsplit", &subobj, &maxsplit)) return NULL; if (maxsplit < 0) - maxsplit = INT_MAX; + maxsplit = PY_SSIZE_T_MAX; if (subobj == Py_None) return rsplit_whitespace(s, len, maxsplit); if (PyString_Check(subobj)) { @@ -1685,7 +1685,7 @@ sz += PyString_GET_SIZE(item); if (i != 0) sz += seplen; - if (sz < old_sz || sz > INT_MAX) { + if (sz < old_sz || sz > PY_SSIZE_T_MAX) { PyErr_SetString(PyExc_OverflowError, "join() is too long for a Python string"); Py_DECREF(seq); @@ -1746,7 +1746,7 @@ { const char *s = PyString_AS_STRING(self), *sub; Py_ssize_t len = PyString_GET_SIZE(self); - Py_ssize_t n, i = 0, last = INT_MAX; + Py_ssize_t n, i = 0, last = PY_SSIZE_T_MAX; PyObject *subobj; /* XXX ssize_t i */ @@ -2158,7 +2158,7 @@ { const char *s = PyString_AS_STRING(self), *sub, *t; Py_ssize_t len = PyString_GET_SIZE(self), n; - Py_ssize_t i = 0, last = INT_MAX; + Py_ssize_t i = 0, last = PY_SSIZE_T_MAX; Py_ssize_t m, r; PyObject *subobj; @@ -2446,7 +2446,7 @@ /* find length of output string */ nfound = (pat_len > 0) ? mymemcnt(str, len, pat, pat_len) : len + 1; if (count < 0) - count = INT_MAX; + count = PY_SSIZE_T_MAX; else if (nfound > count) nfound = count; if (nfound == 0) @@ -2595,7 +2595,7 @@ const char* prefix; Py_ssize_t plen; Py_ssize_t start = 0; - Py_ssize_t end = INT_MAX; + Py_ssize_t end = PY_SSIZE_T_MAX; PyObject *subobj; if (!PyArg_ParseTuple(args, "O|O&O&:startswith", &subobj, @@ -2646,7 +2646,7 @@ const char* suffix; Py_ssize_t slen; Py_ssize_t start = 0; - Py_ssize_t end = INT_MAX; + Py_ssize_t end = PY_SSIZE_T_MAX; PyObject *subobj; if (!PyArg_ParseTuple(args, "O|O&O&:endswith", &subobj, @@ -3701,7 +3701,7 @@ } buf = PyString_AsString(result); llen = PyString_Size(result); - if (llen > INT_MAX) { + if (llen > PY_SSIZE_T_MAX) { PyErr_SetString(PyExc_ValueError, "string too large in _PyString_FormatLong"); return NULL; } From python-checkins at python.org Thu Apr 13 09:24:51 2006 From: python-checkins at python.org (martin.v.loewis) Date: Thu, 13 Apr 2006 09:24:51 +0200 (CEST) Subject: [Python-checkins] r45343 - python/trunk/Objects/unicodeobject.c Message-ID: <20060413072451.578A21E4009@bag.python.org> Author: martin.v.loewis Date: Thu Apr 13 09:24:50 2006 New Revision: 45343 Modified: python/trunk/Objects/unicodeobject.c Log: Remove another INT_MAX limitation Modified: python/trunk/Objects/unicodeobject.c ============================================================================== --- python/trunk/Objects/unicodeobject.c (original) +++ python/trunk/Objects/unicodeobject.c Thu Apr 13 09:24:50 2006 @@ -225,8 +225,7 @@ */ unicode->str[0] = 0; unicode->str[length] = 0; - assert(lengthlength = (int)length; + unicode->length = length; unicode->hash = -1; unicode->defenc = NULL; return unicode; From python-checkins at python.org Thu Apr 13 09:28:29 2006 From: python-checkins at python.org (martin.v.loewis) Date: Thu, 13 Apr 2006 09:28:29 +0200 (CEST) Subject: [Python-checkins] r45344 - python/trunk/Doc/api/intro.tex Message-ID: <20060413072829.8EF451E4009@bag.python.org> Author: martin.v.loewis Date: Thu Apr 13 09:28:29 2006 New Revision: 45344 Modified: python/trunk/Doc/api/intro.tex Log: Stop claiming that Py_Finalize releases all memory. Fixes part of #1445210. Modified: python/trunk/Doc/api/intro.tex ============================================================================== --- python/trunk/Doc/api/intro.tex (original) +++ python/trunk/Doc/api/intro.tex Thu Apr 13 09:28:29 2006 @@ -572,8 +572,11 @@ Sometimes, it is desirable to ``uninitialize'' Python. For instance, the application may want to start over (make another call to \cfunction{Py_Initialize()}) or the application is simply done with its -use of Python and wants to free all memory allocated by Python. This +use of Python and wants to free memory allocated by Python. This can be accomplished by calling \cfunction{Py_Finalize()}. The function \cfunction{Py_IsInitialized()}\ttindex{Py_IsInitialized()} returns true if Python is currently in the initialized state. More information about these functions is given in a later chapter. +Notice that \cfunction{Py_Finalize} does \emph{not} free all memory +allocated by the Python interpreter, e.g. memory allocated by extension +modules currently cannot be released. From python-checkins at python.org Thu Apr 13 09:29:19 2006 From: python-checkins at python.org (georg.brandl) Date: Thu, 13 Apr 2006 09:29:19 +0200 (CEST) Subject: [Python-checkins] r45345 - python/trunk/Modules/_testcapimodule.c Message-ID: <20060413072919.235C91E4009@bag.python.org> Author: georg.brandl Date: Thu Apr 13 09:29:18 2006 New Revision: 45345 Modified: python/trunk/Modules/_testcapimodule.c Log: Add PY_SSIZE_T_MIN/MAX to _testcapi. Modified: python/trunk/Modules/_testcapimodule.c ============================================================================== --- python/trunk/Modules/_testcapimodule.c (original) +++ python/trunk/Modules/_testcapimodule.c Thu Apr 13 09:29:18 2006 @@ -704,8 +704,10 @@ PyModule_AddObject(m, "ULONG_MAX", PyLong_FromUnsignedLong(ULONG_MAX)); PyModule_AddObject(m, "INT_MIN", PyInt_FromLong(INT_MIN)); PyModule_AddObject(m, "LONG_MIN", PyInt_FromLong(LONG_MIN)); + PyModule_AddObject(m, "PY_SSIZE_T_MIN", PyInt_FromSsize_t(PY_SSIZE_T_MIN)); PyModule_AddObject(m, "INT_MAX", PyInt_FromLong(INT_MAX)); PyModule_AddObject(m, "LONG_MAX", PyInt_FromLong(LONG_MAX)); + PyModule_AddObject(m, "PY_SSIZE_T_MAX", PyInt_FromSsize_t(PY_SSIZE_T_MAX)); TestError = PyErr_NewException("_testcapi.error", NULL, NULL); Py_INCREF(TestError); From python-checkins at python.org Thu Apr 13 09:34:10 2006 From: python-checkins at python.org (martin.v.loewis) Date: Thu, 13 Apr 2006 09:34:10 +0200 (CEST) Subject: [Python-checkins] r45346 - python/trunk/Modules/stropmodule.c Message-ID: <20060413073410.41DA31E4009@bag.python.org> Author: martin.v.loewis Date: Thu Apr 13 09:34:09 2006 New Revision: 45346 Modified: python/trunk/Modules/stropmodule.c Log: Replace INT_MAX with PY_SSIZE_T_MAX. Modified: python/trunk/Modules/stropmodule.c ============================================================================== --- python/trunk/Modules/stropmodule.c (original) +++ python/trunk/Modules/stropmodule.c Thu Apr 13 09:34:09 2006 @@ -333,7 +333,7 @@ { char *s, *sub; Py_ssize_t len, n, j; - Py_ssize_t i = 0, last = INT_MAX; + Py_ssize_t i = 0, last = PY_SSIZE_T_MAX; WARN; if (!PyArg_ParseTuple(args, "t#t#|nn:rfind", &s, &len, &sub, &n, &i, &last)) @@ -647,7 +647,7 @@ { char *s, *sub; Py_ssize_t len, n; - Py_ssize_t i = 0, last = INT_MAX; + Py_ssize_t i = 0, last = PY_SSIZE_T_MAX; Py_ssize_t m, r; WARN; @@ -1078,7 +1078,7 @@ /* find length of output string */ nfound = mymemcnt(str, len, pat, pat_len); if (count < 0) - count = INT_MAX; + count = PY_SSIZE_T_MAX; else if (nfound > count) nfound = count; if (nfound == 0) From python-checkins at python.org Thu Apr 13 09:37:25 2006 From: python-checkins at python.org (martin.v.loewis) Date: Thu, 13 Apr 2006 09:37:25 +0200 (CEST) Subject: [Python-checkins] r45347 - python/trunk/Objects/fileobject.c Message-ID: <20060413073725.931091E4009@bag.python.org> Author: martin.v.loewis Date: Thu Apr 13 09:37:25 2006 New Revision: 45347 Modified: python/trunk/Objects/fileobject.c Log: Replace INT_MAX with PY_SSIZE_T_MAX where string length are concerned. Modified: python/trunk/Objects/fileobject.c ============================================================================== --- python/trunk/Objects/fileobject.c (original) +++ python/trunk/Objects/fileobject.c Thu Apr 13 09:37:25 2006 @@ -823,7 +823,7 @@ buffersize = new_buffersize(f, (size_t)0); else buffersize = bytesrequested; - if (buffersize > INT_MAX) { + if (buffersize > PY_SSIZE_T_MAX) { PyErr_SetString(PyExc_OverflowError, "requested number of bytes is more than a Python string can hold"); return NULL; @@ -1098,7 +1098,7 @@ assert(*(pvend-1) == '\0'); increment = total_v_size >> 2; /* mild exponential growth */ total_v_size += increment; - if (total_v_size > INT_MAX) { + if (total_v_size > PY_SSIZE_T_MAX) { PyErr_SetString(PyExc_OverflowError, "line is longer than a Python string can hold"); Py_DECREF(v); @@ -1209,7 +1209,7 @@ used_v_size = total_v_size; increment = total_v_size >> 2; /* mild exponential growth */ total_v_size += increment; - if (total_v_size > INT_MAX) { + if (total_v_size > PY_SSIZE_T_MAX) { PyErr_SetString(PyExc_OverflowError, "line is longer than a Python string can hold"); Py_DECREF(v); @@ -1401,7 +1401,7 @@ /* Need a larger buffer to fit this line */ nfilled += nread; buffersize *= 2; - if (buffersize > INT_MAX) { + if (buffersize > PY_SSIZE_T_MAX) { PyErr_SetString(PyExc_OverflowError, "line is longer than a Python string can hold"); goto error; From buildbot at python.org Thu Apr 13 09:43:20 2006 From: buildbot at python.org (buildbot at python.org) Date: Thu, 13 Apr 2006 07:43:20 +0000 Subject: [Python-checkins] buildbot failure in amd64 gentoo trunk Message-ID: <20060413074320.50FB01E4009@bag.python.org> The Buildbot has detected a new failure of amd64 gentoo trunk. Full details are available at: http://www.python.org/dev/buildbot/all/amd64%2520gentoo%2520trunk/builds/448 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: anthony.baxter,georg.brandl,martin.v.loewis BUILD FAILED: failed compile sincerely, -The Buildbot From python-checkins at python.org Thu Apr 13 09:52:28 2006 From: python-checkins at python.org (martin.v.loewis) Date: Thu, 13 Apr 2006 09:52:28 +0200 (CEST) Subject: [Python-checkins] r45348 - in python/trunk: Modules/functionalmodule.c Objects/listobject.c Python/bltinmodule.c Python/codecs.c Python/modsupport.c Message-ID: <20060413075228.C5E6A1E4009@bag.python.org> Author: martin.v.loewis Date: Thu Apr 13 09:52:27 2006 New Revision: 45348 Modified: python/trunk/Modules/functionalmodule.c python/trunk/Objects/listobject.c python/trunk/Python/bltinmodule.c python/trunk/Python/codecs.c python/trunk/Python/modsupport.c Log: Replace INT_MAX with PY_SSIZE_T_MAX. Modified: python/trunk/Modules/functionalmodule.c ============================================================================== --- python/trunk/Modules/functionalmodule.c (original) +++ python/trunk/Modules/functionalmodule.c Thu Apr 13 09:52:27 2006 @@ -48,7 +48,7 @@ pto->fn = func; Py_INCREF(func); - pto->args = PyTuple_GetSlice(args, 1, INT_MAX); + pto->args = PyTuple_GetSlice(args, 1, PY_SSIZE_T_MAX); if (pto->args == NULL) { pto->kw = NULL; Py_DECREF(pto); Modified: python/trunk/Objects/listobject.c ============================================================================== --- python/trunk/Objects/listobject.c (original) +++ python/trunk/Objects/listobject.c Thu Apr 13 09:52:27 2006 @@ -181,7 +181,7 @@ PyErr_BadInternalCall(); return -1; } - if (n == INT_MAX) { + if (n == PY_SSIZE_T_MAX) { PyErr_SetString(PyExc_OverflowError, "cannot add more objects to list"); return -1; @@ -221,7 +221,7 @@ Py_ssize_t n = PyList_GET_SIZE(self); assert (v != NULL); - if (n == INT_MAX) { + if (n == PY_SSIZE_T_MAX) { PyErr_SetString(PyExc_OverflowError, "cannot add more objects to list"); return -1; Modified: python/trunk/Python/bltinmodule.c ============================================================================== --- python/trunk/Python/bltinmodule.c (original) +++ python/trunk/Python/bltinmodule.c Thu Apr 13 09:52:27 2006 @@ -1746,14 +1746,13 @@ } else { /* strip trailing '\n' */ size_t len = strlen(s); - if (len > INT_MAX) { + if (len > PY_SSIZE_T_MAX) { PyErr_SetString(PyExc_OverflowError, "[raw_]input: input too long"); result = NULL; } else { - result = PyString_FromStringAndSize(s, - (int)(len-1)); + result = PyString_FromStringAndSize(s, len-1); } } PyMem_FREE(s); Modified: python/trunk/Python/codecs.c ============================================================================== --- python/trunk/Python/codecs.c (original) +++ python/trunk/Python/codecs.c Thu Apr 13 09:52:27 2006 @@ -56,12 +56,12 @@ char *p; PyObject *v; - if (len > INT_MAX) { - PyErr_SetString(PyExc_OverflowError, "string is too large"); - return NULL; - } + if (len > PY_SSIZE_T_MAX) { + PyErr_SetString(PyExc_OverflowError, "string is too large"); + return NULL; + } - v = PyString_FromStringAndSize(NULL, (int)len); + v = PyString_FromStringAndSize(NULL, len); if (v == NULL) return NULL; p = PyString_AS_STRING(v); Modified: python/trunk/Python/modsupport.c ============================================================================== --- python/trunk/Python/modsupport.c (original) +++ python/trunk/Python/modsupport.c Thu Apr 13 09:52:27 2006 @@ -407,7 +407,7 @@ else { if (n < 0) { size_t m = strlen(str); - if (m > INT_MAX) { + if (m > PY_SSIZE_T_MAX) { PyErr_SetString(PyExc_OverflowError, "string too long for Python string"); return NULL; From python-checkins at python.org Thu Apr 13 09:59:31 2006 From: python-checkins at python.org (georg.brandl) Date: Thu, 13 Apr 2006 09:59:31 +0200 (CEST) Subject: [Python-checkins] r45349 - in python/trunk: Lib/test/test_getargs2.py Modules/_testcapimodule.c Python/getargs.c Message-ID: <20060413075931.8E8B41E4002@bag.python.org> Author: georg.brandl Date: Thu Apr 13 09:59:30 2006 New Revision: 45349 Modified: python/trunk/Lib/test/test_getargs2.py python/trunk/Modules/_testcapimodule.c python/trunk/Python/getargs.c Log: Add a test for Py_ssize_t. Correct typo in getargs.c. Modified: python/trunk/Lib/test/test_getargs2.py ============================================================================== --- python/trunk/Lib/test/test_getargs2.py (original) +++ python/trunk/Lib/test/test_getargs2.py Thu Apr 13 09:59:30 2006 @@ -48,7 +48,7 @@ VERY_LARGE = 0xFF0000121212121212121242L from _testcapi import UCHAR_MAX, USHRT_MAX, UINT_MAX, ULONG_MAX, INT_MAX, \ - INT_MIN, LONG_MIN, LONG_MAX + INT_MIN, LONG_MIN, LONG_MAX, PY_SSIZE_T_MIN, PY_SSIZE_T_MAX # fake, they are not defined in Python's header files LLONG_MAX = 2**63-1 @@ -182,6 +182,23 @@ self.failUnlessEqual(42, getargs_l(42L)) self.assertRaises(OverflowError, getargs_l, VERY_LARGE) + def test_n(self): + from _testcapi import getargs_n + # n returns 'Py_ssize_t', and does range checking + # (PY_SSIZE_T_MIN ... PY_SSIZE_T_MAX) + self.failUnlessEqual(3, getargs_n(3.14)) + self.failUnlessEqual(99, getargs_n(Long())) + self.failUnlessEqual(99, getargs_n(Int())) + + self.assertRaises(OverflowError, getargs_n, PY_SSIZE_T_MIN-1) + self.failUnlessEqual(PY_SSIZE_T_MIN, getargs_n(PY_SSIZE_T_MIN)) + self.failUnlessEqual(PY_SSIZE_T_MAX, getargs_n(PY_SSIZE_T_MAX)) + self.assertRaises(OverflowError, getargs_n, PY_SSIZE_T_MAX+1) + + self.failUnlessEqual(42, getargs_n(42)) + self.failUnlessEqual(42, getargs_n(42L)) + self.assertRaises(OverflowError, getargs_n, VERY_LARGE) + class LongLong_TestCase(unittest.TestCase): def test_L(self): Modified: python/trunk/Modules/_testcapimodule.c ============================================================================== --- python/trunk/Modules/_testcapimodule.c (original) +++ python/trunk/Modules/_testcapimodule.c Thu Apr 13 09:59:30 2006 @@ -360,6 +360,15 @@ return PyLong_FromLong(value); } +static PyObject * +getargs_n(PyObject *self, PyObject *args) +{ + Py_ssize_t value; + if (!PyArg_ParseTuple(args, "n", &value)) + return NULL; + return PyInt_FromSsize_t(value); +} + #ifdef HAVE_LONG_LONG static PyObject * getargs_L(PyObject *self, PyObject *args) @@ -661,17 +670,18 @@ {"test_k_code", (PyCFunction)test_k_code, METH_NOARGS}, {"test_null_strings", (PyCFunction)test_null_strings, METH_NOARGS}, - {"getargs_b", (PyCFunction)getargs_b, METH_VARARGS}, - {"getargs_B", (PyCFunction)getargs_B, METH_VARARGS}, - {"getargs_H", (PyCFunction)getargs_H, METH_VARARGS}, - {"getargs_I", (PyCFunction)getargs_I, METH_VARARGS}, - {"getargs_k", (PyCFunction)getargs_k, METH_VARARGS}, - {"getargs_i", (PyCFunction)getargs_i, METH_VARARGS}, - {"getargs_l", (PyCFunction)getargs_l, METH_VARARGS}, + {"getargs_b", getargs_b, METH_VARARGS}, + {"getargs_B", getargs_B, METH_VARARGS}, + {"getargs_H", getargs_H, METH_VARARGS}, + {"getargs_I", getargs_I, METH_VARARGS}, + {"getargs_k", getargs_k, METH_VARARGS}, + {"getargs_i", getargs_i, METH_VARARGS}, + {"getargs_l", getargs_l, METH_VARARGS}, + {"getargs_n", getargs_n, METH_VARARGS}, #ifdef HAVE_LONG_LONG - {"getargs_L", (PyCFunction)getargs_L, METH_VARARGS}, - {"getargs_K", (PyCFunction)getargs_K, METH_VARARGS}, - {"test_longlong_api", (PyCFunction)test_longlong_api, METH_NOARGS}, + {"getargs_L", getargs_L, METH_VARARGS}, + {"getargs_K", getargs_K, METH_VARARGS}, + {"test_longlong_api", test_longlong_api, METH_NOARGS}, {"test_L_code", (PyCFunction)test_L_code, METH_NOARGS}, {"codec_incrementalencoder", (PyCFunction)codec_incrementalencoder, METH_VARARGS}, @@ -682,7 +692,7 @@ {"test_u_code", (PyCFunction)test_u_code, METH_NOARGS}, #endif #ifdef WITH_THREAD - {"_test_thread_state", (PyCFunction)test_thread_state, METH_VARARGS}, + {"_test_thread_state", test_thread_state, METH_VARARGS}, #endif {NULL, NULL} /* sentinel */ }; Modified: python/trunk/Python/getargs.c ============================================================================== --- python/trunk/Python/getargs.c (original) +++ python/trunk/Python/getargs.c Thu Apr 13 09:59:30 2006 @@ -647,10 +647,10 @@ Py_ssize_t *p = va_arg(*p_va, Py_ssize_t *); Py_ssize_t ival; if (float_argument_error(arg)) - return converterr("integer", arg, msgbuf, bufsize); + return converterr("integer", arg, msgbuf, bufsize); ival = PyInt_AsSsize_t(arg); if (ival == -1 && PyErr_Occurred()) - return converterr("integer", arg, msgbuf, bufsize); + return converterr("integer", arg, msgbuf, bufsize); *p = ival; break; } From buildbot at python.org Thu Apr 13 10:03:01 2006 From: buildbot at python.org (buildbot at python.org) Date: Thu, 13 Apr 2006 08:03:01 +0000 Subject: [Python-checkins] buildbot failure in ia64 Debian unstable trunk Message-ID: <20060413080301.3F9A21E4002@bag.python.org> The Buildbot has detected a new failure of ia64 Debian unstable trunk. Full details are available at: http://www.python.org/dev/buildbot/all/ia64%2520Debian%2520unstable%2520trunk/builds/111 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: anthony.baxter,georg.brandl,martin.v.loewis BUILD FAILED: failed compile sincerely, -The Buildbot From python-checkins at python.org Thu Apr 13 10:04:57 2006 From: python-checkins at python.org (georg.brandl) Date: Thu, 13 Apr 2006 10:04:57 +0200 (CEST) Subject: [Python-checkins] r45350 - python/trunk/Doc/tut/tut.tex Message-ID: <20060413080457.1AB5F1E4002@bag.python.org> Author: georg.brandl Date: Thu Apr 13 10:04:56 2006 New Revision: 45350 Modified: python/trunk/Doc/tut/tut.tex Log: Add two entries about how to actually clear a list. Modified: python/trunk/Doc/tut/tut.tex ============================================================================== --- python/trunk/Doc/tut/tut.tex (original) +++ python/trunk/Doc/tut/tut.tex Thu Apr 13 10:04:56 2006 @@ -1012,7 +1012,7 @@ \end{verbatim} Assignment to slices is also possible, and this can even change the size -of the list: +of the list or clear it entirely: \begin{verbatim} >>> # Replace some items: @@ -1027,9 +1027,14 @@ ... a[1:1] = ['bletch', 'xyzzy'] >>> a [123, 'bletch', 'xyzzy', 1234] ->>> a[:0] = a # Insert (a copy of) itself at the beginning +>>> # Insert (a copy of) itself at the beginning +>>> a[:0] = a >>> a [123, 'bletch', 'xyzzy', 1234, 123, 'bletch', 'xyzzy', 1234] +>>> # Clear the list: replace all items with an empty list +>>> a[:] = [] +>>> a +[] \end{verbatim} The built-in function \function{len()} also applies to lists: @@ -2011,9 +2016,9 @@ There is a way to remove an item from a list given its index instead of its value: the \keyword{del} statement. This differs from the \method{pop()}) method which returns a value. The \keyword{del} -statement can also be used to -remove slices from a list (which we did earlier by assignment of an -empty list to the slice). For example: +statement can also be used to remove slices from a list or clear the +entire list (which we did earlier by assignment of an empty list to +the slice). For example: \begin{verbatim} >>> a = [-1, 1, 66.25, 333, 333, 1234.5] @@ -2023,6 +2028,9 @@ >>> del a[2:4] >>> a [1, 66.25, 1234.5] +>>> del a[:] +>>> a +[] \end{verbatim} \keyword{del} can also be used to delete entire variables: From python-checkins at python.org Thu Apr 13 10:33:59 2006 From: python-checkins at python.org (thomas.heller) Date: Thu, 13 Apr 2006 10:33:59 +0200 (CEST) Subject: [Python-checkins] r45351 - in python/trunk: PC/dllbase_nt.txt PCbuild/_ctypes.vcproj Message-ID: <20060413083359.47C4C1E4002@bag.python.org> Author: thomas.heller Date: Thu Apr 13 10:33:58 2006 New Revision: 45351 Modified: python/trunk/PC/dllbase_nt.txt python/trunk/PCbuild/_ctypes.vcproj Log: Allocate the 0x1D1A0000 base adress for the _ctypes extension module. Export the COM entry point functions. Note that there are several conflicting base addresses defined for other extensions, also not all are listed in dllbase_nt.txt. Modified: python/trunk/PC/dllbase_nt.txt ============================================================================== --- python/trunk/PC/dllbase_nt.txt (original) +++ python/trunk/PC/dllbase_nt.txt Thu Apr 13 10:33:58 2006 @@ -33,6 +33,7 @@ - bZ2 1D170000 - 1D180000 - datetime 1D180000 - 1D190000 pyd removed in 2.4 - _csv 1D190000 - 1D1A0000 pyd removed in 2.4 + - _ctypes 1D1A0000 - 1D1B0000 Other extension modules - win32api 1e200000 - 1e220000 Modified: python/trunk/PCbuild/_ctypes.vcproj ============================================================================== --- python/trunk/PCbuild/_ctypes.vcproj (original) +++ python/trunk/PCbuild/_ctypes.vcproj Thu Apr 13 10:33:58 2006 @@ -33,12 +33,14 @@ Name="VCCustomBuildTool"/> Author: martin.v.loewis Date: Thu Apr 13 10:37:17 2006 New Revision: 45352 Modified: python/trunk/Objects/stringobject.c Log: Change maxsplit types to Py_ssize_t. Modified: python/trunk/Objects/stringobject.c ============================================================================== --- python/trunk/Objects/stringobject.c (original) +++ python/trunk/Objects/stringobject.c Thu Apr 13 10:37:17 2006 @@ -1407,11 +1407,11 @@ { Py_ssize_t len = PyString_GET_SIZE(self), n, i, j; int err; - int maxsplit = -1; + Py_ssize_t maxsplit = -1; const char *s = PyString_AS_STRING(self), *sub; PyObject *list, *item, *subobj = Py_None; - if (!PyArg_ParseTuple(args, "|Oi:split", &subobj, &maxsplit)) + if (!PyArg_ParseTuple(args, "|On:split", &subobj, &maxsplit)) return NULL; if (maxsplit < 0) maxsplit = PY_SSIZE_T_MAX; @@ -1548,11 +1548,11 @@ { Py_ssize_t len = PyString_GET_SIZE(self), n, i, j; int err; - int maxsplit = -1; + Py_ssize_t maxsplit = -1; const char *s = PyString_AS_STRING(self), *sub; PyObject *list, *item, *subobj = Py_None; - if (!PyArg_ParseTuple(args, "|Oi:rsplit", &subobj, &maxsplit)) + if (!PyArg_ParseTuple(args, "|On:rsplit", &subobj, &maxsplit)) return NULL; if (maxsplit < 0) maxsplit = PY_SSIZE_T_MAX; From python-checkins at python.org Thu Apr 13 10:52:57 2006 From: python-checkins at python.org (martin.v.loewis) Date: Thu, 13 Apr 2006 10:52:57 +0200 (CEST) Subject: [Python-checkins] r45353 - python/trunk/Objects/stringobject.c Message-ID: <20060413085257.0127C1E4002@bag.python.org> Author: martin.v.loewis Date: Thu Apr 13 10:52:56 2006 New Revision: 45353 Modified: python/trunk/Objects/stringobject.c Log: Change more occurrences of maxsplit to Py_ssize_t. Modified: python/trunk/Objects/stringobject.c ============================================================================== --- python/trunk/Objects/stringobject.c (original) +++ python/trunk/Objects/stringobject.c Thu Apr 13 10:52:56 2006 @@ -1332,7 +1332,7 @@ Py_DECREF(str); static PyObject * -split_whitespace(const char *s, Py_ssize_t len, int maxsplit) +split_whitespace(const char *s, Py_ssize_t len, Py_ssize_t maxsplit) { Py_ssize_t i, j; PyObject *str; @@ -1366,7 +1366,7 @@ } static PyObject * -split_char(const char *s, Py_ssize_t len, char ch, int maxcount) +split_char(const char *s, Py_ssize_t len, char ch, Py_ssize_t maxcount) { register Py_ssize_t i, j; PyObject *str; @@ -1472,7 +1472,7 @@ } static PyObject * -rsplit_whitespace(const char *s, Py_ssize_t len, int maxsplit) +rsplit_whitespace(const char *s, Py_ssize_t len, Py_ssize_t maxsplit) { Py_ssize_t i, j; PyObject *str; @@ -1506,7 +1506,7 @@ } static PyObject * -rsplit_char(const char *s, Py_ssize_t len, char ch, int maxcount) +rsplit_char(const char *s, Py_ssize_t len, char ch, Py_ssize_t maxcount) { register Py_ssize_t i, j; PyObject *str; From buildbot at python.org Thu Apr 13 10:58:24 2006 From: buildbot at python.org (buildbot at python.org) Date: Thu, 13 Apr 2006 08:58:24 +0000 Subject: [Python-checkins] buildbot failure in alpha Tru64 5.1 trunk Message-ID: <20060413085824.6247C1E4002@bag.python.org> The Buildbot has detected a new failure of alpha Tru64 5.1 trunk. Full details are available at: http://www.python.org/dev/buildbot/all/alpha%2520Tru64%25205.1%2520trunk/builds/200 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: anthony.baxter,georg.brandl,martin.v.loewis,thomas.heller BUILD FAILED: failed compile sincerely, -The Buildbot From buildbot at python.org Thu Apr 13 11:36:55 2006 From: buildbot at python.org (buildbot at python.org) Date: Thu, 13 Apr 2006 09:36:55 +0000 Subject: [Python-checkins] buildbot warnings in ia64 Debian unstable trunk Message-ID: <20060413093655.DC4AC1E4002@bag.python.org> The Buildbot has detected a new failure of ia64 Debian unstable trunk. Full details are available at: http://www.python.org/dev/buildbot/all/ia64%2520Debian%2520unstable%2520trunk/builds/116 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: martin.v.loewis Build Had Warnings: warnings test sincerely, -The Buildbot From python-checkins at python.org Thu Apr 13 11:37:02 2006 From: python-checkins at python.org (skip.montanaro) Date: Thu, 13 Apr 2006 11:37:02 +0200 (CEST) Subject: [Python-checkins] r45354 - in python/trunk: Include/asdl.h Python/compile.c Message-ID: <20060413093702.DF7861E4002@bag.python.org> Author: skip.montanaro Date: Thu Apr 13 11:37:01 2006 New Revision: 45354 Modified: python/trunk/Include/asdl.h python/trunk/Python/compile.c Log: Use union to discriminate pointer types from enum/int types. Modified: python/trunk/Include/asdl.h ============================================================================== --- python/trunk/Include/asdl.h (original) +++ python/trunk/Include/asdl.h Thu Apr 13 11:37:01 2006 @@ -19,18 +19,22 @@ typedef struct { int size; - void *elements[1]; + union { + void *elements[1]; + unsigned int enum_type[1]; + } elt; } asdl_seq; asdl_seq *asdl_seq_new(int size, PyArena *arena); -#define asdl_seq_GET(S, I) (S)->elements[(I)] +#define asdl_seq_GET(S, I) (S)->elt.elements[(I)] +#define asdl_seq_GET_ENUM(S, I) (S)->elt.enum_type[(I)] #define asdl_seq_LEN(S) ((S) == NULL ? 0 : (S)->size) #ifdef Py_DEBUG #define asdl_seq_SET(S, I, V) { \ int _asdl_i = (I); \ assert((S) && _asdl_i < (S)->size); \ - (S)->elements[_asdl_i] = (V); \ + (S)->elt.elements[_asdl_i] = (V); \ } #else #define asdl_seq_SET(S, I, V) (S)->elements[I] = (V) Modified: python/trunk/Python/compile.c ============================================================================== --- python/trunk/Python/compile.c (original) +++ python/trunk/Python/compile.c Thu Apr 13 11:37:01 2006 @@ -3066,10 +3066,8 @@ for (i = 1; i < n; i++) { ADDOP(c, DUP_TOP); ADDOP(c, ROT_THREE); - /* XXX We're casting a void* to cmpop_ty in the next stmt. */ ADDOP_I(c, COMPARE_OP, - cmpop((cmpop_ty)( CMPCAST asdl_seq_GET( - e->v.Compare.ops, i - 1)))); + cmpop((cmpop_ty)asdl_seq_GET_ENUM(e->v.Compare.ops, i - 1))); ADDOP_JREL(c, JUMP_IF_FALSE, cleanup); NEXT_BLOCK(c); ADDOP(c, POP_TOP); @@ -3080,8 +3078,7 @@ VISIT(c, expr, (expr_ty)asdl_seq_GET(e->v.Compare.comparators, n - 1)); ADDOP_I(c, COMPARE_OP, /* XXX We're casting a void* to cmpop_ty in the next stmt. */ - cmpop((cmpop_ty)( CMPCAST asdl_seq_GET(e->v.Compare.ops, - n - 1)))); + cmpop((cmpop_ty)asdl_seq_GET_ENUM(e->v.Compare.ops, n - 1))); if (n > 1) { basicblock *end = compiler_new_block(c); if (end == NULL) From buildbot at python.org Thu Apr 13 11:42:57 2006 From: buildbot at python.org (buildbot at python.org) Date: Thu, 13 Apr 2006 09:42:57 +0000 Subject: [Python-checkins] buildbot failure in amd64 gentoo trunk Message-ID: <20060413094258.339C31E4002@bag.python.org> The Buildbot has detected a new failure of amd64 gentoo trunk. Full details are available at: http://www.python.org/dev/buildbot/all/amd64%2520gentoo%2520trunk/builds/457 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: skip.montanaro BUILD FAILED: failed compile sincerely, -The Buildbot From buildbot at python.org Thu Apr 13 11:43:10 2006 From: buildbot at python.org (buildbot at python.org) Date: Thu, 13 Apr 2006 09:43:10 +0000 Subject: [Python-checkins] buildbot failure in ia64 Debian unstable trunk Message-ID: <20060413094310.9265C1E4002@bag.python.org> The Buildbot has detected a new failure of ia64 Debian unstable trunk. Full details are available at: http://www.python.org/dev/buildbot/all/ia64%2520Debian%2520unstable%2520trunk/builds/117 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: skip.montanaro BUILD FAILED: failed compile sincerely, -The Buildbot From buildbot at python.org Thu Apr 13 11:45:16 2006 From: buildbot at python.org (buildbot at python.org) Date: Thu, 13 Apr 2006 09:45:16 +0000 Subject: [Python-checkins] buildbot failure in alpha Tru64 5.1 trunk Message-ID: <20060413094516.E7DC11E4033@bag.python.org> The Buildbot has detected a new failure of alpha Tru64 5.1 trunk. Full details are available at: http://www.python.org/dev/buildbot/all/alpha%2520Tru64%25205.1%2520trunk/builds/202 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: skip.montanaro BUILD FAILED: failed compile sincerely, -The Buildbot From python-checkins at python.org Thu Apr 13 11:48:28 2006 From: python-checkins at python.org (skip.montanaro) Date: Thu, 13 Apr 2006 11:48:28 +0200 (CEST) Subject: [Python-checkins] r45355 - in python/trunk: Include/asdl.h Python/compile.c Message-ID: <20060413094828.D78171E4002@bag.python.org> Author: skip.montanaro Date: Thu Apr 13 11:48:28 2006 New Revision: 45355 Modified: python/trunk/Include/asdl.h python/trunk/Python/compile.c Log: revert - breaks build of Python/ast.c w/ gcc Modified: python/trunk/Include/asdl.h ============================================================================== --- python/trunk/Include/asdl.h (original) +++ python/trunk/Include/asdl.h Thu Apr 13 11:48:28 2006 @@ -19,22 +19,18 @@ typedef struct { int size; - union { - void *elements[1]; - unsigned int enum_type[1]; - } elt; + void *elements[1]; } asdl_seq; asdl_seq *asdl_seq_new(int size, PyArena *arena); -#define asdl_seq_GET(S, I) (S)->elt.elements[(I)] -#define asdl_seq_GET_ENUM(S, I) (S)->elt.enum_type[(I)] +#define asdl_seq_GET(S, I) (S)->elements[(I)] #define asdl_seq_LEN(S) ((S) == NULL ? 0 : (S)->size) #ifdef Py_DEBUG #define asdl_seq_SET(S, I, V) { \ int _asdl_i = (I); \ assert((S) && _asdl_i < (S)->size); \ - (S)->elt.elements[_asdl_i] = (V); \ + (S)->elements[_asdl_i] = (V); \ } #else #define asdl_seq_SET(S, I, V) (S)->elements[I] = (V) Modified: python/trunk/Python/compile.c ============================================================================== --- python/trunk/Python/compile.c (original) +++ python/trunk/Python/compile.c Thu Apr 13 11:48:28 2006 @@ -3066,8 +3066,10 @@ for (i = 1; i < n; i++) { ADDOP(c, DUP_TOP); ADDOP(c, ROT_THREE); + /* XXX We're casting a void* to cmpop_ty in the next stmt. */ ADDOP_I(c, COMPARE_OP, - cmpop((cmpop_ty)asdl_seq_GET_ENUM(e->v.Compare.ops, i - 1))); + cmpop((cmpop_ty)( CMPCAST asdl_seq_GET( + e->v.Compare.ops, i - 1)))); ADDOP_JREL(c, JUMP_IF_FALSE, cleanup); NEXT_BLOCK(c); ADDOP(c, POP_TOP); @@ -3078,7 +3080,8 @@ VISIT(c, expr, (expr_ty)asdl_seq_GET(e->v.Compare.comparators, n - 1)); ADDOP_I(c, COMPARE_OP, /* XXX We're casting a void* to cmpop_ty in the next stmt. */ - cmpop((cmpop_ty)asdl_seq_GET_ENUM(e->v.Compare.ops, n - 1))); + cmpop((cmpop_ty)( CMPCAST asdl_seq_GET(e->v.Compare.ops, + n - 1)))); if (n > 1) { basicblock *end = compiler_new_block(c); if (end == NULL) From buildbot at python.org Thu Apr 13 13:10:03 2006 From: buildbot at python.org (buildbot at python.org) Date: Thu, 13 Apr 2006 11:10:03 +0000 Subject: [Python-checkins] buildbot failure in alpha Tru64 5.1 trunk Message-ID: <20060413111003.81FA61E4002@bag.python.org> The Buildbot has detected a new failure of alpha Tru64 5.1 trunk. Full details are available at: http://www.python.org/dev/buildbot/all/alpha%2520Tru64%25205.1%2520trunk/builds/205 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: The web-page 'force build' button was pressed by 'Martin von Loewis': clean Build Source Stamp: [branch clean] HEAD Blamelist: BUILD FAILED: failed svn sincerely, -The Buildbot From python-checkins at python.org Thu Apr 13 13:40:30 2006 From: python-checkins at python.org (andrew.kuchling) Date: Thu, 13 Apr 2006 13:40:30 +0200 (CEST) Subject: [Python-checkins] r45356 - python/trunk/Doc/dist/dist.tex Message-ID: <20060413114030.9487D1E4080@bag.python.org> Author: andrew.kuchling Date: Thu Apr 13 13:40:29 2006 New Revision: 45356 Modified: python/trunk/Doc/dist/dist.tex Log: Add missing word Modified: python/trunk/Doc/dist/dist.tex ============================================================================== --- python/trunk/Doc/dist/dist.tex (original) +++ python/trunk/Doc/dist/dist.tex Thu Apr 13 13:40:29 2006 @@ -1467,7 +1467,7 @@ \lineii{\%description (section)}{\option{long\_description}} \end{tableii} -Additionally, there many options in \file{.spec} files that don't have +Additionally, there are many options in \file{.spec} files that don't have corresponding options in the setup script. Most of these are handled through options to the \command{bdist\_rpm} command as follows: From python-checkins at python.org Thu Apr 13 13:40:43 2006 From: python-checkins at python.org (andrew.kuchling) Date: Thu, 13 Apr 2006 13:40:43 +0200 (CEST) Subject: [Python-checkins] r45357 - python/branches/release24-maint/Doc/dist/dist.tex Message-ID: <20060413114043.263621E412D@bag.python.org> Author: andrew.kuchling Date: Thu Apr 13 13:40:42 2006 New Revision: 45357 Modified: python/branches/release24-maint/Doc/dist/dist.tex Log: Add missing word Modified: python/branches/release24-maint/Doc/dist/dist.tex ============================================================================== --- python/branches/release24-maint/Doc/dist/dist.tex (original) +++ python/branches/release24-maint/Doc/dist/dist.tex Thu Apr 13 13:40:42 2006 @@ -1391,7 +1391,7 @@ \lineii{\%description (section)}{\option{long\_description}} \end{tableii} -Additionally, there many options in \file{.spec} files that don't have +Additionally, there are many options in \file{.spec} files that don't have corresponding options in the setup script. Most of these are handled through options to the \command{bdist\_rpm} command as follows: From python-checkins at python.org Thu Apr 13 13:51:08 2006 From: python-checkins at python.org (andrew.kuchling) Date: Thu, 13 Apr 2006 13:51:08 +0200 (CEST) Subject: [Python-checkins] r45358 - python/trunk/Doc/whatsnew/whatsnew25.tex Message-ID: <20060413115108.B611A1E4002@bag.python.org> Author: andrew.kuchling Date: Thu Apr 13 13:51:07 2006 New Revision: 45358 Modified: python/trunk/Doc/whatsnew/whatsnew25.tex Log: Describe sys.subversion, Py_GetBuildInfo() Add metadata example Modified: python/trunk/Doc/whatsnew/whatsnew25.tex ============================================================================== --- python/trunk/Doc/whatsnew/whatsnew25.tex (original) +++ python/trunk/Doc/whatsnew/whatsnew25.tex Thu Apr 13 13:51:07 2006 @@ -227,7 +227,16 @@ possible to look up an entry in the package index, determine the dependencies for a package, and download the required packages. -% XXX put example here +\begin{verbatim} +VERSION = '1.0' +setup(name='PyPackage', + version=VERSION, + requires=['numarray', 'zlib (>=1.1.4)'], + obsoletes=['OldPackage'] + download_url=('http://www.example.com/pypackage/dist/pkg-%s.tar.gz' + % VERSION), + ) +\end{verbatim} \begin{seealso} @@ -1070,7 +1079,18 @@ shadow password database on systems that support it. % XXX give example -% XXX patch #1382163: sys.subversion, Py_GetBuildNumber() +\item The Python developers switched from CVS to Subversion during the 2.5 +development process. Information about the exact build version is +available as the \code{sys.subversion} variable, a 3-tuple +of \code{(\var{interpreter-name}, \var{branch-name}, \var{revision-range})}. +For example, at the time of writing +my copy of 2.5 was reporting \code{('CPython', 'trunk', '45313:45315')}. + +This information is also available to C extensions via the +\cfunction{Py_GetBuildInfo()} function that returns a +string of build information like this: +\code{"trunk:45355:45356M, Apr 13 2006, 07:42:19"}. +(Contributed by Barry Warsaw.) \item The \class{TarFile} class in the \module{tarfile} module now has an \method{extractall()} method that extracts all members from the @@ -1106,6 +1126,7 @@ %====================================================================== % whole new modules get described in subsections here +%====================================================================== \subsection{The ctypes package} The \module{ctypes} package, written by Thomas Heller, has been added @@ -1179,8 +1200,6 @@ Python wrappers atop a library accessed through \module{ctypes} instead of extension modules, now that \module{ctypes} is included with core Python. -% XXX write introduction - \begin{seealso} \seeurl{http://starship.python.net/crew/theller/ctypes/} @@ -1188,6 +1207,8 @@ \end{seealso} + +%====================================================================== \subsection{The ElementTree package} A subset of Fredrik Lundh's ElementTree library for processing XML has @@ -1298,6 +1319,7 @@ \end{seealso} +%====================================================================== \subsection{The hashlib package} A new \module{hashlib} module has been added to replace the @@ -1346,6 +1368,7 @@ This module was contributed by Gregory P. Smith. +%====================================================================== \subsection{The sqlite3 package} The pysqlite module (\url{http://www.pysqlite.org}), a wrapper for the @@ -1525,6 +1548,13 @@ add and remove elements, and \cfunction{PySet_Contains} and \cfunction{PySet_Size} to examine the set's state. +\item C code can now obtain information about the exact revision +of the Python interpreter by calling the +\cfunction{Py_GetBuildInfo()} function that returns a +string of build information like this: +\code{"trunk:45355:45356M, Apr 13 2006, 07:42:19"}. +(Contributed by Barry Warsaw.) + \item The \cfunction{PyRange_New()} function was removed. It was never documented, never used in the core code, and had dangerously lax error checking. From buildbot at python.org Thu Apr 13 13:51:16 2006 From: buildbot at python.org (buildbot at python.org) Date: Thu, 13 Apr 2006 11:51:16 +0000 Subject: [Python-checkins] buildbot warnings in ia64 Debian unstable trunk Message-ID: <20060413115116.A6E351E4002@bag.python.org> The Buildbot has detected a new failure of ia64 Debian unstable trunk. Full details are available at: http://www.python.org/dev/buildbot/all/ia64%2520Debian%2520unstable%2520trunk/builds/120 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: The web-page 'rebuild' button was pressed by '': Build Source Stamp: [branch trunk] HEAD Blamelist: skip.montanaro Build Had Warnings: warnings test sincerely, -The Buildbot From python-checkins at python.org Thu Apr 13 14:09:09 2006 From: python-checkins at python.org (andrew.kuchling) Date: Thu, 13 Apr 2006 14:09:09 +0200 (CEST) Subject: [Python-checkins] r45359 - python/trunk/Doc/whatsnew/whatsnew25.tex Message-ID: <20060413120909.057C51E4002@bag.python.org> Author: andrew.kuchling Date: Thu Apr 13 14:09:08 2006 New Revision: 45359 Modified: python/trunk/Doc/whatsnew/whatsnew25.tex Log: Include more detail on Coverity results and add a link; minor edits Modified: python/trunk/Doc/whatsnew/whatsnew25.tex ============================================================================== --- python/trunk/Doc/whatsnew/whatsnew25.tex (original) +++ python/trunk/Doc/whatsnew/whatsnew25.tex Thu Apr 13 14:09:08 2006 @@ -370,8 +370,8 @@ Until Python 2.5, the \keyword{try} statement came in two flavours. You could use a \keyword{finally} block to ensure that code -is always executed, or a number of \keyword{except} blocks to catch an -exception. You couldn't combine both \keyword{except} blocks and a +is always executed, or one or more \keyword{except} blocks to catch +specific exceptions. You couldn't combine both \keyword{except} blocks and a \keyword{finally} block, because generating the right bytecode for the combined version was complicated and it wasn't clear what the semantics of the combined should be. @@ -949,7 +949,7 @@ %====================================================================== \section{New, Improved, and Deprecated Modules} -As usual, Python's standard library received a number of enhancements and +As usual, Python's standard library received many enhancements and bug fixes. Here's a partial list of the most notable changes, sorted alphabetically by module name. Consult the \file{Misc/NEWS} file in the source tree for a more @@ -1022,7 +1022,7 @@ (Contributed by Raymond Hettinger.) -\item The \module{os} module underwent a number of changes. The +\item The \module{os} module underwent several changes. The \member{stat_float_times} variable now defaults to true, meaning that \function{os.stat()} will now return time values as floats. (This doesn't necessarily mean that \function{os.stat()} will return times @@ -1590,7 +1590,7 @@ Peters.) Note that this change means extension modules need to be more careful -with how they allocate memory. Python's API has a number of different +with how they allocate memory. Python's API has many different functions for allocating memory that are grouped into families. For example, \cfunction{PyMem_Malloc()}, \cfunction{PyMem_Realloc()}, and \cfunction{PyMem_Free()} are one family that allocates raw memory, @@ -1608,9 +1608,10 @@ \item Coverity, a company that markets a source code analysis tool called Prevent, provided the results of their examination of the Python - source code. The analysis found a number of refcounting bugs, often - in error-handling code. These bugs have been fixed. - % XXX provide reference? + source code. The analysis found about 60 bugs that + were quickly fixed. Many of the bugs were refcounting problems, often + occurring in error-handling code. See + \url{http://scan.coverity.com} for the statistics. \end{itemize} From python-checkins at python.org Thu Apr 13 14:29:44 2006 From: python-checkins at python.org (martin.v.loewis) Date: Thu, 13 Apr 2006 14:29:44 +0200 (CEST) Subject: [Python-checkins] r45360 - in python/trunk: Include/Python-ast.h Include/asdl.h Parser/asdl_c.py Python/Python-ast.c Python/asdl.c Python/ast.c Python/compile.c Message-ID: <20060413122944.EF84E1E4002@bag.python.org> Author: martin.v.loewis Date: Thu Apr 13 14:29:43 2006 New Revision: 45360 Modified: python/trunk/Include/Python-ast.h python/trunk/Include/asdl.h python/trunk/Parser/asdl_c.py python/trunk/Python/Python-ast.c python/trunk/Python/asdl.c python/trunk/Python/ast.c python/trunk/Python/compile.c Log: Introduce asdl_int_seq, to hold cmpop_ty. Modified: python/trunk/Include/Python-ast.h ============================================================================== --- python/trunk/Include/Python-ast.h (original) +++ python/trunk/Include/Python-ast.h Thu Apr 13 14:29:43 2006 @@ -240,7 +240,7 @@ struct { expr_ty left; - asdl_seq *ops; + asdl_int_seq *ops; asdl_seq *comparators; } Compare; @@ -409,7 +409,7 @@ expr_ty GeneratorExp(expr_ty elt, asdl_seq * generators, int lineno, int col_offset, PyArena *arena); expr_ty Yield(expr_ty value, int lineno, int col_offset, PyArena *arena); -expr_ty Compare(expr_ty left, asdl_seq * ops, asdl_seq * comparators, int +expr_ty Compare(expr_ty left, asdl_int_seq * ops, asdl_seq * comparators, int lineno, int col_offset, PyArena *arena); expr_ty Call(expr_ty func, asdl_seq * args, asdl_seq * keywords, expr_ty starargs, expr_ty kwargs, int lineno, int col_offset, PyArena Modified: python/trunk/Include/asdl.h ============================================================================== --- python/trunk/Include/asdl.h (original) +++ python/trunk/Include/asdl.h Thu Apr 13 14:29:43 2006 @@ -22,7 +22,13 @@ void *elements[1]; } asdl_seq; +typedef struct { + int size; + int elements[1]; +} asdl_int_seq; + asdl_seq *asdl_seq_new(int size, PyArena *arena); +asdl_int_seq *asdl_int_seq_new(int size, PyArena *arena); #define asdl_seq_GET(S, I) (S)->elements[(I)] #define asdl_seq_LEN(S) ((S) == NULL ? 0 : (S)->size) Modified: python/trunk/Parser/asdl_c.py ============================================================================== --- python/trunk/Parser/asdl_c.py (original) +++ python/trunk/Parser/asdl_c.py Thu Apr 13 14:29:43 2006 @@ -188,7 +188,10 @@ ctype = get_c_type(field.type) name = field.name if field.seq: - self.emit("asdl_seq *%(name)s;" % locals(), depth) + if field.type.value in ('cmpop',): + self.emit("asdl_int_seq *%(name)s;" % locals(), depth) + else: + self.emit("asdl_seq *%(name)s;" % locals(), depth) else: self.emit("%(ctype)s %(name)s;" % locals(), depth) @@ -234,7 +237,10 @@ name = f.name # XXX should extend get_c_type() to handle this if f.seq: - ctype = "asdl_seq *" + if f.type.value in ('cmpop',): + ctype = "asdl_int_seq *" + else: + ctype = "asdl_seq *" else: ctype = get_c_type(f.type) args.append((ctype, name, f.opt or f.seq)) @@ -681,7 +687,7 @@ self.emit("if (!value) goto failed;", depth+1) self.emit("for(i = 0; i < n; i++)", depth+1) # This cannot fail, so no need for error handling - self.emit("PyList_SET_ITEM(value, i, ast2obj_cmpop((cmpop_ty)(int)asdl_seq_GET(%s, i)));" % value, + self.emit("PyList_SET_ITEM(value, i, ast2obj_cmpop((cmpop_ty)asdl_seq_GET(%s, i)));" % value, depth+2, reflow=False) self.emit("}", depth) else: Modified: python/trunk/Python/Python-ast.c ============================================================================== --- python/trunk/Python/Python-ast.c (original) +++ python/trunk/Python/Python-ast.c Thu Apr 13 14:29:43 2006 @@ -1503,8 +1503,8 @@ } expr_ty -Compare(expr_ty left, asdl_seq * ops, asdl_seq * comparators, int lineno, int - col_offset, PyArena *arena) +Compare(expr_ty left, asdl_int_seq * ops, asdl_seq * comparators, int lineno, + int col_offset, PyArena *arena) { expr_ty p; if (!left) { @@ -2503,7 +2503,7 @@ value = PyList_New(n); if (!value) goto failed; for(i = 0; i < n; i++) - PyList_SET_ITEM(value, i, ast2obj_cmpop((cmpop_ty)(int)asdl_seq_GET(o->v.Compare.ops, i))); + PyList_SET_ITEM(value, i, ast2obj_cmpop((cmpop_ty)asdl_seq_GET(o->v.Compare.ops, i))); } if (!value) goto failed; if (PyObject_SetAttrString(result, "ops", value) == -1) Modified: python/trunk/Python/asdl.c ============================================================================== --- python/trunk/Python/asdl.c (original) +++ python/trunk/Python/asdl.c Thu Apr 13 14:29:43 2006 @@ -8,7 +8,24 @@ size_t n = sizeof(asdl_seq) + (size ? (sizeof(void *) * (size - 1)) : 0); - seq = (asdl_seq *)PyArena_Malloc(arena, n); + seq = (asdl_seq *)PyArena_Malloc(arena, n); + if (!seq) { + PyErr_NoMemory(); + return NULL; + } + memset(seq, 0, n); + seq->size = size; + return seq; +} + +asdl_int_seq * +asdl_int_seq_new(int size, PyArena *arena) +{ + asdl_seq *seq = NULL; + size_t n = sizeof(asdl_seq) + + (size ? (sizeof(int) * (size - 1)) : 0); + + seq = (asdl_seq *)PyArena_Malloc(arena, n); if (!seq) { PyErr_NoMemory(); return NULL; Modified: python/trunk/Python/ast.c ============================================================================== --- python/trunk/Python/ast.c (original) +++ python/trunk/Python/ast.c Thu Apr 13 14:29:43 2006 @@ -1600,8 +1600,9 @@ } else { expr_ty expression; - asdl_seq *ops, *cmps; - ops = asdl_seq_new(NCH(n) / 2, c->c_arena); + asdl_int_seq *ops; + asdl_seq *cmps; + ops = asdl_int_seq_new(NCH(n) / 2, c->c_arena); if (!ops) return NULL; cmps = asdl_seq_new(NCH(n) / 2, c->c_arena); @@ -1609,7 +1610,6 @@ return NULL; } for (i = 1; i < NCH(n); i += 2) { - /* XXX cmpop_ty is just an enum */ cmpop_ty newoperator; newoperator = ast_for_comp_op(CHILD(n, i)); @@ -1622,7 +1622,7 @@ return NULL; } - asdl_seq_SET(ops, i / 2, (void *)(Py_uintptr_t)newoperator); + asdl_seq_SET(ops, i / 2, newoperator); asdl_seq_SET(cmps, i / 2, expression); } expression = ast_for_expr(c, CHILD(n, 0)); Modified: python/trunk/Python/compile.c ============================================================================== --- python/trunk/Python/compile.c (original) +++ python/trunk/Python/compile.c Thu Apr 13 14:29:43 2006 @@ -3058,17 +3058,11 @@ VISIT(c, expr, (expr_ty)asdl_seq_GET(e->v.Compare.comparators, 0)); } -#ifdef __cplusplus -#define CMPCAST (intptr_t) -#else -#define CMPCAST -#endif for (i = 1; i < n; i++) { ADDOP(c, DUP_TOP); ADDOP(c, ROT_THREE); - /* XXX We're casting a void* to cmpop_ty in the next stmt. */ ADDOP_I(c, COMPARE_OP, - cmpop((cmpop_ty)( CMPCAST asdl_seq_GET( + cmpop((cmpop_ty)(asdl_seq_GET( e->v.Compare.ops, i - 1)))); ADDOP_JREL(c, JUMP_IF_FALSE, cleanup); NEXT_BLOCK(c); @@ -3079,9 +3073,7 @@ } VISIT(c, expr, (expr_ty)asdl_seq_GET(e->v.Compare.comparators, n - 1)); ADDOP_I(c, COMPARE_OP, - /* XXX We're casting a void* to cmpop_ty in the next stmt. */ - cmpop((cmpop_ty)( CMPCAST asdl_seq_GET(e->v.Compare.ops, - n - 1)))); + cmpop((cmpop_ty)(asdl_seq_GET(e->v.Compare.ops, n - 1)))); if (n > 1) { basicblock *end = compiler_new_block(c); if (end == NULL) From python-checkins at python.org Thu Apr 13 14:37:22 2006 From: python-checkins at python.org (andrew.kuchling) Date: Thu, 13 Apr 2006 14:37:22 +0200 (CEST) Subject: [Python-checkins] r45361 - python/trunk/Doc/whatsnew/whatsnew25.tex Message-ID: <20060413123722.2F3181E4066@bag.python.org> Author: andrew.kuchling Date: Thu Apr 13 14:37:21 2006 New Revision: 45361 Modified: python/trunk/Doc/whatsnew/whatsnew25.tex Log: Add some items Modified: python/trunk/Doc/whatsnew/whatsnew25.tex ============================================================================== --- python/trunk/Doc/whatsnew/whatsnew25.tex (original) +++ python/trunk/Doc/whatsnew/whatsnew25.tex Thu Apr 13 14:37:21 2006 @@ -958,22 +958,26 @@ \begin{itemize} -% collections.deque now has .remove() +% XXX collections.deque now has .remove() % collections.defaultdict % the cPickle module no longer accepts the deprecated None option in the % args tuple returned by __reduce__(). -% csv module improvements +% XXX csv module improvements -% datetime.datetime() now has a strptime class method which can be used to +% XXX datetime.datetime() now has a strptime class method which can be used to % create datetime object using a string and format. -% fileinput: opening hook used to control how files are opened. +% XXX fileinput: opening hook used to control how files are opened. % .input() now has a mode parameter % now has a fileno() function % accepts Unicode filenames +\item The \module{audioop} module now supports the a-LAW encoding, +and the code for u-LAW encoding has been improved. (Contributed by +Lars Immisch.) + \item In the \module{gc} module, the new \function{get_count()} function returns a 3-tuple containing the current collection counts for the three GC generations. This is accounting information for the garbage @@ -1563,9 +1567,15 @@ %====================================================================== -%\subsection{Port-Specific Changes} +\subsection{Port-Specific Changes} + +\begin{itemize} -%Platform-specific changes go here. +\item MacOS X (10.3 and higher): dynamic loading of modules +now uses the \cfunction{dlopen()} function instead of MacOS-specific +functions. + +\end{itemize} %====================================================================== From python-checkins at python.org Thu Apr 13 14:49:39 2006 From: python-checkins at python.org (andrew.kuchling) Date: Thu, 13 Apr 2006 14:49:39 +0200 (CEST) Subject: [Python-checkins] r45362 - python/trunk/Doc/whatsnew/whatsnew25.tex Message-ID: <20060413124939.006811E4002@bag.python.org> Author: andrew.kuchling Date: Thu Apr 13 14:49:39 2006 New Revision: 45362 Modified: python/trunk/Doc/whatsnew/whatsnew25.tex Log: [Bug #1464571] Mention that generator's .gi_frame can now be None Modified: python/trunk/Doc/whatsnew/whatsnew25.tex ============================================================================== --- python/trunk/Doc/whatsnew/whatsnew25.tex (original) +++ python/trunk/Doc/whatsnew/whatsnew25.tex Thu Apr 13 14:49:39 2006 @@ -412,7 +412,7 @@ %====================================================================== -\section{PEP 342: New Generator Features} +\section{PEP 342: New Generator Features\label{section-generators}} Python 2.5 adds a simple way to pass values \emph{into} a generator. As introduced in Python 2.3, generators only produce output; once a @@ -541,7 +541,7 @@ The addition of the \method{close()} method has one side effect that isn't obvious. \method{close()} is called when a generator is garbage-collected, so this means the generator's code gets one last -chance to run before the generator is destroyed, and this last chance +chance to run before the generator is destroyed. This last chance means that \code{try...finally} statements in generators can now be guaranteed to work; the \keyword{finally} clause will now always get a chance to run. The syntactic restriction that you couldn't mix @@ -552,6 +552,11 @@ described by PEP 343. We'll look at this new statement in the following section. +Another even more esoteric effect of this change: previously, the +\member{gi_frame} attribute of a generator was always a frame object. +It's now possible for \member{gi_frame} to be \code{None} +once the generator has been exhausted. + \begin{seealso} \seepep{342}{Coroutines via Enhanced Generators}{PEP written by @@ -1641,6 +1646,11 @@ \item The \module{pickle} module no longer uses the deprecated \var{bin} parameter. +\item Previously, the \member{gi_frame} attribute of a generator +was always a frame object. Because of the \pep{342} changes +described in section~\ref{section-generators}, it's now possible +for \member{gi_frame} to be \code{None}. + \item C API: Many functions now use \ctype{Py_ssize_t} instead of \ctype{int} to allow processing more data on 64-bit machines. Extension code may need to make From python-checkins at python.org Thu Apr 13 15:02:42 2006 From: python-checkins at python.org (andrew.kuchling) Date: Thu, 13 Apr 2006 15:02:42 +0200 (CEST) Subject: [Python-checkins] r45363 - python/trunk/Doc/whatsnew/whatsnew25.tex Message-ID: <20060413130242.A74471E4002@bag.python.org> Author: andrew.kuchling Date: Thu Apr 13 15:02:42 2006 New Revision: 45363 Modified: python/trunk/Doc/whatsnew/whatsnew25.tex Log: Typo fix Modified: python/trunk/Doc/whatsnew/whatsnew25.tex ============================================================================== --- python/trunk/Doc/whatsnew/whatsnew25.tex (original) +++ python/trunk/Doc/whatsnew/whatsnew25.tex Thu Apr 13 15:02:42 2006 @@ -352,7 +352,7 @@ The \module{runpy} module implements a more sophisticated import mechanism so that it's now possible to run modules in a package such as \module{pychecker.checker}. The module also supports alternative -import mechanisms such as the \module{zipimport} module. (This means +import mechanisms such as the \module{zipimport} module. This means you can add a .zip archive's path to \code{sys.path} and then use the \programopt{-m} switch to execute code from the archive. From python-checkins at python.org Thu Apr 13 15:08:58 2006 From: python-checkins at python.org (martin.v.loewis) Date: Thu, 13 Apr 2006 15:08:58 +0200 (CEST) Subject: [Python-checkins] r45364 - python/trunk/Python/asdl.c Message-ID: <20060413130858.714C41E4002@bag.python.org> Author: martin.v.loewis Date: Thu Apr 13 15:08:58 2006 New Revision: 45364 Modified: python/trunk/Python/asdl.c Log: Fix type errors. Modified: python/trunk/Python/asdl.c ============================================================================== --- python/trunk/Python/asdl.c (original) +++ python/trunk/Python/asdl.c Thu Apr 13 15:08:58 2006 @@ -21,11 +21,11 @@ asdl_int_seq * asdl_int_seq_new(int size, PyArena *arena) { - asdl_seq *seq = NULL; + asdl_int_seq *seq = NULL; size_t n = sizeof(asdl_seq) + (size ? (sizeof(int) * (size - 1)) : 0); - seq = (asdl_seq *)PyArena_Malloc(arena, n); + seq = (asdl_int_seq *)PyArena_Malloc(arena, n); if (!seq) { PyErr_NoMemory(); return NULL; From python-checkins at python.org Thu Apr 13 15:36:15 2006 From: python-checkins at python.org (david.goodger) Date: Thu, 13 Apr 2006 15:36:15 +0200 (CEST) Subject: [Python-checkins] r45365 - peps/trunk/pep-0304.txt Message-ID: <20060413133615.C0E831E4002@bag.python.org> Author: david.goodger Date: Thu Apr 13 15:36:15 2006 New Revision: 45365 Modified: peps/trunk/pep-0304.txt Log: URL update Modified: peps/trunk/pep-0304.txt ============================================================================== --- peps/trunk/pep-0304.txt (original) +++ peps/trunk/pep-0304.txt Thu Apr 13 15:36:15 2006 @@ -330,7 +330,7 @@ (http://mail.python.org/pipermail/python-dev/2003-January/032060.html) .. [5] PEP 302, New Import Hooks, van Rossum and Moore - (http://www.python.org/dev/peps/pep-0302.html) + (http://www.python.org/dev/peps/pep-0302) .. [6] patch 677103, PYTHONBYTECODEBASE patch (PEP 304), Montanaro (http://www.python.org/sf/677103) From python-checkins at python.org Thu Apr 13 15:36:24 2006 From: python-checkins at python.org (david.goodger) Date: Thu, 13 Apr 2006 15:36:24 +0200 (CEST) Subject: [Python-checkins] r45366 - peps/trunk/pep-0000.txt peps/trunk/pep-0359.txt peps/trunk/pep-3002.txt Message-ID: <20060413133624.C82F81E4002@bag.python.org> Author: david.goodger Date: Thu Apr 13 15:36:24 2006 New Revision: 45366 Added: peps/trunk/pep-0359.txt (contents, props changed) peps/trunk/pep-3002.txt (contents, props changed) Modified: peps/trunk/pep-0000.txt Log: added PEPs 359 (The "make Statement) and 3002 (Procedure for Backwards-Incompatible Changes) by Steven Bethard Modified: peps/trunk/pep-0000.txt ============================================================================== --- peps/trunk/pep-0000.txt (original) +++ peps/trunk/pep-0000.txt Thu Apr 13 15:36:24 2006 @@ -45,6 +45,7 @@ P 347 Migrating the Python CVS to Subversion von L?wis P 3000 Python 3000 GvR P 3001 Reviewing and improving stdlib modules Brandl + P 3002 Procedure for Backwards-Incompatible Changes Bethard I 3099 Things that will Not Change in Python 3000 Brandl Other Informational PEPs @@ -102,6 +103,7 @@ S 354 Enumerations in Python Finney S 355 Path - Object oriented filesystem paths Lindqvist S 358 The "bytes" Object Schemenauer + S 359 The "make" Statement Bethard S 754 IEEE 754 Floating Point Special Values Warnes Finished PEPs (done, implemented in Subversion) @@ -415,10 +417,12 @@ I 356 Python 2.5 Release Schedule Norwitz, et al SF 357 Allowing Any Object to be Used for Slicing Oliphant S 358 The "bytes" Object Schemenauer + S 359 The "make" Statement Bethard SR 666 Reject Foolish Indentation Creighton S 754 IEEE 754 Floating Point Special Values Warnes P 3000 Python 3000 GvR P 3001 Reviewing and improving stdlib modules Brandl + P 3002 Procedure for Backwards-Incompatible Changes Bethard I 3099 Things that will Not Change in Python 3000 Brandl I 3100 Python 3.0 Plans Kuchling, Cannon @@ -449,6 +453,7 @@ Batista, Facundo facundo at taniquetil.com.ar Baxter, Anthony anthony at interlink.com.au Bellman, Thomas bellman+pep-divmod at lysator.liu.se + Bethard, Steven steven.bethard at gmail.com Brandl, Georg g.brandl at gmx.net Cannon, Brett brett at python.org Carlson, Josiah jcarlson at uci.edu Added: peps/trunk/pep-0359.txt ============================================================================== --- (empty file) +++ peps/trunk/pep-0359.txt Thu Apr 13 15:36:24 2006 @@ -0,0 +1,274 @@ +PEP: 359 +Title: The "make" Statement +Version: $Revision$ +Last-Modified: $Date$ +Author: Steven Bethard +Status: Draft +Type: Standards Track +Content-Type: text/x-rst +Created: 05-Apr-2006 +Python-Version: 2.6 +Post-History: 05-Apr-2006, 06-Apr-2006 + + +Abstract +======== + +This PEP proposes a generalization of the class-declaration syntax, +the ``make`` statement. The proposed syntax and semantics parallel +the syntax for class definition, and so:: + + make : + + +is translated into the assignment:: + + = ("", , ) + +where ```` is the dict created by executing ````. +The PEP is based on a suggestion [1]_ from Michele Simionato on the +python-dev list. + + +Motivation +========== + +Class statements provide two nice facilities to Python: + + (1) They are the standard Python means of creating a namespace. All + statements within a class body are executed, and the resulting + local name bindings are passed as a dict to the metaclass. + + (2) They encourage DRY (don't repeat yourself) by allowing the class + being created to know the name it is being assigned. + +Thus in a simple class statement like:: + + class C(object): + x = 1 + def foo(self): + return 'bar' + +the metaclass (``type``) gets called with something like:: + + C = type('C', (object,), {'x':1, 'foo':}) + +The class statement is just syntactic sugar for the above assignment +statement, but clearly a very useful sort of syntactic sugar. It +avoids not only the repetition of ``C``, but also simplifies the +creation of the dict by allowing it to be expressed as a series of +statements. + +Historically, type instances (a.k.a. class objects) have been the only +objects blessed with this sort of syntactic support. But other sorts +of objects could benefit from such support. For example, property +objects take three function arguments, but because the property type +cannot be passed a namespace, these functions, though relevant only to +the property, must be declared before it and then passed as arguments +to the property call, e.g.:: + + class C(object): + ... + def get_x(self): + ... + def set_x(self): + ... + x = property(get_x, set_x, ...) + +There have been a few recipes [2]_ trying to work around this +behavior, but with the new make statement (and an appropriate +definition of property), the getter and setter functions can be +defined in the property's namespace like:: + + class C(object): + ... + make property x: + def get(self): + ... + def set(self): + ... + +The definition of such a property callable could be as simple as:: + + def property(name, args, namespace): + fget = namespace.get('get') + fset = namespace.get('set') + fdel = namespace.get('delete') + doc = namespace.get('__doc__') + return __builtin__.property(fget, fset, fdel, doc) + +Of course, properties are only one of the many possible uses of the +make statement. The make statement is useful in essentially any +situation where a name is associated with a namespace. So, for +example, namespaces could be created as simply as:: + + make namespace ns: + """This creates a namespace named ns with a badger attribute + and a spam function""" + + badger = 42 + + def spam(): + ... + +And if Python acquires interfaces, given an appropriately defined +``interface`` callable, the make statement can support interface +creation through the syntax:: + + make interface C(...): + ... + +This would mean that interface systems like that of Zope would no +longer have to abuse the class syntax to create proper interface +instances. + + +Specification +============= + +Python will translate a make statement:: + + make : + + +into the assignment:: + + = ("", , ) + +where ```` is the dict created by executing ````. +The ```` expression is optional; if not present, an empty tuple +will be assumed. + +A patch is available implementing these semantics [3]_. + +The make statement introduces a new keyword, ``make``. Thus in Python +2.6, the make statement will have to be enabled using ``from +__future__ import make_statement``. + + +Open Issues +=========== + +Does the ``make`` keyword break too much code? Originally, the make +statement used the keyword ``create`` (a suggestion due to Nick +Coghlan). However, investigations into the standard library [4]_ and +Zope+Plone code [5]_ revealed that ``create`` would break a lot more +code, so ``make`` was adopted as the keyword instead. However, there +are still a few instances where ``make`` would break code. Is there a +better keyword for the statement? + +********** + +Currently, there are not many functions which have the signature +``(name, args, kwargs)``. That means that something like:: + + make dict params: + x = 1 + y = 2 + +is currently impossible because the dict constructor has a different +signature. Does this sort of thing need to be supported? One +suggestion, by Carl Banks, would be to add a ``__make__`` magic method +that would be called before ``__call__``. For types, the ``__make__`` +method would be identical to ``__call__`` (and thus unnecessary), but +dicts could support the make statement by defining a ``__make__`` +method on the dict type that looks something like:: + + def __make__(cls, name, args, kwargs): + return cls(**kwargs) + +Of course, rather than adding another magic method, the dict type +could just grow a classmethod something like ``dict.fromblock`` that +could be used like:: + + make dict.fromblock params: + x = 1 + y = 2 + + +Optional Extensions +=================== + +Remove the make keyword +------------------------- + +It might be possible to remove the make keyword so that such +statements would begin with the callable being called, e.g.:: + + namespace ns: + badger = 42 + def spam(): + ... + + interface C(...): + ... + +However, almost all other Python statements begin with a keyword, and +removing the keyword would make it harder to look up this construct in +the documentation. Additionally, this would add some complexity in +the grammar and so far I (Steven Bethard) have not been able to +implement the feature without the keyword. + + +Removing __metaclass__ in Python 3000 +------------------------------------- + +As a side-effect of its generality, the make statement mostly +eliminates the need for the ``__metaclass__`` attribute in class +objects. Thus in Python 3000, instead of:: + + class : + __metaclass__ = + + +metaclasses could be supported by using the metaclass as the callable +in a make statement:: + + make : + + +Removing the ``__metaclass__`` hook would simplify the BUILD_CLASS +opcode a bit. + + +Removing class statements in Python 3000 +---------------------------------------- + +In the most extreme application of make statements, the class +statement itself could be deprecated in favor of ``make type`` +statements. + + +References +========== + +.. [1] Michele Simionato's original suggestion + (http://mail.python.org/pipermail/python-dev/2005-October/057435.html) + +.. [2] Namespace-based property recipe + (http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/442418) + +.. [3] Make Statement patch + (http://ucsu.colorado.edu/~bethard/py/make_statement.patch) + +.. [4] Instances of create in the stdlib + (http://mail.python.org/pipermail/python-list/2006-April/335159.html) + +.. [5] Instances of create in Zope+Plone + (http://mail.python.org/pipermail/python-list/2006-April/335284.html) + + +Copyright +========= + +This document has been placed in the public domain. + + +.. + Local Variables: + mode: indented-text + indent-tabs-mode: nil + sentence-end-double-space: t + fill-column: 70 + coding: utf-8 + End: Added: peps/trunk/pep-3002.txt ============================================================================== --- (empty file) +++ peps/trunk/pep-3002.txt Thu Apr 13 15:36:24 2006 @@ -0,0 +1,164 @@ +PEP: 3002 +Title: Procedure for Backwards-Incompatible Changes +Version: $Revision$ +Last-Modified: $Date$ +Author: Steven Bethard +Status: Draft +Type: Process +Content-Type: text/x-rst +Created: 03-Mar-2006 +Post-History: 03-Mar-2006 + + +Abstract +======== + +This PEP describes the procedure for changes to Python that are +backwards-incompatible between the Python 2.X series and Python 3000. +All such changes must be documented by an appropriate Python 3000 PEP +and must be accompanied by code that can identify when pieces of +Python 2.X code may be problematic in Python 3000. + + +Rationale +========= + +Python 3000 will introduce a number of backwards-incompatible changes +to Python, mainly to streamline the language and to remove some +previous design mistakes. But Python 3000 is not intended to be a new +and completely different language from the Python 2.X series, and it +is expected that much of the Python user community will make the +transition to Python 3000 when it becomes available. + +To encourage this transition, it is crucial to provide a clear and +complete guide on how to upgrade Python 2.X code to Python 3000 code. +Thus, for any backwards-incompatible change, two things are required: + +* An official Python Enhancement Proposal (PEP) +* Code that can identify pieces of Python 2.X code that may be + problematic in Python 3000 + + +Python Enchancement Proposals +============================= + +Every backwards-incompatible change must be accompanied by a PEP. +This PEP should follow the usual PEP guidelines and explain the +purpose and reasoning behind the backwards incompatible change. In +addition to the usual PEP sections, all PEPs proposing +backwards-incompatible changes must include an additional section: +Compatibility Issues. This section should describe what is backwards +incompatible about the proposed change to Python, and the major sorts +of breakage to be expected. + +While PEPs must still be evaluated on a case-by-case basis, a PEP may +be inappropriate for Python 3000 if its Compatibility Issues section +implies any of the following: + +* Most or all instances of a Python 2.X construct are incorrect in + Python 3000, and most or all instances of the Python 3000 construct + are incorrect in Python 2.X. + + So for example, changing the meaning of the for-loop else-clause + from "executed when the loop was not broken out of" to "executed + when the loop had zero iterations" would mean that all Python 2.X + for-loop else-clauses would be broken, and there would be no way to + use a for-loop else-clause in a Python-3000-appropriate manner. + Thus a PEP for such an idea would likely be rejected. + +* Many instances of a Python 2.X construct are incorrect in Python + 3000 and the PEP fails to demonstrate real-world use-cases for the + changes. + + Backwards incompatible changes are allowed in Python 3000, but not + to excess. A PEP that proposes backwards-incompatible changes + should provide good examples of code that visibly benefits from the + changes. + +PEP-writing is time-consuming, so when a number of +backwards-incompatible changes are closely related, they should be +proposed in the same PEP. Such PEPs will likely have longer +Compatibility Issues sections however, since they must now describe +the sorts of breakage expected from *all* the proposed changes. + + +Identifying Problematic Code +============================ + +In addition to the PEP requirement, backwards incompatible changes to +Python must also be accompanied by code that can identify pieces of +Python 2.X code that may be problematic in Python 3.0. + +This PEP proposes to house this code in tools/scripts/python3warn.py. +Thus PEPs for backwards incompatible changes should include a patch to +this file that produces the appropriate warnings. Code in +python3warn.py should be written to the latest version of Python 2.X +(not Python 3000) so that Python 2.X users will be able to run the +program without having Python 3000 installed. + +Currently, it seems too stringent to require that the code in +python3warn.py identify all changes perfectly. Thus it is permissable +if a backwards-incompatible PEP's python3warn.py code produces a +number of false-positives (warning that a piece of code might be +invalid in Python 3000 when it's actually still okay). However, +false-negatives (not issuing a warning for code that will do the +wrong thing in Python 3000) should be avoided whenever possible -- +users of python3warn.py should be reasonably confident that they have +been warned about the vast majority of incompatibilities. + +So for example, a PEP proposing that ``dict.items()`` be modified to +return an iterator instead of a list might add code like the following +to python3warn.py:: + + items_in_for = re.compile(r'for\s+\w+\s+in\s+\w+\.items\(\):') + ... + for i, line in enumerate(file_lines): + ... + if '.items()' in line and not items_in_for.search(line): + message = 'dict.items() call may expect list at line %i' + warnings.warn(message % i) + +This would issue a warning any time a ``.items()`` method was called +and not immediately iterated over in a for-loop. Clearly this will +issue a number of false-positive warnings (e.g. ``d2 = +dict(d.items())``), but the number of false-negative warnings should +be relatively low. + + +Optional Extensions +=================== + +Instead of the python3warn.py script, a branch of Python 3000 could be +maintained that added warnings at all the appropriate points in the +code-base. PEPs proposing backwards-incompatible changes would then +provide patches to the Python-3000-warn branch instead of to +python3warn.py. With such a branch, the warnings issued could be +near-perfect and Python users could be confident that their code was +correct Python 3000 code by first running it on the Python-3000-warn +branch and fixing all the warnings. + +At the moment, however, this PEP opts for the weaker measure +(python3warn.py) as it is expected that maintaining a Python-3000-warn +branch will be too much of a time drain. + + +References +========== + +TBD + + +Copyright +========= + +This document has been placed in the public domain. + + +.. + Local Variables: + mode: indented-text + indent-tabs-mode: nil + sentence-end-double-space: t + fill-column: 70 + coding: utf-8 + End: From dynkin at gmail.com Thu Apr 13 17:28:32 2006 From: dynkin at gmail.com (George Yoshida) Date: Fri, 14 Apr 2006 00:28:32 +0900 Subject: [Python-checkins] r45329 - python/trunk/Doc/whatsnew/whatsnew25.tex In-Reply-To: References: <20060413020445.29D061E4009@bag.python.org> <2f188ee80604122005n37b11481w1d2f9850a4a337f3@mail.gmail.com> Message-ID: <2f188ee80604130828x47cf5f4u3214c533323ea3c@mail.gmail.com> On 4/13/06, Neal Norwitz wrote: > First off, thanks for spotting this and all the other issues. I > checked in a fix. > Second, I'm really tired of checking in these fixes. :-) > How would you like to check them in yourself next time? OK, maybe I had bothered others too much and now is the time to commit them by myself. I'd be happy to be a committer. Please tell me the instruction to get the svn access. And please list my name as one of the LaTexification member in this page: http://www.python.org/dev/doc/ I'll probably work on the documentation most actively. FYI, my sourceforge account name is "quiver". Thanks in advance. -- george From guido at python.org Thu Apr 13 17:58:07 2006 From: guido at python.org (Guido van Rossum) Date: Thu, 13 Apr 2006 16:58:07 +0100 Subject: [Python-checkins] r43299 - python/trunk/Lib/traceback.py In-Reply-To: References: <20060324221055.5853B1E4021@bag.python.org> Message-ID: In any case, one part of the recently debated and rolled-back change by georg should be reinstated: the issubclass() check should mention BaseException. Test case that is now broken: >>> try: raise KeyboardInterrupt ... except: import traceback; traceback.print_exc() ... Traceback (most recent call last): File "", line 1, in >>> This should have printed KeyboardInterrupt instead of . --Guido On 4/13/06, Neal Norwitz wrote: > ping > > On 3/24/06, Neal Norwitz wrote: > > How about a test case? > > > > Is this just for strings? If so, should you just check for basestring instead of type? > > > > n > > -- > > > > On 3/24/06, phillip.eby wrote: > > > Author: phillip.eby > > > Date: Fri Mar 24 23:10:54 2006 > > > New Revision: 43299 > > > > > > Modified: > > > python/trunk/Lib/traceback.py > > > Log: > > > Revert r42719, because the isinstance() check wasn't redundant; formatting a > > > string exception was causing a TypeError. > > > > > > > > > Modified: python/trunk/Lib/traceback.py > > > ============================================================================== > > > --- python/trunk/Lib/traceback.py (original) > > > +++ python/trunk/Lib/traceback.py Fri Mar 24 23:10:54 2006 > > > @@ -158,7 +158,7 @@ > > > """ > > > list = [] > > > if (type(etype) == types.ClassType > > > - or issubclass(etype, Exception)): > > > + or (isinstance(etype, type) and issubclass(etype, Exception))): > > > stype = etype.__name__ > > > else: > > > stype = etype > > > _______________________________________________ > > > Python-checkins mailing list > > > Python-checkins at python.org > > > http://mail.python.org/mailman/listinfo/python-checkins > > > > > > _______________________________________________ > Python-checkins mailing list > Python-checkins at python.org > http://mail.python.org/mailman/listinfo/python-checkins > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From jimjjewett at gmail.com Thu Apr 13 18:45:08 2006 From: jimjjewett at gmail.com (Jim Jewett) Date: Thu, 13 Apr 2006 12:45:08 -0400 Subject: [Python-checkins] r45334 - python/trunk/Lib/test/leakers/test_gen1.py python/trunk/Lib/test/leakers/test_generator_cycle.py python/trunk/Lib/test/leakers/test_tee.py In-Reply-To: <20060413043540.51D6F1E404D@bag.python.org> References: <20060413043540.51D6F1E404D@bag.python.org> Message-ID: Should these really be removed, or just added to the regular tests -- to ensure that the leakage doesn't get worse. -jJ On 4/13/06, neal.norwitz wrote: > Author: neal.norwitz > Date: Thu Apr 13 06:35:36 2006 > New Revision: 45334 > > Added: > python/trunk/Lib/test/leakers/test_gen1.py (contents, props changed) > Removed: > python/trunk/Lib/test/leakers/test_generator_cycle.py > python/trunk/Lib/test/leakers/test_tee.py > Log: > Remove tests that no longer leak. There is still one leaking generator test > > Added: python/trunk/Lib/test/leakers/test_gen1.py > ============================================================================== > --- (empty file) > +++ python/trunk/Lib/test/leakers/test_gen1.py Thu Apr 13 06:35:36 2006 > @@ -0,0 +1,19 @@ > +import gc > + > +# Taken from test_generators > + > +def f(): > + try: > + yield > + except GeneratorExit: > + yield "foo!" > + > +def inner_leak(): > + g = f() > + g.next() > + > +def leak(): > + inner_leak() > + gc.collect() > + gc.collect() > + gc.collect() > > Deleted: /python/trunk/Lib/test/leakers/test_generator_cycle.py > ============================================================================== > --- /python/trunk/Lib/test/leakers/test_generator_cycle.py Thu Apr 13 06:35:36 2006 > +++ (empty file) > @@ -1,10 +0,0 @@ > - > -# This leaks since the introduction of yield-expr and the use of generators > -# as coroutines, trunk revision 39239. The cycle-GC doesn't seem to pick up > -# the cycle, or decides it can't clean it up. > - > -def leak(): > - def gen(): > - while True: > - yield g > - g = gen() > > Deleted: /python/trunk/Lib/test/leakers/test_tee.py > ============================================================================== > --- /python/trunk/Lib/test/leakers/test_tee.py Thu Apr 13 06:35:36 2006 > +++ (empty file) > @@ -1,20 +0,0 @@ > - > -# Test case taken from test_itertools > -# See http://mail.python.org/pipermail/python-dev/2005-November/058339.html > -# When this is fixed remember to remove from LEAKY_TESTS in Misc/build.sh. > - > -from itertools import tee > - > -def leak(): > - def fib(): > - def yield_identity_forever(g): > - while 1: > - yield g > - def _fib(): > - for i in yield_identity_forever(head): > - yield i > - head, tail, result = tee(_fib(), 3) > - return result > - > - x = fib() > - x.next() > _______________________________________________ > Python-checkins mailing list > Python-checkins at python.org > http://mail.python.org/mailman/listinfo/python-checkins > From pje at telecommunity.com Thu Apr 13 18:48:50 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 13 Apr 2006 12:48:50 -0400 Subject: [Python-checkins] r43299 - python/trunk/Lib/traceback.py In-Reply-To: References: <20060324221055.5853B1E4021@bag.python.org> Message-ID: <5.1.1.6.0.20060413124741.01fe5c98@mail.telecommunity.com> At 10:26 PM 4/12/2006 -0700, Neal Norwitz wrote: >ping > >On 3/24/06, Neal Norwitz wrote: > > How about a test case? The author of the broken code didn't run the full test suite, which includes formatting of string exceptions already. From fdrake at acm.org Thu Apr 13 19:37:59 2006 From: fdrake at acm.org (Fred L. Drake, Jr.) Date: Thu, 13 Apr 2006 13:37:59 -0400 Subject: [Python-checkins] r45329 - python/trunk/Doc/whatsnew/whatsnew25.tex In-Reply-To: <2f188ee80604130828x47cf5f4u3214c533323ea3c@mail.gmail.com> References: <20060413020445.29D061E4009@bag.python.org> <2f188ee80604130828x47cf5f4u3214c533323ea3c@mail.gmail.com> Message-ID: <200604131337.59193.fdrake@acm.org> On Thursday 13 April 2006 11:28, George Yoshida wrote: > And please list my name as one of the LaTexification member in > this page: > http://www.python.org/dev/doc/ > I'll probably work on the documentation most actively. I've taken care of this; thanks! -Fred -- Fred L. Drake, Jr. From martin at v.loewis.de Thu Apr 13 19:50:47 2006 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 13 Apr 2006 19:50:47 +0200 Subject: [Python-checkins] r45329 - python/trunk/Doc/whatsnew/whatsnew25.tex In-Reply-To: <2f188ee80604130828x47cf5f4u3214c533323ea3c@mail.gmail.com> References: <20060413020445.29D061E4009@bag.python.org> <2f188ee80604122005n37b11481w1d2f9850a4a337f3@mail.gmail.com> <2f188ee80604130828x47cf5f4u3214c533323ea3c@mail.gmail.com> Message-ID: <443E8F77.8090403@v.loewis.de> George Yoshida wrote: > OK, maybe I had bothered others too much and now is the time to > commit them by myself. I'd be happy to be a committer. > Please tell me the instruction to get the svn access. Sure. Please send me your SSH public key. Are you ok with "george.yoshida" as the committer name? Regards, Martin From python-checkins at python.org Thu Apr 13 21:16:14 2006 From: python-checkins at python.org (martin.v.loewis) Date: Thu, 13 Apr 2006 21:16:14 +0200 (CEST) Subject: [Python-checkins] r45367 - python/trunk/Python/pyarena.c Message-ID: <20060413191614.2E9D31E401C@bag.python.org> Author: martin.v.loewis Date: Thu Apr 13 21:16:13 2006 New Revision: 45367 Modified: python/trunk/Python/pyarena.c Log: Force 8-alignment of memory blocks, as needed on 64-bit machines that require pointers to be aligned (e.g. IA64) Modified: python/trunk/Python/pyarena.c ============================================================================== --- python/trunk/Python/pyarena.c (original) +++ python/trunk/Python/pyarena.c Thu Apr 13 21:16:13 2006 @@ -12,6 +12,11 @@ */ #define DEFAULT_BLOCK_SIZE 8192 +#define ALIGNMENT 8 +#define ALIGNMENT_SHIFT 3 +#define ALIGNMENT_MASK (ALIGNMENT - 1) +#define ROUNDUP(x) (((x) + ALIGNMENT_MASK) & ~ALIGNMENT_MASK) + typedef struct _block { /* Total number of bytes owned by this block available to pass out. * Read-only after initialization. The first such byte starts at @@ -82,7 +87,8 @@ b->ab_size = size; b->ab_mem = (void *)(b + 1); b->ab_next = NULL; - b->ab_offset = 0; + b->ab_offset = ROUNDUP((Py_uintptr_t)(b->ab_mem)) - + (Py_uintptr_t)(b->ab_mem); return b; } @@ -100,6 +106,7 @@ { void *p; assert(b); + size = ROUNDUP(size); if (b->ab_offset + size > b->ab_size) { /* If we need to allocate more memory than will fit in the default block, allocate a one-off block that is From python-checkins at python.org Thu Apr 13 21:19:02 2006 From: python-checkins at python.org (gregory.p.smith) Date: Thu, 13 Apr 2006 21:19:02 +0200 (CEST) Subject: [Python-checkins] r45368 - in python/trunk: Doc/lib/libbsddb.tex Misc/NEWS README setup.py Message-ID: <20060413191902.967BD1E4009@bag.python.org> Author: gregory.p.smith Date: Thu Apr 13 21:19:01 2006 New Revision: 45368 Modified: python/trunk/Doc/lib/libbsddb.tex python/trunk/Misc/NEWS python/trunk/README python/trunk/setup.py Log: raise the minimum supported BerkeleyDB version to 3.3 and add notes to news about this and a couple other recent fixes. Modified: python/trunk/Doc/lib/libbsddb.tex ============================================================================== --- python/trunk/Doc/lib/libbsddb.tex (original) +++ python/trunk/Doc/lib/libbsddb.tex Thu Apr 13 21:19:01 2006 @@ -15,9 +15,8 @@ serialize them somehow, typically using \function{marshal.dumps()} or \function{pickle.dumps}. -Starting with Python 2.3 the \module{bsddb} module requires the -Berkeley DB library version 3.2 or later (it is known to work with 3.2 -through 4.3 at the time of this writing). +The \module{bsddb} module requires a Berkeley DB library version from +3.3 thru 4.4. \begin{seealso} \seeurl{http://pybsddb.sourceforge.net/}{Website with documentation Modified: python/trunk/Misc/NEWS ============================================================================== --- python/trunk/Misc/NEWS (original) +++ python/trunk/Misc/NEWS Thu Apr 13 21:19:01 2006 @@ -44,7 +44,16 @@ - Bug #1467952: os.listdir() now correctly raises an error if readdir() fails with an error condition. -- Fix bsddb.db.DBError derived exceptions so they can be unpickled. +- Fixed bsddb.db.DBError derived exceptions so they can be unpickled. + +- Bug #1117761: bsddb.*open() no longer raises an exception when using + the cachesize parameter. + +- Bug #1149413: bsddb.*open() no longer raises an exception when using + a temporary db (file=None) with the 'n' flag to truncate on open. + +- Bug #1332852: bsddb module minimum BerkeleyDB version raised to 3.3 + as older versions cause excessive test failures. Library ------- Modified: python/trunk/README ============================================================================== --- python/trunk/README (original) +++ python/trunk/README Thu Apr 13 21:19:01 2006 @@ -719,14 +719,13 @@ Beginning with Python version 2.3, the PyBsddb package was adopted into Python as the bsddb package, exposing a set of package-level functions which provide -backwards-compatible behavior. Only versions 3.1 through 4.1 of +backwards-compatible behavior. Only versions 3.3 through 4.4 of Sleepycat's libraries provide the necessary API, so older versions aren't supported through this interface. The old bsddb module has been retained as bsddb185, though it is not built by default. Users wishing to use it will have to tweak Modules/Setup to build it. The dbm module will still be built against the Sleepycat libraries if -other preferred alternatives (ndbm, gdbm) are not found, though -versions of the Sleepycat library prior to 3.1 are not considered. +other preferred alternatives (ndbm, gdbm) are not found. Building the sqlite3 module --------------------------- Modified: python/trunk/setup.py ============================================================================== --- python/trunk/setup.py (original) +++ python/trunk/setup.py Thu Apr 13 21:19:01 2006 @@ -571,14 +571,13 @@ # Sleepycat Berkeley DB interface. http://www.sleepycat.com # - # This requires the Sleepycat DB code. The earliest supported version - # of that library is 3.2, the latest supported version is 4.4. A list - # of available releases can be found at - # - # http://www.sleepycat.com/update/index.html + # This requires the Sleepycat DB code. The supported versions + # are set below. Visit http://www.sleepycat.com/ to download + # a release. Most open source OSes come with one or more + # versions of BerkeleyDB already installed. max_db_ver = (4, 4) - min_db_ver = (3, 2) + min_db_ver = (3, 3) db_setup_debug = False # verbose debug prints from this script? # construct a list of paths to look for the header file in on From python-checkins at python.org Thu Apr 13 21:24:48 2006 From: python-checkins at python.org (gregory.p.smith) Date: Thu, 13 Apr 2006 21:24:48 +0200 (CEST) Subject: [Python-checkins] r45369 - python/branches/release24-maint/Misc/NEWS Message-ID: <20060413192448.A43A61E4009@bag.python.org> Author: gregory.p.smith Date: Thu Apr 13 21:24:47 2006 New Revision: 45369 Modified: python/branches/release24-maint/Misc/NEWS Log: mention bsddb open cachesize parameter bugfix Modified: python/branches/release24-maint/Misc/NEWS ============================================================================== --- python/branches/release24-maint/Misc/NEWS (original) +++ python/branches/release24-maint/Misc/NEWS Thu Apr 13 21:24:47 2006 @@ -20,6 +20,9 @@ - Fix bsddb.db.DBError derived exceptions so they can be unpickled. +- Bug #1117761: bsddb.*open() no longer raises an exception when using + the cachesize parameter. + Library ------- From neal at metaslash.com Thu Apr 13 23:04:53 2006 From: neal at metaslash.com (Neal Norwitz) Date: Thu, 13 Apr 2006 17:04:53 -0400 Subject: [Python-checkins] Python Regression Test Failures refleak (1) Message-ID: <20060413210453.GA27083@python.psfb.org> test_cmd_line leaked [0, -17, 0] references test_generators leaked [1, 1, 1] references test_threadedtempfile leaked [-85, 0, 0] references test_threading_local leaked [26, 24, -154] references test_urllib2 leaked [-121, 88, 99] references From theller at python.net Thu Apr 13 23:22:23 2006 From: theller at python.net (Thomas Heller) Date: Thu, 13 Apr 2006 23:22:23 +0200 Subject: [Python-checkins] Python Regression Test Failures refleak (1) In-Reply-To: <20060413210453.GA27083@python.psfb.org> References: <20060413210453.GA27083@python.psfb.org> Message-ID: <443EC10F.9040707@python.net> > test_cmd_line leaked [0, -17, 0] references > test_generators leaked [1, 1, 1] references > test_threadedtempfile leaked [-85, 0, 0] references > test_threading_local leaked [26, 24, -154] references > test_urllib2 leaked [-121, 88, 99] references Neal, does this mean that test_ctypes does not leak anymore, or is it somehow excluded from this list? Thanks, Thomas From tim.peters at gmail.com Fri Apr 14 00:42:40 2006 From: tim.peters at gmail.com (Tim Peters) Date: Thu, 13 Apr 2006 18:42:40 -0400 Subject: [Python-checkins] Python Regression Test Failures refleak (1) In-Reply-To: <443EC10F.9040707@python.net> References: <20060413210453.GA27083@python.psfb.org> <443EC10F.9040707@python.net> Message-ID: <1f7befae0604131542v3ef6d68buf902ef51af305d3b@mail.gmail.com> [Thomas Heller] > Neal, does this mean that test_ctypes does not leak anymore, or is it > somehow excluded from this list? I'm guessing, but I believe Neal uses Misc/build.sh for this, and ctypes is now in its LEAKY_TESTS pattern of test names to skip: LEAKY_TESTS="test_(cmd_line|ctypes|filecmp|socket|threadedtempfile|threading|threading_local|urllib2)" From python-checkins at python.org Fri Apr 14 00:58:43 2006 From: python-checkins at python.org (tim.peters) Date: Fri, 14 Apr 2006 00:58:43 +0200 (CEST) Subject: [Python-checkins] r45370 - python/trunk/Lib/test/test_getargs2.py Message-ID: <20060413225843.969BB1E400B@bag.python.org> Author: tim.peters Date: Fri Apr 14 00:58:42 2006 New Revision: 45370 Modified: python/trunk/Lib/test/test_getargs2.py Log: Whitespace normalization. Modified: python/trunk/Lib/test/test_getargs2.py ============================================================================== --- python/trunk/Lib/test/test_getargs2.py (original) +++ python/trunk/Lib/test/test_getargs2.py Fri Apr 14 00:58:42 2006 @@ -184,7 +184,7 @@ def test_n(self): from _testcapi import getargs_n - # n returns 'Py_ssize_t', and does range checking + # n returns 'Py_ssize_t', and does range checking # (PY_SSIZE_T_MIN ... PY_SSIZE_T_MAX) self.failUnlessEqual(3, getargs_n(3.14)) self.failUnlessEqual(99, getargs_n(Long())) From python-checkins at python.org Fri Apr 14 01:12:24 2006 From: python-checkins at python.org (tim.peters) Date: Fri, 14 Apr 2006 01:12:24 +0200 (CEST) Subject: [Python-checkins] r45371 - python/trunk/Lib/test/regrtest.py Message-ID: <20060413231224.C6D5A1E400B@bag.python.org> Author: tim.peters Date: Fri Apr 14 01:12:24 2006 New Revision: 45371 Modified: python/trunk/Lib/test/regrtest.py Log: When using -R, lots of "*** DocTestRunner.merge:" nuisance messages appear. Get rid of them by nuking doctest's default DocTestRunner instance as part of cleanup(). Also cleanup() before running the first test repetition (the test was run once before we get into the -R branch). Modified: python/trunk/Lib/test/regrtest.py ============================================================================== --- python/trunk/Lib/test/regrtest.py (original) +++ python/trunk/Lib/test/regrtest.py Fri Apr 14 01:12:24 2006 @@ -520,7 +520,7 @@ import gc def cleanup(): import _strptime, linecache, warnings, dircache - import urlparse, urllib, urllib2, mimetypes + import urlparse, urllib, urllib2, mimetypes, doctest from distutils.dir_util import _path_created _path_created.clear() warnings.filters[:] = fs @@ -537,6 +537,7 @@ dircache.reset() linecache.clearcache() mimetypes._default_mime_types() + doctest.master = None if indirect_test: def run_the_test(): indirect_test() @@ -548,6 +549,7 @@ print >> sys.stderr, "beginning", repcount, "repetitions" print >> sys.stderr, \ ("1234567890"*(repcount//10 + 1))[:repcount] + cleanup() for i in range(repcount): rc = sys.gettotalrefcount() run_the_test() From buildbot at python.org Fri Apr 14 01:36:30 2006 From: buildbot at python.org (buildbot at python.org) Date: Thu, 13 Apr 2006 23:36:30 +0000 Subject: [Python-checkins] buildbot warnings in x86 gentoo trunk Message-ID: <20060413233630.7BF471E400B@bag.python.org> The Buildbot has detected a new failure of x86 gentoo trunk. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520gentoo%2520trunk/builds/449 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: tim.peters Build Had Warnings: warnings test sincerely, -The Buildbot From buildbot at python.org Fri Apr 14 01:49:28 2006 From: buildbot at python.org (buildbot at python.org) Date: Thu, 13 Apr 2006 23:49:28 +0000 Subject: [Python-checkins] buildbot warnings in x86 OpenBSD trunk Message-ID: <20060413234928.264391E400B@bag.python.org> The Buildbot has detected a new failure of x86 OpenBSD trunk. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520OpenBSD%2520trunk/builds/286 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: tim.peters Build Had Warnings: warnings test sincerely, -The Buildbot From python-checkins at python.org Fri Apr 14 05:53:34 2006 From: python-checkins at python.org (neal.norwitz) Date: Fri, 14 Apr 2006 05:53:34 +0200 (CEST) Subject: [Python-checkins] r45372 - python/trunk/Python/pyarena.c Message-ID: <20060414035334.09C7D1E4004@bag.python.org> Author: neal.norwitz Date: Fri Apr 14 05:53:34 2006 New Revision: 45372 Modified: python/trunk/Python/pyarena.c Log: ALIGNMENT_SHIFT is not used Modified: python/trunk/Python/pyarena.c ============================================================================== --- python/trunk/Python/pyarena.c (original) +++ python/trunk/Python/pyarena.c Fri Apr 14 05:53:34 2006 @@ -13,7 +13,6 @@ #define DEFAULT_BLOCK_SIZE 8192 #define ALIGNMENT 8 -#define ALIGNMENT_SHIFT 3 #define ALIGNMENT_MASK (ALIGNMENT - 1) #define ROUNDUP(x) (((x) + ALIGNMENT_MASK) & ~ALIGNMENT_MASK) From dynkin at gmail.com Fri Apr 14 05:55:59 2006 From: dynkin at gmail.com (George Yoshida) Date: Fri, 14 Apr 2006 12:55:59 +0900 Subject: [Python-checkins] r45329 - python/trunk/Doc/whatsnew/whatsnew25.tex In-Reply-To: <200604131337.59193.fdrake@acm.org> References: <20060413020445.29D061E4009@bag.python.org> <2f188ee80604130828x47cf5f4u3214c533323ea3c@mail.gmail.com> <200604131337.59193.fdrake@acm.org> Message-ID: <2f188ee80604132055t67abb5fag90cb73b0b68a75ae@mail.gmail.com> On 4/14/06, Fred L. Drake, Jr. wrote: > On Thursday 13 April 2006 11:28, George Yoshida wrote: > > And please list my name as one of the LaTexification member in > > this page: > > http://www.python.org/dev/doc/ > > I'll probably work on the documentation most actively. > > I've taken care of this; thanks! Fred, I'd like to ask you one more thing. Can you grant me sourceforge manager privilege so I can handle bug/patch tasks? Login users are basically only allowed to comment on them. -- george From nnorwitz at gmail.com Fri Apr 14 06:03:49 2006 From: nnorwitz at gmail.com (Neal Norwitz) Date: Thu, 13 Apr 2006 21:03:49 -0700 Subject: [Python-checkins] Python Regression Test Failures refleak (1) In-Reply-To: <1f7befae0604131542v3ef6d68buf902ef51af305d3b@mail.gmail.com> References: <20060413210453.GA27083@python.psfb.org> <443EC10F.9040707@python.net> <1f7befae0604131542v3ef6d68buf902ef51af305d3b@mail.gmail.com> Message-ID: On 4/13/06, Tim Peters wrote: > [Thomas Heller] > > Neal, does this mean that test_ctypes does not leak anymore, or is it > > somehow excluded from this list? > > I'm guessing, but I believe Neal uses Misc/build.sh for this, and > ctypes is now in its LEAKY_TESTS pattern of test names to skip: > > LEAKY_TESTS="test_(cmd_line|ctypes|filecmp|socket|threadedtempfile|threading|threading_local|urllib2)" Tim's right, the leak report comes from Misc/build.sh (same with building the docs and other emails sent to python-checkins). ctypes leaks inconsistently which is why it's in the LEAKY_TESTS. In this case it didn't show any leaks. I don't know if the leaks will return. When the leaks aren't consistent and report the same number, it usually means there are references elsewhere that may or may not be cleaned up. However, ctypes still leaks when reloaded. We probably ought to have a test for each module that just does reload(module) and see if that leaks. I wouldn't be surprised if several C modules leak. >>> import ctypes, gc [30740 refs] >>> reload(ctypes) ; gc.collect() ; gc.collect() ; gc.collect() 20 0 0 [32090 refs] >>> reload(ctypes) ; gc.collect() ; gc.collect() ; gc.collect() 103 13 0 [32576 refs] >>> reload(ctypes) ; gc.collect() ; gc.collect() ; gc.collect() 103 13 0 [33048 refs] >>> reload(ctypes) ; gc.collect() ; gc.collect() ; gc.collect() 103 13 0 [33520 refs] >>> reload(ctypes) ; gc.collect() ; gc.collect() ; gc.collect() 103 13 0 [33992 refs] Cheers, n From tim.peters at gmail.com Fri Apr 14 06:53:01 2006 From: tim.peters at gmail.com (Tim Peters) Date: Fri, 14 Apr 2006 00:53:01 -0400 Subject: [Python-checkins] r45329 - python/trunk/Doc/whatsnew/whatsnew25.tex In-Reply-To: <2f188ee80604132055t67abb5fag90cb73b0b68a75ae@mail.gmail.com> References: <20060413020445.29D061E4009@bag.python.org> <2f188ee80604130828x47cf5f4u3214c533323ea3c@mail.gmail.com> <200604131337.59193.fdrake@acm.org> <2f188ee80604132055t67abb5fag90cb73b0b68a75ae@mail.gmail.com> Message-ID: <1f7befae0604132153i13cd233fu1880985ba8525497@mail.gmail.com> [George Yoshida] > Fred, I'd like to ask you one more thing. Fred paid me to do it instead ;-) > Can you grant me sourceforge manager privilege so I can handle > bug/patch tasks? > Login users are basically only allowed to comment on them. I added SF name "quiver" to the SF Python project as a tracker admin. The SF permission system has gotten so convoluted I'm not sure that's enough, so scream at me if you're still getting blocked. From python-checkins at python.org Fri Apr 14 06:55:00 2006 From: python-checkins at python.org (tim.peters) Date: Fri, 14 Apr 2006 06:55:00 +0200 (CEST) Subject: [Python-checkins] r45373 - python/trunk/Misc/developers.txt Message-ID: <20060414045500.7F8361E4004@bag.python.org> Author: tim.peters Date: Fri Apr 14 06:54:58 2006 New Revision: 45373 Modified: python/trunk/Misc/developers.txt Log: Added George Yoshida. Modified: python/trunk/Misc/developers.txt ============================================================================== --- python/trunk/Misc/developers.txt (original) +++ python/trunk/Misc/developers.txt Fri Apr 14 06:54:58 2006 @@ -17,6 +17,12 @@ Permissions History ------------------- +- George Yoshida (SF name "quiver") added to the SourceForge Python + project 14 Apr 2006, by Tim Peters, as a tracker admin. See + contemporaneous python-checkins thread with the unlikely Subject: + + r45329 - python/trunk/Doc/whatsnew/whatsnew25.tex + - Ronald Oussoren was given SVN access on 3 Mar 2006 by NCN, for Mac related work. From python-checkins at python.org Fri Apr 14 07:20:29 2006 From: python-checkins at python.org (neal.norwitz) Date: Fri, 14 Apr 2006 07:20:29 +0200 (CEST) Subject: [Python-checkins] r45374 - in python/trunk: Misc/NEWS Python/getargs.c Message-ID: <20060414052029.8503B1E4004@bag.python.org> Author: neal.norwitz Date: Fri Apr 14 07:20:28 2006 New Revision: 45374 Modified: python/trunk/Misc/NEWS python/trunk/Python/getargs.c Log: SF Bug #1454485, array.array('u') could crash the interpreter when passing a string. Martin already fixed the actual crash by ensuring Py_UNICODE is unsigned. As discussed on python-dev, this fix removes the possibility of creating a unicode string from a raw buffer. There is an outstanding question of how to fix the crash in 2.4. Modified: python/trunk/Misc/NEWS ============================================================================== --- python/trunk/Misc/NEWS (original) +++ python/trunk/Misc/NEWS Fri Apr 14 07:20:28 2006 @@ -12,6 +12,13 @@ Core and builtins ----------------- +- Bug #1454485, array.array('u') could crash the interpreter. This was + due to PyArgs_ParseTuple(args, 'u#', ...) trying to convert buffers (strings) + to unicode when it didn't make sense. 'u#' now requires a unicode string. + +- Py_UNICODE is unsigned. It was always documented as unsigned, but + due to a bug had a signed value in previous versions. + - Patch #837242: ``id()`` of any Python object always gives a positive number now, which might be a long integer. ``PyLong_FromVoidPtr`` and ``PyLong_AsVoidPtr`` have been changed accordingly. Note that it has Modified: python/trunk/Python/getargs.c ============================================================================== --- python/trunk/Python/getargs.c (original) +++ python/trunk/Python/getargs.c Fri Apr 14 07:20:28 2006 @@ -1042,11 +1042,8 @@ STORE_SIZE(PyUnicode_GET_SIZE(arg)); } else { - char *buf; - Py_ssize_t count = convertbuffer(arg, p, &buf); - if (count < 0) - return converterr(buf, arg, msgbuf, bufsize); - STORE_SIZE(count/(sizeof(Py_UNICODE))); + return converterr("cannot convert raw buffers", + arg, msgbuf, bufsize); } format++; } else { From python-checkins at python.org Fri Apr 14 07:35:40 2006 From: python-checkins at python.org (neal.norwitz) Date: Fri, 14 Apr 2006 07:35:40 +0200 (CEST) Subject: [Python-checkins] r45375 - python/trunk/Lib/test/leakers/README.txt Message-ID: <20060414053540.657C01E4004@bag.python.org> Author: neal.norwitz Date: Fri Apr 14 07:35:39 2006 New Revision: 45375 Modified: python/trunk/Lib/test/leakers/README.txt Log: Update notes to address cyclic garbage and what should happen when a test is fixed. Modified: python/trunk/Lib/test/leakers/README.txt ============================================================================== --- python/trunk/Lib/test/leakers/README.txt (original) +++ python/trunk/Lib/test/leakers/README.txt Fri Apr 14 07:35:39 2006 @@ -5,6 +5,15 @@ doesn't increase, the bug has been fixed and the file should be removed from the repository. +Note: be careful to check for cyclic garbage. Sometimes it may be helpful +to define the leak function like: + +def leak(): + def inner_leak(): + # this is the function that leaks, but also creates cycles + inner_leak() + gc.collect() ; gc.collect() ; gc.collect() + Here's an example interpreter session for test_gestalt which still leaks: >>> from test.leakers.test_gestalt import leak @@ -17,3 +26,7 @@ [28940 refs] >>> +Once the leak is fixed, the test case should be moved into an appropriate +test (even if it was originally from the test suite). This ensures the +regression doesn't happen again. And if it does, it should be easier +to track down. From buildbot at python.org Fri Apr 14 07:54:50 2006 From: buildbot at python.org (buildbot at python.org) Date: Fri, 14 Apr 2006 05:54:50 +0000 Subject: [Python-checkins] buildbot warnings in x86 Ubuntu dapper (icc) trunk Message-ID: <20060414055451.1939B1E4004@bag.python.org> The Buildbot has detected a new failure of x86 Ubuntu dapper (icc) trunk. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520Ubuntu%2520dapper%2520%2528icc%2529%2520trunk/builds/122 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: tim.peters Build Had Warnings: warnings test sincerely, -The Buildbot From python-checkins at python.org Fri Apr 14 08:11:09 2006 From: python-checkins at python.org (neal.norwitz) Date: Fri, 14 Apr 2006 08:11:09 +0200 (CEST) Subject: [Python-checkins] r45376 - python/trunk/Lib/test/test_generators.py Message-ID: <20060414061109.4AC681E4004@bag.python.org> Author: neal.norwitz Date: Fri Apr 14 08:11:08 2006 New Revision: 45376 Modified: python/trunk/Lib/test/test_generators.py Log: Move the old test_generator_cycle.py which leaked but was removed into the test Modified: python/trunk/Lib/test/test_generators.py ============================================================================== --- python/trunk/Lib/test/test_generators.py (original) +++ python/trunk/Lib/test/test_generators.py Fri Apr 14 08:11:08 2006 @@ -739,6 +739,22 @@ """ +leak_test1 = """ + +This test leaked at one point due to generator finalization/destruction. +It was copied from Lib/test/leakers/test_generator_cycle.py before the file +was removed. + +>>> def leak(): +... def gen(): +... while True: +... yield g +... g = gen() + +>>> leak() + +""" + # syntax_tests mostly provokes SyntaxErrors. Also fiddling with #if 0 # hackery. @@ -1755,6 +1771,7 @@ "pep": pep_tests, "email": email_tests, "fun": fun_tests, + "leak1": leak_test1, "syntax": syntax_tests, "conjoin": conjoin_tests, "weakref": weakref_tests, From python-checkins at python.org Fri Apr 14 08:33:25 2006 From: python-checkins at python.org (neal.norwitz) Date: Fri, 14 Apr 2006 08:33:25 +0200 (CEST) Subject: [Python-checkins] r45377 - python/trunk/Lib/test/leakers/test_tee.py Message-ID: <20060414063325.805211E4004@bag.python.org> Author: neal.norwitz Date: Fri Apr 14 08:33:24 2006 New Revision: 45377 Added: python/trunk/Lib/test/leakers/test_tee.py (contents, props changed) Log: Restore test tee with some modifications. The test case came from test_generators, not test_itertools. Ensure there's no cyclic garbage we are counting. This is weird because it leaks, then reaches a limit: python.exe -i test_tee.py >>> leak() 0 [26633 refs] >>> leak() 0 [26658 refs] >>> leak() 0 [26683 refs] >>> leak() 0 [26708 refs] >>> leak() 0 [26708 refs] >>> leak() 0 [26708 refs] >>> leak() 0 Added: python/trunk/Lib/test/leakers/test_tee.py ============================================================================== --- (empty file) +++ python/trunk/Lib/test/leakers/test_tee.py Fri Apr 14 08:33:24 2006 @@ -0,0 +1,25 @@ + +# Test case taken from test_generators +# See http://mail.python.org/pipermail/python-dev/2005-November/058339.html + +from itertools import tee +import gc + +def leak(): + def inner(): + def fib(): + def yield_identity_forever(g): + while 1: + yield g + def _fib(): + for i in yield_identity_forever(head): + yield i + head, tail, result = tee(_fib(), 3) + return result + + x = fib() + x.next() + inner() + gc.collect() ; gc.collect() + # this is expected to return 0 + return gc.collect() From python-checkins at python.org Fri Apr 14 08:35:47 2006 From: python-checkins at python.org (neal.norwitz) Date: Fri, 14 Apr 2006 08:35:47 +0200 (CEST) Subject: [Python-checkins] r45378 - python/trunk/Lib/test/crashers/README Message-ID: <20060414063547.53D8D1E4019@bag.python.org> Author: neal.norwitz Date: Fri Apr 14 08:35:46 2006 New Revision: 45378 Modified: python/trunk/Lib/test/crashers/README Log: Copy note from leakers README here too. We want to keep all test cases. Modified: python/trunk/Lib/test/crashers/README ============================================================================== --- python/trunk/Lib/test/crashers/README (original) +++ python/trunk/Lib/test/crashers/README Fri Apr 14 08:35:46 2006 @@ -13,3 +13,8 @@ Put as much info into a docstring or comments to help determine the cause of the failure. Particularly note if the cause is system or environment dependent and what the variables are. + +Once the crash is fixed, the test case should be moved into an appropriate +test (even if it was originally from the test suite). This ensures the +regression doesn't happen again. And if it does, it should be easier +to track down. From buildbot at python.org Fri Apr 14 09:41:00 2006 From: buildbot at python.org (buildbot at python.org) Date: Fri, 14 Apr 2006 07:41:00 +0000 Subject: [Python-checkins] buildbot warnings in g4 osx.4 trunk Message-ID: <20060414074100.A84551E400B@bag.python.org> The Buildbot has detected a new failure of g4 osx.4 trunk. Full details are available at: http://www.python.org/dev/buildbot/all/g4%2520osx.4%2520trunk/builds/398 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: neal.norwitz Build Had Warnings: warnings test sincerely, -The Buildbot From python-checkins at python.org Fri Apr 14 11:08:44 2006 From: python-checkins at python.org (martin.v.loewis) Date: Fri, 14 Apr 2006 11:08:44 +0200 (CEST) Subject: [Python-checkins] r45379 - in python/trunk: Include/abstract.h Include/modsupport.h Objects/abstract.c Objects/stringobject.c Objects/unicodeobject.c Python/exceptions.c Python/getargs.c Python/modsupport.c Message-ID: <20060414090844.1CE6F1E400B@bag.python.org> Author: martin.v.loewis Date: Fri Apr 14 11:08:42 2006 New Revision: 45379 Modified: python/trunk/Include/abstract.h python/trunk/Include/modsupport.h python/trunk/Objects/abstract.c python/trunk/Objects/stringobject.c python/trunk/Objects/unicodeobject.c python/trunk/Python/exceptions.c python/trunk/Python/getargs.c python/trunk/Python/modsupport.c Log: Make Py_BuildValue, PyObject_CallFunction and PyObject_CallMethod aware of PY_SSIZE_T_CLEAN. Modified: python/trunk/Include/abstract.h ============================================================================== --- python/trunk/Include/abstract.h (original) +++ python/trunk/Include/abstract.h Fri Apr 14 11:08:42 2006 @@ -4,6 +4,11 @@ extern "C" { #endif +#ifdef PY_SSIZE_T_CLEAN +#define PyObject_CallFunction _PyObject_CallFunction_SizeT +#define PyObject_CallMethod _PyObject_CallMethod_SizeT +#endif + /* Abstract Object Interface (many thanks to Jim Fulton) */ /* Modified: python/trunk/Include/modsupport.h ============================================================================== --- python/trunk/Include/modsupport.h (original) +++ python/trunk/Include/modsupport.h Fri Apr 14 11:08:42 2006 @@ -17,21 +17,11 @@ #define PyArg_ParseTupleAndKeywords _PyArg_ParseTupleAndKeywords_SizeT #define PyArg_VaParse _PyArg_VaParse_SizeT #define PyArg_VaParseTupleAndKeywords _PyArg_VaParseTupleAndKeywords_SizeT -#define PyArg_BuildValue _PyArg_BuildValue_SizeT -#define PyArg_VaBuildValue _PyArg_VaBuildValue_SizeT +#define Py_BuildValue _Py_BuildValue_SizeT +#define Py_VaBuildValue _Py_VaBuildValue_SizeT #else -#ifdef HAVE_DECLSPEC_DLL -PyAPI_FUNC(int) _PyArg_Parse_SizeT(PyObject *, char *, ...); -PyAPI_FUNC(int) _PyArg_ParseTuple_SizeT(PyObject *, char *, ...); -PyAPI_FUNC(int) _PyArg_ParseTupleAndKeywords_SizeT(PyObject *, PyObject *, - const char *, char **, ...); -PyAPI_FUNC(PyObject *) _Py_BuildValue_SizeT(const char *, ...); -PyAPI_FUNC(int) _PyArg_VaParse_SizeT(PyObject *, char *, va_list); -PyAPI_FUNC(int) _PyArg_VaParseTupleAndKeywords_SizeT(PyObject *, PyObject *, - const char *, char **, va_list); PyAPI_FUNC(PyObject *) _Py_VaBuildValue_SizeT(const char *, va_list); #endif -#endif PyAPI_FUNC(int) PyArg_Parse(PyObject *, const char *, ...); PyAPI_FUNC(int) PyArg_ParseTuple(PyObject *, const char *, ...); Modified: python/trunk/Objects/abstract.c ============================================================================== --- python/trunk/Objects/abstract.c (original) +++ python/trunk/Objects/abstract.c Fri Apr 14 11:08:42 2006 @@ -10,6 +10,14 @@ #define HASINDEX(o) PyType_HasFeature((o)->ob_type, Py_TPFLAGS_HAVE_INDEX) +#ifdef HAVE_DECLSPEC_DLL +PyAPI_FUNC(PyObject *) _PyObject_CallFunction_SizeT(PyObject *callable_object, + char *format, ...); +PyAPI_FUNC(PyObject *) _PyObject_CallMethod_SizeT(PyObject *o, char *m, + char *format, ...); +#endif + + /* Shorthands to return certain errors */ static PyObject * @@ -1800,11 +1808,37 @@ return NULL; } +static PyObject* +call_function_tail(PyObject *callable, PyObject *args) +{ + PyObject *retval; + + if (args == NULL) + return NULL; + + if (!PyTuple_Check(args)) { + PyObject *a; + + a = PyTuple_New(1); + if (a == NULL) { + Py_DECREF(args); + return NULL; + } + PyTuple_SET_ITEM(a, 0, args); + args = a; + } + retval = PyObject_Call(callable, args, NULL); + + Py_DECREF(args); + + return retval; +} + PyObject * PyObject_CallFunction(PyObject *callable, char *format, ...) { va_list va; - PyObject *args, *retval; + PyObject *args; if (callable == NULL) return null_error(); @@ -1817,31 +1851,34 @@ else args = PyTuple_New(0); - if (args == NULL) - return NULL; + return call_function_tail(callable, args); +} - if (!PyTuple_Check(args)) { - PyObject *a; +PyObject * +_PyObject_CallFunction_SizeT(PyObject *callable, char *format, ...) +{ + va_list va; + PyObject *args; - a = PyTuple_New(1); - if (a == NULL) - return NULL; - if (PyTuple_SetItem(a, 0, args) < 0) - return NULL; - args = a; - } - retval = PyObject_Call(callable, args, NULL); + if (callable == NULL) + return null_error(); - Py_DECREF(args); + if (format && *format) { + va_start(va, format); + args = _Py_VaBuildValue_SizeT(format, va); + va_end(va); + } + else + args = PyTuple_New(0); - return retval; + return call_function_tail(callable, args); } PyObject * PyObject_CallMethod(PyObject *o, char *name, char *format, ...) { va_list va; - PyObject *args = NULL; + PyObject *args; PyObject *func = NULL; PyObject *retval = NULL; @@ -1867,24 +1904,49 @@ else args = PyTuple_New(0); - if (!args) - goto exit; + retval = call_function_tail(func, args); - if (!PyTuple_Check(args)) { - PyObject *a; + exit: + /* args gets consumed in call_function_tail */ + Py_XDECREF(func); - a = PyTuple_New(1); - if (a == NULL) - goto exit; - if (PyTuple_SetItem(a, 0, args) < 0) - goto exit; - args = a; + return retval; +} + +PyObject * +_PyObject_CallMethod_SizeT(PyObject *o, char *name, char *format, ...) +{ + va_list va; + PyObject *args; + PyObject *func = NULL; + PyObject *retval = NULL; + + if (o == NULL || name == NULL) + return null_error(); + + func = PyObject_GetAttrString(o, name); + if (func == NULL) { + PyErr_SetString(PyExc_AttributeError, name); + return 0; + } + + if (!PyCallable_Check(func)) { + type_error("call of non-callable attribute"); + goto exit; + } + + if (format && *format) { + va_start(va, format); + args = _Py_VaBuildValue_SizeT(format, va); + va_end(va); } + else + args = PyTuple_New(0); - retval = PyObject_Call(func, args, NULL); + retval = call_function_tail(func, args); exit: - Py_XDECREF(args); + /* args gets consumed in call_function_tail */ Py_XDECREF(func); return retval; Modified: python/trunk/Objects/stringobject.c ============================================================================== --- python/trunk/Objects/stringobject.c (original) +++ python/trunk/Objects/stringobject.c Fri Apr 14 11:08:42 2006 @@ -1,5 +1,6 @@ /* String object implementation */ +#define PY_SSIZE_T_CLEAN #include "Python.h" #include Modified: python/trunk/Objects/unicodeobject.c ============================================================================== --- python/trunk/Objects/unicodeobject.c (original) +++ python/trunk/Objects/unicodeobject.c Fri Apr 14 11:08:42 2006 @@ -36,6 +36,7 @@ */ +#define PY_SSIZE_T_CLEAN #include "Python.h" #include "unicodeobject.h" Modified: python/trunk/Python/exceptions.c ============================================================================== --- python/trunk/Python/exceptions.c (original) +++ python/trunk/Python/exceptions.c Fri Apr 14 11:08:42 2006 @@ -14,6 +14,7 @@ * Copyright (c) 1998-2000 by Secret Labs AB. All rights reserved. */ +#define PY_SSIZE_T_CLEAN #include "Python.h" #include "osdefs.h" @@ -1450,8 +1451,8 @@ assert(length < INT_MAX); assert(start < INT_MAX); assert(end < INT_MAX); - return PyObject_CallFunction(PyExc_UnicodeDecodeError, "ss#iis", - encoding, object, (int)length, (int)start, (int)end, reason); + return PyObject_CallFunction(PyExc_UnicodeDecodeError, "ss#nns", + encoding, object, length, start, end, reason); } @@ -1565,7 +1566,7 @@ const Py_UNICODE *object, Py_ssize_t length, Py_ssize_t start, Py_ssize_t end, const char *reason) { - return PyObject_CallFunction(PyExc_UnicodeTranslateError, "u#iis", + return PyObject_CallFunction(PyExc_UnicodeTranslateError, "u#nns", object, length, start, end, reason); } #endif Modified: python/trunk/Python/getargs.c ============================================================================== --- python/trunk/Python/getargs.c (original) +++ python/trunk/Python/getargs.c Fri Apr 14 11:08:42 2006 @@ -18,6 +18,18 @@ int PyArg_VaParseTupleAndKeywords(PyObject *, PyObject *, const char *, char **, va_list); +#ifdef HAVE_DECLSPEC_DLL +/* Export functions */ +PyAPI_FUNC(int) _PyArg_Parse_SizeT(PyObject *, char *, ...); +PyAPI_FUNC(int) _PyArg_ParseTuple_SizeT(PyObject *, char *, ...); +PyAPI_FUNC(int) _PyArg_ParseTupleAndKeywords_SizeT(PyObject *, PyObject *, + const char *, char **, ...); +PyAPI_FUNC(PyObject *) _Py_BuildValue_SizeT(const char *, ...); +PyAPI_FUNC(int) _PyArg_VaParse_SizeT(PyObject *, char *, va_list); +PyAPI_FUNC(int) _PyArg_VaParseTupleAndKeywords_SizeT(PyObject *, PyObject *, + const char *, char **, va_list); +#endif + #define FLAG_COMPAT 1 #define FLAG_SIZE_T 2 Modified: python/trunk/Python/modsupport.c ============================================================================== --- python/trunk/Python/modsupport.c (original) +++ python/trunk/Python/modsupport.c Fri Apr 14 11:08:42 2006 @@ -3,8 +3,14 @@ #include "Python.h" +#define FLAG_SIZE_T 1 typedef double va_double; +static PyObject *va_build_value(const char *, va_list, int); +#ifdef HAVE_DECLSPEC_DLL +PyAPI_FUNC(PyObject *) _Py_BuildValue_SizeT(const char *, ...); +#endif + /* Package context -- the full module name for package imports */ char *_Py_PackageContext = NULL; @@ -146,14 +152,14 @@ /* Generic function to create a value -- the inverse of getargs() */ /* After an original idea and first implementation by Steven Miale */ -static PyObject *do_mktuple(const char**, va_list *, int, int); -static PyObject *do_mklist(const char**, va_list *, int, int); -static PyObject *do_mkdict(const char**, va_list *, int, int); -static PyObject *do_mkvalue(const char**, va_list *); +static PyObject *do_mktuple(const char**, va_list *, int, int, int); +static PyObject *do_mklist(const char**, va_list *, int, int, int); +static PyObject *do_mkdict(const char**, va_list *, int, int, int); +static PyObject *do_mkvalue(const char**, va_list *, int); static PyObject * -do_mkdict(const char **p_format, va_list *p_va, int endchar, int n) +do_mkdict(const char **p_format, va_list *p_va, int endchar, int n, int flags) { PyObject *d; int i; @@ -167,13 +173,13 @@ for (i = 0; i < n; i+= 2) { PyObject *k, *v; int err; - k = do_mkvalue(p_format, p_va); + k = do_mkvalue(p_format, p_va, flags); if (k == NULL) { itemfailed = 1; Py_INCREF(Py_None); k = Py_None; } - v = do_mkvalue(p_format, p_va); + v = do_mkvalue(p_format, p_va, flags); if (v == NULL) { itemfailed = 1; Py_INCREF(Py_None); @@ -199,7 +205,7 @@ } static PyObject * -do_mklist(const char **p_format, va_list *p_va, int endchar, int n) +do_mklist(const char **p_format, va_list *p_va, int endchar, int n, int flags) { PyObject *v; int i; @@ -212,7 +218,7 @@ /* Note that we can't bail immediately on error as this will leak refcounts on any 'N' arguments. */ for (i = 0; i < n; i++) { - PyObject *w = do_mkvalue(p_format, p_va); + PyObject *w = do_mkvalue(p_format, p_va, flags); if (w == NULL) { itemfailed = 1; Py_INCREF(Py_None); @@ -249,7 +255,7 @@ #endif static PyObject * -do_mktuple(const char **p_format, va_list *p_va, int endchar, int n) +do_mktuple(const char **p_format, va_list *p_va, int endchar, int n, int flags) { PyObject *v; int i; @@ -261,7 +267,7 @@ /* Note that we can't bail immediately on error as this will leak refcounts on any 'N' arguments. */ for (i = 0; i < n; i++) { - PyObject *w = do_mkvalue(p_format, p_va); + PyObject *w = do_mkvalue(p_format, p_va, flags); if (w == NULL) { itemfailed = 1; Py_INCREF(Py_None); @@ -286,21 +292,21 @@ } static PyObject * -do_mkvalue(const char **p_format, va_list *p_va) +do_mkvalue(const char **p_format, va_list *p_va, int flags) { for (;;) { switch (*(*p_format)++) { case '(': return do_mktuple(p_format, p_va, ')', - countformat(*p_format, ')')); + countformat(*p_format, ')'), flags); case '[': return do_mklist(p_format, p_va, ']', - countformat(*p_format, ']')); + countformat(*p_format, ']'), flags); case '{': return do_mkdict(p_format, p_va, '}', - countformat(*p_format, '}')); + countformat(*p_format, '}'), flags); case 'b': case 'B': @@ -351,10 +357,13 @@ { PyObject *v; Py_UNICODE *u = va_arg(*p_va, Py_UNICODE *); - int n; + Py_ssize_t n; if (**p_format == '#') { ++*p_format; - n = va_arg(*p_va, int); + if (flags & FLAG_SIZE_T) + n = va_arg(*p_va, Py_ssize_t); + else + n = va_arg(*p_va, int); } else n = -1; @@ -393,10 +402,13 @@ { PyObject *v; char *str = va_arg(*p_va, char *); - int n; + Py_ssize_t n; if (**p_format == '#') { ++*p_format; - n = va_arg(*p_va, int); + if (flags & FLAG_SIZE_T) + n = va_arg(*p_va, Py_ssize_t); + else + n = va_arg(*p_va, int); } else n = -1; @@ -472,7 +484,18 @@ va_list va; PyObject* retval; va_start(va, format); - retval = Py_VaBuildValue(format, va); + retval = va_build_value(format, va, 0); + va_end(va); + return retval; +} + +PyObject * +_Py_BuildValue_SizeT(const char *format, ...) +{ + va_list va; + PyObject* retval; + va_start(va, format); + retval = va_build_value(format, va, FLAG_SIZE_T); va_end(va); return retval; } @@ -480,6 +503,18 @@ PyObject * Py_VaBuildValue(const char *format, va_list va) { + return va_build_value(format, va, 0); +} + +PyObject * +_Py_VaBuildValue_SizeT(const char *format, va_list va) +{ + return va_build_value(format, va, FLAG_SIZE_T); +} + +static PyObject * +va_build_value(const char *format, va_list va, int flags) +{ const char *f = format; int n = countformat(f, '\0'); va_list lva; @@ -501,8 +536,8 @@ return Py_None; } if (n == 1) - return do_mkvalue(&f, &lva); - return do_mktuple(&f, &lva, '\0', n); + return do_mkvalue(&f, &lva, flags); + return do_mktuple(&f, &lva, '\0', n, flags); } From buildbot at python.org Fri Apr 14 11:16:06 2006 From: buildbot at python.org (buildbot at python.org) Date: Fri, 14 Apr 2006 09:16:06 +0000 Subject: [Python-checkins] buildbot failure in x86 XP-2 trunk Message-ID: <20060414091606.6662A1E4037@bag.python.org> The Buildbot has detected a new failure of x86 XP-2 trunk. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520XP-2%2520trunk/builds/265 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: neal.norwitz BUILD FAILED: failed failed slave lost sincerely, -The Buildbot From python-checkins at python.org Fri Apr 14 12:29:56 2006 From: python-checkins at python.org (andrew.kuchling) Date: Fri, 14 Apr 2006 12:29:56 +0200 (CEST) Subject: [Python-checkins] r45380 - python/trunk/Doc/whatsnew/whatsnew25.tex Message-ID: <20060414102956.EC6981E400B@bag.python.org> Author: andrew.kuchling Date: Fri Apr 14 12:29:55 2006 New Revision: 45380 Modified: python/trunk/Doc/whatsnew/whatsnew25.tex Log: Add an item and a reminder Modified: python/trunk/Doc/whatsnew/whatsnew25.tex ============================================================================== --- python/trunk/Doc/whatsnew/whatsnew25.tex (original) +++ python/trunk/Doc/whatsnew/whatsnew25.tex Fri Apr 14 12:29:55 2006 @@ -5,6 +5,7 @@ % Fix XXX comments % The easy_install stuff % Stateful codec changes +% cProfile % Count up the patches and bugs \title{What's New in Python 2.5} @@ -1021,6 +1022,11 @@ (Contributed by Raymond Hettinger.) +\item The \module{nis} module now supports accessing domains other +than the system default domain by supplying a \var{domain} argument to +the \function{nis.match()} and \function{nis.maps()} functions. +(Contributed by Ben Bell.) + \item The \module{operator} module's \function{itemgetter()} and \function{attrgetter()} functions now support multiple fields. A call such as \code{operator.attrgetter('a', 'b')} @@ -1136,6 +1142,9 @@ % whole new modules get described in subsections here %====================================================================== +%\subsection{The cProfile module} + +%====================================================================== \subsection{The ctypes package} The \module{ctypes} package, written by Thomas Heller, has been added From python-checkins at python.org Fri Apr 14 14:07:42 2006 From: python-checkins at python.org (andrew.kuchling) Date: Fri, 14 Apr 2006 14:07:42 +0200 (CEST) Subject: [Python-checkins] r45382 - python/trunk/Doc/lib/libprofile.tex Message-ID: <20060414120742.3CC8A1E4038@bag.python.org> Author: andrew.kuchling Date: Fri Apr 14 14:07:41 2006 New Revision: 45382 Modified: python/trunk/Doc/lib/libprofile.tex Log: Typo fix Modified: python/trunk/Doc/lib/libprofile.tex ============================================================================== --- python/trunk/Doc/lib/libprofile.tex (original) +++ python/trunk/Doc/lib/libprofile.tex Fri Apr 14 14:07:41 2006 @@ -124,7 +124,7 @@ %\end{description} -\section{Instant Users Manual \label{profile-instant}} +\section{Instant User's Manual \label{profile-instant}} This section is provided for users that ``don't want to read the manual.'' It provides a very brief overview, and allows a user to From buildbot at python.org Fri Apr 14 14:18:47 2006 From: buildbot at python.org (buildbot at python.org) Date: Fri, 14 Apr 2006 12:18:47 +0000 Subject: [Python-checkins] buildbot failure in x86 cygwin trunk Message-ID: <20060414121847.AEBC21E400B@bag.python.org> The Buildbot has detected a new failure of x86 cygwin trunk. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520cygwin%2520trunk/builds/141 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: andrew.kuchling BUILD FAILED: failed compile sincerely, -The Buildbot From python-checkins at python.org Fri Apr 14 14:35:55 2006 From: python-checkins at python.org (martin.v.loewis) Date: Fri, 14 Apr 2006 14:35:55 +0200 (CEST) Subject: [Python-checkins] r45383 - peps/trunk/pep-0353.txt Message-ID: <20060414123555.327501E400B@bag.python.org> Author: martin.v.loewis Date: Fri Apr 14 14:35:54 2006 New Revision: 45383 Modified: peps/trunk/pep-0353.txt Log: Add PyObject_CallFunction and PyObject_CallMethod to those with new conversion semantics. Modified: peps/trunk/pep-0353.txt ============================================================================== --- peps/trunk/pep-0353.txt (original) +++ peps/trunk/pep-0353.txt Fri Apr 14 14:35:54 2006 @@ -72,7 +72,8 @@ writebufferproc, segcountproc, and charbufferproc. A new conversion code 'n' is introduced for PyArg_ParseTuple -and Py_BuildValue, which operates on Py_ssize_t. +Py_BuildValue, PyObject_CallFunction and PyObject_CallMethod. +This code operates on Py_ssize_t. The conversion codes 's#' and 't#' will output Py_ssize_t if the macro PY_SSIZE_T_CLEAN is defined before Python.h From python-checkins at python.org Fri Apr 14 14:41:20 2006 From: python-checkins at python.org (andrew.kuchling) Date: Fri, 14 Apr 2006 14:41:20 +0200 (CEST) Subject: [Python-checkins] r45384 - python/trunk/Doc/whatsnew/whatsnew25.tex Message-ID: <20060414124120.640BB1E400B@bag.python.org> Author: andrew.kuchling Date: Fri Apr 14 14:41:19 2006 New Revision: 45384 Modified: python/trunk/Doc/whatsnew/whatsnew25.tex Log: Add more items Modified: python/trunk/Doc/whatsnew/whatsnew25.tex ============================================================================== --- python/trunk/Doc/whatsnew/whatsnew25.tex (original) +++ python/trunk/Doc/whatsnew/whatsnew25.tex Fri Apr 14 14:41:19 2006 @@ -857,10 +857,29 @@ \begin{itemize} +\item The \class{dict} type has a new hook for letting subclasses +provide a default value when a key isn't contained in the dictionary. +When a key isn't found, the dictionary's +\method{__missing__(\var{key})} +method will be called. This hook is used to implement +the new \class{defaultdict} class in the \module{collections} +module. The following example defines a dictionary +that returns zero for any missing key: + +\begin{verbatim} +class zerodict (dict): + def __missing__ (self, key): + return 0 + +d = zerodict({1:1, 2:2}) +print d[1], d[2] # Prints 1, 2 +print d[3], d[4] # Prints 0, 0 +\end{verbatim} + \item The \function{min()} and \function{max()} built-in functions gained a \code{key} keyword argument analogous to the \code{key} -argument for \method{sort()}. This argument supplies a function -that takes a single argument and is called for every value in the list; +argument for \method{sort()}. This argument supplies a function that +takes a single argument and is called for every value in the list; \function{min()}/\function{max()} will return the element with the smallest/largest return value from this function. For example, to find the longest string in a list, you can do: @@ -903,8 +922,6 @@ \end{verbatim} (Implemented by Brett Cannon.) -% XXX __missing__ hook in dictionaries - \end{itemize} @@ -964,9 +981,6 @@ \begin{itemize} -% XXX collections.deque now has .remove() -% collections.defaultdict - % the cPickle module no longer accepts the deprecated None option in the % args tuple returned by __reduce__(). @@ -984,6 +998,55 @@ and the code for u-LAW encoding has been improved. (Contributed by Lars Immisch.) +\item The \module{collections} module gained a new type, +\class{defaultdict}, that subclasses the standard \class{dict} +type. The new type mostly behaves like a dictionary but constructs a +default value when a key isn't present, automatically adding it to the +dictionary for the requested key value. + +The first argument to \class{defaultdict}'s constructor is a factory +function that gets called whenever a key is requested but not found. +This factory function receives no arguments, so you can use built-in +type constructors such as \function{list()} or \function{int()}. For +example, +you can make an index of words based on their initial letter like this: + +\begin{verbatim} +words = """Nel mezzo del cammin di nostra vita +mi ritrovai per una selva oscura +che la diritta via era smarrita""".lower().split() + +index = defaultdict(list) + +for w in words: + init_letter = w[0] + index[init_letter].append(w) +\end{verbatim} + +Printing \code{index} results in the following output: + +\begin{verbatim} +defaultdict(, {'c': ['cammin', 'che'], 'e': ['era'], + 'd': ['del', 'di', 'diritta'], 'm': ['mezzo', 'mi'], + 'l': ['la'], 'o': ['oscura'], 'n': ['nel', 'nostra'], + 'p': ['per'], 's': ['selva', 'smarrita'], + 'r': ['ritrovai'], 'u': ['una'], 'v': ['vita', 'via']} +\end{verbatim} + +The \class{deque} double-ended queue type supplied by the +\module{collections} module now has a \method{remove(\var{value})} +method that removes the first occurrence of \var{value} in the queue, +raising \exception{ValueError} if the value isn't found. + +\item The \module{cProfile} module is a C implementation of +the existing \module{profile} module that has much lower overhead. +The module's interface is the same as \module{profile}: you run +\code{cProfile.run('main()')} to profile a function, can save profile +data to a file, etc. It's not yet known if the Hotshot profiler, +which is also written in C but doesn't match the \module{profile} +module's interface, will continue to be maintained in future versions +of Python. (Contributed by Armin Rigo.) + \item In the \module{gc} module, the new \function{get_count()} function returns a 3-tuple containing the current collection counts for the three GC generations. This is accounting information for the garbage @@ -1142,9 +1205,6 @@ % whole new modules get described in subsections here %====================================================================== -%\subsection{The cProfile module} - -%====================================================================== \subsection{The ctypes package} The \module{ctypes} package, written by Thomas Heller, has been added From python-checkins at python.org Fri Apr 14 14:42:09 2006 From: python-checkins at python.org (andrew.kuchling) Date: Fri, 14 Apr 2006 14:42:09 +0200 (CEST) Subject: [Python-checkins] r45385 - python/trunk/Doc/lib/libcollections.tex Message-ID: <20060414124209.C0D201E400B@bag.python.org> Author: andrew.kuchling Date: Fri Apr 14 14:42:09 2006 New Revision: 45385 Modified: python/trunk/Doc/lib/libcollections.tex Log: Typo fixes Modified: python/trunk/Doc/lib/libcollections.tex ============================================================================== --- python/trunk/Doc/lib/libcollections.tex (original) +++ python/trunk/Doc/lib/libcollections.tex Fri Apr 14 14:42:09 2006 @@ -279,7 +279,7 @@ When each key is encountered for the first time, it is not already in the mapping; so an entry is automatically created using the \member{default_factory} function which returns an empty \class{list}. The -\method{list.append()} operation then attaches the value the new list. When +\method{list.append()} operation then attaches the value to the new list. When keys are encountered again, the look-up proceeds normally (returning the list for that key) and the \method{list.append()} operation adds another value to the list. This technique is simpler and faster than an equivalent technique @@ -308,7 +308,7 @@ [('i', 4), ('p', 2), ('s', 4), ('m', 1)] \end{verbatim} -When a letter in first encountered, it is missing from the mapping, so the +When a letter is first encountered, it is missing from the mapping, so the \member{default_factory} function calls \function{int()} to supply a default count of zero. The increment operation then builds of the count for each letter. This technique makes counting simpler and faster than an equivalent From buildbot at python.org Fri Apr 14 14:46:13 2006 From: buildbot at python.org (buildbot at python.org) Date: Fri, 14 Apr 2006 12:46:13 +0000 Subject: [Python-checkins] buildbot warnings in g4 osx.4 trunk Message-ID: <20060414124614.0345A1E400B@bag.python.org> The Buildbot has detected a new failure of g4 osx.4 trunk. Full details are available at: http://www.python.org/dev/buildbot/all/g4%2520osx.4%2520trunk/builds/402 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: andrew.kuchling Build Had Warnings: warnings test sincerely, -The Buildbot From buildbot at python.org Fri Apr 14 15:03:19 2006 From: buildbot at python.org (buildbot at python.org) Date: Fri, 14 Apr 2006 13:03:19 +0000 Subject: [Python-checkins] buildbot warnings in amd64 gentoo trunk Message-ID: <20060414130319.1E6DA1E400B@bag.python.org> The Buildbot has detected a new failure of amd64 gentoo trunk. Full details are available at: http://www.python.org/dev/buildbot/all/amd64%2520gentoo%2520trunk/builds/479 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: andrew.kuchling Build Had Warnings: warnings test sincerely, -The Buildbot From python-checkins at python.org Fri Apr 14 16:03:55 2006 From: python-checkins at python.org (walter.doerwald) Date: Fri, 14 Apr 2006 16:03:55 +0200 (CEST) Subject: [Python-checkins] r45386 - python/trunk/Lib/test/test_codecs.py Message-ID: <20060414140355.C7D0E1E400B@bag.python.org> Author: walter.doerwald Date: Fri Apr 14 16:03:55 2006 New Revision: 45386 Modified: python/trunk/Lib/test/test_codecs.py Log: Call encode()/decode() with final==True as the last call in the incremental codec tests. Modified: python/trunk/Lib/test/test_codecs.py ============================================================================== --- python/trunk/Lib/test/test_codecs.py (original) +++ python/trunk/Lib/test/test_codecs.py Fri Apr 14 16:03:55 2006 @@ -1044,20 +1044,24 @@ encodedresult = "" for c in s: encodedresult += encoder.encode(c) + encodedresult += encoder.encode(u"", True) decoder = codecs.getincrementaldecoder(encoding)() decodedresult = u"" for c in encodedresult: decodedresult += decoder.decode(c) + decodedresult += decoder.decode("", True) self.assertEqual(decodedresult, s, "%r != %r (encoding=%r)" % (decodedresult, s, encoding)) # check C API encodedresult = "" for c in s: encodedresult += cencoder.encode(c) + encodedresult += cencoder.encode(u"", True) cdecoder = _testcapi.codec_incrementaldecoder(encoding) decodedresult = u"" for c in encodedresult: decodedresult += cdecoder.decode(c) + decodedresult += cdecoder.decode("", True) self.assertEqual(decodedresult, s, "%r != %r (encoding=%r)" % (decodedresult, s, encoding)) # check iterencode()/iterdecode() From buildbot at python.org Fri Apr 14 16:29:40 2006 From: buildbot at python.org (buildbot at python.org) Date: Fri, 14 Apr 2006 14:29:40 +0000 Subject: [Python-checkins] buildbot warnings in x86 XP trunk Message-ID: <20060414142940.94E101E4010@bag.python.org> The Buildbot has detected a new failure of x86 XP trunk. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520XP%2520trunk/builds/422 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: walter.doerwald Build Had Warnings: warnings failed slave lost sincerely, -The Buildbot From python-checkins at python.org Fri Apr 14 16:34:28 2006 From: python-checkins at python.org (martin.v.loewis) Date: Fri, 14 Apr 2006 16:34:28 +0200 (CEST) Subject: [Python-checkins] r45387 - in python/trunk: Makefile.pre.in Misc/NEWS Modules/ccpython.cc README configure configure.in Message-ID: <20060414143428.07EEC1E400B@bag.python.org> Author: martin.v.loewis Date: Fri Apr 14 16:34:26 2006 New Revision: 45387 Removed: python/trunk/Modules/ccpython.cc Modified: python/trunk/Makefile.pre.in python/trunk/Misc/NEWS python/trunk/README python/trunk/configure python/trunk/configure.in Log: Patch #1324762: Change --with-cxx to --with-cxx-main. Modified: python/trunk/Makefile.pre.in ============================================================================== --- python/trunk/Makefile.pre.in (original) +++ python/trunk/Makefile.pre.in Fri Apr 14 16:34:26 2006 @@ -30,6 +30,7 @@ CC= @CC@ CXX= @CXX@ +MAINCC= @MAINCC@ LINKCC= @LINKCC@ AR= @AR@ RANLIB= @RANLIB@ @@ -157,7 +158,6 @@ SYSLIBS= $(LIBM) $(LIBC) SHLIBS= @SHLIBS@ -MAINOBJ= @MAINOBJ@ THREADOBJ= @THREADOBJ@ DLINCLDIR= @DLINCLDIR@ DYNLOADFILE= @DYNLOADFILE@ @@ -326,9 +326,9 @@ all: $(BUILDPYTHON) oldsharedmods sharedmods # Build the interpreter -$(BUILDPYTHON): Modules/$(MAINOBJ) $(LIBRARY) $(LDLIBRARY) +$(BUILDPYTHON): Modules/python.o $(LIBRARY) $(LDLIBRARY) $(LINKCC) $(LDFLAGS) $(LINKFORSHARED) -o $@ \ - Modules/$(MAINOBJ) \ + Modules/python.o \ $(BLDLIBRARY) $(LIBS) $(MODLIBS) $(SYSLIBS) $(LDLAST) platform: $(BUILDPYTHON) @@ -448,8 +448,8 @@ -DVPATH='"$(VPATH)"' \ -o $@ $(srcdir)/Modules/getpath.c -Modules/ccpython.o: $(srcdir)/Modules/ccpython.cc - $(CXX) -c $(PY_CFLAGS) -o $@ $(srcdir)/Modules/ccpython.cc +Modules/python.o: $(srcdir)/Modules/python.c + $(MAINCC) -c $(PY_CFLAGS) -o $@ $(srcdir)/Modules/python.c $(GRAMMAR_H) $(GRAMMAR_C): $(PGEN) $(GRAMMAR_INPUT) @@ -537,7 +537,7 @@ Include/weakrefobject.h \ pyconfig.h -$(LIBRARY_OBJS) $(MODOBJS) Modules/$(MAINOBJ): $(PYTHON_HEADERS) +$(LIBRARY_OBJS) $(MODOBJS) Modules/python.o: $(PYTHON_HEADERS) ###################################################################### @@ -813,7 +813,7 @@ fi; \ fi $(INSTALL_DATA) Modules/config.c $(DESTDIR)$(LIBPL)/config.c - $(INSTALL_DATA) Modules/$(MAINOBJ) $(DESTDIR)$(LIBPL)/$(MAINOBJ) + $(INSTALL_DATA) Modules/python.o $(DESTDIR)$(LIBPL)/python.o $(INSTALL_DATA) $(srcdir)/Modules/config.c.in $(DESTDIR)$(LIBPL)/config.c.in $(INSTALL_DATA) Makefile $(DESTDIR)$(LIBPL)/Makefile $(INSTALL_DATA) Modules/Setup $(DESTDIR)$(LIBPL)/Setup Modified: python/trunk/Misc/NEWS ============================================================================== --- python/trunk/Misc/NEWS (original) +++ python/trunk/Misc/NEWS Fri Apr 14 16:34:26 2006 @@ -82,6 +82,11 @@ Build ----- +- Patch #1324762:Remove ccpython.cc; replace --with-cxx with + --with-cxx-main. Link with C++ compiler only if --with-cxx-main was + specified. (Can be overriden by explicitly setting LINKCC.) Decouple + CXX from --with-cxx-main, see description in README. + - Patch #1429775: Link extension modules with the shared libpython. C API Deleted: /python/trunk/Modules/ccpython.cc ============================================================================== --- /python/trunk/Modules/ccpython.cc Fri Apr 14 16:34:26 2006 +++ (empty file) @@ -1,11 +0,0 @@ -/* Minimal main program -- everything is loaded from the library */ - -#include "Python.h" - -extern "C" -DL_EXPORT(int) Py_Main( int argc, char *argv[] ); - -int main( int argc, char *argv[] ) -{ - return Py_Main(argc, argv); -} Modified: python/trunk/README ============================================================================== --- python/trunk/README (original) +++ python/trunk/README Fri Apr 14 16:34:26 2006 @@ -1045,13 +1045,35 @@ --with-libs='libs': Add 'libs' to the LIBS that the python interpreter is linked against. ---with-cxx=: Some C++ compilers require that main() is - compiled with the C++ if there is any C++ code in the application. - Specifically, g++ on a.out systems may require that to support - construction of global objects. With this option, the main() function - of Python will be compiled with ; use that only if you - plan to use C++ extension modules, and if your compiler requires - compilation of main() as a C++ program. +--with-cxx-main=: If you plan to use C++ extension modules, + then -- on some platforms -- you need to compile python's main() + function with the C++ compiler. With this option, make will use + to compile main() *and* to link the python executable. + It is likely that the resulting executable depends on the C++ + runtime library of . (The default is --without-cxx-main.) + + There are platforms that do not require you to build Python + with a C++ compiler in order to use C++ extension modules. + E.g., x86 Linux with ELF shared binaries and GCC 3.x, 4.x is such + a platform. We recommend that you configure Python + --without-cxx-main on those platforms because a mismatch + between the C++ compiler version used to build Python and to + build a C++ extension module is likely to cause a crash at + runtime. + + The Python installation also stores the variable CXX that + determines, e.g., the C++ compiler distutils calls by default + to build C++ extensions. If you set CXX on the configure command + line to any string of non-zero length, then configure won't + change CXX. If you do not preset CXX but pass + --with-cxx-main=, then configure sets CXX=. + In all other cases, configure looks for a C++ compiler by + some common names (c++, g++, gcc, CC, cxx, cc++, cl) and sets + CXX to the first compiler it finds. If it does not find any + C++ compiler, then it sets CXX="". + + Similarly, if you want to change the command used to link the + python executable, then set LINKCC on the configure command line. --with-pydebug: Enable additional debugging code to help track down Modified: python/trunk/configure ============================================================================== --- python/trunk/configure (original) +++ python/trunk/configure Fri Apr 14 16:34:26 2006 @@ -1,5 +1,5 @@ #! /bin/sh -# From configure.in Revision: 45278 . +# From configure.in Revision: 45328 . # Guess values for system-dependent variables and create Makefiles. # Generated by GNU Autoconf 2.59 for python 2.5. # @@ -312,7 +312,7 @@ # include #endif" -ac_subst_vars='SHELL PATH_SEPARATOR PACKAGE_NAME PACKAGE_TARNAME PACKAGE_VERSION PACKAGE_STRING PACKAGE_BUGREPORT exec_prefix prefix program_transform_name bindir sbindir libexecdir datadir sysconfdir sharedstatedir localstatedir libdir includedir oldincludedir infodir mandir build_alias host_alias target_alias DEFS ECHO_C ECHO_N ECHO_T LIBS VERSION SOVERSION CONFIG_ARGS PYTHONFRAMEWORK PYTHONFRAMEWORKDIR PYTHONFRAMEWORKPREFIX PYTHONFRAMEWORKINSTALLDIR MACHDEP SGI_ABI EXTRAPLATDIR EXTRAMACHDEPPATH CONFIGURE_MACOSX_DEPLOYMENT_TARGET CXX MAINOBJ EXEEXT CC CFLAGS LDFLAGS CPPFLAGS ac_ct_CC OBJEXT CPP EGREP BUILDEXEEXT LIBRARY LDLIBRARY DLLLIBRARY BLDLIBRARY LDLIBRARYDIR INSTSONAME RUNSHARED LINKCC RANLIB ac_ct_RANLIB AR SVNVERSION INSTALL_PROGRAM INSTALL_SCRIPT INSTALL_DATA LN OPT BASECFLAGS OTHER_LIBTOOL_OPT LIBTOOL_CRUFT SO LDSHARED BLDSHARED CCSHARED LINKFORSHARED CFLAGSFORSHARED SHLIBS USE_SIGNAL_MODULE SIGNAL_OBJS USE_THREAD_MODULE LDLAST THREADOBJ DLINCLDIR DYNLOADFILE MACHDEP_OBJS TRUE LIBOBJS HAVE_GETHOSTBYNAME_R_6_ARG HAVE_GETHOSTBYNAME_R_5_ARG HAVE_GETHOSTBYNAME_R_3_ARG HAVE_GETHOSTBYNAME_R HAVE_GETHOSTBYNAME LIBM LIBC UNICODE_OBJS THREADHEADERS SRCDIRS LTLIBOBJS' +ac_subst_vars='SHELL PATH_SEPARATOR PACKAGE_NAME PACKAGE_TARNAME PACKAGE_VERSION PACKAGE_STRING PACKAGE_BUGREPORT exec_prefix prefix program_transform_name bindir sbindir libexecdir datadir sysconfdir sharedstatedir localstatedir libdir includedir oldincludedir infodir mandir build_alias host_alias target_alias DEFS ECHO_C ECHO_N ECHO_T LIBS VERSION SOVERSION CONFIG_ARGS PYTHONFRAMEWORK PYTHONFRAMEWORKDIR PYTHONFRAMEWORKPREFIX PYTHONFRAMEWORKINSTALLDIR MACHDEP SGI_ABI EXTRAPLATDIR EXTRAMACHDEPPATH CONFIGURE_MACOSX_DEPLOYMENT_TARGET CC CFLAGS LDFLAGS CPPFLAGS ac_ct_CC EXEEXT OBJEXT CXX MAINCC CPP EGREP BUILDEXEEXT LIBRARY LDLIBRARY DLLLIBRARY BLDLIBRARY LDLIBRARYDIR INSTSONAME RUNSHARED LINKCC RANLIB ac_ct_RANLIB AR SVNVERSION INSTALL_PROGRAM INSTALL_SCRIPT INSTALL_DATA LN OPT BASECFLAGS OTHER_LIBTOOL_OPT LIBTOOL_CRUFT SO LDSHARED BLDSHARED CCSHARED LINKFORSHARED CFLAGSFORSHARED SHLIBS USE_SIGNAL_MODULE SIGNAL_OBJS USE_THREAD_MODULE LDLAST THREADOBJ DLINCLDIR DYNLOADFILE MACHDEP_OBJS TRUE LIBOBJS HAVE_GETHOSTBYNAME_R_6_ARG HAVE_GETHOSTBYNAME_R_5_ARG HAVE_GETHOSTBYNAME_R_3_ARG HAVE_GETHOSTBYNAME_R HAVE_GETHOSTBYNAME LIBM LIBC UNICODE_OBJS THREADHEADERS SRCDIRS LTLIBOBJS' ac_subst_files='' # Initialize some variables set by options. @@ -859,7 +859,9 @@ --with-PACKAGE[=ARG] use PACKAGE [ARG=yes] --without-PACKAGE do not use PACKAGE (same as --with-PACKAGE=no) --without-gcc never use gcc - --with-cxx= enable C++ support + --with-cxx-main= + compile main() and link python executable with C++ + compiler --with-suffix=.exe set executable suffix --with-pydebug build with Py_DEBUG defined --with-libs='lib1 ...' link against additional libs @@ -1679,258 +1681,6 @@ echo "$as_me:$LINENO: result: $without_gcc" >&5 echo "${ECHO_T}$without_gcc" >&6 - - -MAINOBJ=python.o -echo "$as_me:$LINENO: checking for --with-cxx=" >&5 -echo $ECHO_N "checking for --with-cxx=... $ECHO_C" >&6 - -# Check whether --with-cxx or --without-cxx was given. -if test "${with_cxx+set}" = set; then - withval="$with_cxx" - - check_cxx=no - case $withval in - no) CXX= - with_cxx=no;; - *) CXX=$withval - MAINOBJ=ccpython.o - with_cxx=$withval;; - esac -else - - with_cxx=no - check_cxx=yes - -fi; -echo "$as_me:$LINENO: result: $with_cxx" >&5 -echo "${ECHO_T}$with_cxx" >&6 - -if test "$with_cxx" = "yes" -then - { { echo "$as_me:$LINENO: error: must supply a compiler when using --with-cxx" >&5 -echo "$as_me: error: must supply a compiler when using --with-cxx" >&2;} - { (exit 1); exit 1; }; } -fi - - - - -if test "$check_cxx" = "yes" -then - for ac_prog in $CCC c++ g++ gcc CC cxx cc++ cl -do - # Extract the first word of "$ac_prog", so it can be a program name with args. -set dummy $ac_prog; ac_word=$2 -echo "$as_me:$LINENO: checking for $ac_word" >&5 -echo $ECHO_N "checking for $ac_word... $ECHO_C" >&6 -if test "${ac_cv_prog_CXX+set}" = set; then - echo $ECHO_N "(cached) $ECHO_C" >&6 -else - if test -n "$CXX"; then - ac_cv_prog_CXX="$CXX" # Let the user override the test. -else -as_save_IFS=$IFS; IFS=$PATH_SEPARATOR -for as_dir in $PATH -do - IFS=$as_save_IFS - test -z "$as_dir" && as_dir=. - for ac_exec_ext in '' $ac_executable_extensions; do - if $as_executable_p "$as_dir/$ac_word$ac_exec_ext"; then - ac_cv_prog_CXX="$ac_prog" - echo "$as_me:$LINENO: found $as_dir/$ac_word$ac_exec_ext" >&5 - break 2 - fi -done -done - -fi -fi -CXX=$ac_cv_prog_CXX -if test -n "$CXX"; then - echo "$as_me:$LINENO: result: $CXX" >&5 -echo "${ECHO_T}$CXX" >&6 -else - echo "$as_me:$LINENO: result: no" >&5 -echo "${ECHO_T}no" >&6 -fi - - test -n "$CXX" && break -done -test -n "$CXX" || CXX="notfound" - - if test "$CXX" = "notfound" - then - CXX= - else - ac_ext=cc -ac_cpp='$CXXCPP $CPPFLAGS' -ac_compile='$CXX -c $CXXFLAGS $CPPFLAGS conftest.$ac_ext >&5' -ac_link='$CXX -o conftest$ac_exeext $CXXFLAGS $CPPFLAGS $LDFLAGS conftest.$ac_ext $LIBS >&5' -ac_compiler_gnu=$ac_cv_cxx_compiler_gnu - cat >conftest.$ac_ext <<_ACEOF -/* confdefs.h. */ -_ACEOF -cat confdefs.h >>conftest.$ac_ext -cat >>conftest.$ac_ext <<_ACEOF -/* end confdefs.h. */ - -int -main () -{ - - ; - return 0; -} -_ACEOF -ac_clean_files_save=$ac_clean_files -ac_clean_files="$ac_clean_files a.out a.exe b.out" -# Try to create an executable without -o first, disregard a.out. -# It will help us diagnose broken compilers, and finding out an intuition -# of exeext. -echo "$as_me:$LINENO: checking for C++ compiler default output file name" >&5 -echo $ECHO_N "checking for C++ compiler default output file name... $ECHO_C" >&6 -ac_link_default=`echo "$ac_link" | sed 's/ -o *conftest[^ ]*//'` -if { (eval echo "$as_me:$LINENO: \"$ac_link_default\"") >&5 - (eval $ac_link_default) 2>&5 - ac_status=$? - echo "$as_me:$LINENO: \$? = $ac_status" >&5 - (exit $ac_status); }; then - # Find the output, starting from the most likely. This scheme is -# not robust to junk in `.', hence go to wildcards (a.*) only as a last -# resort. - -# Be careful to initialize this variable, since it used to be cached. -# Otherwise an old cache value of `no' led to `EXEEXT = no' in a Makefile. -ac_cv_exeext= -# b.out is created by i960 compilers. -for ac_file in a_out.exe a.exe conftest.exe a.out conftest a.* conftest.* b.out -do - test -f "$ac_file" || continue - case $ac_file in - *.$ac_ext | *.xcoff | *.tds | *.d | *.pdb | *.xSYM | *.bb | *.bbg | *.o | *.obj ) - ;; - conftest.$ac_ext ) - # This is the source file. - ;; - [ab].out ) - # We found the default executable, but exeext='' is most - # certainly right. - break;; - *.* ) - ac_cv_exeext=`expr "$ac_file" : '[^.]*\(\..*\)'` - # FIXME: I believe we export ac_cv_exeext for Libtool, - # but it would be cool to find out if it's true. Does anybody - # maintain Libtool? --akim. - export ac_cv_exeext - break;; - * ) - break;; - esac -done -else - echo "$as_me: failed program was:" >&5 -sed 's/^/| /' conftest.$ac_ext >&5 - -{ { echo "$as_me:$LINENO: error: C++ compiler cannot create executables -See \`config.log' for more details." >&5 -echo "$as_me: error: C++ compiler cannot create executables -See \`config.log' for more details." >&2;} - { (exit 77); exit 77; }; } -fi - -ac_exeext=$ac_cv_exeext -echo "$as_me:$LINENO: result: $ac_file" >&5 -echo "${ECHO_T}$ac_file" >&6 - -# Check the compiler produces executables we can run. If not, either -# the compiler is broken, or we cross compile. -echo "$as_me:$LINENO: checking whether the C++ compiler works" >&5 -echo $ECHO_N "checking whether the C++ compiler works... $ECHO_C" >&6 -# FIXME: These cross compiler hacks should be removed for Autoconf 3.0 -# If not cross compiling, check that we can run a simple program. -if test "$cross_compiling" != yes; then - if { ac_try='./$ac_file' - { (eval echo "$as_me:$LINENO: \"$ac_try\"") >&5 - (eval $ac_try) 2>&5 - ac_status=$? - echo "$as_me:$LINENO: \$? = $ac_status" >&5 - (exit $ac_status); }; }; then - cross_compiling=no - else - if test "$cross_compiling" = maybe; then - cross_compiling=yes - else - { { echo "$as_me:$LINENO: error: cannot run C++ compiled programs. -If you meant to cross compile, use \`--host'. -See \`config.log' for more details." >&5 -echo "$as_me: error: cannot run C++ compiled programs. -If you meant to cross compile, use \`--host'. -See \`config.log' for more details." >&2;} - { (exit 1); exit 1; }; } - fi - fi -fi -echo "$as_me:$LINENO: result: yes" >&5 -echo "${ECHO_T}yes" >&6 - -rm -f a.out a.exe conftest$ac_cv_exeext b.out -ac_clean_files=$ac_clean_files_save -# Check the compiler produces executables we can run. If not, either -# the compiler is broken, or we cross compile. -echo "$as_me:$LINENO: checking whether we are cross compiling" >&5 -echo $ECHO_N "checking whether we are cross compiling... $ECHO_C" >&6 -echo "$as_me:$LINENO: result: $cross_compiling" >&5 -echo "${ECHO_T}$cross_compiling" >&6 - -echo "$as_me:$LINENO: checking for suffix of executables" >&5 -echo $ECHO_N "checking for suffix of executables... $ECHO_C" >&6 -if { (eval echo "$as_me:$LINENO: \"$ac_link\"") >&5 - (eval $ac_link) 2>&5 - ac_status=$? - echo "$as_me:$LINENO: \$? = $ac_status" >&5 - (exit $ac_status); }; then - # If both `conftest.exe' and `conftest' are `present' (well, observable) -# catch `conftest.exe'. For instance with Cygwin, `ls conftest' will -# work properly (i.e., refer to `conftest.exe'), while it won't with -# `rm'. -for ac_file in conftest.exe conftest conftest.*; do - test -f "$ac_file" || continue - case $ac_file in - *.$ac_ext | *.xcoff | *.tds | *.d | *.pdb | *.xSYM | *.bb | *.bbg | *.o | *.obj ) ;; - *.* ) ac_cv_exeext=`expr "$ac_file" : '[^.]*\(\..*\)'` - export ac_cv_exeext - break;; - * ) break;; - esac -done -else - { { echo "$as_me:$LINENO: error: cannot compute suffix of executables: cannot compile and link -See \`config.log' for more details." >&5 -echo "$as_me: error: cannot compute suffix of executables: cannot compile and link -See \`config.log' for more details." >&2;} - { (exit 1); exit 1; }; } -fi - -rm -f conftest$ac_cv_exeext -echo "$as_me:$LINENO: result: $ac_cv_exeext" >&5 -echo "${ECHO_T}$ac_cv_exeext" >&6 - -rm -f conftest.$ac_ext -EXEEXT=$ac_cv_exeext -ac_exeext=$EXEEXT - - ac_ext=c -ac_cpp='$CPP $CPPFLAGS' -ac_compile='$CC -c $CFLAGS $CPPFLAGS conftest.$ac_ext >&5' -ac_link='$CC -o conftest$ac_exeext $CFLAGS $CPPFLAGS $LDFLAGS conftest.$ac_ext $LIBS >&5' -ac_compiler_gnu=$ac_cv_c_compiler_gnu - - - - fi -fi - # If the user switches compilers, we can't believe the cache if test ! -z "$ac_cv_prog_CC" -a ! -z "$CC" -a "$CC" != "$ac_cv_prog_CC" then @@ -2872,6 +2622,190 @@ ac_compiler_gnu=$ac_cv_c_compiler_gnu + + +echo "$as_me:$LINENO: checking for --with-cxx-main=" >&5 +echo $ECHO_N "checking for --with-cxx-main=... $ECHO_C" >&6 + +# Check whether --with-cxx_main or --without-cxx_main was given. +if test "${with_cxx_main+set}" = set; then + withval="$with_cxx_main" + + + case $withval in + no) with_cxx_main=no + MAINCC='$(CC)';; + yes) with_cxx_main=yes + MAINCC='$(CXX)';; + *) with_cxx_main=yes + MAINCC=$withval + if test -z "$CXX" + then + CXX=$withval + fi;; + esac +else + + with_cxx_main=no + MAINCC='$(CC)' + +fi; +echo "$as_me:$LINENO: result: $with_cxx_main" >&5 +echo "${ECHO_T}$with_cxx_main" >&6 + +preset_cxx="$CXX" +if test -z "$CXX" +then + case "$CC" in + gcc) # Extract the first word of "g++", so it can be a program name with args. +set dummy g++; ac_word=$2 +echo "$as_me:$LINENO: checking for $ac_word" >&5 +echo $ECHO_N "checking for $ac_word... $ECHO_C" >&6 +if test "${ac_cv_path_CXX+set}" = set; then + echo $ECHO_N "(cached) $ECHO_C" >&6 +else + case $CXX in + [\\/]* | ?:[\\/]*) + ac_cv_path_CXX="$CXX" # Let the user override the test with a path. + ;; + *) + as_save_IFS=$IFS; IFS=$PATH_SEPARATOR +for as_dir in notfound +do + IFS=$as_save_IFS + test -z "$as_dir" && as_dir=. + for ac_exec_ext in '' $ac_executable_extensions; do + if $as_executable_p "$as_dir/$ac_word$ac_exec_ext"; then + ac_cv_path_CXX="$as_dir/$ac_word$ac_exec_ext" + echo "$as_me:$LINENO: found $as_dir/$ac_word$ac_exec_ext" >&5 + break 2 + fi +done +done + + test -z "$ac_cv_path_CXX" && ac_cv_path_CXX="g++" + ;; +esac +fi +CXX=$ac_cv_path_CXX + +if test -n "$CXX"; then + echo "$as_me:$LINENO: result: $CXX" >&5 +echo "${ECHO_T}$CXX" >&6 +else + echo "$as_me:$LINENO: result: no" >&5 +echo "${ECHO_T}no" >&6 +fi + ;; + cc) # Extract the first word of "c++", so it can be a program name with args. +set dummy c++; ac_word=$2 +echo "$as_me:$LINENO: checking for $ac_word" >&5 +echo $ECHO_N "checking for $ac_word... $ECHO_C" >&6 +if test "${ac_cv_path_CXX+set}" = set; then + echo $ECHO_N "(cached) $ECHO_C" >&6 +else + case $CXX in + [\\/]* | ?:[\\/]*) + ac_cv_path_CXX="$CXX" # Let the user override the test with a path. + ;; + *) + as_save_IFS=$IFS; IFS=$PATH_SEPARATOR +for as_dir in notfound +do + IFS=$as_save_IFS + test -z "$as_dir" && as_dir=. + for ac_exec_ext in '' $ac_executable_extensions; do + if $as_executable_p "$as_dir/$ac_word$ac_exec_ext"; then + ac_cv_path_CXX="$as_dir/$ac_word$ac_exec_ext" + echo "$as_me:$LINENO: found $as_dir/$ac_word$ac_exec_ext" >&5 + break 2 + fi +done +done + + test -z "$ac_cv_path_CXX" && ac_cv_path_CXX="c++" + ;; +esac +fi +CXX=$ac_cv_path_CXX + +if test -n "$CXX"; then + echo "$as_me:$LINENO: result: $CXX" >&5 +echo "${ECHO_T}$CXX" >&6 +else + echo "$as_me:$LINENO: result: no" >&5 +echo "${ECHO_T}no" >&6 +fi + ;; + esac + if test "$CXX" = "notfound" + then + CXX="" + fi +fi +if test -z "$CXX" +then + for ac_prog in $CCC c++ g++ gcc CC cxx cc++ cl +do + # Extract the first word of "$ac_prog", so it can be a program name with args. +set dummy $ac_prog; ac_word=$2 +echo "$as_me:$LINENO: checking for $ac_word" >&5 +echo $ECHO_N "checking for $ac_word... $ECHO_C" >&6 +if test "${ac_cv_prog_CXX+set}" = set; then + echo $ECHO_N "(cached) $ECHO_C" >&6 +else + if test -n "$CXX"; then + ac_cv_prog_CXX="$CXX" # Let the user override the test. +else +as_save_IFS=$IFS; IFS=$PATH_SEPARATOR +for as_dir in $PATH +do + IFS=$as_save_IFS + test -z "$as_dir" && as_dir=. + for ac_exec_ext in '' $ac_executable_extensions; do + if $as_executable_p "$as_dir/$ac_word$ac_exec_ext"; then + ac_cv_prog_CXX="$ac_prog" + echo "$as_me:$LINENO: found $as_dir/$ac_word$ac_exec_ext" >&5 + break 2 + fi +done +done + +fi +fi +CXX=$ac_cv_prog_CXX +if test -n "$CXX"; then + echo "$as_me:$LINENO: result: $CXX" >&5 +echo "${ECHO_T}$CXX" >&6 +else + echo "$as_me:$LINENO: result: no" >&5 +echo "${ECHO_T}no" >&6 +fi + + test -n "$CXX" && break +done +test -n "$CXX" || CXX="notfound" + + if test "$CXX" = "notfound" + then + CXX="" + fi +fi +if test "$preset_cxx" != "$CXX" +then + { echo "$as_me:$LINENO: WARNING: + + By default, distutils will build C++ extension modules with \"$CXX\". + If this is not intended, then set CXX on the configure command line. + " >&5 +echo "$as_me: WARNING: + + By default, distutils will build C++ extension modules with \"$CXX\". + If this is not intended, then set CXX on the configure command line. + " >&2;} +fi + + # checks for UNIX variants that set C preprocessor variables ac_ext=c @@ -3274,22 +3208,7 @@ echo $ECHO_N "checking LINKCC... $ECHO_C" >&6 if test -z "$LINKCC" then - if test -z "$CXX"; then - LINKCC="\$(PURIFY) \$(CC)" - else - echo 'extern "C" void foo();int main(){foo();}' > conftest_a.cc - $CXX -c conftest_a.cc # 2>&5 - echo 'void foo(){}' > conftest_b.$ac_ext - $CC -c conftest_b.$ac_ext # 2>&5 - if $CC -o conftest$ac_exeext conftest_a.$ac_objext conftest_b.$ac_objext 2>&5 \ - && test -s conftest$ac_exeext && ./conftest$ac_exeext - then - LINKCC="\$(PURIFY) \$(CC)" - else - LINKCC="\$(PURIFY) \$(CXX)" - fi - rm -fr conftest* - fi + LINKCC='$(PURIFY) $(MAINCC)' case $ac_sys_system in AIX*) exp_extra="\"\"" @@ -22485,15 +22404,15 @@ s, at EXTRAPLATDIR@,$EXTRAPLATDIR,;t t s, at EXTRAMACHDEPPATH@,$EXTRAMACHDEPPATH,;t t s, at CONFIGURE_MACOSX_DEPLOYMENT_TARGET@,$CONFIGURE_MACOSX_DEPLOYMENT_TARGET,;t t -s, at CXX@,$CXX,;t t -s, at MAINOBJ@,$MAINOBJ,;t t -s, at EXEEXT@,$EXEEXT,;t t s, at CC@,$CC,;t t s, at CFLAGS@,$CFLAGS,;t t s, at LDFLAGS@,$LDFLAGS,;t t s, at CPPFLAGS@,$CPPFLAGS,;t t s, at ac_ct_CC@,$ac_ct_CC,;t t +s, at EXEEXT@,$EXEEXT,;t t s, at OBJEXT@,$OBJEXT,;t t +s, at CXX@,$CXX,;t t +s, at MAINCC@,$MAINCC,;t t s, at CPP@,$CPP,;t t s, at EGREP@,$EGREP,;t t s, at BUILDEXEEXT@,$BUILDEXEEXT,;t t Modified: python/trunk/configure.in ============================================================================== --- python/trunk/configure.in (original) +++ python/trunk/configure.in Fri Apr 14 16:34:26 2006 @@ -313,64 +313,69 @@ esac]) AC_MSG_RESULT($without_gcc) +# If the user switches compilers, we can't believe the cache +if test ! -z "$ac_cv_prog_CC" -a ! -z "$CC" -a "$CC" != "$ac_cv_prog_CC" +then + AC_MSG_ERROR([cached CC is different -- throw away $cache_file +(it is also a good idea to do 'make clean' before compiling)]) +fi + +AC_PROG_CC + AC_SUBST(CXX) -AC_SUBST(MAINOBJ) -MAINOBJ=python.o -AC_MSG_CHECKING(for --with-cxx=) -AC_ARG_WITH(cxx, - AC_HELP_STRING(--with-cxx=, enable C++ support), +AC_SUBST(MAINCC) +AC_MSG_CHECKING(for --with-cxx-main=) +AC_ARG_WITH(cxx_main, + AC_HELP_STRING([--with-cxx-main=], + [compile main() and link python executable with C++ compiler]), [ - check_cxx=no + case $withval in - no) CXX= - with_cxx=no;; - *) CXX=$withval - MAINOBJ=ccpython.o - with_cxx=$withval;; + no) with_cxx_main=no + MAINCC='$(CC)';; + yes) with_cxx_main=yes + MAINCC='$(CXX)';; + *) with_cxx_main=yes + MAINCC=$withval + if test -z "$CXX" + then + CXX=$withval + fi;; esac], [ - with_cxx=no - check_cxx=yes + with_cxx_main=no + MAINCC='$(CC)' ]) -AC_MSG_RESULT($with_cxx) +AC_MSG_RESULT($with_cxx_main) -if test "$with_cxx" = "yes" +preset_cxx="$CXX" +if test -z "$CXX" then - AC_MSG_ERROR([must supply a compiler when using --with-cxx]) + case "$CC" in + gcc) AC_PATH_PROG(CXX, [g++], [g++], [notfound]) ;; + cc) AC_PATH_PROG(CXX, [c++], [c++], [notfound]) ;; + esac + if test "$CXX" = "notfound" + then + CXX="" + fi fi - -dnl The following fragment works similar to AC_PROG_CXX. -dnl It does not fail if CXX is not found, and it is not executed if -dnl --without-cxx was given. -dnl Finally, it does not test whether CXX is g++. - -dnl Autoconf 2.5x does not have AC_PROG_CXX_WORKS anymore -ifdef([AC_PROG_CXX_WORKS],[], - [AC_DEFUN([AC_PROG_CXX_WORKS], - [AC_LANG_PUSH(C++)dnl - _AC_COMPILER_EXEEXT - AC_LANG_POP() - ] -)]) - -if test "$check_cxx" = "yes" +if test -z "$CXX" then AC_CHECK_PROGS(CXX, $CCC c++ g++ gcc CC cxx cc++ cl, notfound) if test "$CXX" = "notfound" then - CXX= - else - AC_PROG_CXX_WORKS + CXX="" fi fi - -# If the user switches compilers, we can't believe the cache -if test ! -z "$ac_cv_prog_CC" -a ! -z "$CC" -a "$CC" != "$ac_cv_prog_CC" +if test "$preset_cxx" != "$CXX" then - AC_MSG_ERROR([cached CC is different -- throw away $cache_file -(it is also a good idea to do 'make clean' before compiling)]) + AC_MSG_WARN([ + + By default, distutils will build C++ extension modules with "$CXX". + If this is not intended, then set CXX on the configure command line. + ]) fi -AC_PROG_CC # checks for UNIX variants that set C preprocessor variables AC_AIX @@ -480,22 +485,7 @@ AC_MSG_CHECKING(LINKCC) if test -z "$LINKCC" then - if test -z "$CXX"; then - LINKCC="\$(PURIFY) \$(CC)" - else - echo 'extern "C" void foo();int main(){foo();}' > conftest_a.cc - $CXX -c conftest_a.cc # 2>&5 - echo 'void foo(){}' > conftest_b.$ac_ext - $CC -c conftest_b.$ac_ext # 2>&5 - if $CC -o conftest$ac_exeext conftest_a.$ac_objext conftest_b.$ac_objext 2>&5 \ - && test -s conftest$ac_exeext && ./conftest$ac_exeext - then - LINKCC="\$(PURIFY) \$(CC)" - else - LINKCC="\$(PURIFY) \$(CXX)" - fi - rm -fr conftest* - fi + LINKCC='$(PURIFY) $(MAINCC)' case $ac_sys_system in AIX*) exp_extra="\"\"" From python-checkins at python.org Fri Apr 14 16:54:19 2006 From: python-checkins at python.org (andrew.kuchling) Date: Fri, 14 Apr 2006 16:54:19 +0200 (CEST) Subject: [Python-checkins] r45388 - python/trunk/Misc/NEWS Message-ID: <20060414145419.4E76C1E400B@bag.python.org> Author: andrew.kuchling Date: Fri Apr 14 16:54:18 2006 New Revision: 45388 Modified: python/trunk/Misc/NEWS Log: Typo fix Modified: python/trunk/Misc/NEWS ============================================================================== --- python/trunk/Misc/NEWS (original) +++ python/trunk/Misc/NEWS Fri Apr 14 16:54:18 2006 @@ -84,7 +84,7 @@ - Patch #1324762:Remove ccpython.cc; replace --with-cxx with --with-cxx-main. Link with C++ compiler only if --with-cxx-main was - specified. (Can be overriden by explicitly setting LINKCC.) Decouple + specified. (Can be overridden by explicitly setting LINKCC.) Decouple CXX from --with-cxx-main, see description in README. - Patch #1429775: Link extension modules with the shared libpython. From python-checkins at python.org Fri Apr 14 16:58:30 2006 From: python-checkins at python.org (armin.rigo) Date: Fri, 14 Apr 2006 16:58:30 +0200 (CEST) Subject: [Python-checkins] r45389 - python/trunk/Lib/test/leakers/test_selftype.py Message-ID: <20060414145830.7DD1B1E400B@bag.python.org> Author: armin.rigo Date: Fri Apr 14 16:58:30 2006 New Revision: 45389 Added: python/trunk/Lib/test/leakers/test_selftype.py Log: Show case: reference cycles involving only the ob_type field are rather uncommon but possible. Inspired by SF bug 1469629. Added: python/trunk/Lib/test/leakers/test_selftype.py ============================================================================== --- (empty file) +++ python/trunk/Lib/test/leakers/test_selftype.py Fri Apr 14 16:58:30 2006 @@ -0,0 +1,13 @@ +# Reference cycles involving only the ob_type field are rather uncommon +# but possible. Inspired by SF bug 1469629. + +import gc + +def leak(): + class T(type): + pass + class U(type): + __metaclass__ = T + U.__class__ = U + del U + gc.collect(); gc.collect(); gc.collect() From python-checkins at python.org Fri Apr 14 17:02:33 2006 From: python-checkins at python.org (martin.v.loewis) Date: Fri, 14 Apr 2006 17:02:33 +0200 (CEST) Subject: [Python-checkins] r45390 - in python/trunk: Makefile.pre.in Parser/asdl_c.py Message-ID: <20060414150233.68C241E400B@bag.python.org> Author: martin.v.loewis Date: Fri Apr 14 17:02:32 2006 New Revision: 45390 Modified: python/trunk/Makefile.pre.in python/trunk/Parser/asdl_c.py Log: Patch #1355883: Build Python-ast.c and Python-ast.h independently. Fixes #1355883. Modified: python/trunk/Makefile.pre.in ============================================================================== --- python/trunk/Makefile.pre.in (original) +++ python/trunk/Makefile.pre.in Fri Apr 14 17:02:32 2006 @@ -219,13 +219,15 @@ ########################################################################## # AST -AST_H= $(srcdir)/Include/Python-ast.h -AST_C= $(srcdir)/Python/Python-ast.c +AST_H_DIR= $(srcdir)/Include +AST_H= $(AST_H_DIR)/Python-ast.h +AST_C_DIR= $(srcdir)/Python +AST_C= $(AST_C_DIR)/Python-ast.c AST_ASDL= $(srcdir)/Parser/Python.asdl ASDLGEN_FILES= $(srcdir)/Parser/asdl.py $(srcdir)/Parser/asdl_c.py # XXX Note that a build now requires Python exist before the build starts -ASDLGEN= $(srcdir)/Parser/asdl_c.py -h $(srcdir)/Include -c $(srcdir)/Python +ASDLGEN= $(srcdir)/Parser/asdl_c.py ########################################################################## # Python @@ -465,9 +467,12 @@ Parser/tokenizer_pgen.o: $(srcdir)/Parser/tokenizer.c -$(AST_H) $(AST_C): $(AST_ASDL) $(ASDLGEN_FILES) - $(ASDLGEN) $(AST_ASDL) +$(AST_H): $(AST_ASDL) $(ASDLGEN_FILES) + $(ASDLGEN) -h $(AST_H_DIR) $(AST_ASDL) +$(AST_C): $(AST_ASDL) $(ASDLGEN_FILES) + $(ASDLGEN) -c $(AST_C_DIR) $(AST_ASDL) + Python/compile.o Python/symtable.o: $(GRAMMAR_H) $(AST_H) Python/getplatform.o: $(srcdir)/Python/getplatform.c Modified: python/trunk/Parser/asdl_c.py ============================================================================== --- python/trunk/Parser/asdl_c.py (original) +++ python/trunk/Parser/asdl_c.py Fri Apr 14 17:02:32 2006 @@ -726,39 +726,35 @@ sys.exit(1) if INC_DIR: p = "%s/%s-ast.h" % (INC_DIR, mod.name) - else: - p = "%s-ast.h" % mod.name - f = open(p, "wb") - print >> f, auto_gen_msg - print >> f, '#include "asdl.h"\n' - c = ChainOfVisitors(TypeDefVisitor(f), - StructVisitor(f), - PrototypeVisitor(f), - ) - c.visit(mod) - print >>f, "PyObject* PyAST_mod2obj(mod_ty t);" - f.close() + f = open(p, "wb") + print >> f, auto_gen_msg + print >> f, '#include "asdl.h"\n' + c = ChainOfVisitors(TypeDefVisitor(f), + StructVisitor(f), + PrototypeVisitor(f), + ) + c.visit(mod) + print >>f, "PyObject* PyAST_mod2obj(mod_ty t);" + f.close() if SRC_DIR: p = os.path.join(SRC_DIR, str(mod.name) + "-ast.c") - else: - p = "%s-ast.c" % mod.name - f = open(p, "wb") - print >> f, auto_gen_msg - print >> f, '#include "Python.h"' - print >> f, '#include "%s-ast.h"' % mod.name - print >> f - print >>f, "static PyTypeObject* AST_type;" - v = ChainOfVisitors( - PyTypesDeclareVisitor(f), - PyTypesVisitor(f), - FunctionVisitor(f), - ObjVisitor(f), - ASTModuleVisitor(f), - PartingShots(f), - ) - v.visit(mod) - f.close() + f = open(p, "wb") + print >> f, auto_gen_msg + print >> f, '#include "Python.h"' + print >> f, '#include "%s-ast.h"' % mod.name + print >> f + print >>f, "static PyTypeObject* AST_type;" + v = ChainOfVisitors( + PyTypesDeclareVisitor(f), + PyTypesVisitor(f), + FunctionVisitor(f), + ObjVisitor(f), + ASTModuleVisitor(f), + PartingShots(f), + ) + v.visit(mod) + f.close() if __name__ == "__main__": import sys @@ -767,6 +763,9 @@ INC_DIR = '' SRC_DIR = '' opts, args = getopt.getopt(sys.argv[1:], "h:c:") + if len(opts) != 1: + print "Must specify exactly one output file" + sys.exit(1) for o, v in opts: if o == '-h': INC_DIR = v @@ -774,4 +773,5 @@ SRC_DIR = v if len(args) != 1: print "Must specify single input file" + sys.exit(1) main(args[0]) From python-checkins at python.org Fri Apr 14 17:07:46 2006 From: python-checkins at python.org (martin.v.loewis) Date: Fri, 14 Apr 2006 17:07:46 +0200 (CEST) Subject: [Python-checkins] r45391 - python/trunk/Modules/_hotshot.c Message-ID: <20060414150746.C9C741E400B@bag.python.org> Author: martin.v.loewis Date: Fri Apr 14 17:07:46 2006 New Revision: 45391 Modified: python/trunk/Modules/_hotshot.c Log: Patch #1470300: Port _hotshot to QNX4.25 port. Modified: python/trunk/Modules/_hotshot.c ============================================================================== --- python/trunk/Modules/_hotshot.c (original) +++ python/trunk/Modules/_hotshot.c Fri Apr 14 17:07:46 2006 @@ -26,7 +26,7 @@ #ifndef HAVE_GETTIMEOFDAY #error "This module requires gettimeofday() on non-Windows platforms!" #endif -#if (defined(PYOS_OS2) && defined(PYCC_GCC)) +#if (defined(PYOS_OS2) && defined(PYCC_GCC)) || defined(__QNX__) #include #else #include @@ -918,7 +918,7 @@ #endif } #if defined(MS_WINDOWS) || defined(PYOS_OS2) || \ - defined(__VMS) + defined(__VMS) || defined (__QNX__) rusage_diff = -1; #else { From python-checkins at python.org Fri Apr 14 17:16:16 2006 From: python-checkins at python.org (martin.v.loewis) Date: Fri, 14 Apr 2006 17:16:16 +0200 (CEST) Subject: [Python-checkins] r45392 - python/trunk/configure python/trunk/configure.in Message-ID: <20060414151616.B259F1E4027@bag.python.org> Author: martin.v.loewis Date: Fri Apr 14 17:16:15 2006 New Revision: 45392 Modified: python/trunk/configure python/trunk/configure.in Log: Patch #1045620: Prepend Modules/ before Setup in stdout. Modified: python/trunk/configure ============================================================================== --- python/trunk/configure (original) +++ python/trunk/configure Fri Apr 14 17:16:15 2006 @@ -1,5 +1,5 @@ #! /bin/sh -# From configure.in Revision: 45328 . +# From configure.in Revision: 45387 . # Guess values for system-dependent variables and create Makefiles. # Generated by GNU Autoconf 2.59 for python 2.5. # @@ -22956,13 +22956,13 @@ fi -echo "creating Setup" +echo "creating Modules/Setup" if test ! -f Modules/Setup then cp $srcdir/Modules/Setup.dist Modules/Setup fi -echo "creating Setup.local" +echo "creating Modules/Setup.local" if test ! -f Modules/Setup.local then echo "# Edit this file for local setup changes" >Modules/Setup.local Modified: python/trunk/configure.in ============================================================================== --- python/trunk/configure.in (original) +++ python/trunk/configure.in Fri Apr 14 17:16:15 2006 @@ -3221,13 +3221,13 @@ AC_CONFIG_FILES(Makefile.pre Modules/Setup.config) AC_OUTPUT -echo "creating Setup" +echo "creating Modules/Setup" if test ! -f Modules/Setup then cp $srcdir/Modules/Setup.dist Modules/Setup fi -echo "creating Setup.local" +echo "creating Modules/Setup.local" if test ! -f Modules/Setup.local then echo "# Edit this file for local setup changes" >Modules/Setup.local From python-checkins at python.org Fri Apr 14 17:22:27 2006 From: python-checkins at python.org (walter.doerwald) Date: Fri, 14 Apr 2006 17:22:27 +0200 (CEST) Subject: [Python-checkins] r45393 - python/trunk/Lib/encodings/idna.py Message-ID: <20060414152227.9EE9A1E400B@bag.python.org> Author: walter.doerwald Date: Fri Apr 14 17:22:27 2006 New Revision: 45393 Modified: python/trunk/Lib/encodings/idna.py Log: Make raise statements PEP 8 compatible. Modified: python/trunk/Lib/encodings/idna.py ============================================================================== --- python/trunk/Lib/encodings/idna.py (original) +++ python/trunk/Lib/encodings/idna.py Fri Apr 14 17:22:27 2006 @@ -35,7 +35,7 @@ stringprep.in_table_c7(c) or \ stringprep.in_table_c8(c) or \ stringprep.in_table_c9(c): - raise UnicodeError, "Invalid character %s" % repr(c) + raise UnicodeError("Invalid character %r" % c) # Check bidi RandAL = map(stringprep.in_table_d1, label) @@ -48,14 +48,14 @@ # 2) If a string contains any RandALCat character, the string # MUST NOT contain any LCat character. if filter(stringprep.in_table_d2, label): - raise UnicodeError, "Violation of BIDI requirement 2" + raise UnicodeError("Violation of BIDI requirement 2") # 3) If a string contains any RandALCat character, a # RandALCat character MUST be the first character of the # string, and a RandALCat character MUST be the last # character of the string. if not RandAL[0] or not RandAL[-1]: - raise UnicodeError, "Violation of BIDI requirement 3" + raise UnicodeError("Violation of BIDI requirement 3") return label @@ -70,7 +70,7 @@ # Skip to step 8. if 0 < len(label) < 64: return label - raise UnicodeError, "label too long" + raise UnicodeError("label too long") # Step 2: nameprep label = nameprep(label) @@ -85,11 +85,11 @@ # Skip to step 8. if 0 < len(label) < 64: return label - raise UnicodeError, "label too long" + raise UnicodeError("label too long") # Step 5: Check ACE prefix if label.startswith(uace_prefix): - raise UnicodeError, "Label starts with ACE prefix" + raise UnicodeError("Label starts with ACE prefix") # Step 6: Encode with PUNYCODE label = label.encode("punycode") @@ -100,7 +100,7 @@ # Step 8: Check size if 0 < len(label) < 64: return label - raise UnicodeError, "label too long" + raise UnicodeError("label too long") def ToUnicode(label): # Step 1: Check for ASCII @@ -119,7 +119,7 @@ try: label = label.encode("ascii") except UnicodeError: - raise UnicodeError, "Invalid character in IDN label" + raise UnicodeError("Invalid character in IDN label") # Step 3: Check for ACE prefix if not label.startswith(ace_prefix): return unicode(label, "ascii") @@ -136,7 +136,7 @@ # Step 7: Compare the result of step 6 with the one of step 3 # label2 will already be in lower case. if label.lower() != label2: - raise UnicodeError, ("IDNA does not round-trip", label, label2) + raise UnicodeError("IDNA does not round-trip", label, label2) # Step 8: return the result of step 5 return result @@ -148,7 +148,7 @@ if errors != 'strict': # IDNA is quite clear that implementations must be strict - raise UnicodeError, "unsupported error handling "+errors + raise UnicodeError("unsupported error handling "+errors) if not input: return "", 0 @@ -168,7 +168,7 @@ def decode(self,input,errors='strict'): if errors != 'strict': - raise UnicodeError, "Unsupported error handling "+errors + raise UnicodeError("Unsupported error handling "+errors) if not input: return u"", 0 From python-checkins at python.org Fri Apr 14 17:40:55 2006 From: python-checkins at python.org (walter.doerwald) Date: Fri, 14 Apr 2006 17:40:55 +0200 (CEST) Subject: [Python-checkins] r45394 - python/trunk/Lib/codecs.py Message-ID: <20060414154055.0677B1E4091@bag.python.org> Author: walter.doerwald Date: Fri Apr 14 17:40:54 2006 New Revision: 45394 Modified: python/trunk/Lib/codecs.py Log: Fix wrong attribute name. Modified: python/trunk/Lib/codecs.py ============================================================================== --- python/trunk/Lib/codecs.py (original) +++ python/trunk/Lib/codecs.py Fri Apr 14 17:40:54 2006 @@ -233,7 +233,7 @@ def reset(self): IncrementalDecoder.reset(self) - self.bytebuffer = "" + self.buffer = "" # # The StreamWriter and StreamReader class provide generic working From buildbot at python.org Fri Apr 14 17:45:09 2006 From: buildbot at python.org (buildbot at python.org) Date: Fri, 14 Apr 2006 15:45:09 +0000 Subject: [Python-checkins] buildbot warnings in g4 osx.4 trunk Message-ID: <20060414154509.5AEE71E400B@bag.python.org> The Buildbot has detected a new failure of g4 osx.4 trunk. Full details are available at: http://www.python.org/dev/buildbot/all/g4%2520osx.4%2520trunk/builds/405 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: martin.v.loewis Build Had Warnings: warnings test sincerely, -The Buildbot From python-checkins at python.org Fri Apr 14 17:58:03 2006 From: python-checkins at python.org (martin.v.loewis) Date: Fri, 14 Apr 2006 17:58:03 +0200 (CEST) Subject: [Python-checkins] r45395 - python/trunk/Doc/api/memory.tex python/trunk/Doc/api/newtypes.tex Message-ID: <20060414155803.DF5BC1E400B@bag.python.org> Author: martin.v.loewis Date: Fri Apr 14 17:58:03 2006 New Revision: 45395 Modified: python/trunk/Doc/api/memory.tex python/trunk/Doc/api/newtypes.tex Log: Patch #702933: Undocument PyObject_NEW, PyObject_NEW_VAR, and PyObject_DEL. Modified: python/trunk/Doc/api/memory.tex ============================================================================== --- python/trunk/Doc/api/memory.tex (original) +++ python/trunk/Doc/api/memory.tex Fri Apr 14 17:58:03 2006 @@ -195,9 +195,7 @@ In addition to the functions aimed at handling raw memory blocks from the Python heap, objects in Python are allocated and released with \cfunction{PyObject_New()}, \cfunction{PyObject_NewVar()} and -\cfunction{PyObject_Del()}, or with their corresponding macros -\cfunction{PyObject_NEW()}, \cfunction{PyObject_NEW_VAR()} and -\cfunction{PyObject_DEL()}. +\cfunction{PyObject_Del()}. These will be explained in the next chapter on defining and implementing new object types in C. Modified: python/trunk/Doc/api/newtypes.tex ============================================================================== --- python/trunk/Doc/api/newtypes.tex (original) +++ python/trunk/Doc/api/newtypes.tex Fri Apr 14 17:58:03 2006 @@ -62,23 +62,6 @@ after this call as the memory is no longer a valid Python object. \end{cfuncdesc} -\begin{cfuncdesc}{\var{TYPE}*}{PyObject_NEW}{TYPE, PyTypeObject *type} - Macro version of \cfunction{PyObject_New()}, to gain performance at - the expense of safety. This does not check \var{type} for a \NULL{} - value. -\end{cfuncdesc} - -\begin{cfuncdesc}{\var{TYPE}*}{PyObject_NEW_VAR}{TYPE, PyTypeObject *type, - Py_ssize_t size} - Macro version of \cfunction{PyObject_NewVar()}, to gain performance - at the expense of safety. This does not check \var{type} for a - \NULL{} value. -\end{cfuncdesc} - -\begin{cfuncdesc}{void}{PyObject_DEL}{PyObject *op} - Macro version of \cfunction{PyObject_Del()}. -\end{cfuncdesc} - \begin{cfuncdesc}{PyObject*}{Py_InitModule}{char *name, PyMethodDef *methods} Create a new module object based on a name and table of functions, From dynkin at gmail.com Fri Apr 14 18:33:07 2006 From: dynkin at gmail.com (George Yoshida) Date: Sat, 15 Apr 2006 01:33:07 +0900 Subject: [Python-checkins] r45329 - python/trunk/Doc/whatsnew/whatsnew25.tex In-Reply-To: <1f7befae0604132153i13cd233fu1880985ba8525497@mail.gmail.com> References: <20060413020445.29D061E4009@bag.python.org> <2f188ee80604130828x47cf5f4u3214c533323ea3c@mail.gmail.com> <200604131337.59193.fdrake@acm.org> <2f188ee80604132055t67abb5fag90cb73b0b68a75ae@mail.gmail.com> <1f7befae0604132153i13cd233fu1880985ba8525497@mail.gmail.com> Message-ID: <2f188ee80604140933h268148d5i2031d5f36dd5703c@mail.gmail.com> On 4/14/06, Tim Peters wrote: > > Can you grant me sourceforge manager privilege so I can handle > > bug/patch tasks? > > Login users are basically only allowed to comment on them. > > I added SF name "quiver" to the SF Python project as a tracker admin. > The SF permission system has gotten so convoluted I'm not sure that's > enough, so scream at me if you're still getting blocked. > Thanks Tim. I really appreciate it. -- george From python-checkins at python.org Fri Apr 14 19:00:37 2006 From: python-checkins at python.org (walter.doerwald) Date: Fri, 14 Apr 2006 19:00:37 +0200 (CEST) Subject: [Python-checkins] r45396 - python/trunk/Lib/encodings/idna.py Message-ID: <20060414170037.18B391E402E@bag.python.org> Author: walter.doerwald Date: Fri Apr 14 19:00:36 2006 New Revision: 45396 Modified: python/trunk/Lib/encodings/idna.py Log: Make error message less misleading for u"a..b".encode("idna"). Modified: python/trunk/Lib/encodings/idna.py ============================================================================== --- python/trunk/Lib/encodings/idna.py (original) +++ python/trunk/Lib/encodings/idna.py Fri Apr 14 19:00:36 2006 @@ -70,7 +70,7 @@ # Skip to step 8. if 0 < len(label) < 64: return label - raise UnicodeError("label too long") + raise UnicodeError("label empty or too long") # Step 2: nameprep label = nameprep(label) @@ -85,7 +85,7 @@ # Skip to step 8. if 0 < len(label) < 64: return label - raise UnicodeError("label too long") + raise UnicodeError("label empty or too long") # Step 5: Check ACE prefix if label.startswith(uace_prefix): @@ -100,7 +100,7 @@ # Step 8: Check size if 0 < len(label) < 64: return label - raise UnicodeError("label too long") + raise UnicodeError("label empty or too long") def ToUnicode(label): # Step 1: Check for ASCII From python-checkins at python.org Fri Apr 14 19:31:03 2006 From: python-checkins at python.org (phillip.eby) Date: Fri, 14 Apr 2006 19:31:03 +0200 (CEST) Subject: [Python-checkins] r45397 - in sandbox/branches/setuptools-0.6: ez_setup.py release.sh setup.py setuptools/__init__.py version version.dat Message-ID: <20060414173103.5BFDA1E400B@bag.python.org> Author: phillip.eby Date: Fri Apr 14 19:31:02 2006 New Revision: 45397 Added: sandbox/branches/setuptools-0.6/ - copied from r45395, sandbox/trunk/setuptools/ sandbox/branches/setuptools-0.6/release.sh (contents, props changed) sandbox/branches/setuptools-0.6/version (contents, props changed) sandbox/branches/setuptools-0.6/version.dat (contents, props changed) Modified: sandbox/branches/setuptools-0.6/ez_setup.py sandbox/branches/setuptools-0.6/setup.py sandbox/branches/setuptools-0.6/setuptools/__init__.py Log: Creating a branch for the "stable" 0.6 development line of setuptools, so that it can continue forward to 0.6b1 and onward, while using the trunk for 0.7 development and Python 2.5 snapshots thereof. Modified: sandbox/branches/setuptools-0.6/ez_setup.py ============================================================================== --- sandbox/trunk/setuptools/ez_setup.py (original) +++ sandbox/branches/setuptools-0.6/ez_setup.py Fri Apr 14 19:31:02 2006 @@ -14,7 +14,7 @@ This file can also be run as a script to install or upgrade setuptools. """ import sys -DEFAULT_VERSION = "0.6a12" +DEFAULT_VERSION = "0.6b1" DEFAULT_URL = "http://cheeseshop.python.org/packages/%s/s/setuptools/" % sys.version[:3] md5_data = { Added: sandbox/branches/setuptools-0.6/release.sh ============================================================================== --- (empty file) +++ sandbox/branches/setuptools-0.6/release.sh Fri Apr 14 19:31:02 2006 @@ -0,0 +1,22 @@ +#!/bin/sh + +# This script is for PJE's working environment only, to upload +# releases to PyPI, telecommunity, eby-sarna SVN, update local +# project checkouts, etc. +# +# If your initials aren't PJE, don't run it. :) +# + +export VERSION="0.6b1" + +wpython setup.py -q source && \ +cpython setup.py -q binary && \ +python ez_setup.py --md5update dist/setuptools-$VERSION*-py2.?.egg && \ + scp ez_setup.py virtual-python.py t3:web/PEAK/dist/ && \ + cp ez_setup.py ~/projects/ez_setup/__init__.py && \ + svn ci -m "Update ez_setup for setuptools $VERSION" \ + ~/projects/ez_setup/__init__.py && \ + svn up ~/projects/*/ez_setup + +# XXX update wiki pages from EasyInstall.txt, setuptools.txt, & +# pkg_resources.txt Modified: sandbox/branches/setuptools-0.6/setup.py ============================================================================== --- sandbox/trunk/setuptools/setup.py (original) +++ sandbox/branches/setuptools-0.6/setup.py Fri Apr 14 19:31:02 2006 @@ -19,7 +19,7 @@ d = {}; execfile(convert_path('setuptools/command/__init__.py'), d) SETUP_COMMANDS = d['__all__'] -VERSION = "0.6a12" +VERSION = "0.6b1" from setuptools import setup, find_packages import sys scripts = [] Modified: sandbox/branches/setuptools-0.6/setuptools/__init__.py ============================================================================== --- sandbox/trunk/setuptools/setuptools/__init__.py (original) +++ sandbox/branches/setuptools-0.6/setuptools/__init__.py Fri Apr 14 19:31:02 2006 @@ -7,7 +7,7 @@ from distutils.util import convert_path import os.path -__version__ = '0.6a12' +__version__ = '0.6b1' __all__ = [ 'setup', 'Distribution', 'Feature', 'Command', 'Extension', 'Require', 'find_packages' Added: sandbox/branches/setuptools-0.6/version ============================================================================== --- (empty file) +++ sandbox/branches/setuptools-0.6/version Fri Apr 14 19:31:02 2006 @@ -0,0 +1,48 @@ +#!/usr/local/bin/invoke /usr/local/bin/c6peak version-config + +# This is a PEAK 'version' tool configuration file, that's +# also executable. PJE uses it to bump version numbers in +# the various parts of the project without having to edit them +# by hand. The current version is stored in the version.dat +# file. + +# These are not the droids you're looking for. You can go on +# about your business... + + + DefaultFormat full + part major + part minor + part status choice alpha beta "release candidate" final + part build + part date timestamp + + + trailer remap status "a%(build)s" "b%(build)s" "c%(build)s" "%(dot-maint)s" + dot-maint optional build ".%(build)s" + full "%(major)s.%(minor)s %(status)s %(build)s" + short "%(major)s.%(minor)s%(trailer)s" + + + + + Name setuptools + + + File setup.py + File ez_setup.py + Match 'VERSION = "%(short)s"' + + + + File release.sh + Match 'VERSION="%(short)s"' + + + + File setuptools/__init__.py + Match "__version__ = '%(short)s'" + + + + Added: sandbox/branches/setuptools-0.6/version.dat ============================================================================== --- (empty file) +++ sandbox/branches/setuptools-0.6/version.dat Fri Apr 14 19:31:02 2006 @@ -0,0 +1,6 @@ +[setuptools] +status = 'beta' +major = 0 +build = 1 +minor = 6 + From python-checkins at python.org Fri Apr 14 20:02:13 2006 From: python-checkins at python.org (phillip.eby) Date: Fri, 14 Apr 2006 20:02:13 +0200 (CEST) Subject: [Python-checkins] r45398 - sandbox/branches/setuptools-0.6/setuptools.txt Message-ID: <20060414180213.678CC1E401D@bag.python.org> Author: phillip.eby Date: Fri Apr 14 20:02:12 2006 New Revision: 45398 Modified: sandbox/branches/setuptools-0.6/setuptools.txt Log: Make docs & PyPI list separate locations for 0.6 and 0.7 development versions. Modified: sandbox/branches/setuptools-0.6/setuptools.txt ============================================================================== --- sandbox/branches/setuptools-0.6/setuptools.txt (original) +++ sandbox/branches/setuptools-0.6/setuptools.txt Fri Apr 14 20:02:12 2006 @@ -63,7 +63,10 @@ extensions, using simple "entry points" declared in a project's setup script. In addition to the PyPI downloads, the development version of ``setuptools`` -is available from the `Python SVN sandbox`_. +is available from the `Python SVN sandbox`_, and in-development versions of the +`0.6 branch`_ are available as well. + +.. _0.6 branch: http://svn.python.org/projects/sandbox/branches/setuptools-0.6/#egg=setuptools-dev06 .. _Python SVN sandbox: http://svn.python.org/projects/sandbox/trunk/setuptools/#egg=setuptools-dev From python-checkins at python.org Fri Apr 14 20:09:09 2006 From: python-checkins at python.org (phillip.eby) Date: Fri, 14 Apr 2006 20:09:09 +0200 (CEST) Subject: [Python-checkins] r45399 - sandbox/branches/setuptools-0.6/EasyInstall.txt Message-ID: <20060414180909.D10C71E4005@bag.python.org> Author: phillip.eby Date: Fri Apr 14 20:09:09 2006 New Revision: 45399 Modified: sandbox/branches/setuptools-0.6/EasyInstall.txt Log: Correct version # in release notes Modified: sandbox/branches/setuptools-0.6/EasyInstall.txt ============================================================================== --- sandbox/branches/setuptools-0.6/EasyInstall.txt (original) +++ sandbox/branches/setuptools-0.6/EasyInstall.txt Fri Apr 14 20:09:09 2006 @@ -1097,7 +1097,7 @@ Release Notes/Change History ============================ -0.6a12 +0.6b1 * Added automatic retry for Sourceforge mirrors. The new download process is to first just try dl.sourceforge.net, then randomly select mirror IPs and remove ones that fail, until something works. The removed IPs stay removed From python-checkins at python.org Fri Apr 14 20:10:51 2006 From: python-checkins at python.org (phillip.eby) Date: Fri, 14 Apr 2006 20:10:51 +0200 (CEST) Subject: [Python-checkins] r45400 - in sandbox/trunk/setuptools: EasyInstall.txt ez_setup.py pkg_resources.txt setup.py setuptools.txt setuptools/__init__.py Message-ID: <20060414181051.B17111E403D@bag.python.org> Author: phillip.eby Date: Fri Apr 14 20:10:50 2006 New Revision: 45400 Modified: sandbox/trunk/setuptools/EasyInstall.txt sandbox/trunk/setuptools/ez_setup.py sandbox/trunk/setuptools/pkg_resources.txt sandbox/trunk/setuptools/setup.py sandbox/trunk/setuptools/setuptools.txt sandbox/trunk/setuptools/setuptools/__init__.py Log: Bump trunk version # to 0.7a1.dev, and clear out old version history from release notes. Modified: sandbox/trunk/setuptools/EasyInstall.txt ============================================================================== --- sandbox/trunk/setuptools/EasyInstall.txt (original) +++ sandbox/trunk/setuptools/EasyInstall.txt Fri Apr 14 20:10:50 2006 @@ -1093,354 +1093,15 @@ set, if you haven't already got this set up on your machine. - Release Notes/Change History ============================ -0.6a12 +0.7a1 * Added automatic retry for Sourceforge mirrors. The new download process is to first just try dl.sourceforge.net, then randomly select mirror IPs and remove ones that fail, until something works. The removed IPs stay removed for the remainder of the run. -0.6a11 - * Process ``dependency_links.txt`` if found in a distribution, by adding the - URLs to the list for scanning. - - * Use relative paths in ``.pth`` files when eggs are being installed to the - same directory as the ``.pth`` file. This maximizes portability of the - target directory when building applications that contain eggs. - - * Added ``easy_install-N.N`` script(s) for convenience when using multiple - Python versions. - - * Added automatic handling of installation conflicts. Eggs are now shifted to - the front of sys.path, in an order consistent with where they came from, - making EasyInstall seamlessly co-operate with system package managers. - - The ``--delete-conflicting`` and ``--ignore-conflicts-at-my-risk`` options - are now no longer necessary, and will generate warnings at the end of a - run if you use them. - - * Don't recursively traverse subdirectories given to ``--find-links``. - -0.6a10 - * Added exhaustive testing of the install directory, including a spawn test - for ``.pth`` file support, and directory writability/existence checks. This - should virtually eliminate the need to set or configure ``--site-dirs``. - - * Added ``--prefix`` option for more do-what-I-mean-ishness in the absence of - RTFM-ing. :) - - * Enhanced ``PYTHONPATH`` support so that you don't have to put any eggs on it - manually to make it work. ``--multi-version`` is no longer a silent - default; you must explicitly use it if installing to a non-PYTHONPATH, - non-"site" directory. - - * Expand ``$variables`` used in the ``--site-dirs``, ``--build-directory``, - ``--install-dir``, and ``--script-dir`` options, whether on the command line - or in configuration files. - - * Improved SourceForge mirror processing to work faster and be less affected - by transient HTML changes made by SourceForge. - - * PyPI searches now use the exact spelling of requirements specified on the - command line or in a project's ``install_requires``. Previously, a - normalized form of the name was used, which could lead to unnecessary - full-index searches when a project's name had an underscore (``_``) in it. - - * EasyInstall can now download bare ``.py`` files and wrap them in an egg, - as long as you include an ``#egg=name-version`` suffix on the URL, or if - the ``.py`` file is listed as the "Download URL" on the project's PyPI page. - This allows third parties to "package" trivial Python modules just by - linking to them (e.g. from within their own PyPI page or download links - page). - - * The ``--always-copy`` option now skips "system" and "development" eggs since - they can't be reliably copied. Note that this may cause EasyInstall to - choose an older version of a package than what you expected, or it may cause - downloading and installation of a fresh version of what's already installed. - - * The ``--find-links`` option previously scanned all supplied URLs and - directories as early as possible, but now only directories and direct - archive links are scanned immediately. URLs are not retrieved unless a - package search was already going to go online due to a package not being - available locally, or due to the use of the ``--update`` or ``-U`` option. - - * Fixed the annoying ``--help-commands`` wart. - -0.6a9 - * Fixed ``.pth`` file processing picking up nested eggs (i.e. ones inside - "baskets") when they weren't explicitly listed in the ``.pth`` file. - - * If more than one URL appears to describe the exact same distribution, prefer - the shortest one. This helps to avoid "table of contents" CGI URLs like the - ones on effbot.org. - - * Quote arguments to python.exe (including python's path) to avoid problems - when Python (or a script) is installed in a directory whose name contains - spaces on Windows. - - * Support full roundtrip translation of eggs to and from ``bdist_wininst`` - format. Running ``bdist_wininst`` on a setuptools-based package wraps the - egg in an .exe that will safely install it as an egg (i.e., with metadata - and entry-point wrapper scripts), and ``easy_install`` can turn the .exe - back into an ``.egg`` file or directory and install it as such. - -0.6a8 - * Update for changed SourceForge mirror format - - * Fixed not installing dependencies for some packages fetched via Subversion - - * Fixed dependency installation with ``--always-copy`` not using the same - dependency resolution procedure as other operations. - - * Fixed not fully removing temporary directories on Windows, if a Subversion - checkout left read-only files behind - - * Fixed some problems building extensions when Pyrex was installed, especially - with Python 2.4 and/or packages using SWIG. - -0.6a7 - * Fixed not being able to install Windows script wrappers using Python 2.3 - -0.6a6 - * Added support for "traditional" PYTHONPATH-based non-root installation, and - also the convenient ``virtual-python.py`` script, based on a contribution - by Ian Bicking. The setuptools egg now contains a hacked ``site`` module - that makes the PYTHONPATH-based approach work with .pth files, so that you - can get the full EasyInstall feature set on such installations. - - * Added ``--no-deps`` and ``--allow-hosts`` options. - - * Improved Windows ``.exe`` script wrappers so that the script can have the - same name as a module without confusing Python. - - * Changed dependency processing so that it's breadth-first, allowing a - depender's preferences to override those of a dependee, to prevent conflicts - when a lower version is acceptable to the dependee, but not the depender. - Also, ensure that currently installed/selected packages aren't given - precedence over ones desired by a package being installed, which could - cause conflict errors. - -0.6a3 - * Improved error message when trying to use old ways of running - ``easy_install``. Removed the ability to run via ``python -m`` or by - running ``easy_install.py``; ``easy_install`` is the command to run on all - supported platforms. - - * Improved wrapper script generation and runtime initialization so that a - VersionConflict doesn't occur if you later install a competing version of a - needed package as the default version of that package. - - * Fixed a problem parsing version numbers in ``#egg=`` links. - -0.6a2 - * EasyInstall can now install "console_scripts" defined by packages that use - ``setuptools`` and define appropriate entry points. On Windows, console - scripts get an ``.exe`` wrapper so you can just type their name. On other - platforms, the scripts are installed without a file extension. - - * Using ``python -m easy_install`` or running ``easy_install.py`` is now - DEPRECATED, since an ``easy_install`` wrapper is now available on all - platforms. - -0.6a1 - * EasyInstall now does MD5 validation of downloads from PyPI, or from any link - that has an "#md5=..." trailer with a 32-digit lowercase hex md5 digest. - - * EasyInstall now handles symlinks in target directories by removing the link, - rather than attempting to overwrite the link's destination. This makes it - easier to set up an alternate Python "home" directory (as described above in - the `Non-Root Installation`_ section). - - * Added support for handling MacOS platform information in ``.egg`` filenames, - based on a contribution by Kevin Dangoor. You may wish to delete and - reinstall any eggs whose filename includes "darwin" and "Power_Macintosh", - because the format for this platform information has changed so that minor - OS X upgrades (such as 10.4.1 to 10.4.2) do not cause eggs built with a - previous OS version to become obsolete. - - * easy_install's dependency processing algorithms have changed. When using - ``--always-copy``, it now ensures that dependencies are copied too. When - not using ``--always-copy``, it tries to use a single resolution loop, - rather than recursing. - - * Fixed installing extra ``.pyc`` or ``.pyo`` files for scripts with ``.py`` - extensions. - - * Added ``--site-dirs`` option to allow adding custom "site" directories. - Made ``easy-install.pth`` work in platform-specific alternate site - directories (e.g. ``~/Library/Python/2.x/site-packages`` on Mac OS X). - - * If you manually delete the current version of a package, the next run of - EasyInstall against the target directory will now remove the stray entry - from the ``easy-install.pth`` file. - - * EasyInstall now recognizes URLs with a ``#egg=project_name`` fragment ID - as pointing to the named project's source checkout. Such URLs have a lower - match precedence than any other kind of distribution, so they'll only be - used if they have a higher version number than any other available - distribution, or if you use the ``--editable`` option. The ``#egg`` - fragment can contain a version if it's formatted as ``#egg=proj-ver``, - where ``proj`` is the project name, and ``ver`` is the version number. You - *must* use the format for these values that the ``bdist_egg`` command uses; - i.e., all non-alphanumeric runs must be condensed to single underscore - characters. - - * Added the ``--editable`` option; see `Editing and Viewing Source Packages`_ - above for more info. Also, slightly changed the behavior of the - ``--build-directory`` option. - - * Fixed the setup script sandbox facility not recognizing certain paths as - valid on case-insensitive platforms. - -0.5a12 - * Fix ``python -m easy_install`` not working due to setuptools being installed - as a zipfile. Update safety scanner to check for modules that might be used - as ``python -m`` scripts. - - * Misc. fixes for win32.exe support, including changes to support Python 2.4's - changed ``bdist_wininst`` format. - -0.5a10 - * Put the ``easy_install`` module back in as a module, as it's needed for - ``python -m`` to run it! - - * Allow ``--find-links/-f`` to accept local directories or filenames as well - as URLs. - -0.5a9 - * EasyInstall now automatically detects when an "unmanaged" package or - module is going to be on ``sys.path`` ahead of a package you're installing, - thereby preventing the newer version from being imported. By default, it - will abort installation to alert you of the problem, but there are also - new options (``--delete-conflicting`` and ``--ignore-conflicts-at-my-risk``) - available to change the default behavior. (Note: this new feature doesn't - take effect for egg files that were built with older ``setuptools`` - versions, because they lack the new metadata file required to implement it.) - - * The ``easy_install`` distutils command now uses ``DistutilsError`` as its - base error type for errors that should just issue a message to stderr and - exit the program without a traceback. - - * EasyInstall can now be given a path to a directory containing a setup - script, and it will attempt to build and install the package there. - - * EasyInstall now performs a safety analysis on module contents to determine - whether a package is likely to run in zipped form, and displays - information about what modules may be doing introspection that would break - when running as a zipfile. - - * Added the ``--always-unzip/-Z`` option, to force unzipping of packages that - would ordinarily be considered safe to unzip, and changed the meaning of - ``--zip-ok/-z`` to "always leave everything zipped". - -0.5a8 - * There is now a separate documentation page for `setuptools`_; revision - history that's not specific to EasyInstall has been moved to that page. - - .. _setuptools: http://peak.telecommunity.com/DevCenter/setuptools - -0.5a5 - * Made ``easy_install`` a standard ``setuptools`` command, moving it from - the ``easy_install`` module to ``setuptools.command.easy_install``. Note - that if you were importing or extending it, you must now change your imports - accordingly. ``easy_install.py`` is still installed as a script, but not as - a module. - -0.5a4 - * Added ``--always-copy/-a`` option to always copy needed packages to the - installation directory, even if they're already present elsewhere on - sys.path. (In previous versions, this was the default behavior, but now - you must request it.) - - * Added ``--upgrade/-U`` option to force checking PyPI for latest available - version(s) of all packages requested by name and version, even if a matching - version is available locally. - - * Added automatic installation of dependencies declared by a distribution - being installed. These dependencies must be listed in the distribution's - ``EGG-INFO`` directory, so the distribution has to have declared its - dependencies by using setuptools. If a package has requirements it didn't - declare, you'll still have to deal with them yourself. (E.g., by asking - EasyInstall to find and install them.) - - * Added the ``--record`` option to ``easy_install`` for the benefit of tools - that run ``setup.py install --record=filename`` on behalf of another - packaging system.) - -0.5a3 - * Fixed not setting script permissions to allow execution. - - * Improved sandboxing so that setup scripts that want a temporary directory - (e.g. pychecker) can still run in the sandbox. - -0.5a2 - * Fix stupid stupid refactoring-at-the-last-minute typos. :( - -0.5a1 - * Added support for converting ``.win32.exe`` installers to eggs on the fly. - EasyInstall will now recognize such files by name and install them. - - * Fixed a problem with picking the "best" version to install (versions were - being sorted as strings, rather than as parsed values) - -0.4a4 - * Added support for the distutils "verbose/quiet" and "dry-run" options, as - well as the "optimize" flag. - - * Support downloading packages that were uploaded to PyPI (by scanning all - links on package pages, not just the homepage/download links). - -0.4a3 - * Add progress messages to the search/download process so that you can tell - what URLs it's reading to find download links. (Hopefully, this will help - people report out-of-date and broken links to package authors, and to tell - when they've asked for a package that doesn't exist.) - -0.4a2 - * Added support for installing scripts - - * Added support for setting options via distutils configuration files, and - using distutils' default options as a basis for EasyInstall's defaults. - - * Renamed ``--scan-url/-s`` to ``--find-links/-f`` to free up ``-s`` for the - script installation directory option. - - * Use ``urllib2`` instead of ``urllib``, to allow use of ``https:`` URLs if - Python includes SSL support. - -0.4a1 - * Added ``--scan-url`` and ``--index-url`` options, to scan download pages - and search PyPI for needed packages. - -0.3a4 - * Restrict ``--build-directory=DIR/-b DIR`` option to only be used with single - URL installs, to avoid running the wrong setup.py. - -0.3a3 - * Added ``--build-directory=DIR/-b DIR`` option. - - * Added "installation report" that explains how to use 'require()' when doing - a multiversion install or alternate installation directory. - - * Added SourceForge mirror auto-select (Contributed by Ian Bicking) - - * Added "sandboxing" that stops a setup script from running if it attempts to - write to the filesystem outside of the build area - - * Added more workarounds for packages with quirky ``install_data`` hacks - -0.3a2 - * Added subversion download support for ``svn:`` and ``svn+`` URLs, as well as - automatic recognition of HTTP subversion URLs (Contributed by Ian Bicking) - - * Misc. bug fixes - -0.3a1 - * Initial release. - Future Plans ============ Modified: sandbox/trunk/setuptools/ez_setup.py ============================================================================== --- sandbox/trunk/setuptools/ez_setup.py (original) +++ sandbox/trunk/setuptools/ez_setup.py Fri Apr 14 20:10:50 2006 @@ -14,7 +14,7 @@ This file can also be run as a script to install or upgrade setuptools. """ import sys -DEFAULT_VERSION = "0.6a12" +DEFAULT_VERSION = "0.7a1" DEFAULT_URL = "http://cheeseshop.python.org/packages/%s/s/setuptools/" % sys.version[:3] md5_data = { Modified: sandbox/trunk/setuptools/pkg_resources.txt ============================================================================== --- sandbox/trunk/setuptools/pkg_resources.txt (original) +++ sandbox/trunk/setuptools/pkg_resources.txt Fri Apr 14 20:10:50 2006 @@ -1652,224 +1652,5 @@ Release Notes/Change History ---------------------------- -0.6a11 - * Added ``ExtractionError`` and ``ResourceManager.extraction_error()`` so that - cache permission problems get a more user-friendly explanation of the - problem, and so that programs can catch and handle extraction errors if they - need to. - -0.6a10 - * Added the ``extras`` attribute to ``Distribution``, the ``find_plugins()`` - method to ``WorkingSet``, and the ``__add__()`` and ``__iadd__()`` methods - to ``Environment``. - - * ``safe_name()`` now allows dots in project names. - - * There is a new ``to_filename()`` function that escapes project names and - versions for safe use in constructing egg filenames from a Distribution - object's metadata. - - * Added ``Distribution.clone()`` method, and keyword argument support to other - ``Distribution`` constructors. - - * Added the ``DEVELOP_DIST`` precedence, and automatically assign it to - eggs using ``.egg-info`` format. - -0.6a9 - * Don't raise an error when an invalid (unfinished) distribution is found - unless absolutely necessary. Warn about skipping invalid/unfinished eggs - when building an Environment. - - * Added support for ``.egg-info`` files or directories with version/platform - information embedded in the filename, so that system packagers have the - option of including ``PKG-INFO`` files to indicate the presence of a - system-installed egg, without needing to use ``.egg`` directories, zipfiles, - or ``.pth`` manipulation. - - * Changed ``parse_version()`` to remove dashes before pre-release tags, so - that ``0.2-rc1`` is considered an *older* version than ``0.2``, and is equal - to ``0.2rc1``. The idea that a dash *always* meant a post-release version - was highly non-intuitive to setuptools users and Python developers, who - seem to want to use ``-rc`` version numbers a lot. - -0.6a8 - * Fixed a problem with ``WorkingSet.resolve()`` that prevented version - conflicts from being detected at runtime. - - * Improved runtime conflict warning message to identify a line in the user's - program, rather than flagging the ``warn()`` call in ``pkg_resources``. - - * Avoid giving runtime conflict warnings for namespace packages, even if they - were declared by a different package than the one currently being activated. - - * Fix path insertion algorithm for case-insensitive filesystems. - - * Fixed a problem with nested namespace packages (e.g. ``peak.util``) not - being set as an attribute of their parent package. - -0.6a6 - * Activated distributions are now inserted in ``sys.path`` (and the working - set) just before the directory that contains them, instead of at the end. - This allows e.g. eggs in ``site-packages`` to override unmanaged modules in - the same location, and allows eggs found earlier on ``sys.path`` to override - ones found later. - - * When a distribution is activated, it now checks whether any contained - non-namespace modules have already been imported and issues a warning if - a conflicting module has already been imported. - - * Changed dependency processing so that it's breadth-first, allowing a - depender's preferences to override those of a dependee, to prevent conflicts - when a lower version is acceptable to the dependee, but not the depender. - - * Fixed a problem extracting zipped files on Windows, when the egg in question - has had changed contents but still has the same version number. - -0.6a4 - * Fix a bug in ``WorkingSet.resolve()`` that was introduced in 0.6a3. - -0.6a3 - * Added ``safe_extra()`` parsing utility routine, and use it for Requirement, - EntryPoint, and Distribution objects' extras handling. - -0.6a1 - * Enhanced performance of ``require()`` and related operations when all - requirements are already in the working set, and enhanced performance of - directory scanning for distributions. - - * Fixed some problems using ``pkg_resources`` w/PEP 302 loaders other than - ``zipimport``, and the previously-broken "eager resource" support. - - * Fixed ``pkg_resources.resource_exists()`` not working correctly, along with - some other resource API bugs. - - * Many API changes and enhancements: - - * Added ``EntryPoint``, ``get_entry_map``, ``load_entry_point``, and - ``get_entry_info`` APIs for dynamic plugin discovery. - - * ``list_resources`` is now ``resource_listdir`` (and it actually works) - - * Resource API functions like ``resource_string()`` that accepted a package - name and resource name, will now also accept a ``Requirement`` object in - place of the package name (to allow access to non-package data files in - an egg). - - * ``get_provider()`` will now accept a ``Requirement`` instance or a module - name. If it is given a ``Requirement``, it will return a corresponding - ``Distribution`` (by calling ``require()`` if a suitable distribution - isn't already in the working set), rather than returning a metadata and - resource provider for a specific module. (The difference is in how - resource paths are interpreted; supplying a module name means resources - path will be module-relative, rather than relative to the distribution's - root.) - - * ``Distribution`` objects now implement the ``IResourceProvider`` and - ``IMetadataProvider`` interfaces, so you don't need to reference the (no - longer available) ``metadata`` attribute to get at these interfaces. - - * ``Distribution`` and ``Requirement`` both have a ``project_name`` - attribute for the project name they refer to. (Previously these were - ``name`` and ``distname`` attributes.) - - * The ``path`` attribute of ``Distribution`` objects is now ``location``, - because it isn't necessarily a filesystem path (and hasn't been for some - time now). The ``location`` of ``Distribution`` objects in the filesystem - should always be normalized using ``pkg_resources.normalize_path()``; all - of the setuptools and EasyInstall code that generates distributions from - the filesystem (including ``Distribution.from_filename()``) ensure this - invariant, but if you use a more generic API like ``Distribution()`` or - ``Distribution.from_location()`` you should take care that you don't - create a distribution with an un-normalized filesystem path. - - * ``Distribution`` objects now have an ``as_requirement()`` method that - returns a ``Requirement`` for the distribution's project name and version. - - * Distribution objects no longer have an ``installed_on()`` method, and the - ``install_on()`` method is now ``activate()`` (but may go away altogether - soon). The ``depends()`` method has also been renamed to ``requires()``, - and ``InvalidOption`` is now ``UnknownExtra``. - - * ``find_distributions()`` now takes an additional argument called ``only``, - that tells it to only yield distributions whose location is the passed-in - path. (It defaults to False, so that the default behavior is unchanged.) - - * ``AvailableDistributions`` is now called ``Environment``, and the - ``get()``, ``__len__()``, and ``__contains__()`` methods were removed, - because they weren't particularly useful. ``__getitem__()`` no longer - raises ``KeyError``; it just returns an empty list if there are no - distributions for the named project. - - * The ``resolve()`` method of ``Environment`` is now a method of - ``WorkingSet`` instead, and the ``best_match()`` method now uses a working - set instead of a path list as its second argument. - - * There is a new ``pkg_resources.add_activation_listener()`` API that lets - you register a callback for notifications about distributions added to - ``sys.path`` (including the distributions already on it). This is - basically a hook for extensible applications and frameworks to be able to - search for plugin metadata in distributions added at runtime. - -0.5a13 - * Fixed a bug in resource extraction from nested packages in a zipped egg. - -0.5a12 - * Updated extraction/cache mechanism for zipped resources to avoid inter- - process and inter-thread races during extraction. The default cache - location can now be set via the ``PYTHON_EGGS_CACHE`` environment variable, - and the default Windows cache is now a ``Python-Eggs`` subdirectory of the - current user's "Application Data" directory, if the ``PYTHON_EGGS_CACHE`` - variable isn't set. - -0.5a10 - * Fix a problem with ``pkg_resources`` being confused by non-existent eggs on - ``sys.path`` (e.g. if a user deletes an egg without removing it from the - ``easy-install.pth`` file). - - * Fix a problem with "basket" support in ``pkg_resources``, where egg-finding - never actually went inside ``.egg`` files. - - * Made ``pkg_resources`` import the module you request resources from, if it's - not already imported. - -0.5a4 - * ``pkg_resources.AvailableDistributions.resolve()`` and related methods now - accept an ``installer`` argument: a callable taking one argument, a - ``Requirement`` instance. The callable must return a ``Distribution`` - object, or ``None`` if no distribution is found. This feature is used by - EasyInstall to resolve dependencies by recursively invoking itself. - -0.4a4 - * Fix problems with ``resource_listdir()``, ``resource_isdir()`` and resource - directory extraction for zipped eggs. - -0.4a3 - * Fixed scripts not being able to see a ``__file__`` variable in ``__main__`` - - * Fixed a problem with ``resource_isdir()`` implementation that was introduced - in 0.4a2. - -0.4a1 - * Fixed a bug in requirements processing for exact versions (i.e. ``==`` and - ``!=``) when only one condition was included. - - * Added ``safe_name()`` and ``safe_version()`` APIs to clean up handling of - arbitrary distribution names and versions found on PyPI. - -0.3a4 - * ``pkg_resources`` now supports resource directories, not just the resources - in them. In particular, there are ``resource_listdir()`` and - ``resource_isdir()`` APIs. - - * ``pkg_resources`` now supports "egg baskets" -- .egg zipfiles which contain - multiple distributions in subdirectories whose names end with ``.egg``. - Having such a "basket" in a directory on ``sys.path`` is equivalent to - having the individual eggs in that directory, but the contained eggs can - be individually added (or not) to ``sys.path``. Currently, however, there - is no automated way to create baskets. - - * Namespace package manipulation is now protected by the Python import lock. - -0.3a1 - * Initial release. +XXX Starting fresh for 0.7 Modified: sandbox/trunk/setuptools/setup.py ============================================================================== --- sandbox/trunk/setuptools/setup.py (original) +++ sandbox/trunk/setuptools/setup.py Fri Apr 14 20:10:50 2006 @@ -19,7 +19,7 @@ d = {}; execfile(convert_path('setuptools/command/__init__.py'), d) SETUP_COMMANDS = d['__all__'] -VERSION = "0.6a12" +VERSION = "0.7a1" from setuptools import setup, find_packages import sys scripts = [] Modified: sandbox/trunk/setuptools/setuptools.txt ============================================================================== --- sandbox/trunk/setuptools/setuptools.txt (original) +++ sandbox/trunk/setuptools/setuptools.txt Fri Apr 14 20:10:50 2006 @@ -63,7 +63,10 @@ extensions, using simple "entry points" declared in a project's setup script. In addition to the PyPI downloads, the development version of ``setuptools`` -is available from the `Python SVN sandbox`_. +is available from the `Python SVN sandbox`_, and in-development versions of the +`0.6 branch`_ are available as well. + +.. _0.6 branch: http://svn.python.org/projects/sandbox/branches/setuptools-0.6/#egg=setuptools-dev06 .. _Python SVN sandbox: http://svn.python.org/projects/sandbox/trunk/setuptools/#egg=setuptools-dev @@ -1316,28 +1319,6 @@ other copies will ever be loaded!) -TRANSITIONAL NOTE -~~~~~~~~~~~~~~~~~ - -Setuptools 0.6a automatically calls ``declare_namespace()`` for you at runtime, -but the 0.7a versions will *not*. This is because the automatic declaration -feature has some negative side effects, such as needing to import all namespace -packages during the initialization of the ``pkg_resources`` runtime, and also -the need for ``pkg_resources`` to be explicitly imported before any namespace -packages work at all. Beginning with the 0.7a releases, you'll be responsible -for including your own declaration lines, and the automatic declaration feature -will be dropped to get rid of the negative side effects. - -During the remainder of the 0.6 development cycle, therefore, setuptools will -warn you about missing ``declare_namespace()`` calls in your ``__init__.py`` -files, and you should correct these as soon as possible before setuptools 0.7a1 -is released. Namespace packages without declaration lines will not work -correctly once a user has upgraded to setuptools 0.7a1, so it's important that -you make this change now in order to avoid having your code break in the field. -Our apologies for the inconvenience, and thank you for your patience. - - - Tagging and "Daily Build" or "Snapshot" Releases ------------------------------------------------ @@ -2495,341 +2476,7 @@ Release Notes/Change History ---------------------------- -0.6a11 - * Added ``test_loader`` keyword to support custom test loaders - - * Added ``setuptools.file_finders`` entry point group to allow implementing - revision control plugins. - - * Added ``--identity`` option to ``upload`` command. - - * Added ``dependency_links`` to allow specifying URLs for ``--find-links``. - - * Enhanced test loader to scan packages as well as modules, and call - ``additional_tests()`` if present to get non-unittest tests. - - * Support namespace packages in conjunction with system packagers, by omitting - the installation of any ``__init__.py`` files for namespace packages, and - adding a special ``.pth`` file to create a working package in - ``sys.modules``. - - * Made ``--single-version-externally-managed`` automatic when ``--root`` is - used, so that most system packagers won't require special support for - setuptools. - - * Fixed ``setup_requires``, ``tests_require``, etc. not using ``setup.cfg`` or - other configuration files for their option defaults when installing, and - also made the install use ``--multi-version`` mode so that the project - directory doesn't need to support .pth files. - - * ``MANIFEST.in`` is now forcibly closed when any errors occur while reading - it. Previously, the file could be left open and the actual error would be - masked by problems trying to remove the open file on Windows systems. - -0.6a10 - * Fixed the ``develop`` command ignoring ``--find-links``. - -0.6a9 - * The ``sdist`` command no longer uses the traditional ``MANIFEST`` file to - create source distributions. ``MANIFEST.in`` is still read and processed, - as are the standard defaults and pruning. But the manifest is built inside - the project's ``.egg-info`` directory as ``SOURCES.txt``, and it is rebuilt - every time the ``egg_info`` command is run. - - * Added the ``include_package_data`` keyword to ``setup()``, allowing you to - automatically include any package data listed in revision control or - ``MANIFEST.in`` - - * Added the ``exclude_package_data`` keyword to ``setup()``, allowing you to - trim back files included via the ``package_data`` and - ``include_package_data`` options. - - * Fixed ``--tag-svn-revision`` not working when run from a source - distribution. - - * Added warning for namespace packages with missing ``declare_namespace()`` - - * Added ``tests_require`` keyword to ``setup()``, so that e.g. packages - requiring ``nose`` to run unit tests can make this dependency optional - unless the ``test`` command is run. - - * Made all commands that use ``easy_install`` respect its configuration - options, as this was causing some problems with ``setup.py install``. - - * Added an ``unpack_directory()`` driver to ``setuptools.archive_util``, so - that you can process a directory tree through a processing filter as if it - were a zipfile or tarfile. - - * Added an internal ``install_egg_info`` command to use as part of old-style - ``install`` operations, that installs an ``.egg-info`` directory with the - package. - - * Added a ``--single-version-externally-managed`` option to the ``install`` - command so that you can more easily wrap a "flat" egg in a system package. - - * Enhanced ``bdist_rpm`` so that it installs single-version eggs that - don't rely on a ``.pth`` file. The ``--no-egg`` option has been removed, - since all RPMs are now built in a more backwards-compatible format. - - * Support full roundtrip translation of eggs to and from ``bdist_wininst`` - format. Running ``bdist_wininst`` on a setuptools-based package wraps the - egg in an .exe that will safely install it as an egg (i.e., with metadata - and entry-point wrapper scripts), and ``easy_install`` can turn the .exe - back into an ``.egg`` file or directory and install it as such. - - -0.6a8 - * Fixed some problems building extensions when Pyrex was installed, especially - with Python 2.4 and/or packages using SWIG. - - * Made ``develop`` command accept all the same options as ``easy_install``, - and use the ``easy_install`` command's configuration settings as defaults. - - * Made ``egg_info --tag-svn-revision`` fall back to extracting the revision - number from ``PKG-INFO`` in case it is being run on a source distribution of - a snapshot taken from a Subversion-based project. - - * Automatically detect ``.dll``, ``.so`` and ``.dylib`` files that are being - installed as data, adding them to ``native_libs.txt`` automatically. - - * Fixed some problems with fresh checkouts of projects that don't include - ``.egg-info/PKG-INFO`` under revision control and put the project's source - code directly in the project directory. If such a package had any - requirements that get processed before the ``egg_info`` command can be run, - the setup scripts would fail with a "Missing 'Version:' header and/or - PKG-INFO file" error, because the egg runtime interpreted the unbuilt - metadata in a directory on ``sys.path`` (i.e. the current directory) as - being a corrupted egg. Setuptools now monkeypatches the distribution - metadata cache to pretend that the egg has valid version information, until - it has a chance to make it actually be so (via the ``egg_info`` command). - -0.6a5 - * Fixed missing gui/cli .exe files in distribution. Fixed bugs in tests. - -0.6a3 - * Added ``gui_scripts`` entry point group to allow installing GUI scripts - on Windows and other platforms. (The special handling is only for Windows; - other platforms are treated the same as for ``console_scripts``.) - -0.6a2 - * Added ``console_scripts`` entry point group to allow installing scripts - without the need to create separate script files. On Windows, console - scripts get an ``.exe`` wrapper so you can just type their name. On other - platforms, the scripts are written without a file extension. - -0.6a1 - * Added support for building "old-style" RPMs that don't install an egg for - the target package, using a ``--no-egg`` option. - - * The ``build_ext`` command now works better when using the ``--inplace`` - option and multiple Python versions. It now makes sure that all extensions - match the current Python version, even if newer copies were built for a - different Python version. - - * The ``upload`` command no longer attaches an extra ``.zip`` when uploading - eggs, as PyPI now supports egg uploads without trickery. - - * The ``ez_setup`` script/module now displays a warning before downloading - the setuptools egg, and attempts to check the downloaded egg against an - internal MD5 checksum table. - - * Fixed the ``--tag-svn-revision`` option of ``egg_info`` not finding the - latest revision number; it was using the revision number of the directory - containing ``setup.py``, not the highest revision number in the project. - - * Added ``eager_resources`` setup argument - - * The ``sdist`` command now recognizes Subversion "deleted file" entries and - does not include them in source distributions. - - * ``setuptools`` now embeds itself more thoroughly into the distutils, so that - other distutils extensions (e.g. py2exe, py2app) will subclass setuptools' - versions of things, rather than the native distutils ones. - - * Added ``entry_points`` and ``setup_requires`` arguments to ``setup()``; - ``setup_requires`` allows you to automatically find and download packages - that are needed in order to *build* your project (as opposed to running it). - - * ``setuptools`` now finds its commands, ``setup()`` argument validators, and - metadata writers using entry points, so that they can be extended by - third-party packages. See `Creating distutils Extensions`_ above for more - details. - - * The vestigial ``depends`` command has been removed. It was never finished - or documented, and never would have worked without EasyInstall - which it - pre-dated and was never compatible with. - -0.5a12 - * The zip-safety scanner now checks for modules that might be used with - ``python -m``, and marks them as unsafe for zipping, since Python 2.4 can't - handle ``-m`` on zipped modules. - -0.5a11 - * Fix breakage of the "develop" command that was caused by the addition of - ``--always-unzip`` to the ``easy_install`` command. - -0.5a9 - * Include ``svn:externals`` directories in source distributions as well as - normal subversion-controlled files and directories. - - * Added ``exclude=patternlist`` option to ``setuptools.find_packages()`` - - * Changed --tag-svn-revision to include an "r" in front of the revision number - for better readability. - - * Added ability to build eggs without including source files (except for any - scripts, of course), using the ``--exclude-source-files`` option to - ``bdist_egg``. - - * ``setup.py install`` now automatically detects when an "unmanaged" package - or module is going to be on ``sys.path`` ahead of a package being installed, - thereby preventing the newer version from being imported. If this occurs, - a warning message is output to ``sys.stderr``, but installation proceeds - anyway. The warning message informs the user what files or directories - need deleting, and advises them they can also use EasyInstall (with the - ``--delete-conflicting`` option) to do it automatically. - - * The ``egg_info`` command now adds a ``top_level.txt`` file to the metadata - directory that lists all top-level modules and packages in the distribution. - This is used by the ``easy_install`` command to find possibly-conflicting - "unmanaged" packages when installing the distribution. - - * Added ``zip_safe`` and ``namespace_packages`` arguments to ``setup()``. - Added package analysis to determine zip-safety if the ``zip_safe`` flag - is not given, and advise the author regarding what code might need changing. - - * Fixed the swapped ``-d`` and ``-b`` options of ``bdist_egg``. - -0.5a8 - * The "egg_info" command now always sets the distribution metadata to "safe" - forms of the distribution name and version, so that distribution files will - be generated with parseable names (i.e., ones that don't include '-' in the - name or version). Also, this means that if you use the various ``--tag`` - options of "egg_info", any distributions generated will use the tags in the - version, not just egg distributions. - - * Added support for defining command aliases in distutils configuration files, - under the "[aliases]" section. To prevent recursion and to allow aliases to - call the command of the same name, a given alias can be expanded only once - per command-line invocation. You can define new aliases with the "alias" - command, either for the local, global, or per-user configuration. - - * Added "rotate" command to delete old distribution files, given a set of - patterns to match and the number of files to keep. (Keeps the most - recently-modified distribution files matching each pattern.) - - * Added "saveopts" command that saves all command-line options for the current - invocation to the local, global, or per-user configuration file. Useful for - setting defaults without having to hand-edit a configuration file. - - * Added a "setopt" command that sets a single option in a specified distutils - configuration file. - -0.5a7 - * Added "upload" support for egg and source distributions, including a bug - fix for "upload" and a temporary workaround for lack of .egg support in - PyPI. - -0.5a6 - * Beefed up the "sdist" command so that if you don't have a MANIFEST.in, it - will include all files under revision control (CVS or Subversion) in the - current directory, and it will regenerate the list every time you create a - source distribution, not just when you tell it to. This should make the - default "do what you mean" more often than the distutils' default behavior - did, while still retaining the old behavior in the presence of MANIFEST.in. - - * Fixed the "develop" command always updating .pth files, even if you - specified ``-n`` or ``--dry-run``. - - * Slightly changed the format of the generated version when you use - ``--tag-build`` on the "egg_info" command, so that you can make tagged - revisions compare *lower* than the version specified in setup.py (e.g. by - using ``--tag-build=dev``). - -0.5a5 - * Added ``develop`` command to ``setuptools``-based packages. This command - installs an ``.egg-link`` pointing to the package's source directory, and - script wrappers that ``execfile()`` the source versions of the package's - scripts. This lets you put your development checkout(s) on sys.path without - having to actually install them. (To uninstall the link, use - use ``setup.py develop --uninstall``.) - - * Added ``egg_info`` command to ``setuptools``-based packages. This command - just creates or updates the "projectname.egg-info" directory, without - building an egg. (It's used by the ``bdist_egg``, ``test``, and ``develop`` - commands.) - - * Enhanced the ``test`` command so that it doesn't install the package, but - instead builds any C extensions in-place, updates the ``.egg-info`` - metadata, adds the source directory to ``sys.path``, and runs the tests - directly on the source. This avoids an "unmanaged" installation of the - package to ``site-packages`` or elsewhere. - - * Made ``easy_install`` a standard ``setuptools`` command, moving it from - the ``easy_install`` module to ``setuptools.command.easy_install``. Note - that if you were importing or extending it, you must now change your imports - accordingly. ``easy_install.py`` is still installed as a script, but not as - a module. - -0.5a4 - * Setup scripts using setuptools can now list their dependencies directly in - the setup.py file, without having to manually create a ``depends.txt`` file. - The ``install_requires`` and ``extras_require`` arguments to ``setup()`` - are used to create a dependencies file automatically. If you are manually - creating ``depends.txt`` right now, please switch to using these setup - arguments as soon as practical, because ``depends.txt`` support will be - removed in the 0.6 release cycle. For documentation on the new arguments, - see the ``setuptools.dist.Distribution`` class. - - * Setup scripts using setuptools now always install using ``easy_install`` - internally, for ease of uninstallation and upgrading. - -0.5a1 - * Added support for "self-installation" bootstrapping. Packages can now - include ``ez_setup.py`` in their source distribution, and add the following - to their ``setup.py``, in order to automatically bootstrap installation of - setuptools as part of their setup process:: - - from ez_setup import use_setuptools - use_setuptools() - - from setuptools import setup - # etc... - -0.4a2 - * Added ``ez_setup.py`` installer/bootstrap script to make initial setuptools - installation easier, and to allow distributions using setuptools to avoid - having to include setuptools in their source distribution. - - * All downloads are now managed by the ``PackageIndex`` class (which is now - subclassable and replaceable), so that embedders can more easily override - download logic, give download progress reports, etc. The class has also - been moved to the new ``setuptools.package_index`` module. - - * The ``Installer`` class no longer handles downloading, manages a temporary - directory, or tracks the ``zip_ok`` option. Downloading is now handled - by ``PackageIndex``, and ``Installer`` has become an ``easy_install`` - command class based on ``setuptools.Command``. - - * There is a new ``setuptools.sandbox.run_setup()`` API to invoke a setup - script in a directory sandbox, and a new ``setuptools.archive_util`` module - with an ``unpack_archive()`` API. These were split out of EasyInstall to - allow reuse by other tools and applications. - - * ``setuptools.Command`` now supports reinitializing commands using keyword - arguments to set/reset options. Also, ``Command`` subclasses can now set - their ``command_consumes_arguments`` attribute to ``True`` in order to - receive an ``args`` option containing the rest of the command line. - -0.3a2 - * Added new options to ``bdist_egg`` to allow tagging the egg's version number - with a subversion revision number, the current date, or an explicit tag - value. Run ``setup.py bdist_egg --help`` to get more information. - - * Misc. bug fixes - -0.3a1 - * Initial release. +XXX starting fresh for 0.7 Mailing list Modified: sandbox/trunk/setuptools/setuptools/__init__.py ============================================================================== --- sandbox/trunk/setuptools/setuptools/__init__.py (original) +++ sandbox/trunk/setuptools/setuptools/__init__.py Fri Apr 14 20:10:50 2006 @@ -7,7 +7,7 @@ from distutils.util import convert_path import os.path -__version__ = '0.6a12' +__version__ = '0.7a1' __all__ = [ 'setup', 'Distribution', 'Feature', 'Command', 'Extension', 'Require', 'find_packages' From python-checkins at python.org Fri Apr 14 20:25:40 2006 From: python-checkins at python.org (walter.doerwald) Date: Fri, 14 Apr 2006 20:25:40 +0200 (CEST) Subject: [Python-checkins] r45401 - in python/trunk/Lib: codecs.py encodings/idna.py test/test_codecs.py Message-ID: <20060414182540.5CA661E402A@bag.python.org> Author: walter.doerwald Date: Fri Apr 14 20:25:39 2006 New Revision: 45401 Modified: python/trunk/Lib/codecs.py python/trunk/Lib/encodings/idna.py python/trunk/Lib/test/test_codecs.py Log: Add a BufferedIncrementalEncoder class that can be used for implementing an incremental encoder that must retain part of the data between calls to the encode() method. Fix the incremental encoder and decoder for the IDNA encoding. This closes SF patch #1453235. Modified: python/trunk/Lib/codecs.py ============================================================================== --- python/trunk/Lib/codecs.py (original) +++ python/trunk/Lib/codecs.py Fri Apr 14 20:25:39 2006 @@ -181,6 +181,33 @@ Resets the encoder to the initial state. """ +class BufferedIncrementalEncoder(IncrementalEncoder): + """ + This subclass of IncrementalEncoder can be used as the baseclass for an + incremental encoder if the encoder must keep some of the output in a + buffer between calls to encode(). + """ + def __init__(self, errors='strict'): + IncrementalEncoder.__init__(self, errors) + self.buffer = "" # unencoded input that is kept between calls to encode() + + def _buffer_encode(self, input, errors, final): + # Overwrite this method in subclasses: It must encode input + # and return an (output, length consumed) tuple + raise NotImplementedError + + def encode(self, input, final=False): + # encode input (taking the buffer into account) + data = self.buffer + input + (result, consumed) = self._buffer_encode(data, self.errors, final) + # keep unencoded input until the next call + self.buffer = data[consumed:] + return result + + def reset(self): + IncrementalEncoder.reset(self) + self.buffer = "" + class IncrementalDecoder(object): """ An IncrementalDecoder decodes an input in multiple steps. The input can be Modified: python/trunk/Lib/encodings/idna.py ============================================================================== --- python/trunk/Lib/encodings/idna.py (original) +++ python/trunk/Lib/encodings/idna.py Fri Apr 14 20:25:39 2006 @@ -194,13 +194,79 @@ return u".".join(result)+trailing_dot, len(input) -class IncrementalEncoder(codecs.IncrementalEncoder): - def encode(self, input, final=False): - return Codec().encode(input, self.errors)[0] - -class IncrementalDecoder(codecs.IncrementalDecoder): - def decode(self, input, final=False): - return Codec().decode(input, self.errors)[0] +class IncrementalEncoder(codecs.BufferedIncrementalEncoder): + def _buffer_encode(self, input, errors, final): + if errors != 'strict': + # IDNA is quite clear that implementations must be strict + raise UnicodeError("unsupported error handling "+errors) + + if not input: + return ("", 0) + + labels = dots.split(input) + trailing_dot = u'' + if labels: + if not labels[-1]: + trailing_dot = '.' + del labels[-1] + elif not final: + # Keep potentially unfinished label until the next call + del labels[-1] + if labels: + trailing_dot = '.' + + result = [] + size = 0 + for label in labels: + result.append(ToASCII(label)) + if size: + size += 1 + size += len(label) + + # Join with U+002E + result = ".".join(result) + trailing_dot + size += len(trailing_dot) + return (result, size) + +class IncrementalDecoder(codecs.BufferedIncrementalDecoder): + def _buffer_decode(self, input, errors, final): + if errors != 'strict': + raise UnicodeError("Unsupported error handling "+errors) + + if not input: + return (u"", 0) + + # IDNA allows decoding to operate on Unicode strings, too. + if isinstance(input, unicode): + labels = dots.split(input) + else: + # Must be ASCII string + input = str(input) + unicode(input, "ascii") + labels = input.split(".") + + trailing_dot = u'' + if labels: + if not labels[-1]: + trailing_dot = u'.' + del labels[-1] + elif not final: + # Keep potentially unfinished label until the next call + del labels[-1] + if labels: + trailing_dot = u'.' + + result = [] + size = 0 + for label in labels: + result.append(ToUnicode(label)) + if size: + size += 1 + size += len(label) + + result = u".".join(result) + trailing_dot + size += len(trailing_dot) + return (result, size) class StreamWriter(Codec,codecs.StreamWriter): pass Modified: python/trunk/Lib/test/test_codecs.py ============================================================================== --- python/trunk/Lib/test/test_codecs.py (original) +++ python/trunk/Lib/test/test_codecs.py Fri Apr 14 20:25:39 2006 @@ -781,9 +781,18 @@ except Exception,e: raise test_support.TestFailed("Test 3.%d: %s" % (pos+1, str(e))) -class CodecTest(unittest.TestCase): - def test_builtin(self): +class IDNACodecTest(unittest.TestCase): + def test_builtin_decode(self): self.assertEquals(unicode("python.org", "idna"), u"python.org") + self.assertEquals(unicode("python.org.", "idna"), u"python.org.") + self.assertEquals(unicode("xn--pythn-mua.org", "idna"), u"pyth\xf6n.org") + self.assertEquals(unicode("xn--pythn-mua.org.", "idna"), u"pyth\xf6n.org.") + + def test_builtin_encode(self): + self.assertEquals(u"python.org".encode("idna"), "python.org") + self.assertEquals("python.org.".encode("idna"), "python.org.") + self.assertEquals(u"pyth\xf6n.org".encode("idna"), "xn--pythn-mua.org") + self.assertEquals(u"pyth\xf6n.org.".encode("idna"), "xn--pythn-mua.org.") def test_stream(self): import StringIO @@ -791,6 +800,64 @@ r.read(3) self.assertEquals(r.read(), u"") + def test_incremental_decode(self): + self.assertEquals( + "".join(codecs.iterdecode("python.org", "idna")), + u"python.org" + ) + self.assertEquals( + "".join(codecs.iterdecode("python.org.", "idna")), + u"python.org." + ) + self.assertEquals( + "".join(codecs.iterdecode("xn--pythn-mua.org.", "idna")), + u"pyth\xf6n.org." + ) + self.assertEquals( + "".join(codecs.iterdecode("xn--pythn-mua.org.", "idna")), + u"pyth\xf6n.org." + ) + + decoder = codecs.getincrementaldecoder("idna")() + self.assertEquals(decoder.decode("xn--xam", ), u"") + self.assertEquals(decoder.decode("ple-9ta.o", ), u"\xe4xample.") + self.assertEquals(decoder.decode(u"rg"), u"") + self.assertEquals(decoder.decode(u"", True), u"org") + + decoder.reset() + self.assertEquals(decoder.decode("xn--xam", ), u"") + self.assertEquals(decoder.decode("ple-9ta.o", ), u"\xe4xample.") + self.assertEquals(decoder.decode("rg."), u"org.") + self.assertEquals(decoder.decode("", True), u"") + + def test_incremental_encode(self): + self.assertEquals( + "".join(codecs.iterencode(u"python.org", "idna")), + "python.org" + ) + self.assertEquals( + "".join(codecs.iterencode(u"python.org.", "idna")), + "python.org." + ) + self.assertEquals( + "".join(codecs.iterencode(u"pyth\xf6n.org.", "idna")), + "xn--pythn-mua.org." + ) + self.assertEquals( + "".join(codecs.iterencode(u"pyth\xf6n.org.", "idna")), + "xn--pythn-mua.org." + ) + + encoder = codecs.getincrementalencoder("idna")() + self.assertEquals(encoder.encode(u"\xe4x"), "") + self.assertEquals(encoder.encode(u"ample.org"), "xn--xample-9ta.") + self.assertEquals(encoder.encode(u"", True), "org") + + encoder.reset() + self.assertEquals(encoder.encode(u"\xe4x"), "") + self.assertEquals(encoder.encode(u"ample.org."), "xn--xample-9ta.org.") + self.assertEquals(encoder.encode(u"", True), "") + class CodecsModuleTest(unittest.TestCase): def test_decode(self): @@ -1158,7 +1225,7 @@ PunycodeTest, UnicodeInternalTest, NameprepTest, - CodecTest, + IDNACodecTest, CodecsModuleTest, StreamReaderTest, Str2StrTest, From python-checkins at python.org Fri Apr 14 20:34:14 2006 From: python-checkins at python.org (tim.peters) Date: Fri, 14 Apr 2006 20:34:14 +0200 (CEST) Subject: [Python-checkins] r45402 - python/trunk/Lib/test/leakers/test_tee.py Message-ID: <20060414183414.B877F1E4005@bag.python.org> Author: tim.peters Date: Fri Apr 14 20:34:14 2006 New Revision: 45402 Modified: python/trunk/Lib/test/leakers/test_tee.py Log: Whitespace normalization. Modified: python/trunk/Lib/test/leakers/test_tee.py ============================================================================== --- python/trunk/Lib/test/leakers/test_tee.py (original) +++ python/trunk/Lib/test/leakers/test_tee.py Fri Apr 14 20:34:14 2006 @@ -6,20 +6,20 @@ import gc def leak(): - def inner(): - def fib(): - def yield_identity_forever(g): - while 1: - yield g - def _fib(): - for i in yield_identity_forever(head): - yield i - head, tail, result = tee(_fib(), 3) - return result + def inner(): + def fib(): + def yield_identity_forever(g): + while 1: + yield g + def _fib(): + for i in yield_identity_forever(head): + yield i + head, tail, result = tee(_fib(), 3) + return result - x = fib() - x.next() - inner() - gc.collect() ; gc.collect() - # this is expected to return 0 - return gc.collect() + x = fib() + x.next() + inner() + gc.collect() ; gc.collect() + # this is expected to return 0 + return gc.collect() From python-checkins at python.org Fri Apr 14 21:13:26 2006 From: python-checkins at python.org (phillip.eby) Date: Fri, 14 Apr 2006 21:13:26 +0200 (CEST) Subject: [Python-checkins] r45403 - in sandbox/trunk/setuptools: pkg_resources.py pkg_resources.txt setuptools.txt setuptools/command/build_py.py Message-ID: <20060414191326.19FB61E4005@bag.python.org> Author: phillip.eby Date: Fri Apr 14 21:13:24 2006 New Revision: 45403 Modified: sandbox/trunk/setuptools/pkg_resources.py sandbox/trunk/setuptools/pkg_resources.txt sandbox/trunk/setuptools/setuptools.txt sandbox/trunk/setuptools/setuptools/command/build_py.py Log: Don't eagerly import namespace packages. This was the big reason for branching to 0.7 now, as I wanted this wart gone before anything went into Python 2.5. But it's gone now, yay! Modified: sandbox/trunk/setuptools/pkg_resources.py ============================================================================== --- sandbox/trunk/setuptools/pkg_resources.py (original) +++ sandbox/trunk/setuptools/pkg_resources.py Fri Apr 14 21:13:24 2006 @@ -2045,8 +2045,8 @@ self.insert_on(path) if path is sys.path: fixup_namespace_packages(self.location) - map(declare_namespace, self._get_metadata('namespace_packages.txt')) - + for pkg in self._get_metadata('namespace_packages.txt'): + if pkg in sys.modules: declare_namespace(pkg) def egg_name(self): """Return what this distribution's standard .egg filename should be""" Modified: sandbox/trunk/setuptools/pkg_resources.txt ============================================================================== --- sandbox/trunk/setuptools/pkg_resources.txt (original) +++ sandbox/trunk/setuptools/pkg_resources.txt Fri Apr 14 21:13:24 2006 @@ -121,13 +121,16 @@ A namespace package is a package that only contains other packages and modules, with no direct contents of its own. Such packages can be split across -multiple, separately-packaged distributions. Normally, you do not need to use -the namespace package APIs directly; instead you should supply the -``namespace_packages`` argument to ``setup()`` in your project's ``setup.py``. -See the `setuptools documentation on namespace packages`_ for more information. - -However, if for some reason you need to manipulate namespace packages or -directly alter ``sys.path`` at runtime, you may find these APIs useful: +multiple, separately-packaged distributions. They are normally used to split +up large packages produced by a single organization, such as in the ``zope`` +namespace package for Zope Corporation packages, and the ``peak`` namespace +package for the Python Enterprise Application Kit. + +To create a namespace package, you list it in the ``namespace_packages`` +argument to ``setup()``, in your project's ``setup.py``. (See the `setuptools +documentation on namespace packages`_ for more information on this.) Also, +you must add a ``declare_namespace()`` call in the package's ``__init__.py`` +file(s): ``declare_namespace(name)`` Declare that the dotted package name `name` is a "namespace package" whose @@ -140,6 +143,9 @@ is invoked, it checks for the presence of namespace packages and updates their ``__path__`` contents accordingly. +Applications that manipulate namespace packages or directly alter ``sys.path`` +at runtime may also need to use this API function: + ``fixup_namespace_packages(path_item)`` Declare that `path_item` is a newly added item on ``sys.path`` that may need to be used to update existing namespace packages. Ordinarily, this is @@ -1652,5 +1658,10 @@ Release Notes/Change History ---------------------------- -XXX Starting fresh for 0.7 +0.7a1 + * Namespace packages are now imported lazily. That is, the mere declaration + of a namespace package in an egg on ``sys.path`` no longer causes it to be + imported when ``pkg_resources`` is imported. Note that this means that all + of a namespace package's ``__init__.py`` files must include a + ``declare_namespace()`` call. Modified: sandbox/trunk/setuptools/setuptools.txt ============================================================================== --- sandbox/trunk/setuptools/setuptools.txt (original) +++ sandbox/trunk/setuptools/setuptools.txt Fri Apr 14 21:13:24 2006 @@ -2476,7 +2476,11 @@ Release Notes/Change History ---------------------------- -XXX starting fresh for 0.7 +0.7a1 + * Namespace packages' ``__init__.py`` files MUST use ``declare_namespace()`` + in order to ensure that they will be handled properly at runtime. (In 0.6, + it was possible to get away without doing this, but only at the cost of + forcing namespace packages to be imported early, which 0.7 does not do.) Mailing list Modified: sandbox/trunk/setuptools/setuptools/command/build_py.py ============================================================================== --- sandbox/trunk/setuptools/setuptools/command/build_py.py (original) +++ sandbox/trunk/setuptools/setuptools/command/build_py.py Fri Apr 14 21:13:24 2006 @@ -142,12 +142,12 @@ f = open(init_py,'rU') if 'declare_namespace' not in f.read(): - from distutils import log - log.warn( - "WARNING: %s is a namespace package, but its __init__.py does\n" - "not declare_namespace(); setuptools 0.7 will REQUIRE this!\n" - '(See the setuptools manual under "Namespace Packages" for ' - "details.)\n", package + from distutils.errors import DistutilsError + raise DistutilsError( + "Namespace package problem: %s is a namespace package, but its\n" + "__init__.py does not call declare_namespace()! Please fix it.\n" + '(See the setuptools manual under "Namespace Packages" for ' + "details.)\n" % (package,) ) f.close() return init_py From python-checkins at python.org Fri Apr 14 21:17:38 2006 From: python-checkins at python.org (phillip.eby) Date: Fri, 14 Apr 2006 21:17:38 +0200 (CEST) Subject: [Python-checkins] r45404 - sandbox/trunk/setuptools/setuptools.txt Message-ID: <20060414191738.6E6141E4005@bag.python.org> Author: phillip.eby Date: Fri Apr 14 21:17:37 2006 New Revision: 45404 Modified: sandbox/trunk/setuptools/setuptools.txt Log: Namespace package doc tweaks. Modified: sandbox/trunk/setuptools/setuptools.txt ============================================================================== --- sandbox/trunk/setuptools/setuptools.txt (original) +++ sandbox/trunk/setuptools/setuptools.txt Fri Apr 14 21:17:37 2006 @@ -338,8 +338,8 @@ merge such subpackages into a single parent package at runtime, as long as you declare them in each project that contains any subpackages of the namespace package, and as long as the namespace package's ``__init__.py`` - does not contain any code. See the section below on `Namespace Packages`_ - for more information. + does not contain any code other than a namespace declaration. See the + section below on `Namespace Packages`_ for more information. ``test_suite`` A string naming a ``unittest.TestCase`` subclass (or a package or module @@ -1316,7 +1316,7 @@ order to ensure that the namespace will be declared regardless of which project's copy of ``__init__.py`` is loaded first. If the first loaded ``__init__.py`` doesn't declare it, it will never *be* declared, because no -other copies will ever be loaded!) +other copies will ever be loaded! Tagging and "Daily Build" or "Snapshot" Releases From python-checkins at python.org Fri Apr 14 21:38:39 2006 From: python-checkins at python.org (phillip.eby) Date: Fri, 14 Apr 2006 21:38:39 +0200 (CEST) Subject: [Python-checkins] r45405 - in sandbox/trunk/setuptools: EasyInstall.txt easy_install.py setup.py setuptools/command/__init__.py setuptools/command/easy_install.py setuptools/site-patch.py site.py Message-ID: <20060414193839.55B611E4005@bag.python.org> Author: phillip.eby Date: Fri Apr 14 21:38:38 2006 New Revision: 45405 Added: sandbox/trunk/setuptools/setuptools/site-patch.py - copied unchanged from r45281, sandbox/trunk/setuptools/site.py Removed: sandbox/trunk/setuptools/site.py Modified: sandbox/trunk/setuptools/EasyInstall.txt sandbox/trunk/setuptools/easy_install.py sandbox/trunk/setuptools/setup.py sandbox/trunk/setuptools/setuptools/command/__init__.py sandbox/trunk/setuptools/setuptools/command/easy_install.py Log: First round of prepping setuptools for inclusion in Python 2.5: move site.py to setuptools/site-patch.py; reinstate 'python -m easy_install' support; use distutils' "upload" command when running under 2.5. Modified: sandbox/trunk/setuptools/EasyInstall.txt ============================================================================== --- sandbox/trunk/setuptools/EasyInstall.txt (original) +++ sandbox/trunk/setuptools/EasyInstall.txt Fri Apr 14 21:38:38 2006 @@ -331,6 +331,10 @@ 2.4, you can use the ``easy_install-2.3`` or ``easy_install-2.4`` scripts to install packages for Python 2.3 or 2.4, respectively. +Also, if you're working with Python version 2.4 or higher, you can run Python +with ``-m easy_install`` to run that particular Python version's +``easy_install`` command. + Restricting Downloads with ``--allow-hosts`` ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -1097,6 +1101,8 @@ ============================ 0.7a1 + * You can now use "python -m easy_install" with Python 2.4 and above. + * Added automatic retry for Sourceforge mirrors. The new download process is to first just try dl.sourceforge.net, then randomly select mirror IPs and remove ones that fail, until something works. The removed IPs stay removed Modified: sandbox/trunk/setuptools/easy_install.py ============================================================================== --- sandbox/trunk/setuptools/easy_install.py (original) +++ sandbox/trunk/setuptools/easy_install.py Fri Apr 14 21:38:38 2006 @@ -1,15 +1,6 @@ -#!python -"""\ -This script/module exists for backward compatibility only! It will go away -entirely in 0.7. Please start using the 'easy_install' script or .exe instead -of using 'python -m easy_install' or running 'easy_install.py' directly. -""" +"""Run the EasyInstall command""" if __name__ == '__main__': - import sys - print >>sys.stderr, \ - "Please use the 'easy_install' script or executable instead." - print >>sys.stderr, \ - "(i.e., don't include the '.py' extension and don't use 'python -m')" - sys.exit(2) + from setuptools.command.easy_install import main + main() Modified: sandbox/trunk/setuptools/setup.py ============================================================================== --- sandbox/trunk/setuptools/setup.py (original) +++ sandbox/trunk/setuptools/setup.py Fri Apr 14 21:38:38 2006 @@ -23,8 +23,6 @@ from setuptools import setup, find_packages import sys scripts = [] -if sys.platform != "win32": - scripts = ["easy_install.py"] # for backward compatibility only setup( name="setuptools", @@ -38,21 +36,16 @@ keywords = "CPAN PyPI distutils eggs package management", url = "http://peak.telecommunity.com/DevCenter/setuptools", test_suite = 'setuptools.tests', - packages = find_packages(), - package_data = {'setuptools':['*.exe']}, - - py_modules = ['pkg_resources', 'easy_install', 'site'], - - zip_safe = False, # We want 'python -m easy_install' to work, for now :( + package_data = {'setuptools':['*.exe','site-patch.py']}, + py_modules = ['pkg_resources','easy_install'], + zip_safe = (sys.version>="2.5"), # <2.5 needs unzipped for -m to work entry_points = { - "distutils.commands" : [ "%(cmd)s = setuptools.command.%(cmd)s:%(cmd)s" % locals() for cmd in SETUP_COMMANDS ], - "distutils.setup_keywords": [ "eager_resources = setuptools.dist:assert_string_list", "namespace_packages = setuptools.dist:check_nsp", @@ -68,7 +61,6 @@ "dependency_links = setuptools.dist:assert_string_list", "test_loader = setuptools.dist:check_importable", ], - "egg_info.writers": [ "PKG-INFO = setuptools.command.egg_info:write_pkg_info", "requires.txt = setuptools.command.egg_info:write_requirements", @@ -79,16 +71,14 @@ "depends.txt = setuptools.command.egg_info:warn_depends_obsolete", "dependency_links.txt = setuptools.command.egg_info:overwrite_arg", ], - "console_scripts": [ "easy_install = setuptools.command.easy_install:main", "easy_install-%s = setuptools.command.easy_install:main" % sys.version[:3] - ], - + ], "setuptools.file_finders": ["svn_cvs = setuptools.command.sdist:_default_revctrl"] - }, + }, classifiers = [f.strip() for f in """ Development Status :: 3 - Alpha @@ -121,3 +111,13 @@ + + + + + + + + + + Modified: sandbox/trunk/setuptools/setuptools/command/__init__.py ============================================================================== --- sandbox/trunk/setuptools/setuptools/command/__init__.py (original) +++ sandbox/trunk/setuptools/setuptools/command/__init__.py Fri Apr 14 21:38:38 2006 @@ -4,6 +4,11 @@ 'sdist', 'setopt', 'test', 'upload', 'install_egg_info', 'install_scripts', ] +import sys +if sys.version>='2.5': + # In Python 2.5 and above, distutils includes its own upload command + __all__.remove('upload') + from distutils.command.bdist import bdist @@ -11,4 +16,4 @@ bdist.format_command['egg'] = ('bdist_egg', "Python .egg file") bdist.format_commands.append('egg') -del bdist +del bdist, sys Modified: sandbox/trunk/setuptools/setuptools/command/easy_install.py ============================================================================== --- sandbox/trunk/setuptools/setuptools/command/easy_install.py (original) +++ sandbox/trunk/setuptools/setuptools/command/easy_install.py Fri Apr 14 21:38:38 2006 @@ -1073,7 +1073,7 @@ return # already did it, or don't need to sitepy = os.path.join(self.install_dir, "site.py") - source = resource_string(Requirement.parse("setuptools"), "site.py") + source = resource_string("setuptools", "site-patch.py") current = "" if os.path.exists(sitepy): Deleted: /sandbox/trunk/setuptools/site.py ============================================================================== --- /sandbox/trunk/setuptools/site.py Fri Apr 14 21:38:38 2006 +++ (empty file) @@ -1,82 +0,0 @@ -def __boot(): - import sys, imp, os, os.path - PYTHONPATH = os.environ.get('PYTHONPATH') - if PYTHONPATH is None or (sys.platform=='win32' and not PYTHONPATH): - PYTHONPATH = [] - else: - PYTHONPATH = PYTHONPATH.split(os.pathsep) - - pic = getattr(sys,'path_importer_cache',{}) - stdpath = sys.path[len(PYTHONPATH):] - mydir = os.path.dirname(__file__) - #print "searching",stdpath,sys.path - - for item in stdpath: - if item==mydir or not item: - continue # skip if current dir. on Windows, or my own directory - importer = pic.get(item) - if importer is not None: - loader = importer.find_module('site') - if loader is not None: - # This should actually reload the current module - loader.load_module('site') - break - else: - try: - stream, path, descr = imp.find_module('site',[item]) - except ImportError: - continue - if stream is None: - continue - try: - # This should actually reload the current module - imp.load_module('site',stream,path,descr) - finally: - stream.close() - break - else: - raise ImportError("Couldn't find the real 'site' module") - - #print "loaded", __file__ - - known_paths = dict([(makepath(item)[1],1) for item in sys.path]) # 2.2 comp - - oldpos = getattr(sys,'__egginsert',0) # save old insertion position - sys.__egginsert = 0 # and reset the current one - - for item in PYTHONPATH: - addsitedir(item) - - sys.__egginsert += oldpos # restore effective old position - - d,nd = makepath(stdpath[0]) - insert_at = None - new_path = [] - - for item in sys.path: - p,np = makepath(item) - - if np==nd and insert_at is None: - # We've hit the first 'system' path entry, so added entries go here - insert_at = len(new_path) - - if np in known_paths or insert_at is None: - new_path.append(item) - else: - # new path after the insert point, back-insert it - new_path.insert(insert_at, item) - insert_at += 1 - - sys.path[:] = new_path - -if __name__=='site': - __boot() - del __boot - - - - - - - - From python-checkins at python.org Fri Apr 14 22:32:37 2006 From: python-checkins at python.org (andrew.kuchling) Date: Fri, 14 Apr 2006 22:32:37 +0200 (CEST) Subject: [Python-checkins] r45406 - python/trunk/Doc/lib/libcollections.tex Message-ID: <20060414203237.420241E4005@bag.python.org> Author: andrew.kuchling Date: Fri Apr 14 22:32:36 2006 New Revision: 45406 Modified: python/trunk/Doc/lib/libcollections.tex Log: Typo fix Modified: python/trunk/Doc/lib/libcollections.tex ============================================================================== --- python/trunk/Doc/lib/libcollections.tex (original) +++ python/trunk/Doc/lib/libcollections.tex Fri Apr 14 22:32:36 2006 @@ -310,7 +310,7 @@ When a letter is first encountered, it is missing from the mapping, so the \member{default_factory} function calls \function{int()} to supply a default -count of zero. The increment operation then builds of the count for each +count of zero. The increment operation then builds up the count for each letter. This technique makes counting simpler and faster than an equivalent technique using \method{dict.get()}: From python-checkins at python.org Fri Apr 14 22:35:17 2006 From: python-checkins at python.org (andrew.kuchling) Date: Fri, 14 Apr 2006 22:35:17 +0200 (CEST) Subject: [Python-checkins] r45407 - python/trunk/Doc/whatsnew/whatsnew25.tex Message-ID: <20060414203517.7424D1E4005@bag.python.org> Author: andrew.kuchling Date: Fri Apr 14 22:35:17 2006 New Revision: 45407 Modified: python/trunk/Doc/whatsnew/whatsnew25.tex Log: Add an item; better crediting; fix error in SQL example; minor edits Modified: python/trunk/Doc/whatsnew/whatsnew25.tex ============================================================================== --- python/trunk/Doc/whatsnew/whatsnew25.tex (original) +++ python/trunk/Doc/whatsnew/whatsnew25.tex Fri Apr 14 22:35:17 2006 @@ -2,10 +2,10 @@ \usepackage{distutils} % $Id$ -% Fix XXX comments +% Writing context managers % The easy_install stuff % Stateful codec changes -% cProfile +% Fix XXX comments % Count up the patches and bugs \title{What's New in Python 2.5} @@ -1400,7 +1400,8 @@ %====================================================================== \subsection{The hashlib package} -A new \module{hashlib} module has been added to replace the +A new \module{hashlib} module, written by Gregory P. Smith, +has been added to replace the \module{md5} and \module{sha} modules. \module{hashlib} adds support for additional secure hashes (SHA-224, SHA-256, SHA-384, and SHA-512). When available, the module uses OpenSSL for fast platform optimized @@ -1443,26 +1444,25 @@ return the digest value as a binary string or a string of hex digits, and \method{copy()} returns a new hashing object with the same digest state. -This module was contributed by Gregory P. Smith. - %====================================================================== \subsection{The sqlite3 package} The pysqlite module (\url{http://www.pysqlite.org}), a wrapper for the SQLite embedded database, has been added to the standard library under -the package name \module{sqlite3}. SQLite is a C library that -provides a SQL-language database that stores data in disk files -without requiring a separate server process. pysqlite was written by -Gerhard H\"aring, and provides a SQL interface that complies with the -DB-API 2.0 specification described by \pep{249}. This means that it -should be possible to write the first version of your applications -using SQLite for data storage and, if switching to a larger database -such as PostgreSQL or Oracle is necessary, the switch should be -relatively easy. +the package name \module{sqlite3}. + +SQLite is a C library that provides a SQL-language database that +stores data in disk files without requiring a separate server process. +pysqlite was written by Gerhard H\"aring and provides a SQL interface +compliant with the DB-API 2.0 specification described by +\pep{249}. This means that it should be possible to write the first +version of your applications using SQLite for data storage. If +switching to a larger database such as PostgreSQL or Oracle is +later necessary, the switch should be relatively easy. If you're compiling the Python source yourself, note that the source -tree doesn't include the SQLite code itself, only the wrapper module. +tree doesn't include the SQLite code, only the wrapper module. You'll need to have the SQLite libraries and headers installed before compiling Python, and the build process will compile the module when the necessary headers are available. @@ -1491,17 +1491,18 @@ # Insert a row of data c.execute("""insert into stocks - values ('2006-01-05','BUY','RHAT',100, 35.14)""") + values ('2006-01-05','BUY','RHAT',100,35.14)""") \end{verbatim} -Usually your SQL queries will need to reflect the value of Python +Usually your SQL operations will need to use values from Python variables. You shouldn't assemble your query using Python's string operations because doing so is insecure; it makes your program -vulnerable to what's called an SQL injection attack. Instead, use -SQLite's parameter substitution, putting \samp{?} as a placeholder -wherever you want to use a value, and then provide a tuple of values -as the second argument to the cursor's \method{execute()} method. For -example: +vulnerable to an SQL injection attack. + +Instead, use SQLite's parameter substitution. Put \samp{?} as a +placeholder wherever you want to use a value, and then provide a tuple +of values as the second argument to the cursor's \method{execute()} +method. For example: \begin{verbatim} # Never do this -- insecure! @@ -1510,7 +1511,7 @@ # Do this instead t = (symbol,) -c.execute("... where symbol = '?'", t) +c.execute('select * from stocks where symbol=?', ('IBM',)) # Larger example for t in (('2006-03-28', 'BUY', 'IBM', 1000, 45.00), @@ -1540,15 +1541,6 @@ >>> \end{verbatim} -You should also use parameter substitution with SELECT statements: - -\begin{verbatim} ->>> c.execute('select * from stocks where symbol=?', ('IBM',)) ->>> print c.fetchall() -[(u'2006-03-28', u'BUY', u'IBM', 1000, 45.0), - (u'2006-04-06', u'SELL', u'IBM', 500, 53.0)] -\end{verbatim} - For more information about the SQL dialect supported by SQLite, see \url{http://www.sqlite.org}. @@ -1625,6 +1617,7 @@ new set, \cfunction{PySet_Add()} and \cfunction{PySet_Discard()} to add and remove elements, and \cfunction{PySet_Contains} and \cfunction{PySet_Size} to examine the set's state. +(Contributed by Raymond Hettinger.) \item C code can now obtain information about the exact revision of the Python interpreter by calling the @@ -1633,6 +1626,10 @@ \code{"trunk:45355:45356M, Apr 13 2006, 07:42:19"}. (Contributed by Barry Warsaw.) +\item The CPython interpreter is still written in C, but +the code can now be compiled with a {\Cpp} compiler without errors. +(Implemented by Anthony Baxter, Martin von~L\"owis, Skip Montanaro.) + \item The \cfunction{PyRange_New()} function was removed. It was never documented, never used in the core code, and had dangerously lax error checking. From python-checkins at python.org Fri Apr 14 23:23:43 2006 From: python-checkins at python.org (thomas.wouters) Date: Fri, 14 Apr 2006 23:23:43 +0200 (CEST) Subject: [Python-checkins] r45408 - python/trunk/Modules/datetimemodule.c Message-ID: <20060414212343.9752D1E4010@bag.python.org> Author: thomas.wouters Date: Fri Apr 14 23:23:42 2006 New Revision: 45408 Modified: python/trunk/Modules/datetimemodule.c Log: Coverity-found bug: datetime_strptime() failed to check for NULL return from PySequence_GetItem of the time.strptime() result. Not a high probability bug, but not inconceivable either, considering people can provide their own 'time' module. Modified: python/trunk/Modules/datetimemodule.c ============================================================================== --- python/trunk/Modules/datetimemodule.c (original) +++ python/trunk/Modules/datetimemodule.c Fri Apr 14 23:23:42 2006 @@ -3825,6 +3825,10 @@ if (PySequence_Check(obj) && PySequence_Size(obj) >= 6) for (i=0; i < 6; i++) { PyObject *p = PySequence_GetItem(obj, i); + if (p == NULL) { + Py_DECREF(obj); + return NULL; + } if (PyInt_Check(p)) ia[i] = PyInt_AsLong(p); else From buildbot at python.org Fri Apr 14 23:30:03 2006 From: buildbot at python.org (buildbot at python.org) Date: Fri, 14 Apr 2006 21:30:03 +0000 Subject: [Python-checkins] buildbot failure in x86 W2k trunk Message-ID: <20060414213003.80F931E4005@bag.python.org> The Buildbot has detected a new failure of x86 W2k trunk. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520W2k%2520trunk/builds/461 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: thomas.wouters BUILD FAILED: failed failed slave lost sincerely, -The Buildbot From python-checkins at python.org Sat Apr 15 03:02:21 2006 From: python-checkins at python.org (phillip.eby) Date: Sat, 15 Apr 2006 03:02:21 +0200 (CEST) Subject: [Python-checkins] r45409 - in python/trunk: Lib/test/test_generators.py Objects/frameobject.c Objects/genobject.c Message-ID: <20060415010221.4F42D1E4005@bag.python.org> Author: phillip.eby Date: Sat Apr 15 03:02:17 2006 New Revision: 45409 Modified: python/trunk/Lib/test/test_generators.py python/trunk/Objects/frameobject.c python/trunk/Objects/genobject.c Log: Fix SF#1470508: crash in generator cycle finalization. There were two problems: first, PyGen_NeedsFinalizing() had an off-by-one bug that prevented it from ever saying a generator didn't need finalizing, and second, frame objects cleared themselves in a way that caused their owning generator to think they were still executable, causing a double deallocation of objects on the value stack if there was still a loop on the block stack. This revision also removes some unnecessary close() operations from test_generators that are now appropriately handled by the cycle collector. Modified: python/trunk/Lib/test/test_generators.py ============================================================================== --- python/trunk/Lib/test/test_generators.py (original) +++ python/trunk/Lib/test/test_generators.py Sat Apr 15 03:02:17 2006 @@ -421,7 +421,6 @@ ... self.name = name ... self.parent = None ... self.generator = self.generate() -... self.close = self.generator.close ... ... def generate(self): ... while not self.parent: @@ -484,8 +483,6 @@ merged A into G A->G B->G C->G D->G E->G F->G G->G H->G I->G J->G K->G L->G M->G ->>> for s in sets: s.close() # break cycles - """ # Emacs turd ' @@ -593,7 +590,6 @@ ... def __init__(self, g): ... self.sofar = [] ... self.fetch = g.next -... self.close = g.close ... ... def __getitem__(self, i): ... sofar, fetch = self.sofar, self.fetch @@ -624,8 +620,6 @@ [200, 216, 225, 240, 243, 250, 256, 270, 288, 300, 320, 324, 360, 375, 384] [400, 405, 432, 450, 480, 486, 500, 512, 540, 576, 600, 625, 640, 648, 675] ->>> m235.close() - Ye olde Fibonacci generator, LazyList style. >>> def fibgen(a, b): @@ -648,7 +642,6 @@ >>> fib = LazyList(fibgen(1, 2)) >>> firstn(iter(fib), 17) [1, 2, 3, 5, 8, 13, 21, 34, 55, 89, 144, 233, 377, 610, 987, 1597, 2584] ->>> fib.close() Running after your tail with itertools.tee (new in version 2.4) Modified: python/trunk/Objects/frameobject.c ============================================================================== --- python/trunk/Objects/frameobject.c (original) +++ python/trunk/Objects/frameobject.c Sat Apr 15 03:02:17 2006 @@ -454,9 +454,15 @@ static void frame_clear(PyFrameObject *f) { - PyObject **fastlocals, **p; + PyObject **fastlocals, **p, **oldtop; int i, slots; + oldtop = f->f_stacktop; + + /* Before anything else, make sure that this frame is clearly marked + as being defunct! */ + f->f_stacktop = NULL; + Py_XDECREF(f->f_exc_type); f->f_exc_type = NULL; @@ -473,17 +479,13 @@ slots = f->f_nlocals + f->f_ncells + f->f_nfreevars; fastlocals = f->f_localsplus; for (i = slots; --i >= 0; ++fastlocals) { - if (*fastlocals != NULL) { - Py_XDECREF(*fastlocals); - *fastlocals = NULL; - } + Py_CLEAR(*fastlocals); } /* stack */ - if (f->f_stacktop != NULL) { - for (p = f->f_valuestack; p < f->f_stacktop; p++) { - Py_XDECREF(*p); - *p = NULL; + if (oldtop != NULL) { + for (p = f->f_valuestack; p < oldtop; p++) { + Py_CLEAR(*p); } } } Modified: python/trunk/Objects/genobject.c ============================================================================== --- python/trunk/Objects/genobject.c (original) +++ python/trunk/Objects/genobject.c Sat Apr 15 03:02:17 2006 @@ -35,7 +35,7 @@ } _PyObject_GC_UNTRACK(self); - Py_XDECREF(gen->gi_frame); + Py_CLEAR(gen->gi_frame); PyObject_GC_Del(gen); } @@ -130,8 +130,8 @@ "generator ignored GeneratorExit"); return NULL; } - if ( PyErr_ExceptionMatches(PyExc_StopIteration) - || PyErr_ExceptionMatches(PyExc_GeneratorExit) ) + if ( PyErr_ExceptionMatches(PyExc_StopIteration) + || PyErr_ExceptionMatches(PyExc_GeneratorExit) ) { PyErr_Clear(); /* ignore these errors */ Py_INCREF(Py_None); @@ -208,7 +208,7 @@ return next yielded value or raise StopIteration."); static PyObject * -gen_throw(PyGenObject *gen, PyObject *args) +gen_throw(PyGenObject *gen, PyObject *args) { PyObject *typ; PyObject *tb = NULL; @@ -328,7 +328,7 @@ 0, /* tp_getset */ 0, /* tp_base */ 0, /* tp_dict */ - + 0, /* tp_descr_get */ 0, /* tp_descr_set */ 0, /* tp_dictoffset */ @@ -366,15 +366,16 @@ int i; PyFrameObject *f = gen->gi_frame; - if (f == NULL || f->f_stacktop==NULL || f->f_iblock<=0) - return 0; /* no frame or no blockstack == no finalization */ + if (f == NULL || f->f_stacktop == NULL || f->f_iblock <= 0) + return 0; /* no frame or empty blockstack == no finalization */ - for (i=f->f_iblock; i>=0; i--) { + /* Any block type besides a loop requires cleanup. */ + i = f->f_iblock; + while (--i >= 0) { if (f->f_blockstack[i].b_type != SETUP_LOOP) - /* any block type besides a loop requires cleanup */ return 1; } - /* No blocks except loops, it's safe to skip finalization */ + /* No blocks except loops, it's safe to skip finalization. */ return 0; } From buildbot at python.org Sat Apr 15 03:37:08 2006 From: buildbot at python.org (buildbot at python.org) Date: Sat, 15 Apr 2006 01:37:08 +0000 Subject: [Python-checkins] buildbot warnings in x86 XP trunk Message-ID: <20060415013708.443B21E4005@bag.python.org> The Buildbot has detected a new failure of x86 XP trunk. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520XP%2520trunk/builds/432 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: phillip.eby Build Had Warnings: warnings test sincerely, -The Buildbot From python-checkins at python.org Sat Apr 15 03:48:58 2006 From: python-checkins at python.org (tim.peters) Date: Sat, 15 Apr 2006 03:48:58 +0200 (CEST) Subject: [Python-checkins] r45410 - python/trunk/Lib/test/test_generators.py Message-ID: <20060415014858.39A371E4005@bag.python.org> Author: tim.peters Date: Sat Apr 15 03:48:57 2006 New Revision: 45410 Modified: python/trunk/Lib/test/test_generators.py Log: Changed comments to make sense now that the LazyList-based examples no longer require any explicit closing to avoid leaking. That the tee-based examples still do is (I think) still a mystery. Part of the mystery is that gc.garbage remains empty: if it were the case that some generator in a trash cycle said it needed finalization, suppressing collection of that cycle, that generator _would_ show up in gc.garbage. So this is acting more like, e.g., some tp_traverse slot isn't visiting all the pointers it should (in which case the skipped pointer(s) would act like an external root, silently suppressing collection of everything reachable from it(them)). Modified: python/trunk/Lib/test/test_generators.py ============================================================================== --- python/trunk/Lib/test/test_generators.py (original) +++ python/trunk/Lib/test/test_generators.py Sat Apr 15 03:48:57 2006 @@ -700,11 +700,12 @@ iterators, whereupon it is deleted. You can therefore print the hamming sequence during hours without increasing memory usage, or very little. -The beauty of it is that recursive running after their tail FP algorithms +The beauty of it is that recursive running-after-their-tail FP algorithms are quite straightforwardly expressed with this Python idiom. The problem is -that this creates the same kind of reference cycle as the m235() -implementation above, and again we have to explicitly close the innermost -generator to clean up the cycle. +that this creates an uncollectable reference cycle, and we have to explicitly +close the innermost generator to clean up the cycle. +XXX As of 14-Apr-2006, Tim doubts that anyone understands _why_ some cycle +XXX is uncollectable here. Ye olde Fibonacci generator, tee style. @@ -730,6 +731,7 @@ [1, 2, 3, 5, 8, 13, 21, 34, 55, 89, 144, 233, 377, 610, 987, 1597, 2584] >>> closer() +XXX Again the tee-based approach leaks without an explicit close(). """ leak_test1 = """ From python-checkins at python.org Sat Apr 15 04:14:05 2006 From: python-checkins at python.org (tim.peters) Date: Sat, 15 Apr 2006 04:14:05 +0200 (CEST) Subject: [Python-checkins] r45411 - python/trunk/Include/object.h Message-ID: <20060415021405.52D941E4005@bag.python.org> Author: tim.peters Date: Sat Apr 15 04:14:03 2006 New Revision: 45411 Modified: python/trunk/Include/object.h Log: There were no comments explaining what Py_CLEAR() did or why it's important. Now there are ;-) If someone else hasn't already, I'll add a Py_CLEAR cleanup task to the TODO Wiki next. Modified: python/trunk/Include/object.h ============================================================================== --- python/trunk/Include/object.h (original) +++ python/trunk/Include/object.h Sat Apr 15 04:14:03 2006 @@ -645,6 +645,40 @@ else \ _Py_Dealloc((PyObject *)(op)) +/* Safely decref `op` and set `op` to NULL, especially useful in tp_clear + * and tp_dealloc implementatons. + * + * Note that "the obvious" code can be deadly: + * + * Py_XDECREF(op); + * op = NULL; + * + * Typically, `op` is something like self->containee, and `self` is done + * using its `containee` member. In the code sequence above, suppose + * `containee` is non-NULL with a refcount of 1. Its refcount falls to + * 0 on the first line, which can trigger an arbitrary amount of code, + * possibly including finalizers (like __del__ methods or weakref callbacks) + * coded in Python, which in turn can release the GIL and allow other threads + * to run, etc. Such code may even invoke methods of `self` again, or cause + * cyclic gc to trigger, but-- oops! --self->containee still points to the + * object being torn down, and it may be in an insane state while being torn + * down. This has in fact been a rich historic source of miserable (rare & + * hard-to-diagnose) segfaulting (and other) bugs. + * + * The safe way is: + * + * Py_CLEAR(op); + * + * That arranges to set `op` to NULL _before_ decref'ing, so that any code + * triggered as a side-effect of `op` getting torn down no longer believes + * `op` points to a valid object. + * + * There are cases where it's safe to use the naive code, but they're brittle. + * For example, if `op` points to a Python integer, you know that destroying + * one of those can't cause problems -- but in part that relies on that + * Python integers aren't currently weakly referencable. Best practice is + * to use Py_CLEAR() even if you can't think of a reason for why you need to. + */ #define Py_CLEAR(op) \ do { \ if (op) { \ From python-checkins at python.org Sat Apr 15 05:15:29 2006 From: python-checkins at python.org (tim.peters) Date: Sat, 15 Apr 2006 05:15:29 +0200 (CEST) Subject: [Python-checkins] r45412 - python/trunk/Objects/frameobject.c Message-ID: <20060415031529.AD9531E4005@bag.python.org> Author: tim.peters Date: Sat Apr 15 05:15:24 2006 New Revision: 45412 Modified: python/trunk/Objects/frameobject.c Log: Trimmed trailing whitespace. Modified: python/trunk/Objects/frameobject.c ============================================================================== --- python/trunk/Objects/frameobject.c (original) +++ python/trunk/Objects/frameobject.c Sat Apr 15 05:15:24 2006 @@ -1,4 +1,3 @@ - /* Frame object implementation */ #include "Python.h" @@ -333,7 +332,7 @@ Py_XINCREF(v); f->f_trace = v; - + if (v != NULL) f->f_lineno = PyCode_Addr2Line(f->f_code, f->f_lasti); @@ -399,7 +398,7 @@ for (p = f->f_valuestack; p < f->f_stacktop; p++) Py_XDECREF(*p); } - + Py_XDECREF(f->f_back); Py_DECREF(f->f_code); Py_DECREF(f->f_builtins); @@ -459,7 +458,7 @@ oldtop = f->f_stacktop; - /* Before anything else, make sure that this frame is clearly marked + /* Before anything else, make sure that this frame is clearly marked as being defunct! */ f->f_stacktop = NULL; @@ -536,7 +535,7 @@ } PyFrameObject * -PyFrame_New(PyThreadState *tstate, PyCodeObject *code, PyObject *globals, +PyFrame_New(PyThreadState *tstate, PyCodeObject *code, PyObject *globals, PyObject *locals) { PyFrameObject *back = tstate->frame; @@ -565,10 +564,10 @@ builtins = NULL; } if (builtins == NULL) { - /* No builtins! Make up a minimal one + /* No builtins! Make up a minimal one Give them 'None', at least. */ builtins = PyDict_New(); - if (builtins == NULL || + if (builtins == NULL || PyDict_SetItemString( builtins, "None", Py_None) < 0) return NULL; @@ -613,7 +612,7 @@ Py_INCREF(globals); f->f_globals = globals; /* Most functions have CO_NEWLOCALS and CO_OPTIMIZED set. */ - if ((code->co_flags & (CO_NEWLOCALS | CO_OPTIMIZED)) == + if ((code->co_flags & (CO_NEWLOCALS | CO_OPTIMIZED)) == (CO_NEWLOCALS | CO_OPTIMIZED)) locals = NULL; /* PyFrame_FastToLocals() will set. */ else if (code->co_flags & CO_NEWLOCALS) { @@ -761,10 +760,10 @@ && PyTuple_Check(f->f_code->co_freevars))) { return; } - map_to_dict(f->f_code->co_cellvars, + map_to_dict(f->f_code->co_cellvars, PyTuple_GET_SIZE(f->f_code->co_cellvars), locals, fast + f->f_nlocals, 1); - map_to_dict(f->f_code->co_freevars, + map_to_dict(f->f_code->co_freevars, PyTuple_GET_SIZE(f->f_code->co_freevars), locals, fast + f->f_nlocals + f->f_ncells, 1); } @@ -798,12 +797,12 @@ if (!(PyTuple_Check(f->f_code->co_cellvars) && PyTuple_Check(f->f_code->co_freevars))) return; - dict_to_map(f->f_code->co_cellvars, + dict_to_map(f->f_code->co_cellvars, PyTuple_GET_SIZE(f->f_code->co_cellvars), locals, fast + f->f_nlocals, 1, clear); - dict_to_map(f->f_code->co_freevars, + dict_to_map(f->f_code->co_freevars, PyTuple_GET_SIZE(f->f_code->co_freevars), - locals, fast + f->f_nlocals + f->f_ncells, 1, + locals, fast + f->f_nlocals + f->f_ncells, 1, clear); } PyErr_Restore(error_type, error_value, error_traceback); From python-checkins at python.org Sat Apr 15 05:22:47 2006 From: python-checkins at python.org (tim.peters) Date: Sat, 15 Apr 2006 05:22:47 +0200 (CEST) Subject: [Python-checkins] r45413 - in python/trunk: Include/objimpl.h Objects/frameobject.c Message-ID: <20060415032247.69CB11E4005@bag.python.org> Author: tim.peters Date: Sat Apr 15 05:22:46 2006 New Revision: 45413 Modified: python/trunk/Include/objimpl.h python/trunk/Objects/frameobject.c Log: frame_traverse(): Use the standard Py_VISIT macro. Py_VISIT: cast the `op` argument to PyObject* when calling `visit()`. Else the caller has to pay too much attention to this silly detail (e.g., frame_traverse needs to traverse `struct _frame *` and `PyCodeObject *` pointers too). Modified: python/trunk/Include/objimpl.h ============================================================================== --- python/trunk/Include/objimpl.h (original) +++ python/trunk/Include/objimpl.h Sat Apr 15 05:22:46 2006 @@ -303,13 +303,13 @@ * "visit" and "arg". This is intended to keep tp_traverse functions * looking as much alike as possible. */ -#define Py_VISIT(op) \ - do { \ - if (op) { \ - int vret = visit((op), arg); \ - if (vret) \ - return vret; \ - } \ +#define Py_VISIT(op) \ + do { \ + if (op) { \ + int vret = visit((PyObject *)(op), arg); \ + if (vret) \ + return vret; \ + } \ } while (0) /* This is here for the sake of backwards compatibility. Extensions that Modified: python/trunk/Objects/frameobject.c ============================================================================== --- python/trunk/Objects/frameobject.c (original) +++ python/trunk/Objects/frameobject.c Sat Apr 15 05:22:46 2006 @@ -422,30 +422,28 @@ frame_traverse(PyFrameObject *f, visitproc visit, void *arg) { PyObject **fastlocals, **p; - int i, err, slots; -#define VISIT(o) if (o) {if ((err = visit((PyObject *)(o), arg))) return err;} + int i, slots; - VISIT(f->f_back); - VISIT(f->f_code); - VISIT(f->f_builtins); - VISIT(f->f_globals); - VISIT(f->f_locals); - VISIT(f->f_trace); - VISIT(f->f_exc_type); - VISIT(f->f_exc_value); - VISIT(f->f_exc_traceback); + Py_VISIT(f->f_back); + Py_VISIT(f->f_code); + Py_VISIT(f->f_builtins); + Py_VISIT(f->f_globals); + Py_VISIT(f->f_locals); + Py_VISIT(f->f_trace); + Py_VISIT(f->f_exc_type); + Py_VISIT(f->f_exc_value); + Py_VISIT(f->f_exc_traceback); /* locals */ slots = f->f_nlocals + f->f_ncells + f->f_nfreevars; fastlocals = f->f_localsplus; - for (i = slots; --i >= 0; ++fastlocals) { - VISIT(*fastlocals); - } + for (i = slots; --i >= 0; ++fastlocals) + Py_VISIT(*fastlocals); /* stack */ if (f->f_stacktop != NULL) { for (p = f->f_valuestack; p < f->f_stacktop; p++) - VISIT(*p); + Py_VISIT(*p); } return 0; } From python-checkins at python.org Sat Apr 15 05:30:09 2006 From: python-checkins at python.org (tim.peters) Date: Sat, 15 Apr 2006 05:30:09 +0200 (CEST) Subject: [Python-checkins] r45414 - python/trunk/Objects/frameobject.c Message-ID: <20060415033009.737BB1E4015@bag.python.org> Author: tim.peters Date: Sat Apr 15 05:30:08 2006 New Revision: 45414 Modified: python/trunk/Objects/frameobject.c Log: frame_clear(): Explain why it's important to make the frame look dead right at the start. Use Py_CLEAR for four more frame members. Modified: python/trunk/Objects/frameobject.c ============================================================================== --- python/trunk/Objects/frameobject.c (original) +++ python/trunk/Objects/frameobject.c Sat Apr 15 05:30:08 2006 @@ -454,36 +454,29 @@ PyObject **fastlocals, **p, **oldtop; int i, slots; - oldtop = f->f_stacktop; - /* Before anything else, make sure that this frame is clearly marked - as being defunct! */ + * as being defunct! Else, e.g., a generator reachable from this + * frame may also point to this frame, believe itself to still be + * active, and try cleaning up this frame again. + */ + oldtop = f->f_stacktop; f->f_stacktop = NULL; - Py_XDECREF(f->f_exc_type); - f->f_exc_type = NULL; - - Py_XDECREF(f->f_exc_value); - f->f_exc_value = NULL; - - Py_XDECREF(f->f_exc_traceback); - f->f_exc_traceback = NULL; - - Py_XDECREF(f->f_trace); - f->f_trace = NULL; + Py_CLEAR(f->f_exc_type); + Py_CLEAR(f->f_exc_value); + Py_CLEAR(f->f_exc_traceback); + Py_CLEAR(f->f_trace); /* locals */ slots = f->f_nlocals + f->f_ncells + f->f_nfreevars; fastlocals = f->f_localsplus; - for (i = slots; --i >= 0; ++fastlocals) { + for (i = slots; --i >= 0; ++fastlocals) Py_CLEAR(*fastlocals); - } /* stack */ if (oldtop != NULL) { - for (p = f->f_valuestack; p < oldtop; p++) { + for (p = f->f_valuestack; p < oldtop; p++) Py_CLEAR(*p); - } } } From buildbot at python.org Sat Apr 15 05:48:04 2006 From: buildbot at python.org (buildbot at python.org) Date: Sat, 15 Apr 2006 03:48:04 +0000 Subject: [Python-checkins] buildbot failure in x86 XP-2 trunk Message-ID: <20060415034804.4EEF11E4005@bag.python.org> The Buildbot has detected a new failure of x86 XP-2 trunk. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520XP-2%2520trunk/builds/280 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: tim.peters BUILD FAILED: failed failed slave lost sincerely, -The Buildbot From python-checkins at python.org Sat Apr 15 07:32:17 2006 From: python-checkins at python.org (phillip.eby) Date: Sat, 15 Apr 2006 07:32:17 +0200 (CEST) Subject: [Python-checkins] r45415 - sandbox/trunk/setuptools/setuptools/command/build_py.py Message-ID: <20060415053217.D5B131E4005@bag.python.org> Author: phillip.eby Date: Sat Apr 15 07:32:16 2006 New Revision: 45415 Modified: sandbox/trunk/setuptools/setuptools/command/build_py.py Log: Trap absolute paths given as package_dirs, which foul up things terribly. Modified: sandbox/trunk/setuptools/setuptools/command/build_py.py ============================================================================== --- sandbox/trunk/setuptools/setuptools/command/build_py.py (original) +++ sandbox/trunk/setuptools/setuptools/command/build_py.py Sat Apr 15 07:32:16 2006 @@ -84,18 +84,20 @@ self.manifest_files = mf = {} if not self.distribution.include_package_data: return - src_dirs = {} for package in self.packages or (): # Locate package source directory - src_dirs[self.get_package_dir(package)] = package + src_dirs[assert_relative(self.get_package_dir(package))] = package self.run_command('egg_info') ei_cmd = self.get_finalized_command('egg_info') for path in ei_cmd.filelist.files: - if path.endswith('.py'): continue - d,f = os.path.split(path) - while d and d not in src_dirs: + if path.endswith('.py'): + continue + d,f = os.path.split(assert_relative(path)) + prev = None + while d and d!=prev and d not in src_dirs: + prev = d d, df = os.path.split(d) f = os.path.join(df, f) if d in src_dirs: @@ -119,8 +121,6 @@ for filename in filenames ] - - def check_package(self, package, package_dir): """Check namespace packages' __init__ for declare_namespace""" try: @@ -177,19 +177,19 @@ return [f for f in files if f not in bad] - - - - - - - - - - - - - +def assert_relative(path): + if not os.path.isabs(path): + return path + from distutils.errors import DistutilsSetupError + raise DistutilsSetupError( +"""Error: setup script specifies an absolute path: + + %s + +setup() arguments must *always* be /-separated paths relative to the +setup.py directory, *never* absolute paths. +""" % path + ) From python-checkins at python.org Sat Apr 15 07:33:00 2006 From: python-checkins at python.org (phillip.eby) Date: Sat, 15 Apr 2006 07:33:00 +0200 (CEST) Subject: [Python-checkins] r45416 - sandbox/branches/setuptools-0.6/setuptools/command/build_py.py Message-ID: <20060415053300.EFCE31E4005@bag.python.org> Author: phillip.eby Date: Sat Apr 15 07:33:00 2006 New Revision: 45416 Modified: sandbox/branches/setuptools-0.6/setuptools/command/build_py.py Log: Backport absolute path trapping to the 0.6 branch. Modified: sandbox/branches/setuptools-0.6/setuptools/command/build_py.py ============================================================================== --- sandbox/branches/setuptools-0.6/setuptools/command/build_py.py (original) +++ sandbox/branches/setuptools-0.6/setuptools/command/build_py.py Sat Apr 15 07:33:00 2006 @@ -84,18 +84,20 @@ self.manifest_files = mf = {} if not self.distribution.include_package_data: return - src_dirs = {} for package in self.packages or (): # Locate package source directory - src_dirs[self.get_package_dir(package)] = package + src_dirs[assert_relative(self.get_package_dir(package))] = package self.run_command('egg_info') ei_cmd = self.get_finalized_command('egg_info') for path in ei_cmd.filelist.files: - if path.endswith('.py'): continue - d,f = os.path.split(path) - while d and d not in src_dirs: + if path.endswith('.py'): + continue + d,f = os.path.split(assert_relative(path)) + prev = None + while d and d!=prev and d not in src_dirs: + prev = d d, df = os.path.split(d) f = os.path.join(df, f) if d in src_dirs: @@ -119,8 +121,6 @@ for filename in filenames ] - - def check_package(self, package, package_dir): """Check namespace packages' __init__ for declare_namespace""" try: @@ -177,19 +177,19 @@ return [f for f in files if f not in bad] - - - - - - - - - - - - - +def assert_relative(path): + if not os.path.isabs(path): + return path + from distutils.errors import DistutilsSetupError + raise DistutilsSetupError( +"""Error: setup script specifies an absolute path: + + %s + +setup() arguments must *always* be /-separated paths relative to the +setup.py directory, *never* absolute paths. +""" % path + ) From python-checkins at python.org Sat Apr 15 10:13:05 2006 From: python-checkins at python.org (martin.v.loewis) Date: Sat, 15 Apr 2006 10:13:05 +0200 (CEST) Subject: [Python-checkins] r45417 - in python/trunk: Makefile.pre.in Misc/NEWS Misc/python-config.in Message-ID: <20060415081305.BEEF51E4005@bag.python.org> Author: martin.v.loewis Date: Sat Apr 15 10:13:05 2006 New Revision: 45417 Added: python/trunk/Misc/python-config.in (contents, props changed) Modified: python/trunk/Makefile.pre.in python/trunk/Misc/NEWS Log: Patch #1161914: Add python-config. Modified: python/trunk/Makefile.pre.in ============================================================================== --- python/trunk/Makefile.pre.in (original) +++ python/trunk/Makefile.pre.in Sat Apr 15 10:13:05 2006 @@ -826,6 +826,11 @@ $(INSTALL_DATA) Modules/Setup.config $(DESTDIR)$(LIBPL)/Setup.config $(INSTALL_SCRIPT) $(srcdir)/Modules/makesetup $(DESTDIR)$(LIBPL)/makesetup $(INSTALL_SCRIPT) $(srcdir)/install-sh $(DESTDIR)$(LIBPL)/install-sh + # Substitution happens here, as the completely-expanded BINDIR + # is not available in configure + sed -e "s, at BINDIR@,$(BINDIR)," < $(srcdir)/Misc/python-config.in >python-config + $(INSTALL_SCRIPT) python-config $(BINDIR)/python-config + rm python-config @if [ -s Modules/python.exp -a \ "`echo $(MACHDEP) | sed 's/^\(...\).*/\1/'`" = "aix" ]; then \ echo; echo "Installing support files for building shared extension modules on AIX:"; \ Modified: python/trunk/Misc/NEWS ============================================================================== --- python/trunk/Misc/NEWS (original) +++ python/trunk/Misc/NEWS Sat Apr 15 10:13:05 2006 @@ -82,6 +82,7 @@ Build ----- +- Patch #1161914: Add a python-config script. - Patch #1324762:Remove ccpython.cc; replace --with-cxx with --with-cxx-main. Link with C++ compiler only if --with-cxx-main was specified. (Can be overridden by explicitly setting LINKCC.) Decouple Added: python/trunk/Misc/python-config.in ============================================================================== --- (empty file) +++ python/trunk/Misc/python-config.in Sat Apr 15 10:13:05 2006 @@ -0,0 +1,50 @@ +#!@BINDIR@/python + +import sys +import os +import getopt +from distutils import sysconfig + +valid_opts = ['prefix', 'exec-prefix', 'includes', 'libs', 'cflags', + 'ldflags', 'help'] + +def exit_with_usage(code=1): + print >>sys.stderr, "Usage: %s [%s]" % (sys.argv[0], + '|'.join('--'+opt for opt in valid_opts)) + sys.exit(code) + +try: + opts, args = getopt.getopt(sys.argv[1:], '', valid_opts) +except getopt.error: + exit_with_usage() + +if not opts: + exit_with_usage() + +opt = opts[0][0] + +pyver = sysconfig.get_config_var('VERSION') +getvar = sysconfig.get_config_var + +if opt == '--help': + exit_with_usage(0) + +elif opt == '--prefix': + print sysconfig.PREFIX + +elif opt == '--exec-prefix': + print sysconfig.EXEC_PREFIX + +elif opt in ('--includes', '--cflags'): + flags = ['-I'+dir for dir in getvar('INCLDIRSTOMAKE').split()] + if opt == '--cflags': + flags.extend(getvar('CFLAGS').split()) + print ' '.join(flags) + +elif opt in ('--libs', '--ldflags'): + libs = sysconfig.get_config_var('LIBS').split() + libs.append('-lpython'+pyver) + if opt == '--ldflags': + libs.insert(0, '-L' + getvar('LIBPL')) + print ' '.join(libs) + From buildbot at python.org Sat Apr 15 10:34:21 2006 From: buildbot at python.org (buildbot at python.org) Date: Sat, 15 Apr 2006 08:34:21 +0000 Subject: [Python-checkins] buildbot warnings in x86 XP trunk Message-ID: <20060415083421.ACF981E4005@bag.python.org> The Buildbot has detected a new failure of x86 XP trunk. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520XP%2520trunk/builds/437 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: martin.v.loewis Build Had Warnings: warnings failed slave lost sincerely, -The Buildbot From python-checkins at python.org Sat Apr 15 10:35:46 2006 From: python-checkins at python.org (martin.v.loewis) Date: Sat, 15 Apr 2006 10:35:46 +0200 (CEST) Subject: [Python-checkins] r45418 - in python/branches/release24-maint: Misc/NEWS Modules/socketmodule.c Message-ID: <20060415083546.02CF71E4005@bag.python.org> Author: martin.v.loewis Date: Sat Apr 15 10:35:45 2006 New Revision: 45418 Modified: python/branches/release24-maint/Misc/NEWS python/branches/release24-maint/Modules/socketmodule.c Log: Patch #1191065: Fix preprocessor problems on systems where recvfrom is a macro. Modified: python/branches/release24-maint/Misc/NEWS ============================================================================== --- python/branches/release24-maint/Misc/NEWS (original) +++ python/branches/release24-maint/Misc/NEWS Sat Apr 15 10:35:45 2006 @@ -15,6 +15,9 @@ Extension Modules ----------------- +- Patch #1191065: Fix preprocessor problems on systems where recvfrom + is a macro. + - Bug #1467952: os.listdir() now correctly raises an error if readdir() fails with an error condition. Modified: python/branches/release24-maint/Modules/socketmodule.c ============================================================================== --- python/branches/release24-maint/Modules/socketmodule.c (original) +++ python/branches/release24-maint/Modules/socketmodule.c Sat Apr 15 10:35:45 2006 @@ -2161,18 +2161,20 @@ Py_BEGIN_ALLOW_THREADS memset(&addrbuf, 0, addrlen); timeout = internal_select(s, 0); - if (!timeout) - n = recvfrom(s->sock_fd, PyString_AS_STRING(buf), len, flags, + if (!timeout) { #ifndef MS_WINDOWS #if defined(PYOS_OS2) && !defined(PYCC_GCC) - (struct sockaddr *) &addrbuf, &addrlen + n = recvfrom(s->sock_fd, PyString_AS_STRING(buf), len, flags, + (struct sockaddr *) &addrbuf, &addrlen); #else - (void *) &addrbuf, &addrlen + n = recvfrom(s->sock_fd, PyString_AS_STRING(buf), len, flags, + (void *) &addrbuf, &addrlen); #endif #else - (struct sockaddr *) &addrbuf, &addrlen + n = recvfrom(s->sock_fd, PyString_AS_STRING(buf), len, flags, + (struct sockaddr *) &addrbuf, &addrlen); #endif - ); + } Py_END_ALLOW_THREADS if (timeout) { From python-checkins at python.org Sat Apr 15 10:36:00 2006 From: python-checkins at python.org (martin.v.loewis) Date: Sat, 15 Apr 2006 10:36:00 +0200 (CEST) Subject: [Python-checkins] r45419 - in python/trunk: Misc/NEWS Modules/socketmodule.c Message-ID: <20060415083600.700B51E4005@bag.python.org> Author: martin.v.loewis Date: Sat Apr 15 10:35:59 2006 New Revision: 45419 Modified: python/trunk/Misc/NEWS python/trunk/Modules/socketmodule.c Log: Patch #1191065: Fix preprocessor problems on systems where recvfrom is a macro. Modified: python/trunk/Misc/NEWS ============================================================================== --- python/trunk/Misc/NEWS (original) +++ python/trunk/Misc/NEWS Sat Apr 15 10:35:59 2006 @@ -48,6 +48,9 @@ Extension Modules ----------------- +- Patch #1191065: Fix preprocessor problems on systems where recvfrom + is a macro. + - Bug #1467952: os.listdir() now correctly raises an error if readdir() fails with an error condition. Modified: python/trunk/Modules/socketmodule.c ============================================================================== --- python/trunk/Modules/socketmodule.c (original) +++ python/trunk/Modules/socketmodule.c Sat Apr 15 10:35:59 2006 @@ -2208,18 +2208,20 @@ Py_BEGIN_ALLOW_THREADS memset(&addrbuf, 0, addrlen); timeout = internal_select(s, 0); - if (!timeout) - n = recvfrom(s->sock_fd, PyString_AS_STRING(buf), len, flags, + if (!timeout) { #ifndef MS_WINDOWS #if defined(PYOS_OS2) && !defined(PYCC_GCC) - (struct sockaddr *) &addrbuf, &addrlen + n = recvfrom(s->sock_fd, PyString_AS_STRING(buf), len, flags, + (struct sockaddr *) &addrbuf, &addrlen); #else - (void *) &addrbuf, &addrlen + n = recvfrom(s->sock_fd, PyString_AS_STRING(buf), len, flags, + (void *) &addrbuf, &addrlen); #endif #else - (struct sockaddr *) &addrbuf, &addrlen + n = recvfrom(s->sock_fd, PyString_AS_STRING(buf), len, flags, + (struct sockaddr *) &addrbuf, &addrlen); #endif - ); + } Py_END_ALLOW_THREADS if (timeout) { From python-checkins at python.org Sat Apr 15 10:40:55 2006 From: python-checkins at python.org (martin.v.loewis) Date: Sat, 15 Apr 2006 10:40:55 +0200 (CEST) Subject: [Python-checkins] r45420 - in python/branches/release24-maint: Lib/bdb.py Misc/NEWS Message-ID: <20060415084055.0B9661E4005@bag.python.org> Author: martin.v.loewis Date: Sat Apr 15 10:40:53 2006 New Revision: 45420 Modified: python/branches/release24-maint/Lib/bdb.py python/branches/release24-maint/Misc/NEWS Log: Patch #1191700: Adjust column alignment in bdb breakpoint lists. Modified: python/branches/release24-maint/Lib/bdb.py ============================================================================== --- python/branches/release24-maint/Lib/bdb.py (original) +++ python/branches/release24-maint/Lib/bdb.py Sat Apr 15 10:40:53 2006 @@ -479,10 +479,10 @@ else: disp = 'keep ' if self.enabled: - disp = disp + 'yes' + disp = disp + 'yes ' else: - disp = disp + 'no ' - print '%-4dbreakpoint %s at %s:%d' % (self.number, disp, + disp = disp + 'no ' + print '%-4dbreakpoint %s at %s:%d' % (self.number, disp, self.file, self.line) if self.cond: print '\tstop only if %s' % (self.cond,) Modified: python/branches/release24-maint/Misc/NEWS ============================================================================== --- python/branches/release24-maint/Misc/NEWS (original) +++ python/branches/release24-maint/Misc/NEWS Sat Apr 15 10:40:53 2006 @@ -29,6 +29,8 @@ Library ------- +- Patch #1191700: Adjust column alignment in bdb breakpoint lists. + - The email module's parsedate_tz function now sets the daylight savings flag to -1 (unknown) since it can't tell from the date whether it should be set. From python-checkins at python.org Sat Apr 15 10:41:11 2006 From: python-checkins at python.org (martin.v.loewis) Date: Sat, 15 Apr 2006 10:41:11 +0200 (CEST) Subject: [Python-checkins] r45421 - in python/trunk: Lib/bdb.py Misc/NEWS Message-ID: <20060415084111.CEAB71E4005@bag.python.org> Author: martin.v.loewis Date: Sat Apr 15 10:41:11 2006 New Revision: 45421 Modified: python/trunk/Lib/bdb.py python/trunk/Misc/NEWS Log: Patch #1191700: Adjust column alignment in bdb breakpoint lists. Backported to 2.4. Modified: python/trunk/Lib/bdb.py ============================================================================== --- python/trunk/Lib/bdb.py (original) +++ python/trunk/Lib/bdb.py Sat Apr 15 10:41:11 2006 @@ -479,10 +479,10 @@ else: disp = 'keep ' if self.enabled: - disp = disp + 'yes' + disp = disp + 'yes ' else: - disp = disp + 'no ' - print '%-4dbreakpoint %s at %s:%d' % (self.number, disp, + disp = disp + 'no ' + print '%-4dbreakpoint %s at %s:%d' % (self.number, disp, self.file, self.line) if self.cond: print '\tstop only if %s' % (self.cond,) Modified: python/trunk/Misc/NEWS ============================================================================== --- python/trunk/Misc/NEWS (original) +++ python/trunk/Misc/NEWS Sat Apr 15 10:41:11 2006 @@ -68,6 +68,8 @@ Library ------- +- Patch #1191700: Adjust column alignment in bdb breakpoint lists. + - SimpleXMLRPCServer relied on the fcntl module, which is unavailable on Windows. Bug #1469163. From buildbot at python.org Sat Apr 15 12:09:30 2006 From: buildbot at python.org (buildbot at python.org) Date: Sat, 15 Apr 2006 10:09:30 +0000 Subject: [Python-checkins] buildbot warnings in x86 OpenBSD 2.4 Message-ID: <20060415100930.31A901E4005@bag.python.org> The Buildbot has detected a new failure of x86 OpenBSD 2.4. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520OpenBSD%25202.4/builds/60 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch branches/release24-maint] HEAD Blamelist: martin.v.loewis Build Had Warnings: warnings test sincerely, -The Buildbot From python-checkins at python.org Sat Apr 15 14:45:05 2006 From: python-checkins at python.org (martin.v.loewis) Date: Sat, 15 Apr 2006 14:45:05 +0200 (CEST) Subject: [Python-checkins] r45432 - python/trunk/Objects/structseq.c Message-ID: <20060415124505.DC6491E4005@bag.python.org> Author: martin.v.loewis Date: Sat Apr 15 14:45:05 2006 New Revision: 45432 Modified: python/trunk/Objects/structseq.c Log: Unlink the structseq type from the global list of objects before initializing it. It might be linked already if there was a Py_Initialize/Py_Finalize cycle earlier; not unlinking it would break the global list. Modified: python/trunk/Objects/structseq.c ============================================================================== --- python/trunk/Objects/structseq.c (original) +++ python/trunk/Objects/structseq.c Sat Apr 15 14:45:05 2006 @@ -349,6 +349,14 @@ PyMemberDef* members; int n_members, n_unnamed_members, i, k; +#ifdef Py_TRACE_REFS + /* if the type object was chained, unchain it first + before overwriting its storage */ + if (type->_ob_next) { + _Py_ForgetReference((PyObject*)type); + } +#endif + n_unnamed_members = 0; for (i = 0; desc->fields[i].name != NULL; ++i) if (desc->fields[i].name == PyStructSequence_UnnamedField) From python-checkins at python.org Sat Apr 15 14:46:09 2006 From: python-checkins at python.org (martin.v.loewis) Date: Sat, 15 Apr 2006 14:46:09 +0200 (CEST) Subject: [Python-checkins] r45433 - python/trunk/Python/pyarena.c Message-ID: <20060415124609.DBFCA1E4005@bag.python.org> Author: martin.v.loewis Date: Sat Apr 15 14:46:09 2006 New Revision: 45433 Modified: python/trunk/Python/pyarena.c Log: Fix sys.getobjects(0): we get a reference to the arena's "private" list of objects, so there might be two references to that list. Modified: python/trunk/Python/pyarena.c ============================================================================== --- python/trunk/Python/pyarena.c (original) +++ python/trunk/Python/pyarena.c Sat Apr 15 14:46:09 2006 @@ -171,7 +171,10 @@ */ #endif block_free(arena->a_head); + /* This property normally holds, except when the code being compiled + is sys.getobjects(0), in which case there will be two references. assert(arena->a_objects->ob_refcnt == 1); + */ /* Clear all the elements from the list. This is necessary to guarantee that they will be DECREFed. */ From python-checkins at python.org Sat Apr 15 14:47:24 2006 From: python-checkins at python.org (martin.v.loewis) Date: Sat, 15 Apr 2006 14:47:24 +0200 (CEST) Subject: [Python-checkins] r45434 - python/trunk/Objects/setobject.c Message-ID: <20060415124724.1F4951E4005@bag.python.org> Author: martin.v.loewis Date: Sat Apr 15 14:47:23 2006 New Revision: 45434 Modified: python/trunk/Objects/setobject.c Log: Clear dummy and emptyfrozenset, so that we don't have dangling references in case of a Py_Initialize/Py_Finalize cycle. Modified: python/trunk/Objects/setobject.c ============================================================================== --- python/trunk/Objects/setobject.c (original) +++ python/trunk/Objects/setobject.c Sat Apr 15 14:47:23 2006 @@ -978,8 +978,8 @@ so = free_sets[num_free_sets]; PyObject_GC_Del(so); } - Py_XDECREF(dummy); - Py_XDECREF(emptyfrozenset); + Py_CLEAR(dummy); + Py_CLEAR(emptyfrozenset); } static PyObject * From python-checkins at python.org Sat Apr 15 19:28:36 2006 From: python-checkins at python.org (thomas.wouters) Date: Sat, 15 Apr 2006 19:28:36 +0200 (CEST) Subject: [Python-checkins] r45435 - in python/trunk: Modules/_csv.c Modules/cPickle.c Modules/collectionsmodule.c Modules/pyexpat.c Objects/cellobject.c Objects/funcobject.c Objects/typeobject.c Python/traceback.c Message-ID: <20060415172836.BDBAE1E4005@bag.python.org> Author: thomas.wouters Date: Sat Apr 15 19:28:34 2006 New Revision: 45435 Modified: python/trunk/Modules/_csv.c python/trunk/Modules/cPickle.c python/trunk/Modules/collectionsmodule.c python/trunk/Modules/pyexpat.c python/trunk/Objects/cellobject.c python/trunk/Objects/funcobject.c python/trunk/Objects/typeobject.c python/trunk/Python/traceback.c Log: Use Py_CLEAR instead of in-place DECREF/XDECREF or custom macros, for tp_clear methods. Modified: python/trunk/Modules/_csv.c ============================================================================== --- python/trunk/Modules/_csv.c (original) +++ python/trunk/Modules/_csv.c Sat Apr 15 19:28:34 2006 @@ -828,12 +828,9 @@ static int Reader_clear(ReaderObj *self) { - Py_XDECREF(self->dialect); - Py_XDECREF(self->input_iter); - Py_XDECREF(self->fields); - self->dialect = NULL; - self->input_iter = NULL; - self->fields = NULL; + Py_CLEAR(self->dialect); + Py_CLEAR(self->input_iter); + Py_CLEAR(self->fields); return 0; } @@ -1260,10 +1257,8 @@ static int Writer_clear(WriterObj *self) { - Py_XDECREF(self->dialect); - Py_XDECREF(self->writeline); - self->dialect = NULL; - self->writeline = NULL; + Py_CLEAR(self->dialect); + Py_CLEAR(self->writeline); return 0; } Modified: python/trunk/Modules/cPickle.c ============================================================================== --- python/trunk/Modules/cPickle.c (original) +++ python/trunk/Modules/cPickle.c Sat Apr 15 19:28:34 2006 @@ -2931,16 +2931,14 @@ static int Pickler_clear(Picklerobject *self) { -#define CLEAR(SLOT) Py_XDECREF(SLOT); SLOT = NULL; - CLEAR(self->write); - CLEAR(self->memo); - CLEAR(self->fast_memo); - CLEAR(self->arg); - CLEAR(self->file); - CLEAR(self->pers_func); - CLEAR(self->inst_pers_func); - CLEAR(self->dispatch_table); -#undef CLEAR + Py_CLEAR(self->write); + Py_CLEAR(self->memo); + Py_CLEAR(self->fast_memo); + Py_CLEAR(self->arg); + Py_CLEAR(self->file); + Py_CLEAR(self->pers_func); + Py_CLEAR(self->inst_pers_func); + Py_CLEAR(self->dispatch_table); return 0; } @@ -5284,17 +5282,15 @@ static int Unpickler_clear(Unpicklerobject *self) { -#define CLEAR(SLOT) Py_XDECREF(SLOT); SLOT = NULL - CLEAR(self->readline); - CLEAR(self->read); - CLEAR(self->file); - CLEAR(self->memo); - CLEAR(self->stack); - CLEAR(self->pers_func); - CLEAR(self->arg); - CLEAR(self->last_string); - CLEAR(self->find_class); -#undef CLEAR + Py_CLEAR(self->readline); + Py_CLEAR(self->read); + Py_CLEAR(self->file); + Py_CLEAR(self->memo); + Py_CLEAR(self->stack); + Py_CLEAR(self->pers_func); + Py_CLEAR(self->arg); + Py_CLEAR(self->last_string); + Py_CLEAR(self->find_class); return 0; } Modified: python/trunk/Modules/collectionsmodule.c ============================================================================== --- python/trunk/Modules/collectionsmodule.c (original) +++ python/trunk/Modules/collectionsmodule.c Sat Apr 15 19:28:34 2006 @@ -1236,10 +1236,7 @@ static int defdict_tp_clear(defdictobject *dd) { - if (dd->default_factory != NULL) { - Py_DECREF(dd->default_factory); - dd->default_factory = NULL; - } + Py_CLEAR(dd->default_factory); return PyDict_Type.tp_clear((PyObject *)dd); } Modified: python/trunk/Modules/pyexpat.c ============================================================================== --- python/trunk/Modules/pyexpat.c (original) +++ python/trunk/Modules/pyexpat.c Sat Apr 15 19:28:34 2006 @@ -1669,8 +1669,7 @@ xmlparse_clear(xmlparseobject *op) { clear_handlers(op, 0); - Py_XDECREF(op->intern); - op->intern = 0; + Py_CLEAR(op->intern); return 0; } #endif Modified: python/trunk/Objects/cellobject.c ============================================================================== --- python/trunk/Objects/cellobject.c (original) +++ python/trunk/Objects/cellobject.c Sat Apr 15 19:28:34 2006 @@ -81,8 +81,7 @@ static int cell_clear(PyCellObject *op) { - Py_XDECREF(op->ob_ref); - op->ob_ref = NULL; + Py_CLEAR(op->ob_ref); return 0; } Modified: python/trunk/Objects/funcobject.c ============================================================================== --- python/trunk/Objects/funcobject.c (original) +++ python/trunk/Objects/funcobject.c Sat Apr 15 19:28:34 2006 @@ -655,9 +655,7 @@ static int cm_clear(classmethod *cm) { - Py_XDECREF(cm->cm_callable); - cm->cm_callable = NULL; - + Py_CLEAR(cm->cm_callable); return 0; } Modified: python/trunk/Objects/typeobject.c ============================================================================== --- python/trunk/Objects/typeobject.c (original) +++ python/trunk/Objects/typeobject.c Sat Apr 15 19:28:34 2006 @@ -559,8 +559,8 @@ char *addr = (char *)self + mp->offset; PyObject *obj = *(PyObject **)addr; if (obj != NULL) { - Py_DECREF(obj); *(PyObject **)addr = NULL; + Py_DECREF(obj); } } } @@ -2236,13 +2236,6 @@ for heaptypes. */ assert(type->tp_flags & Py_TPFLAGS_HEAPTYPE); -#define CLEAR(SLOT) \ - if (SLOT) { \ - tmp = (PyObject *)(SLOT); \ - SLOT = NULL; \ - Py_DECREF(tmp); \ - } - /* The only field we need to clear is tp_mro, which is part of a hard cycle (its first element is the class itself) that won't be broken otherwise (it's a tuple and tuples don't have a @@ -2268,9 +2261,7 @@ A tuple of strings can't be part of a cycle. */ - CLEAR(type->tp_mro); - -#undef CLEAR + Py_CLEAR(type->tp_mro); return 0; } Modified: python/trunk/Python/traceback.c ============================================================================== --- python/trunk/Python/traceback.c (original) +++ python/trunk/Python/traceback.c Sat Apr 15 19:28:34 2006 @@ -53,10 +53,8 @@ static void tb_clear(PyTracebackObject *tb) { - Py_XDECREF(tb->tb_next); - Py_XDECREF(tb->tb_frame); - tb->tb_next = NULL; - tb->tb_frame = NULL; + Py_CLEAR(tb->tb_next); + Py_CLEAR(tb->tb_frame); } PyTypeObject PyTraceBack_Type = { From python-checkins at python.org Sat Apr 15 19:33:15 2006 From: python-checkins at python.org (thomas.wouters) Date: Sat, 15 Apr 2006 19:33:15 +0200 (CEST) Subject: [Python-checkins] r45436 - python/trunk/Modules/_csv.c Message-ID: <20060415173315.378151E4005@bag.python.org> Author: thomas.wouters Date: Sat Apr 15 19:33:14 2006 New Revision: 45436 Modified: python/trunk/Modules/_csv.c Log: Re-instate backward compatibility by defining Py_CLEAR if it isn't available. Modified: python/trunk/Modules/_csv.c ============================================================================== --- python/trunk/Modules/_csv.c (original) +++ python/trunk/Modules/_csv.c Sat Apr 15 19:33:14 2006 @@ -37,6 +37,20 @@ # define PyMODINIT_FUNC void # endif /* __cplusplus */ #endif + +#ifndef Py_CLEAR +#define Py_CLEAR(op) \ + do { \ + if (op) { \ + PyObject *tmp = (PyObject *)(op); \ + (op) = NULL; \ + Py_DECREF(tmp); \ + } \ + } while (0) +#endif +#ifndef Py_VISIT + + /* end 2.2 compatibility macros */ #define IS_BASESTRING(o) \ From python-checkins at python.org Sat Apr 15 19:36:43 2006 From: python-checkins at python.org (thomas.wouters) Date: Sat, 15 Apr 2006 19:36:43 +0200 (CEST) Subject: [Python-checkins] r45437 - python/trunk/Modules/_csv.c Message-ID: <20060415173643.580761E4005@bag.python.org> Author: thomas.wouters Date: Sat Apr 15 19:36:42 2006 New Revision: 45437 Modified: python/trunk/Modules/_csv.c Log: Remove partial change (don't edit, commit and think at the same time :P) Modified: python/trunk/Modules/_csv.c ============================================================================== --- python/trunk/Modules/_csv.c (original) +++ python/trunk/Modules/_csv.c Sat Apr 15 19:36:42 2006 @@ -48,7 +48,6 @@ } \ } while (0) #endif -#ifndef Py_VISIT /* end 2.2 compatibility macros */ From python-checkins at python.org Sat Apr 15 20:06:55 2006 From: python-checkins at python.org (martin.v.loewis) Date: Sat, 15 Apr 2006 20:06:55 +0200 (CEST) Subject: [Python-checkins] r45438 - in python/trunk: Misc/NEWS PCbuild/db.build PCbuild/python.build PCbuild/readme.txt Message-ID: <20060415180655.A098B1E4005@bag.python.org> Author: martin.v.loewis Date: Sat Apr 15 20:06:54 2006 New Revision: 45438 Added: python/trunk/PCbuild/db.build (contents, props changed) python/trunk/PCbuild/python.build (contents, props changed) Modified: python/trunk/Misc/NEWS python/trunk/PCbuild/readme.txt Log: Patch #1470875: Building Python with MS Free Compiler. Modified: python/trunk/Misc/NEWS ============================================================================== --- python/trunk/Misc/NEWS (original) +++ python/trunk/Misc/NEWS Sat Apr 15 20:06:54 2006 @@ -87,7 +87,10 @@ Build ----- +- Patch #1470875: Building Python with MS Free Compiler + - Patch #1161914: Add a python-config script. + - Patch #1324762:Remove ccpython.cc; replace --with-cxx with --with-cxx-main. Link with C++ compiler only if --with-cxx-main was specified. (Can be overridden by explicitly setting LINKCC.) Decouple Added: python/trunk/PCbuild/db.build ============================================================================== --- (empty file) +++ python/trunk/PCbuild/db.build Sat Apr 15 20:06:54 2006 @@ -0,0 +1,10 @@ + + + + + + + + + + Added: python/trunk/PCbuild/python.build ============================================================================== --- (empty file) +++ python/trunk/PCbuild/python.build Sat Apr 15 20:06:54 2006 @@ -0,0 +1,21 @@ + + + + + + + + + + + + + + + + + + + + + Modified: python/trunk/PCbuild/readme.txt ============================================================================== --- python/trunk/PCbuild/readme.txt (original) +++ python/trunk/PCbuild/readme.txt Sat Apr 15 20:06:54 2006 @@ -273,6 +273,143 @@ to the Itanium configuration; make sure you use the latest version of vsextcomp. +Building Python Using the free MS Toolkit Compiler +-------------------------------------------------- + +The build process for Visual C++ can be used almost unchanged with the free MS +Toolkit Compiler. This provides a way of building Python using freely +available software. + +Requirements + + To build Python, the following tools are required: + + * The Visual C++ Toolkit Compiler + from http://msdn.microsoft.com/visualc/vctoolkit2003/ + * A recent Platform SDK + from http://www.microsoft.com/downloads/details.aspx?FamilyID=484269e2-3b89-47e3-8eb7-1f2be6d7123a + * The .NET 1.1 SDK + from http://www.microsoft.com/downloads/details.aspx?FamilyID=9b3a2ca6-3647-4070-9f41-a333c6b9181d + + [Does anyone have better URLs for the last 2 of these?] + + The toolkit compiler is needed as it is an optimising compiler (the + compiler supplied with the .NET SDK is a non-optimising version). The + platform SDK is needed to provide the Windows header files and libraries + (the Windows 2003 Server SP1 edition, typical install, is known to work - + other configurations or versions are probably fine as well). The .NET 1.1 + SDK is needed because it contains a version of msvcrt.dll which links to + the msvcr71.dll CRT. Note that the .NET 2.0 SDK is NOT acceptable, as it + references msvcr80.dll. + + All of the above items should be installed as normal. + + If you intend to build the openssl (needed for the _ssl extension) you + will need the C runtime sources installed as part of the platform SDK. + + In addition, you will need Nant, available from + http://nant.sourceforge.net. The 0.85 release candidate 3 version is known + to work. This is the latest released version at the time of writing. Later + "nightly build" versions are known NOT to work - it is not clear at + present whether future released versions will work. + +Setting up the environment + + Start a platform SDK "build environment window" from the start menu. The + "Windows XP 32-bit retail" version is known to work. + + Add the following directories to your PATH: + * The toolkit compiler directory + * The SDK "Win64" binaries directory + * The Nant directory + Add to your INCLUDE environment variable: + * The toolkit compiler INCLUDE directory + Add to your LIB environment variable: + * The toolkit compiler LIB directory + * The .NET SDK Visual Studio 2003 VC7\lib directory + + The following commands should set things up as you need them: + + rem Set these values according to where you installed the software + set TOOLKIT=C:\Program Files\Microsoft Visual C++ Toolkit 2003 + set SDK=C:\Program Files\Microsoft Platform SDK + set NET=C:\Program Files\Microsoft Visual Studio .NET 2003 + set NANT=C:\Utils\Nant + + set PATH=%TOOLKIT%\bin;%PATH%;%SDK%\Bin\win64;%NANT%\bin + set INCLUDE=%TOOLKIT%\include;%INCLUDE% + set LIB=%TOOLKIT%\lib;%NET%\VC7\lib;%LIB% + + The "win64" directory from the SDK is added to supply executables such as + "cvtres" and "lib", which are not available elsewhere. The versions in the + "win64" directory are 32-bit programs, so they are fine to use here. + + That's it. To build Python (the core only, no binary extensions which + depend on external libraries) you just need to issue the command + + nant -buildfile:python.build all + + from within the PCBuild directory. + +Extension modules + + To build those extension modules which require external libraries + (_tkinter, bz2, _bsddb, _sqlite3, _ssl) you can follow the instructions + for the Visual Studio build above, with a few minor modifications. These + instructions have only been tested using the sources in the Python + subversion repository - building from original sources should work, but + has not been tested. + + For each extension module you wish to build, you should remove the + associated include line from the excludeprojects section of pc.build. + + The changes required are: + + _tkinter + The tix makefile (tix-8.4.0\win\makefile.vc) must be modified to + remove references to TOOLS32. The relevant lines should be changed to + read: + cc32 = cl.exe + link32 = link.exe + include32 = + The remainder of the build instructions will work as given. + + bz2 + No changes are needed + + _bsddb + The file db.build should be copied from the Python PCBuild directory + to the directory db-4.4.20\build_win32. + + The file db_static.vcproj in db-4.4.20\build_win32 sould be edited to + remove the string "$(SolutionDir)" - this occurs in 2 places, only + relevant for 64-bit builds. (The edit is required as otherwise, nant + wants to read the solution file, which is not in a suitable form). + + The bsddb library can then be build with the command + nant -buildfile:db.build all + run from the db-4.4.20\build_win32 directory. + + _sqlite3 + No changes are needed. However, in order for the tests to succeed, a + copy of sqlite3.dll must be downloaded, and placed alongside + python.exe. + + _ssl + The documented build process works as written. However, it needs a + copy of the file setargv.obj, which is not supplied in the platform + SDK. However, the sources are available (in the crt source code). To + build setargv.obj, proceed as follows: + + Copy setargv.c, cruntime.h and internal.h from %SDK%\src\crt to a + temporary directory. + Compile using "cl /c /I. /MD /D_CRTBLD setargv.c" + Copy the resulting setargv.obj to somewhere on your LIB environment + (%SDK%\lib is a reasonable place). + + With setargv.obj in place, the standard build process should work + fine. + YOUR OWN EXTENSION DLLs ----------------------- If you want to create your own extension module DLL, there's an example From python-checkins at python.org Sat Apr 15 20:14:22 2006 From: python-checkins at python.org (martin.v.loewis) Date: Sat, 15 Apr 2006 20:14:22 +0200 (CEST) Subject: [Python-checkins] r45439 - python/trunk/Python/pystate.c Message-ID: <20060415181422.E94EC1E4005@bag.python.org> Author: martin.v.loewis Date: Sat Apr 15 20:14:21 2006 New Revision: 45439 Modified: python/trunk/Python/pystate.c Log: Zap ZAP. Modified: python/trunk/Python/pystate.c ============================================================================== --- python/trunk/Python/pystate.c (original) +++ python/trunk/Python/pystate.c Sat Apr 15 20:14:21 2006 @@ -23,13 +23,6 @@ #endif -#define ZAP(x) { \ - PyObject *tmp = (PyObject *)(x); \ - (x) = NULL; \ - Py_XDECREF(tmp); \ -} - - #ifdef WITH_THREAD #include "pythread.h" static PyThread_type_lock head_mutex = NULL; /* Protects interp->tstate_head */ @@ -106,12 +99,12 @@ for (p = interp->tstate_head; p != NULL; p = p->next) PyThreadState_Clear(p); HEAD_UNLOCK(); - ZAP(interp->codec_search_path); - ZAP(interp->codec_search_cache); - ZAP(interp->codec_error_registry); - ZAP(interp->modules); - ZAP(interp->sysdict); - ZAP(interp->builtins); + Py_CLEAR(interp->codec_search_path); + Py_CLEAR(interp->codec_search_cache); + Py_CLEAR(interp->codec_error_registry); + Py_CLEAR(interp->modules); + Py_CLEAR(interp->sysdict); + Py_CLEAR(interp->builtins); } @@ -215,23 +208,23 @@ fprintf(stderr, "PyThreadState_Clear: warning: thread still has a frame\n"); - ZAP(tstate->frame); + Py_CLEAR(tstate->frame); - ZAP(tstate->dict); - ZAP(tstate->async_exc); + Py_CLEAR(tstate->dict); + Py_CLEAR(tstate->async_exc); - ZAP(tstate->curexc_type); - ZAP(tstate->curexc_value); - ZAP(tstate->curexc_traceback); - - ZAP(tstate->exc_type); - ZAP(tstate->exc_value); - ZAP(tstate->exc_traceback); + Py_CLEAR(tstate->curexc_type); + Py_CLEAR(tstate->curexc_value); + Py_CLEAR(tstate->curexc_traceback); + + Py_CLEAR(tstate->exc_type); + Py_CLEAR(tstate->exc_value); + Py_CLEAR(tstate->exc_traceback); tstate->c_profilefunc = NULL; tstate->c_tracefunc = NULL; - ZAP(tstate->c_profileobj); - ZAP(tstate->c_traceobj); + Py_CLEAR(tstate->c_profileobj); + Py_CLEAR(tstate->c_traceobj); } @@ -360,7 +353,7 @@ for (p = interp->tstate_head; p != NULL; p = p->next) { if (p->thread_id != id) continue; - ZAP(p->async_exc); + Py_CLEAR(p->async_exc); Py_XINCREF(exc); p->async_exc = exc; count += 1; From buildbot at python.org Sat Apr 15 20:49:01 2006 From: buildbot at python.org (buildbot at python.org) Date: Sat, 15 Apr 2006 18:49:01 +0000 Subject: [Python-checkins] buildbot warnings in x86 XP-2 trunk Message-ID: <20060415184901.22DBE1E4005@bag.python.org> The Buildbot has detected a new failure of x86 XP-2 trunk. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520XP-2%2520trunk/builds/285 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: martin.v.loewis Build Had Warnings: warnings failed slave lost sincerely, -The Buildbot From python-checkins at python.org Sat Apr 15 22:23:54 2006 From: python-checkins at python.org (thomas.heller) Date: Sat, 15 Apr 2006 22:23:54 +0200 (CEST) Subject: [Python-checkins] r45440 - python/trunk/Modules/_ctypes/libffi/src/x86/ffi.c Message-ID: <20060415202354.DA2801E4005@bag.python.org> Author: thomas.heller Date: Sat Apr 15 22:23:54 2006 New Revision: 45440 Modified: python/trunk/Modules/_ctypes/libffi/src/x86/ffi.c Log: Fix for FreeBSD. Fixes http://python.org/sf/1470353. Looks like a libffi bug. Modified: python/trunk/Modules/_ctypes/libffi/src/x86/ffi.c ============================================================================== --- python/trunk/Modules/_ctypes/libffi/src/x86/ffi.c (original) +++ python/trunk/Modules/_ctypes/libffi/src/x86/ffi.c Sat Apr 15 22:23:54 2006 @@ -121,7 +121,7 @@ switch (cif->rtype->type) { case FFI_TYPE_VOID: -#if !defined(X86_WIN32) && !defined(__OpenBSD__) +#if !defined(X86_WIN32) && !defined(__OpenBSD__) && !defined(__FreeBSD__) case FFI_TYPE_STRUCT: #endif case FFI_TYPE_SINT64: @@ -135,7 +135,7 @@ cif->flags = FFI_TYPE_SINT64; break; -#if defined(X86_WIN32) || defined(__OpenBSD__) +#if defined(X86_WIN32) || defined(__OpenBSD__) || defined(__FreeBSD__) case FFI_TYPE_STRUCT: if (cif->rtype->size == 1) { From python-checkins at python.org Sat Apr 15 22:43:22 2006 From: python-checkins at python.org (thomas.heller) Date: Sat, 15 Apr 2006 22:43:22 +0200 (CEST) Subject: [Python-checkins] r45441 - python/trunk/PCbuild/readme.txt Message-ID: <20060415204322.AF7601E4005@bag.python.org> Author: thomas.heller Date: Sat Apr 15 22:43:22 2006 New Revision: 45441 Modified: python/trunk/PCbuild/readme.txt Log: Fix typo. Modified: python/trunk/PCbuild/readme.txt ============================================================================== --- python/trunk/PCbuild/readme.txt (original) +++ python/trunk/PCbuild/readme.txt Sat Apr 15 22:43:22 2006 @@ -381,7 +381,7 @@ The file db.build should be copied from the Python PCBuild directory to the directory db-4.4.20\build_win32. - The file db_static.vcproj in db-4.4.20\build_win32 sould be edited to + The file db_static.vcproj in db-4.4.20\build_win32 should be edited to remove the string "$(SolutionDir)" - this occurs in 2 places, only relevant for 64-bit builds. (The edit is required as otherwise, nant wants to read the solution file, which is not in a suitable form). From buildbot at python.org Sat Apr 15 23:00:13 2006 From: buildbot at python.org (buildbot at python.org) Date: Sat, 15 Apr 2006 21:00:13 +0000 Subject: [Python-checkins] buildbot failure in x86 XP-2 trunk Message-ID: <20060415210013.CB38E1E406D@bag.python.org> The Buildbot has detected a new failure of x86 XP-2 trunk. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520XP-2%2520trunk/builds/288 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: thomas.heller BUILD FAILED: failed compile sincerely, -The Buildbot From python-checkins at python.org Sat Apr 15 23:41:56 2006 From: python-checkins at python.org (thomas.wouters) Date: Sat, 15 Apr 2006 23:41:56 +0200 (CEST) Subject: [Python-checkins] r45442 - python/trunk/Objects/genobject.c Message-ID: <20060415214156.B854C1E4005@bag.python.org> Author: thomas.wouters Date: Sat Apr 15 23:41:56 2006 New Revision: 45442 Modified: python/trunk/Objects/genobject.c Log: - Whitespace normalization - In functions where we already hold the same object in differently typed pointers, use the correctly typed pointer instead of casting the other pointer a second time. Modified: python/trunk/Objects/genobject.c ============================================================================== --- python/trunk/Objects/genobject.c (original) +++ python/trunk/Objects/genobject.c Sat Apr 15 23:41:56 2006 @@ -22,12 +22,11 @@ _PyObject_GC_UNTRACK(gen); if (gen->gi_weakreflist != NULL) - PyObject_ClearWeakRefs((PyObject *) gen); - + PyObject_ClearWeakRefs(self); _PyObject_GC_TRACK(self); - if (gen->gi_frame!=NULL && gen->gi_frame->f_stacktop!=NULL) { + if (gen->gi_frame != NULL && gen->gi_frame->f_stacktop != NULL) { /* Generator is paused, so we need to close */ gen->ob_type->tp_del(self); if (self->ob_refcnt > 0) @@ -54,14 +53,16 @@ } if (f==NULL || f->f_stacktop == NULL) { /* Only set exception if called from send() */ - if (arg && !exc) PyErr_SetNone(PyExc_StopIteration); + if (arg && !exc) + PyErr_SetNone(PyExc_StopIteration); return NULL; } if (f->f_lasti == -1) { if (arg && arg != Py_None) { PyErr_SetString(PyExc_TypeError, - "can't send non-None value to a just-started generator"); + "can't send non-None value to a " + "just-started generator"); return NULL; } } else { @@ -93,7 +94,8 @@ Py_DECREF(result); result = NULL; /* Set exception if not called by gen_iternext() */ - if (arg) PyErr_SetNone(PyExc_StopIteration); + if (arg) + PyErr_SetNone(PyExc_StopIteration); } if (!result || f->f_stacktop == NULL) { @@ -127,11 +129,11 @@ if (retval) { Py_DECREF(retval); PyErr_SetString(PyExc_RuntimeError, - "generator ignored GeneratorExit"); + "generator ignored GeneratorExit"); return NULL; } - if ( PyErr_ExceptionMatches(PyExc_StopIteration) - || PyErr_ExceptionMatches(PyExc_GeneratorExit) ) + if (PyErr_ExceptionMatches(PyExc_StopIteration) + || PyErr_ExceptionMatches(PyExc_GeneratorExit)) { PyErr_Clear(); /* ignore these errors */ Py_INCREF(Py_None); @@ -147,7 +149,7 @@ PyObject *error_type, *error_value, *error_traceback; PyGenObject *gen = (PyGenObject *)self; - if (!gen->gi_frame || gen->gi_frame->f_stacktop==NULL) + if (gen->gi_frame == NULL || gen->gi_frame->f_stacktop == NULL) /* Generator isn't paused, so no need to close */ return; @@ -158,10 +160,10 @@ /* Save the current exception, if any. */ PyErr_Fetch(&error_type, &error_value, &error_traceback); - res = gen_close((PyGenObject *)self, NULL); + res = gen_close(gen, NULL); if (res == NULL) - PyErr_WriteUnraisable((PyObject *)self); + PyErr_WriteUnraisable(self); else Py_DECREF(res); From python-checkins at python.org Sat Apr 15 23:47:12 2006 From: python-checkins at python.org (thomas.wouters) Date: Sat, 15 Apr 2006 23:47:12 +0200 (CEST) Subject: [Python-checkins] r45443 - in python/trunk: Modules/_csv.c Modules/arraymodule.c Modules/cPickle.c Modules/operator.c Modules/pyexpat.c Modules/zipimport.c Objects/cellobject.c Objects/classobject.c Objects/descrobject.c Objects/dictobject.c Objects/enumobject.c Objects/funcobject.c Objects/iterobject.c Objects/listobject.c Objects/methodobject.c Objects/moduleobject.c Objects/tupleobject.c Objects/typeobject.c Objects/weakrefobject.c Python/traceback.c Message-ID: <20060415214712.9EC781E4005@bag.python.org> Author: thomas.wouters Date: Sat Apr 15 23:47:09 2006 New Revision: 45443 Modified: python/trunk/Modules/_csv.c python/trunk/Modules/arraymodule.c python/trunk/Modules/cPickle.c python/trunk/Modules/operator.c python/trunk/Modules/pyexpat.c python/trunk/Modules/zipimport.c python/trunk/Objects/cellobject.c python/trunk/Objects/classobject.c python/trunk/Objects/descrobject.c python/trunk/Objects/dictobject.c python/trunk/Objects/enumobject.c python/trunk/Objects/funcobject.c python/trunk/Objects/iterobject.c python/trunk/Objects/listobject.c python/trunk/Objects/methodobject.c python/trunk/Objects/moduleobject.c python/trunk/Objects/tupleobject.c python/trunk/Objects/typeobject.c python/trunk/Objects/weakrefobject.c python/trunk/Python/traceback.c Log: Use Py_VISIT in all tp_traverse methods, instead of traversing manually or using a custom, nearly-identical macro. This probably changes how some of these functions are compiled, which may result in fractionally slower (or faster) execution. Considering the nature of traversal, visiting much of the address space in unpredictable patterns, I'd argue the code readability and maintainability is well worth it ;P Modified: python/trunk/Modules/_csv.c ============================================================================== --- python/trunk/Modules/_csv.c (original) +++ python/trunk/Modules/_csv.c Sat Apr 15 23:47:09 2006 @@ -48,7 +48,16 @@ } \ } while (0) #endif - +#ifndef Py_VISIT +#define Py_VISIT(op) \ + do { \ + if (op) { \ + int vret = visit((PyObject *)(op), arg); \ + if (vret) \ + return vret; \ + } \ + } while (0) +#endif /* end 2.2 compatibility macros */ @@ -825,16 +834,9 @@ static int Reader_traverse(ReaderObj *self, visitproc visit, void *arg) { - int err; -#define VISIT(SLOT) \ - if (SLOT) { \ - err = visit((PyObject *)(SLOT), arg); \ - if (err) \ - return err; \ - } - VISIT(self->dialect); - VISIT(self->input_iter); - VISIT(self->fields); + Py_VISIT(self->dialect); + Py_VISIT(self->input_iter); + Py_VISIT(self->fields); return 0; } @@ -1255,15 +1257,8 @@ static int Writer_traverse(WriterObj *self, visitproc visit, void *arg) { - int err; -#define VISIT(SLOT) \ - if (SLOT) { \ - err = visit((PyObject *)(SLOT), arg); \ - if (err) \ - return err; \ - } - VISIT(self->dialect); - VISIT(self->writeline); + Py_VISIT(self->dialect); + Py_VISIT(self->writeline); return 0; } Modified: python/trunk/Modules/arraymodule.c ============================================================================== --- python/trunk/Modules/arraymodule.c (original) +++ python/trunk/Modules/arraymodule.c Sat Apr 15 23:47:09 2006 @@ -2061,8 +2061,7 @@ static int arrayiter_traverse(arrayiterobject *it, visitproc visit, void *arg) { - if (it->ao != NULL) - return visit((PyObject *)(it->ao), arg); + Py_VISIT(it->ao); return 0; } Modified: python/trunk/Modules/cPickle.c ============================================================================== --- python/trunk/Modules/cPickle.c (original) +++ python/trunk/Modules/cPickle.c Sat Apr 15 23:47:09 2006 @@ -2909,22 +2909,14 @@ static int Pickler_traverse(Picklerobject *self, visitproc visit, void *arg) { - int err; -#define VISIT(SLOT) \ - if (SLOT) { \ - err = visit((PyObject *)(SLOT), arg); \ - if (err) \ - return err; \ - } - VISIT(self->write); - VISIT(self->memo); - VISIT(self->fast_memo); - VISIT(self->arg); - VISIT(self->file); - VISIT(self->pers_func); - VISIT(self->inst_pers_func); - VISIT(self->dispatch_table); -#undef VISIT + Py_VISIT(self->write); + Py_VISIT(self->memo); + Py_VISIT(self->fast_memo); + Py_VISIT(self->arg); + Py_VISIT(self->file); + Py_VISIT(self->pers_func); + Py_VISIT(self->inst_pers_func); + Py_VISIT(self->dispatch_table); return 0; } @@ -5258,24 +5250,15 @@ static int Unpickler_traverse(Unpicklerobject *self, visitproc visit, void *arg) { - int err; - -#define VISIT(SLOT) \ - if (SLOT) { \ - err = visit((PyObject *)(SLOT), arg); \ - if (err) \ - return err; \ - } - VISIT(self->readline); - VISIT(self->read); - VISIT(self->file); - VISIT(self->memo); - VISIT(self->stack); - VISIT(self->pers_func); - VISIT(self->arg); - VISIT(self->last_string); - VISIT(self->find_class); -#undef VISIT + Py_VISIT(self->readline); + Py_VISIT(self->read); + Py_VISIT(self->file); + Py_VISIT(self->memo); + Py_VISIT(self->stack); + Py_VISIT(self->pers_func); + Py_VISIT(self->arg); + Py_VISIT(self->last_string); + Py_VISIT(self->find_class); return 0; } Modified: python/trunk/Modules/operator.c ============================================================================== --- python/trunk/Modules/operator.c (original) +++ python/trunk/Modules/operator.c Sat Apr 15 23:47:09 2006 @@ -358,8 +358,7 @@ static int itemgetter_traverse(itemgetterobject *ig, visitproc visit, void *arg) { - if (ig->item) - return visit(ig->item, arg); + Py_VISIT(ig->item); return 0; } @@ -497,8 +496,7 @@ static int attrgetter_traverse(attrgetterobject *ag, visitproc visit, void *arg) { - if (ag->attr) - return visit(ag->attr, arg); + Py_VISIT(ag->attr); return 0; } Modified: python/trunk/Modules/pyexpat.c ============================================================================== --- python/trunk/Modules/pyexpat.c (original) +++ python/trunk/Modules/pyexpat.c Sat Apr 15 23:47:09 2006 @@ -1655,13 +1655,8 @@ xmlparse_traverse(xmlparseobject *op, visitproc visit, void *arg) { int i, err; - for (i = 0; handler_info[i].name != NULL; i++) { - if (!op->handlers[i]) - continue; - err = visit(op->handlers[i], arg); - if (err) - return err; - } + for (i = 0; handler_info[i].name != NULL; i++) + Py_VISIT(op->handlers[i]); return 0; } Modified: python/trunk/Modules/zipimport.c ============================================================================== --- python/trunk/Modules/zipimport.c (original) +++ python/trunk/Modules/zipimport.c Sat Apr 15 23:47:09 2006 @@ -170,13 +170,7 @@ zipimporter_traverse(PyObject *obj, visitproc visit, void *arg) { ZipImporter *self = (ZipImporter *)obj; - int err; - - if (self->files != NULL) { - err = visit(self->files, arg); - if (err) - return err; - } + Py_VISIT(self->files); return 0; } Modified: python/trunk/Objects/cellobject.c ============================================================================== --- python/trunk/Objects/cellobject.c (original) +++ python/trunk/Objects/cellobject.c Sat Apr 15 23:47:09 2006 @@ -73,8 +73,7 @@ static int cell_traverse(PyCellObject *op, visitproc visit, void *arg) { - if (op->ob_ref) - return visit(op->ob_ref, arg); + Py_VISIT(op->ob_ref); return 0; } Modified: python/trunk/Objects/classobject.c ============================================================================== --- python/trunk/Objects/classobject.c (original) +++ python/trunk/Objects/classobject.c Sat Apr 15 23:47:09 2006 @@ -404,37 +404,12 @@ static int class_traverse(PyClassObject *o, visitproc visit, void *arg) { - int err; - if (o->cl_bases) { - err = visit(o->cl_bases, arg); - if (err) - return err; - } - if (o->cl_dict) { - err = visit(o->cl_dict, arg); - if (err) - return err; - } - if (o->cl_name) { - err = visit(o->cl_name, arg); - if (err) - return err; - } - if (o->cl_getattr) { - err = visit(o->cl_getattr, arg); - if (err) - return err; - } - if (o->cl_setattr) { - err = visit(o->cl_setattr, arg); - if (err) - return err; - } - if (o->cl_delattr) { - err = visit(o->cl_delattr, arg); - if (err) - return err; - } + Py_VISIT(o->cl_bases); + Py_VISIT(o->cl_dict); + Py_VISIT(o->cl_name); + Py_VISIT(o->cl_getattr); + Py_VISIT(o->cl_setattr); + Py_VISIT(o->cl_delattr); return 0; } @@ -979,17 +954,8 @@ static int instance_traverse(PyInstanceObject *o, visitproc visit, void *arg) { - int err; - if (o->in_class) { - err = visit((PyObject *)(o->in_class), arg); - if (err) - return err; - } - if (o->in_dict) { - err = visit(o->in_dict, arg); - if (err) - return err; - } + Py_VISIT(o->in_class); + Py_VISIT(o->in_dict); return 0; } @@ -2348,22 +2314,9 @@ static int instancemethod_traverse(PyMethodObject *im, visitproc visit, void *arg) { - int err; - if (im->im_func) { - err = visit(im->im_func, arg); - if (err) - return err; - } - if (im->im_self) { - err = visit(im->im_self, arg); - if (err) - return err; - } - if (im->im_class) { - err = visit(im->im_class, arg); - if (err) - return err; - } + Py_VISIT(im->im_func); + Py_VISIT(im->im_self); + Py_VISIT(im->im_class); return 0; } Modified: python/trunk/Objects/descrobject.c ============================================================================== --- python/trunk/Objects/descrobject.c (original) +++ python/trunk/Objects/descrobject.c Sat Apr 15 23:47:09 2006 @@ -377,13 +377,7 @@ descr_traverse(PyObject *self, visitproc visit, void *arg) { PyDescrObject *descr = (PyDescrObject *)self; - int err; - - if (descr->d_type) { - err = visit((PyObject *)(descr->d_type), arg); - if (err) - return err; - } + Py_VISIT(descr->d_type); return 0; } @@ -814,13 +808,7 @@ proxy_traverse(PyObject *self, visitproc visit, void *arg) { proxyobject *pp = (proxyobject *)self; - int err; - - if (pp->dict) { - err = visit(pp->dict, arg); - if (err) - return err; - } + Py_VISIT(pp->dict); return 0; } @@ -999,18 +987,8 @@ wrapper_traverse(PyObject *self, visitproc visit, void *arg) { wrapperobject *wp = (wrapperobject *)self; - int err; - - if (wp->descr) { - err = visit((PyObject *)(wp->descr), arg); - if (err) - return err; - } - if (wp->self) { - err = visit(wp->self, arg); - if (err) - return err; - } + Py_VISIT(wp->descr); + Py_VISIT(wp->self); return 0; } @@ -1237,20 +1215,10 @@ property_traverse(PyObject *self, visitproc visit, void *arg) { propertyobject *pp = (propertyobject *)self; - int err; - -#define VISIT(SLOT) \ - if (pp->SLOT) { \ - err = visit((PyObject *)(pp->SLOT), arg); \ - if (err) \ - return err; \ - } - - VISIT(prop_get); - VISIT(prop_set); - VISIT(prop_del); - VISIT(prop_doc); - + Py_VISIT(pp->prop_get); + Py_VISIT(pp->prop_set); + Py_VISIT(pp->prop_del); + Py_VISIT(pp->prop_doc); return 0; } Modified: python/trunk/Objects/dictobject.c ============================================================================== --- python/trunk/Objects/dictobject.c (original) +++ python/trunk/Objects/dictobject.c Sat Apr 15 23:47:09 2006 @@ -1732,17 +1732,12 @@ dict_traverse(PyObject *op, visitproc visit, void *arg) { Py_ssize_t i = 0; - int err; PyObject *pk; PyObject *pv; while (PyDict_Next(op, &i, &pk, &pv)) { - err = visit(pk, arg); - if (err) - return err; - err = visit(pv, arg); - if (err) - return err; + Py_VISIT(pk); + Py_VISIT(pv); } return 0; } Modified: python/trunk/Objects/enumobject.c ============================================================================== --- python/trunk/Objects/enumobject.c (original) +++ python/trunk/Objects/enumobject.c Sat Apr 15 23:47:09 2006 @@ -49,18 +49,8 @@ static int enum_traverse(enumobject *en, visitproc visit, void *arg) { - int err; - - if (en->en_sit) { - err = visit(en->en_sit, arg); - if (err) - return err; - } - if (en->en_result) { - err = visit(en->en_result, arg); - if (err) - return err; - } + Py_VISIT(en->en_sit); + Py_VISIT(en->en_result); return 0; } @@ -205,8 +195,7 @@ static int reversed_traverse(reversedobject *ro, visitproc visit, void *arg) { - if (ro->seq) - return visit((PyObject *)(ro->seq), arg); + Py_VISIT(ro->seq); return 0; } Modified: python/trunk/Objects/funcobject.c ============================================================================== --- python/trunk/Objects/funcobject.c (original) +++ python/trunk/Objects/funcobject.c Sat Apr 15 23:47:09 2006 @@ -466,47 +466,14 @@ static int func_traverse(PyFunctionObject *f, visitproc visit, void *arg) { - int err; - if (f->func_code) { - err = visit(f->func_code, arg); - if (err) - return err; - } - if (f->func_globals) { - err = visit(f->func_globals, arg); - if (err) - return err; - } - if (f->func_module) { - err = visit(f->func_module, arg); - if (err) - return err; - } - if (f->func_defaults) { - err = visit(f->func_defaults, arg); - if (err) - return err; - } - if (f->func_doc) { - err = visit(f->func_doc, arg); - if (err) - return err; - } - if (f->func_name) { - err = visit(f->func_name, arg); - if (err) - return err; - } - if (f->func_dict) { - err = visit(f->func_dict, arg); - if (err) - return err; - } - if (f->func_closure) { - err = visit(f->func_closure, arg); - if (err) - return err; - } + Py_VISIT(f->func_code); + Py_VISIT(f->func_globals); + Py_VISIT(f->func_module); + Py_VISIT(f->func_defaults); + Py_VISIT(f->func_doc); + Py_VISIT(f->func_name); + Py_VISIT(f->func_dict); + Py_VISIT(f->func_closure); return 0; } @@ -647,9 +614,8 @@ static int cm_traverse(classmethod *cm, visitproc visit, void *arg) { - if (!cm->cm_callable) - return 0; - return visit(cm->cm_callable, arg); + Py_VISIT(cm->cm_callable); + return 0; } static int @@ -806,9 +772,8 @@ static int sm_traverse(staticmethod *sm, visitproc visit, void *arg) { - if (!sm->sm_callable) - return 0; - return visit(sm->sm_callable, arg); + Py_VISIT(sm->sm_callable); + return 0; } static int Modified: python/trunk/Objects/iterobject.c ============================================================================== --- python/trunk/Objects/iterobject.c (original) +++ python/trunk/Objects/iterobject.c Sat Apr 15 23:47:09 2006 @@ -38,9 +38,8 @@ static int iter_traverse(seqiterobject *it, visitproc visit, void *arg) { - if (it->it_seq == NULL) - return 0; - return visit(it->it_seq, arg); + Py_VISIT(it->it_seq); + return 0; } static PyObject * @@ -162,11 +161,8 @@ static int calliter_traverse(calliterobject *it, visitproc visit, void *arg) { - int err; - if (it->it_callable != NULL && (err = visit(it->it_callable, arg))) - return err; - if (it->it_sentinel != NULL && (err = visit(it->it_sentinel, arg))) - return err; + Py_VISIT(it->it_callable); + Py_VISIT(it->it_sentinel); return 0; } Modified: python/trunk/Objects/listobject.c ============================================================================== --- python/trunk/Objects/listobject.c (original) +++ python/trunk/Objects/listobject.c Sat Apr 15 23:47:09 2006 @@ -2276,14 +2276,8 @@ Py_ssize_t i; PyObject *x; - for (i = o->ob_size; --i >= 0; ) { - x = o->ob_item[i]; - if (x != NULL) { - int err = visit(x, arg); - if (err) - return err; - } - } + for (i = o->ob_size; --i >= 0; ) + Py_VISIT(o->ob_item[i]); return 0; } @@ -2779,9 +2773,8 @@ static int listiter_traverse(listiterobject *it, visitproc visit, void *arg) { - if (it->it_seq == NULL) - return 0; - return visit((PyObject *)it->it_seq, arg); + Py_VISIT(it->it_seq); + return 0; } static PyObject * @@ -2898,9 +2891,8 @@ static int listreviter_traverse(listreviterobject *it, visitproc visit, void *arg) { - if (it->it_seq == NULL) - return 0; - return visit((PyObject *)it->it_seq, arg); + Py_VISIT(it->it_seq); + return 0; } static PyObject * Modified: python/trunk/Objects/methodobject.c ============================================================================== --- python/trunk/Objects/methodobject.c (original) +++ python/trunk/Objects/methodobject.c Sat Apr 15 23:47:09 2006 @@ -149,17 +149,8 @@ static int meth_traverse(PyCFunctionObject *m, visitproc visit, void *arg) { - int err; - if (m->m_self != NULL) { - err = visit(m->m_self, arg); - if (err) - return err; - } - if (m->m_module != NULL) { - err = visit(m->m_module, arg); - if (err) - return err; - } + Py_VISIT(m->m_self); + Py_VISIT(m->m_module); return 0; } Modified: python/trunk/Objects/moduleobject.c ============================================================================== --- python/trunk/Objects/moduleobject.c (original) +++ python/trunk/Objects/moduleobject.c Sat Apr 15 23:47:09 2006 @@ -204,8 +204,7 @@ static int module_traverse(PyModuleObject *m, visitproc visit, void *arg) { - if (m->md_dict != NULL) - return visit(m->md_dict, arg); + Py_VISIT(m->md_dict); return 0; } Modified: python/trunk/Objects/tupleobject.c ============================================================================== --- python/trunk/Objects/tupleobject.c (original) +++ python/trunk/Objects/tupleobject.c Sat Apr 15 23:47:09 2006 @@ -438,16 +438,9 @@ tupletraverse(PyTupleObject *o, visitproc visit, void *arg) { Py_ssize_t i; - PyObject *x; - for (i = o->ob_size; --i >= 0; ) { - x = o->ob_item[i]; - if (x != NULL) { - int err = visit(x, arg); - if (err) - return err; - } - } + for (i = o->ob_size; --i >= 0; ) + Py_VISIT(o->ob_item[i]); return 0; } @@ -802,9 +795,8 @@ static int tupleiter_traverse(tupleiterobject *it, visitproc visit, void *arg) { - if (it->it_seq == NULL) - return 0; - return visit((PyObject *)it->it_seq, arg); + Py_VISIT(it->it_seq); + return 0; } static PyObject * Modified: python/trunk/Objects/typeobject.c ============================================================================== --- python/trunk/Objects/typeobject.c (original) +++ python/trunk/Objects/typeobject.c Sat Apr 15 23:47:09 2006 @@ -525,21 +525,15 @@ if (type->tp_dictoffset != base->tp_dictoffset) { PyObject **dictptr = _PyObject_GetDictPtr(self); - if (dictptr && *dictptr) { - int err = visit(*dictptr, arg); - if (err) - return err; - } + if (dictptr && *dictptr) + Py_VISIT(*dictptr); } - if (type->tp_flags & Py_TPFLAGS_HEAPTYPE) { + if (type->tp_flags & Py_TPFLAGS_HEAPTYPE) /* For a heaptype, the instances count as references to the type. Traverse the type so the collector can find cycles involving this link. */ - int err = visit((PyObject *)type, arg); - if (err) - return err; - } + Py_VISIT(type); if (basetraverse) return basetraverse(self, visit, arg); @@ -2198,32 +2192,21 @@ static int type_traverse(PyTypeObject *type, visitproc visit, void *arg) { - int err; - /* Because of type_is_gc(), the collector only calls this for heaptypes. */ assert(type->tp_flags & Py_TPFLAGS_HEAPTYPE); -#define VISIT(SLOT) \ - if (SLOT) { \ - err = visit((PyObject *)(SLOT), arg); \ - if (err) \ - return err; \ - } - - VISIT(type->tp_dict); - VISIT(type->tp_cache); - VISIT(type->tp_mro); - VISIT(type->tp_bases); - VISIT(type->tp_base); + Py_VISIT(type->tp_dict); + Py_VISIT(type->tp_cache); + Py_VISIT(type->tp_mro); + Py_VISIT(type->tp_bases); + Py_VISIT(type->tp_base); /* There's no need to visit type->tp_subclasses or ((PyHeapTypeObject *)type)->ht_slots, because they can't be involved in cycles; tp_subclasses is a list of weak references, and slots is a tuple of strings. */ -#undef VISIT - return 0; } @@ -5805,20 +5788,10 @@ super_traverse(PyObject *self, visitproc visit, void *arg) { superobject *su = (superobject *)self; - int err; - -#define VISIT(SLOT) \ - if (SLOT) { \ - err = visit((PyObject *)(SLOT), arg); \ - if (err) \ - return err; \ - } - - VISIT(su->obj); - VISIT(su->type); - VISIT(su->obj_type); -#undef VISIT + Py_VISIT(su->obj); + Py_VISIT(su->type); + Py_VISIT(su->obj_type); return 0; } Modified: python/trunk/Objects/weakrefobject.c ============================================================================== --- python/trunk/Objects/weakrefobject.c (original) +++ python/trunk/Objects/weakrefobject.c Sat Apr 15 23:47:09 2006 @@ -109,8 +109,7 @@ static int gc_traverse(PyWeakReference *self, visitproc visit, void *arg) { - if (self->wr_callback != NULL) - return visit(self->wr_callback, arg); + Py_VISIT(self->wr_callback); return 0; } Modified: python/trunk/Python/traceback.c ============================================================================== --- python/trunk/Python/traceback.c (original) +++ python/trunk/Python/traceback.c Sat Apr 15 23:47:09 2006 @@ -39,15 +39,9 @@ static int tb_traverse(PyTracebackObject *tb, visitproc visit, void *arg) { - int err = 0; - if (tb->tb_next) { - err = visit((PyObject *)tb->tb_next, arg); - if (err) - return err; - } - if (tb->tb_frame) - err = visit((PyObject *)tb->tb_frame, arg); - return err; + Py_VISIT(tb->tb_next); + Py_VISIT(tb->tb_frame); + return 0; } static void From python-checkins at python.org Sun Apr 16 00:33:15 2006 From: python-checkins at python.org (thomas.wouters) Date: Sun, 16 Apr 2006 00:33:15 +0200 (CEST) Subject: [Python-checkins] r45444 - in python/trunk: Lib/test/test_generators.py Modules/itertoolsmodule.c Message-ID: <20060415223315.9361A1E4005@bag.python.org> Author: thomas.wouters Date: Sun Apr 16 00:33:13 2006 New Revision: 45444 Modified: python/trunk/Lib/test/test_generators.py python/trunk/Modules/itertoolsmodule.c Log: Add missing PyObject_GC_Track call, causing *some* itertools.tee objects to not be tracked by GC. This fixes 254 of test_generators' refleaks on my machine, but I'm sure something else will make them come back :> Not adding a separate test for this kind of cycle, since the existing fib/m235 already test them in more extensive ways than any 'minimal' test has been able to manage. Modified: python/trunk/Lib/test/test_generators.py ============================================================================== --- python/trunk/Lib/test/test_generators.py (original) +++ python/trunk/Lib/test/test_generators.py Sun Apr 16 00:33:13 2006 @@ -668,10 +668,7 @@ all and thereby wasting memory. Thanks to itertools.tee, it is now clear "how to get the internal uses of -m235 to share a single generator". Unfortunately, using generators this way -creates a reference-cycle that the garbage collector (currently) can't clean -up, so we have to explicitly break the cycle (by calling the inner -generator's close() method) +m235 to share a single generator". >>> from itertools import tee >>> def m235(): @@ -683,9 +680,9 @@ ... yield n ... m1 = _m235() ... m2, m3, m5, mRes = tee(m1, 4) -... return m1.close, mRes +... return mRes ->>> closer, it = m235() +>>> it = m235() >>> for i in range(5): ... print firstn(it, 15) [1, 2, 3, 4, 5, 6, 8, 9, 10, 12, 15, 16, 18, 20, 24] @@ -693,7 +690,6 @@ [81, 90, 96, 100, 108, 120, 125, 128, 135, 144, 150, 160, 162, 180, 192] [200, 216, 225, 240, 243, 250, 256, 270, 288, 300, 320, 324, 360, 375, 384] [400, 405, 432, 450, 480, 486, 500, 512, 540, 576, 600, 625, 640, 648, 675] ->>> closer() The "tee" function does just what we want. It internally keeps a generated result for as long as it has not been "consumed" from all of the duplicated @@ -701,11 +697,7 @@ sequence during hours without increasing memory usage, or very little. The beauty of it is that recursive running-after-their-tail FP algorithms -are quite straightforwardly expressed with this Python idiom. The problem is -that this creates an uncollectable reference cycle, and we have to explicitly -close the innermost generator to clean up the cycle. -XXX As of 14-Apr-2006, Tim doubts that anyone understands _why_ some cycle -XXX is uncollectable here. +are quite straightforwardly expressed with this Python idiom. Ye olde Fibonacci generator, tee style. @@ -724,14 +716,11 @@ ... ... realfib = _fib() ... fibHead, fibTail, fibRes = tee(realfib, 3) -... return realfib.close, fibRes +... return fibRes ->>> closer, fibber = fib() ->>> firstn(fibber, 17) +>>> firstn(fib(), 17) [1, 2, 3, 5, 8, 13, 21, 34, 55, 89, 144, 233, 377, 610, 987, 1597, 2584] ->>> closer() -XXX Again the tee-based approach leaks without an explicit close(). """ leak_test1 = """ Modified: python/trunk/Modules/itertoolsmodule.c ============================================================================== --- python/trunk/Modules/itertoolsmodule.c (original) +++ python/trunk/Modules/itertoolsmodule.c Sun Apr 16 00:33:13 2006 @@ -498,6 +498,7 @@ newto->dataobj = to->dataobj; newto->index = to->index; newto->weakreflist = NULL; + PyObject_GC_Track(newto); return (PyObject *)newto; } From python-checkins at python.org Sun Apr 16 00:44:07 2006 From: python-checkins at python.org (thomas.wouters) Date: Sun, 16 Apr 2006 00:44:07 +0200 (CEST) Subject: [Python-checkins] r45445 - python/trunk/Lib/test/test_generators.py Message-ID: <20060415224407.DA5B01E4005@bag.python.org> Author: thomas.wouters Date: Sun Apr 16 00:44:07 2006 New Revision: 45445 Modified: python/trunk/Lib/test/test_generators.py Log: Consolidate 'leak_test1' and 'refleaks_tests', since they both test for the same kind of thing. Modified: python/trunk/Lib/test/test_generators.py ============================================================================== --- python/trunk/Lib/test/test_generators.py (original) +++ python/trunk/Lib/test/test_generators.py Sun Apr 16 00:44:07 2006 @@ -723,22 +723,6 @@ """ -leak_test1 = """ - -This test leaked at one point due to generator finalization/destruction. -It was copied from Lib/test/leakers/test_generator_cycle.py before the file -was removed. - ->>> def leak(): -... def gen(): -... while True: -... yield g -... g = gen() - ->>> leak() - -""" - # syntax_tests mostly provokes SyntaxErrors. Also fiddling with #if 0 # hackery. @@ -1746,6 +1730,21 @@ >>> item = it.next() + + +This test leaked at one point due to generator finalization/destruction. +It was copied from Lib/test/leakers/test_generator_cycle.py before the file +was removed. + +>>> def leak(): +... def gen(): +... while True: +... yield g +... g = gen() + +>>> leak() + + There should be more test_generator-induced refleaks here, after they get fixed. @@ -1755,7 +1754,6 @@ "pep": pep_tests, "email": email_tests, "fun": fun_tests, - "leak1": leak_test1, "syntax": syntax_tests, "conjoin": conjoin_tests, "weakref": weakref_tests, From python-checkins at python.org Sun Apr 16 00:51:27 2006 From: python-checkins at python.org (tim.peters) Date: Sun, 16 Apr 2006 00:51:27 +0200 (CEST) Subject: [Python-checkins] r45446 - python/trunk/Objects/listobject.c python/trunk/Objects/typeobject.c Message-ID: <20060415225127.811A01E4005@bag.python.org> Author: tim.peters Date: Sun Apr 16 00:51:26 2006 New Revision: 45446 Modified: python/trunk/Objects/listobject.c python/trunk/Objects/typeobject.c Log: Remove now-unused variables from tp_traverse and tp_clear methods. Modified: python/trunk/Objects/listobject.c ============================================================================== --- python/trunk/Objects/listobject.c (original) +++ python/trunk/Objects/listobject.c Sun Apr 16 00:51:26 2006 @@ -2274,7 +2274,6 @@ list_traverse(PyListObject *o, visitproc visit, void *arg) { Py_ssize_t i; - PyObject *x; for (i = o->ob_size; --i >= 0; ) Py_VISIT(o->ob_item[i]); Modified: python/trunk/Objects/typeobject.c ============================================================================== --- python/trunk/Objects/typeobject.c (original) +++ python/trunk/Objects/typeobject.c Sun Apr 16 00:51:26 2006 @@ -2213,8 +2213,6 @@ static int type_clear(PyTypeObject *type) { - PyObject *tmp; - /* Because of type_is_gc(), the collector only calls this for heaptypes. */ assert(type->tp_flags & Py_TPFLAGS_HEAPTYPE); From python-checkins at python.org Sun Apr 16 00:59:10 2006 From: python-checkins at python.org (tim.peters) Date: Sun, 16 Apr 2006 00:59:10 +0200 (CEST) Subject: [Python-checkins] r45447 - python/trunk/Objects/genobject.c Message-ID: <20060415225910.B56471E4005@bag.python.org> Author: tim.peters Date: Sun Apr 16 00:59:10 2006 New Revision: 45447 Modified: python/trunk/Objects/genobject.c Log: gen_del(): Looks like much this was copy/pasted from slot_tp_del(), but while the latter had to cater to types that don't participate in GC, we know that generators do. That allows strengthing an assert(). Modified: python/trunk/Objects/genobject.c ============================================================================== --- python/trunk/Objects/genobject.c (original) +++ python/trunk/Objects/genobject.c Sun Apr 16 00:59:10 2006 @@ -185,7 +185,7 @@ _Py_NewReference(self); self->ob_refcnt = refcnt; } - assert(!PyType_IS_GC(self->ob_type) || + assert(PyType_IS_GC(self->ob_type) && _Py_AS_GC(self)->gc.gc_refs != _PyGC_REFS_UNTRACKED); /* If Py_REF_DEBUG, _Py_NewReference bumped _Py_RefTotal, so From buildbot at python.org Sun Apr 16 01:11:39 2006 From: buildbot at python.org (buildbot at python.org) Date: Sat, 15 Apr 2006 23:11:39 +0000 Subject: [Python-checkins] buildbot warnings in ppc Debian unstable trunk Message-ID: <20060415231139.E658E1E4086@bag.python.org> The Buildbot has detected a new failure of ppc Debian unstable trunk. Full details are available at: http://www.python.org/dev/buildbot/all/ppc%2520Debian%2520unstable%2520trunk/builds/160 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: thomas.wouters Build Had Warnings: warnings test sincerely, -The Buildbot From python-checkins at python.org Sun Apr 16 01:27:28 2006 From: python-checkins at python.org (thomas.wouters) Date: Sun, 16 Apr 2006 01:27:28 +0200 (CEST) Subject: [Python-checkins] r45448 - in python/trunk: Lib/test/leakers/test_gen1.py Lib/test/test_generators.py Python/errors.c Message-ID: <20060415232728.DA9AA1E4005@bag.python.org> Author: thomas.wouters Date: Sun Apr 16 01:27:28 2006 New Revision: 45448 Removed: python/trunk/Lib/test/leakers/test_gen1.py Modified: python/trunk/Lib/test/test_generators.py python/trunk/Python/errors.c Log: Add missing DECREF to PyErr_WriteUnraisable(). That function reports exceptions that can't be raised any further, because (for instance) they occur in __del__ methods. The coroutine tests in test_generators was triggering this leak. Remove the leakers' testcase, and add a simpler testcase that explicitly tests this leak to test_generators. test_generators now no longer leaks at all, on my machine. This fix may also solve other leaks, but my full refleakhunting run is still busy, so who knows? Deleted: /python/trunk/Lib/test/leakers/test_gen1.py ============================================================================== --- /python/trunk/Lib/test/leakers/test_gen1.py Sun Apr 16 01:27:28 2006 +++ (empty file) @@ -1,19 +0,0 @@ -import gc - -# Taken from test_generators - -def f(): - try: - yield - except GeneratorExit: - yield "foo!" - -def inner_leak(): - g = f() - g.next() - -def leak(): - inner_leak() - gc.collect() - gc.collect() - gc.collect() Modified: python/trunk/Lib/test/test_generators.py ============================================================================== --- python/trunk/Lib/test/test_generators.py (original) +++ python/trunk/Lib/test/test_generators.py Sun Apr 16 01:27:28 2006 @@ -1745,8 +1745,40 @@ >>> leak() -There should be more test_generator-induced refleaks here, after they get -fixed. + +This test isn't really generator related, but rather exception-in-cleanup +related. The coroutine tests (above) just happen to cause an exception in +the generator's __del__ (tp_del) method. We can also test for this +explicitly, without generators. We do have to redirect stderr to avoid +printing warnings and to doublecheck that we actually tested what we wanted +to test. + +>>> import sys, StringIO +>>> old = sys.stderr +>>> try: +... sys.stderr = StringIO.StringIO() +... class Leaker: +... def __del__(self): +... raise RuntimeError +... +... l = Leaker() +... del l +... err = sys.stderr.getvalue().strip() +... err.startswith( +... "Exception exceptions.RuntimeError: RuntimeError() in <" +... ) +... err.endswith("> ignored") +... len(err.splitlines()) +... finally: +... sys.stderr = old +True +True +1 + + + +These refleak tests should perhaps be in a testfile of their own, +test_generators just happened to be the test that drew these out. """ Modified: python/trunk/Python/errors.c ============================================================================== --- python/trunk/Python/errors.c (original) +++ python/trunk/Python/errors.c Sun Apr 16 01:27:28 2006 @@ -604,6 +604,7 @@ PyFile_WriteString(": ", f); PyFile_WriteObject(v, f, 0); } + Py_DECREF(moduleName); } PyFile_WriteString(" in ", f); PyFile_WriteObject(obj, f, 0); From python-checkins at python.org Sun Apr 16 02:03:00 2006 From: python-checkins at python.org (neal.norwitz) Date: Sun, 16 Apr 2006 02:03:00 +0200 (CEST) Subject: [Python-checkins] r45449 - python/trunk/Modules/pyexpat.c Message-ID: <20060416000300.406A81E4005@bag.python.org> Author: neal.norwitz Date: Sun Apr 16 02:02:59 2006 New Revision: 45449 Modified: python/trunk/Modules/pyexpat.c Log: err is no longer used Modified: python/trunk/Modules/pyexpat.c ============================================================================== --- python/trunk/Modules/pyexpat.c (original) +++ python/trunk/Modules/pyexpat.c Sun Apr 16 02:02:59 2006 @@ -1654,7 +1654,7 @@ static int xmlparse_traverse(xmlparseobject *op, visitproc visit, void *arg) { - int i, err; + int i; for (i = 0; handler_info[i].name != NULL; i++) Py_VISIT(op->handlers[i]); return 0; From buildbot at python.org Sun Apr 16 02:39:21 2006 From: buildbot at python.org (buildbot at python.org) Date: Sun, 16 Apr 2006 00:39:21 +0000 Subject: [Python-checkins] buildbot warnings in alpha Tru64 5.1 trunk Message-ID: <20060416003921.A88C01E4005@bag.python.org> The Buildbot has detected a new failure of alpha Tru64 5.1 trunk. Full details are available at: http://www.python.org/dev/buildbot/all/alpha%2520Tru64%25205.1%2520trunk/builds/243 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: thomas.wouters Build Had Warnings: warnings test sincerely, -The Buildbot From python-checkins at python.org Sun Apr 16 05:28:18 2006 From: python-checkins at python.org (neal.norwitz) Date: Sun, 16 Apr 2006 05:28:18 +0200 (CEST) Subject: [Python-checkins] r45450 - python/trunk/Modules/_sqlite/connection.c Message-ID: <20060416032818.46C731E4005@bag.python.org> Author: neal.norwitz Date: Sun Apr 16 05:28:17 2006 New Revision: 45450 Modified: python/trunk/Modules/_sqlite/connection.c Log: Fix memory leak Modified: python/trunk/Modules/_sqlite/connection.c ============================================================================== --- python/trunk/Modules/_sqlite/connection.c (original) +++ python/trunk/Modules/_sqlite/connection.c Sun Apr 16 05:28:17 2006 @@ -688,9 +688,13 @@ Py_XDECREF(self->isolation_level); + if (self->begin_statement) { + PyMem_Free(self->begin_statement); + self->begin_statement = NULL; + } + if (isolation_level == Py_None) { Py_INCREF(Py_None); - self->begin_statement = NULL; self->isolation_level = Py_None; empty = PyTuple_New(0); From buildbot at python.org Sun Apr 16 05:33:47 2006 From: buildbot at python.org (buildbot at python.org) Date: Sun, 16 Apr 2006 03:33:47 +0000 Subject: [Python-checkins] buildbot failure in x86 W2k trunk Message-ID: <20060416033347.6D84D1E4005@bag.python.org> The Buildbot has detected a new failure of x86 W2k trunk. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520W2k%2520trunk/builds/483 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: neal.norwitz BUILD FAILED: failed compile sincerely, -The Buildbot From python-checkins at python.org Sun Apr 16 05:37:19 2006 From: python-checkins at python.org (neal.norwitz) Date: Sun, 16 Apr 2006 05:37:19 +0200 (CEST) Subject: [Python-checkins] r45451 - python/trunk/Modules/_sqlite/statement.c Message-ID: <20060416033719.AED2C1E4005@bag.python.org> Author: neal.norwitz Date: Sun Apr 16 05:37:19 2006 New Revision: 45451 Modified: python/trunk/Modules/_sqlite/statement.c Log: Fix valgrind problem with invalid memory read Modified: python/trunk/Modules/_sqlite/statement.c ============================================================================== --- python/trunk/Modules/_sqlite/statement.c (original) +++ python/trunk/Modules/_sqlite/statement.c Sun Apr 16 05:37:19 2006 @@ -48,8 +48,6 @@ char* sql_cstr; self->st = NULL; - - self->st = NULL; self->in_use = 0; if (PyString_Check(sql)) { @@ -80,6 +78,7 @@ if (rc == SQLITE_OK && check_remaining_sql(tail)) { (void)sqlite3_finalize(self->st); + self->st = NULL; rc = PYSQLITE_TOO_MUCH_SQL; } From python-checkins at python.org Sun Apr 16 17:11:34 2006 From: python-checkins at python.org (thomas.wouters) Date: Sun, 16 Apr 2006 17:11:34 +0200 (CEST) Subject: [Python-checkins] r45452 - python/trunk/Lib/test/leakers/test_tee.py Message-ID: <20060416151134.6FB301E4005@bag.python.org> Author: thomas.wouters Date: Sun Apr 16 17:11:33 2006 New Revision: 45452 Removed: python/trunk/Lib/test/leakers/test_tee.py Log: This test no longer leaks, and test_generators sufficiently tests it to prevent unreported regression. Deleted: /python/trunk/Lib/test/leakers/test_tee.py ============================================================================== --- /python/trunk/Lib/test/leakers/test_tee.py Sun Apr 16 17:11:33 2006 +++ (empty file) @@ -1,25 +0,0 @@ - -# Test case taken from test_generators -# See http://mail.python.org/pipermail/python-dev/2005-November/058339.html - -from itertools import tee -import gc - -def leak(): - def inner(): - def fib(): - def yield_identity_forever(g): - while 1: - yield g - def _fib(): - for i in yield_identity_forever(head): - yield i - head, tail, result = tee(_fib(), 3) - return result - - x = fib() - x.next() - inner() - gc.collect() ; gc.collect() - # this is expected to return 0 - return gc.collect() From python-checkins at python.org Sun Apr 16 17:22:41 2006 From: python-checkins at python.org (thomas.wouters) Date: Sun, 16 Apr 2006 17:22:41 +0200 (CEST) Subject: [Python-checkins] r45453 - python/trunk/Lib/test/test_timeout.py Message-ID: <20060416152241.B1C931E4005@bag.python.org> Author: thomas.wouters Date: Sun Apr 16 17:22:41 2006 New Revision: 45453 Modified: python/trunk/Lib/test/test_timeout.py Log: Specialcase 'xs4all' (.nl/.net/.com/whatever else we have) as well as 'python.org' when deciding what server to use for the timeout tests; getting tired of seeing the test fail on all my boxes ;P This'll still allow the test to fail for hosts in the XS4ALL network that don't have an 'xs4all' hostname, so maybe it should use a fallback scheme instead. Modified: python/trunk/Lib/test/test_timeout.py ============================================================================== --- python/trunk/Lib/test/test_timeout.py (original) +++ python/trunk/Lib/test/test_timeout.py Sun Apr 16 17:22:41 2006 @@ -113,7 +113,8 @@ # If we are too close to www.python.org, this test will fail. # Pick a host that should be farther away. - if socket.getfqdn().split('.')[-2:] == ['python', 'org']: + if (socket.getfqdn().split('.')[-2:] == ['python', 'org'] or + socket.getfqdn().split('.')[-2] == 'xs4all'): self.addr_remote = ('tut.fi', 80) _t1 = time.time() From python-checkins at python.org Sun Apr 16 17:43:39 2006 From: python-checkins at python.org (thomas.wouters) Date: Sun, 16 Apr 2006 17:43:39 +0200 (CEST) Subject: [Python-checkins] r45454 - python/trunk/Lib/test/test_warnings.py Message-ID: <20060416154339.6187D1E4005@bag.python.org> Author: thomas.wouters Date: Sun Apr 16 17:43:39 2006 New Revision: 45454 Modified: python/trunk/Lib/test/test_warnings.py Log: Make test_warnings play nice with regrtest -R:: now that regrtest doesn't always reload the module (specifically, it doesn't reload if the module has a 'test_main'.) Modified: python/trunk/Lib/test/test_warnings.py ============================================================================== --- python/trunk/Lib/test/test_warnings.py (original) +++ python/trunk/Lib/test/test_warnings.py Sun Apr 16 17:43:39 2006 @@ -82,11 +82,11 @@ self.assertEqual(msg.category, 'UserWarning') def test_main(verbose=None): + # Obscure hack so that this test passes after reloads or repeated calls + # to test_main (regrtest -R). + if '__warningregistry__' in globals(): + del globals()['__warningregistry__'] test_support.run_unittest(TestModule) -# Obscure hack so that this test passes after reloads (regrtest -R). -if '__warningregistry__' in globals(): - del globals()['__warningregistry__'] - if __name__ == "__main__": test_main(verbose=True) From buildbot at python.org Sun Apr 16 18:23:11 2006 From: buildbot at python.org (buildbot at python.org) Date: Sun, 16 Apr 2006 16:23:11 +0000 Subject: [Python-checkins] buildbot warnings in sparc solaris10 gcc trunk Message-ID: <20060416162311.E26081E4005@bag.python.org> The Buildbot has detected a new failure of sparc solaris10 gcc trunk. Full details are available at: http://www.python.org/dev/buildbot/all/sparc%2520solaris10%2520gcc%2520trunk/builds/442 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: thomas.wouters Build Had Warnings: warnings test sincerely, -The Buildbot From python-checkins at python.org Sun Apr 16 18:26:29 2006 From: python-checkins at python.org (thomas.wouters) Date: Sun, 16 Apr 2006 18:26:29 +0200 (CEST) Subject: [Python-checkins] r45455 - python/trunk/Lib/test/test_timeout.py Message-ID: <20060416162629.2F3131E4005@bag.python.org> Author: thomas.wouters Date: Sun Apr 16 18:26:28 2006 New Revision: 45455 Modified: python/trunk/Lib/test/test_timeout.py Log: Make test_timeout not fail on systems with no dots in their fqdn. Modified: python/trunk/Lib/test/test_timeout.py ============================================================================== --- python/trunk/Lib/test/test_timeout.py (original) +++ python/trunk/Lib/test/test_timeout.py Sun Apr 16 18:26:28 2006 @@ -114,7 +114,7 @@ # If we are too close to www.python.org, this test will fail. # Pick a host that should be farther away. if (socket.getfqdn().split('.')[-2:] == ['python', 'org'] or - socket.getfqdn().split('.')[-2] == 'xs4all'): + socket.getfqdn().split('.')[-2:-1] == ['xs4all']): self.addr_remote = ('tut.fi', 80) _t1 = time.time() From python-checkins at python.org Sun Apr 16 20:20:06 2006 From: python-checkins at python.org (andrew.kuchling) Date: Sun, 16 Apr 2006 20:20:06 +0200 (CEST) Subject: [Python-checkins] r45456 - python/trunk/Doc/whatsnew/whatsnew25.tex Message-ID: <20060416182006.549DB1E4007@bag.python.org> Author: andrew.kuchling Date: Sun Apr 16 20:20:05 2006 New Revision: 45456 Modified: python/trunk/Doc/whatsnew/whatsnew25.tex Log: Write most of the 'writing context managers' section. I'd like comments on it, but wait for a few hours before you read it; I'm still revising it and will be tackling contextlib next. Untabify Modified: python/trunk/Doc/whatsnew/whatsnew25.tex ============================================================================== --- python/trunk/Doc/whatsnew/whatsnew25.tex (original) +++ python/trunk/Doc/whatsnew/whatsnew25.tex Sun Apr 16 20:20:05 2006 @@ -323,7 +323,7 @@ package. For example, code in the \module{A.B.C} module can do: \begin{verbatim} -from . import D # Imports A.B.D +from . import D # Imports A.B.D from .. import E # Imports A.E from ..F import G # Imports A.F.G \end{verbatim} @@ -431,7 +431,7 @@ i = 0 while i < maximum: yield i - i += 1 + i += 1 \end{verbatim} When you call \code{counter(10)}, the result is an iterator that @@ -473,11 +473,11 @@ i = 0 while i < maximum: val = (yield i) - # If value provided, change counter + # If value provided, change counter if val is not None: i = val - else: - i += 1 + else: + i += 1 \end{verbatim} And here's an example of changing the counter: @@ -578,33 +578,34 @@ %====================================================================== \section{PEP 343: The 'with' statement} -The \keyword{with} statement allows a clearer -version of code that uses \code{try...finally} blocks +The \keyword{with} statement allows a clearer version of code that +uses \code{try...finally} blocks to ensure that clean-up code is +executed. First, I'll discuss the statement as it will commonly be used, and -then I'll discuss the detailed implementation and how to write objects -(called ``context managers'') that can be used with this statement. -Most people, who will only use \keyword{with} in company with an -existing object, don't need to know these details and can -just use objects that are documented to work as context managers. -Authors of new context managers will need to understand the details of -the underlying implementation. +then a subsection will examine the implementation details and how to +write objects (called ``context managers'') that can be used with this +statement. Most people will only use \keyword{with} in company with +existing objects that are documented to work as context managers, and +don't need to know these details, so you can skip the subsection if +you like. Authors of new context managers will need to understand the +details of the underlying implementation. The \keyword{with} statement is a new control-flow structure whose basic structure is: \begin{verbatim} -with expression as variable: +with expression [as variable]: with-block \end{verbatim} The expression is evaluated, and it should result in a type of object that's called a context manager. The context manager can return a -value that will be bound to the name \var{variable}. (Note carefully: -\var{variable} is \emph{not} assigned the result of \var{expression}. -One method of the context manager is run before \var{with-block} is -executed, and another method is run after the block is done, even if -the block raised an exception. +value that can optionally be bound to the name \var{variable}. (Note +carefully: \var{variable} is \emph{not} assigned the result of +\var{expression}.) One method of the context manager is run before +\var{with-block} is executed, and another method is run after the +block is done, even if the block raised an exception. To enable the statement in Python 2.5, you need to add the following directive to your module: @@ -613,17 +614,22 @@ from __future__ import with_statement \end{verbatim} -Some standard Python objects can now behave as context managers. For -example, file objects: +The statement will always be enabled in Python 2.6. + +Some standard Python objects can now behave as context managers. File +objects are one example: \begin{verbatim} with open('/etc/passwd', 'r') as f: for line in f: print line - -# f has been automatically closed at this point. + ... more processing code ... \end{verbatim} +After this statement has executed, the file object in \var{f} will +have been automatically closed at this point, even if the 'for' loop +raised an exception part-way through the block. + The \module{threading} module's locks and condition variables also support the \keyword{with} statement: @@ -634,7 +640,7 @@ ... \end{verbatim} -The lock is acquired before the block is executed, and released once +The lock is acquired before the block is executed, and always released once the block is complete. The \module{decimal} module's contexts, which encapsulate the desired @@ -644,9 +650,8 @@ \begin{verbatim} import decimal -v1 = decimal.Decimal('578') - # Displays with default precision of 28 digits +v1 = decimal.Decimal('578') print v1.sqrt() with decimal.Context(prec=16): @@ -657,9 +662,170 @@ \subsection{Writing Context Managers} -% XXX write this +Under the hood, the \keyword{with} statement is fairly complicated. +The interface demanded of context managers contains several methods. + +A high-level explanation of the context management protocol is: + +\begin{itemize} +\item The expression is evaluated and should result in an object +that's a context manager, meaning that it has a +\method{__context__()} method. + +\item This object's \method{__context__()} method is called, and must +return a context object. + +\item The context's \method{__enter__()} method is called. +The value returned is assigned to \var{VAR}. If no \code{as \var{VAR}} +clause is present, the value is simply discarded. + +\item The code in \var{BLOCK} is executed. + +\item If \var{BLOCK} raises an exception, the context object's +\method{__exit__(\var{type}, \var{value}, \var{traceback})} is called +with the exception's information, the same values returned by +\function{sys.exc_info()}. The method's return value +controls whether the exception is re-raised: any false value +re-raises the exception, and \code{True} will result in suppressing it. +You'll only rarely want to suppress the exception; the +author of the code containing the \keyword{with} statement will +never realize anything went wrong. + +\item If \var{BLOCK} didn't raise an exception, +the context object's \method{__exit__()} is still called, +but \var{type}, \var{value}, and \var{traceback} are all \code{None}. + +\end{itemize} + +Let's think through an example. I won't present detailed code but +will only sketch the necessary code. The example will be writing a +context manager for a database that supports transactions. + +(For people unfamiliar with database terminology: a set of changes to +the database are grouped into a transaction. Transactions can be +either committed, meaning that all the changes are written into the +database, or rolled back, meaning that the changes are all discarded +and the database is unchanged. See any database textbook for more +information.) +% XXX find a shorter reference? + +Let's assume there's an object representing a database connection. +Our goal will be to let the user write code like this: + +\begin{verbatim} +db_connection = DatabaseConnection() +with db_connection as cursor: + cursor.execute('insert into ...') + cursor.execute('delete from ...') + # ... more operations ... +\end{verbatim} + +The transaction should either be committed if the code in the block +runs flawlessly, or rolled back if there's an exception. + +First, the \class{DatabaseConnection} needs a \method{__context__()} +method. Sometimes an object can be its own context manager and can +simply return \code{self}; the \module{threading} module's lock objects +can do this. For our database example, though, we need to +create a new object; I'll call this class \class{DatabaseContext}. +Our \method{__context__()} must therefore look like this: + +\begin{verbatim} +class DatabaseConnection: + ... + def __context__ (self): + return DatabaseContext(self) + + # Database interface + def cursor (self): + "Returns a cursor object and starts a new transaction" + def commit (self): + "Commits current transaction" + def rollback (self): + "Rolls back current transaction" +\end{verbatim} + +The context needs the connection object so that the connection +object's \method{commit()} or \method{rollback()} methods can be +called: + +\begin{verbatim} +class DatabaseContext: + def __init__ (self, connection): + self.connection = connection +\end{verbatim} + +The \method {__enter__()} method is pretty easy, having only +to start a new transaction. In this example, +the resulting cursor object would be a useful result, +so the method will return it. The user can +then add \code{as cursor} to their \keyword{with} statement +to bind the cursor to a variable name. + +\begin{verbatim} +class DatabaseContext: + ... + def __enter__ (self): + # Code to start a new transaction + cursor = self.connection.cursor() + return cursor +\end{verbatim} + +The \method{__exit__()} method is the most complicated because it's +where most of the work has to be done. The method has to check if an +exception occurred. If there was no exception, the transaction is +committed. The transaction is rolled back if there was an exception. +Here the code will just fall off the end of the function, returning +the default value of \code{None}. \code{None} is false, so the exception +will be re-raised automatically. If you wished, you could be more explicit +and add a \keyword{return} at the marked location. + +\begin{verbatim} +class DatabaseContext: + ... + def __exit__ (self, type, value, tb): + if tb is None: + # No exception, so commit + self.connection.commit() + else: + # Exception occurred, so rollback. + self.connection.rollback() + # return False +\end{verbatim} + +\begin{comment} +% XXX should I give the code, or is the above explanation sufficient? +\pep{343} shows the code generated for a \keyword{with} statement. A +statement such as: + +\begin{verbatim} +with EXPR as VAR: + BLOCK +\end{verbatim} + +is translated into: + +\begin{verbatim} +ctx = (EXPR).__context__() +exit = ctx.__exit__ # Not calling it yet +value = ctx.__enter__() +exc = True +try: + try: + VAR = value # Only if "as VAR" is present + BLOCK + except: + # The exceptional case is handled here + exc = False + if not exit(*sys.exc_info()): + raise +finally: + # The normal and non-local-goto cases are handled here + if exc: + exit(None, None, None) +\end{verbatim} +\end{comment} -This section still needs to be written. The new \module{contextlib} module provides some functions and a decorator that are useful for writing context managers. @@ -670,7 +836,9 @@ \begin{seealso} \seepep{343}{The ``with'' statement}{PEP written by -Guido van Rossum and Nick Coghlan. } +Guido van Rossum and Nick Coghlan. +The PEP shows the code generated for a \keyword{with} statement, +which can be helpful in learning how context managers work.} \end{seealso} @@ -887,7 +1055,7 @@ \begin{verbatim} L = ['medium', 'longest', 'short'] # Prints 'longest' -print max(L, key=len) +print max(L, key=len) # Prints 'short', because lexicographically 'short' has the largest value print max(L) \end{verbatim} @@ -1027,10 +1195,10 @@ \begin{verbatim} defaultdict(, {'c': ['cammin', 'che'], 'e': ['era'], - 'd': ['del', 'di', 'diritta'], 'm': ['mezzo', 'mi'], - 'l': ['la'], 'o': ['oscura'], 'n': ['nel', 'nostra'], - 'p': ['per'], 's': ['selva', 'smarrita'], - 'r': ['ritrovai'], 'u': ['una'], 'v': ['vita', 'via']} + 'd': ['del', 'di', 'diritta'], 'm': ['mezzo', 'mi'], + 'l': ['la'], 'o': ['oscura'], 'n': ['nel', 'nostra'], + 'p': ['per'], 's': ['selva', 'smarrita'], + 'r': ['ritrovai'], 'u': ['una'], 'v': ['vita', 'via']} \end{verbatim} The \class{deque} double-ended queue type supplied by the @@ -1415,15 +1583,15 @@ \begin{verbatim} # Old versions -h = md5.md5() -h = md5.new() +h = md5.md5() +h = md5.new() # New version h = hashlib.md5() # Old versions -h = sha.sha() -h = sha.new() +h = sha.sha() +h = sha.new() # New version h = hashlib.sha1() @@ -1435,7 +1603,7 @@ h = hashlib.sha512() # Alternative form -h = hashlib.new('md5') # Provide algorithm as a string +h = hashlib.new('md5') # Provide algorithm as a string \end{verbatim} Once a hash object has been created, its methods are the same as before: @@ -1515,9 +1683,9 @@ # Larger example for t in (('2006-03-28', 'BUY', 'IBM', 1000, 45.00), - ('2006-04-05', 'BUY', 'MSOFT', 1000, 72.00), - ('2006-04-06', 'SELL', 'IBM', 500, 53.00), - ): + ('2006-04-05', 'BUY', 'MSOFT', 1000, 72.00), + ('2006-04-06', 'SELL', 'IBM', 500, 53.00), + ): c.execute('insert into stocks values (?,?,?,?,?)', t) \end{verbatim} From python-checkins at python.org Sun Apr 16 20:45:12 2006 From: python-checkins at python.org (andrew.kuchling) Date: Sun, 16 Apr 2006 20:45:12 +0200 (CEST) Subject: [Python-checkins] r45457 - python/trunk/Doc/whatsnew/whatsnew25.tex Message-ID: <20060416184512.3B3031E4007@bag.python.org> Author: andrew.kuchling Date: Sun Apr 16 20:45:11 2006 New Revision: 45457 Modified: python/trunk/Doc/whatsnew/whatsnew25.tex Log: Describe contextlib module. (Done for today...) Modified: python/trunk/Doc/whatsnew/whatsnew25.tex ============================================================================== --- python/trunk/Doc/whatsnew/whatsnew25.tex (original) +++ python/trunk/Doc/whatsnew/whatsnew25.tex Sun Apr 16 20:45:11 2006 @@ -585,11 +585,7 @@ First, I'll discuss the statement as it will commonly be used, and then a subsection will examine the implementation details and how to write objects (called ``context managers'') that can be used with this -statement. Most people will only use \keyword{with} in company with -existing objects that are documented to work as context managers, and -don't need to know these details, so you can skip the subsection if -you like. Authors of new context managers will need to understand the -details of the underlying implementation. +statement. The \keyword{with} statement is a new control-flow structure whose basic structure is: @@ -663,7 +659,11 @@ \subsection{Writing Context Managers} Under the hood, the \keyword{with} statement is fairly complicated. -The interface demanded of context managers contains several methods. +Most people will only use \keyword{with} in company with +existing objects that are documented to work as context managers, and +don't need to know these details, so you can skip the following section if +you like. Authors of new context managers will need to understand the +details of the underlying implementation. A high-level explanation of the context management protocol is: @@ -826,19 +826,74 @@ \end{verbatim} \end{comment} +\subsection{The contextlib module\label{module-contextlib}} The new \module{contextlib} module provides some functions and a -decorator that are useful for writing context managers. -Future versions will go into more detail. +decorator that are useful for writing context managers. + +The decorator is called \function{contextmanager}, and lets you write +a simple context manager as a generator. The generator should yield +exactly one value. The code up to the \keyword{yield} will be +executed as the \method{__enter__()} method, and the value yielded +will be the method's return value that will get bound to the variable +in the \keyword{with} statement's \keyword{as} clause, if any. The +code after the \keyword{yield} will be executed in the +\method{__exit__()} method. Any exception raised in the block +will be raised by the \keyword{yield} statement. + +Our database example from the previous section could be written +using this decorator as: + +\begin{verbatim} +from contextlib import contextmanager + + at contextmanager +def db_transaction (connection): + cursor = connection.cursor() + try: + yield cursor + except: + connection.rollback() + raise + else: + connection.commit() + +db = DatabaseConnection() +with db_transaction(db) as cursor: + ... +\end{verbatim} -% XXX describe further +There's a \function{nested(\var{mgr1}, \var{mgr2}, ...)} manager that +combines a number of context managers so you don't need to write +nested \keyword{with} statements. This example +both uses a database transaction and also acquires a thread lock: + +\begin{verbatim} +lock = threading.Lock() +with nested (db_transaction(db), lock) as (cursor, locked): + ... +\end{verbatim} + +Finally, the \function{closing(\var{object})} context manager +returns \var{object} so that it can be bound to a variable, +and calls \code{\var{object}.close()} at the end of the block. + +\begin{verbatim} +with closing(open('/tmp/file', 'r')) as f: + for line in f: + ... +\end{verbatim} \begin{seealso} -\seepep{343}{The ``with'' statement}{PEP written by -Guido van Rossum and Nick Coghlan. -The PEP shows the code generated for a \keyword{with} statement, -which can be helpful in learning how context managers work.} +\seepep{343}{The ``with'' statement}{PEP written by Guido van Rossum +and Nick Coghlan; implemented by Mike Bland, Guido van Rossum, and +Neal Norwitz. The PEP shows the code generated for a \keyword{with} +statement, which can be helpful in learning how context managers +work.} + +\seeurl{../lib/module-contextlib.html}{The documentation +for the \module{contextlib} module.} \end{seealso} @@ -1140,12 +1195,11 @@ %====================================================================== \section{New, Improved, and Deprecated Modules} -As usual, Python's standard library received many enhancements and -bug fixes. Here's a partial list of the most notable changes, sorted -alphabetically by module name. Consult the -\file{Misc/NEWS} file in the source tree for a more -complete list of changes, or look through the SVN logs for all the -details. +The standard library received many enhancements and bug fixes in +Python 2.5. Here's a partial list of the most notable changes, sorted +alphabetically by module name. Consult the \file{Misc/NEWS} file in +the source tree for a more complete list of changes, or look through +the SVN logs for all the details. \begin{itemize} @@ -1206,6 +1260,10 @@ method that removes the first occurrence of \var{value} in the queue, raising \exception{ValueError} if the value isn't found. +\item The \module{contextlib} module contains helper functions for use +with the new \keyword{with} statement. See section~\ref{module-contextlib} +for more about this module. (Contributed by Phillip J. Eby.) + \item The \module{cProfile} module is a C implementation of the existing \module{profile} module that has much lower overhead. The module's interface is the same as \module{profile}: you run From python-checkins at python.org Sun Apr 16 20:55:07 2006 From: python-checkins at python.org (martin.v.loewis) Date: Sun, 16 Apr 2006 20:55:07 +0200 (CEST) Subject: [Python-checkins] r45458 - python/trunk/Misc/SpecialBuilds.txt Message-ID: <20060416185507.E6A8C1E4007@bag.python.org> Author: martin.v.loewis Date: Sun Apr 16 20:55:07 2006 New Revision: 45458 Modified: python/trunk/Misc/SpecialBuilds.txt Log: Update instructions for EXTRA_CFLAGS: configure ignores them; they have to be passed to make. Modified: python/trunk/Misc/SpecialBuilds.txt ============================================================================== --- python/trunk/Misc/SpecialBuilds.txt (original) +++ python/trunk/Misc/SpecialBuilds.txt Sun Apr 16 20:55:07 2006 @@ -1,8 +1,8 @@ This file describes some special Python build types enabled via compile-time preprocessor defines. -It is best to define these options in the EXTRA_CFLAGS environment variable; -``EXTRA_CFLAGS="-DPy_REF_DEBUG" ./configure``. +It is best to define these options in the EXTRA_CFLAGS make variable; +``make EXTRA_CFLAGS="-DPy_REF_DEBUG"``. --------------------------------------------------------------------------- Py_REF_DEBUG introduced in 1.4 From python-checkins at python.org Sun Apr 16 20:55:51 2006 From: python-checkins at python.org (martin.v.loewis) Date: Sun, 16 Apr 2006 20:55:51 +0200 (CEST) Subject: [Python-checkins] r45459 - in python/trunk: Misc/NEWS Modules/_lsprof.c Modules/grpmodule.c Modules/posixmodule.c Modules/pwdmodule.c Modules/resource.c Modules/spwdmodule.c Modules/timemodule.c Message-ID: <20060416185551.D501D1E4007@bag.python.org> Author: martin.v.loewis Date: Sun Apr 16 20:55:50 2006 New Revision: 45459 Modified: python/trunk/Misc/NEWS python/trunk/Modules/_lsprof.c python/trunk/Modules/grpmodule.c python/trunk/Modules/posixmodule.c python/trunk/Modules/pwdmodule.c python/trunk/Modules/resource.c python/trunk/Modules/spwdmodule.c python/trunk/Modules/timemodule.c Log: Initialize structseq types only once. Modified: python/trunk/Misc/NEWS ============================================================================== --- python/trunk/Misc/NEWS (original) +++ python/trunk/Misc/NEWS Sun Apr 16 20:55:50 2006 @@ -12,6 +12,10 @@ Core and builtins ----------------- +- All uses of PyStructSequence_InitType have been changed to initialize + the type objects only once, even if the interpreter is initialized + multiple times. + - Bug #1454485, array.array('u') could crash the interpreter. This was due to PyArgs_ParseTuple(args, 'u#', ...) trying to convert buffers (strings) to unicode when it didn't make sense. 'u#' now requires a unicode string. Modified: python/trunk/Modules/_lsprof.c ============================================================================== --- python/trunk/Modules/_lsprof.c (original) +++ python/trunk/Modules/_lsprof.c Sun Apr 16 20:55:50 2006 @@ -515,6 +515,7 @@ 5 }; +static int initialized; static PyTypeObject StatsEntryType; static PyTypeObject StatsSubEntryType; @@ -857,8 +858,12 @@ return; PyDict_SetItemString(d, "Profiler", (PyObject *)&PyProfiler_Type); - PyStructSequence_InitType(&StatsEntryType, &profiler_entry_desc); - PyStructSequence_InitType(&StatsSubEntryType, &profiler_subentry_desc); + if (!initialized) { + PyStructSequence_InitType(&StatsEntryType, + &profiler_entry_desc); + PyStructSequence_InitType(&StatsSubEntryType, + &profiler_subentry_desc); + } Py_INCREF((PyObject*) &StatsEntryType); Py_INCREF((PyObject*) &StatsSubEntryType); PyModule_AddObject(module, "profiler_entry", @@ -866,4 +871,5 @@ PyModule_AddObject(module, "profiler_subentry", (PyObject*) &StatsSubEntryType); empty_tuple = PyTuple_New(0); + initialized = 1; } Modified: python/trunk/Modules/grpmodule.c ============================================================================== --- python/trunk/Modules/grpmodule.c (original) +++ python/trunk/Modules/grpmodule.c Sun Apr 16 20:55:50 2006 @@ -29,6 +29,7 @@ }; +static int initialized; static PyTypeObject StructGrpType; static PyObject * @@ -174,6 +175,8 @@ if (m == NULL) return; d = PyModule_GetDict(m); - PyStructSequence_InitType(&StructGrpType, &struct_group_type_desc); + if (!initialized) + PyStructSequence_InitType(&StructGrpType, &struct_group_type_desc); PyDict_SetItemString(d, "struct_group", (PyObject *) &StructGrpType); + initialized = 1; } Modified: python/trunk/Modules/posixmodule.c ============================================================================== --- python/trunk/Modules/posixmodule.c (original) +++ python/trunk/Modules/posixmodule.c Sun Apr 16 20:55:50 2006 @@ -981,6 +981,7 @@ 10 }; +static int initialized; static PyTypeObject StatResultType; static PyTypeObject StatVFSResultType; static newfunc structseq_new; @@ -8241,21 +8242,24 @@ posix_putenv_garbage = PyDict_New(); #endif - stat_result_desc.name = MODNAME ".stat_result"; - stat_result_desc.fields[7].name = PyStructSequence_UnnamedField; - stat_result_desc.fields[8].name = PyStructSequence_UnnamedField; - stat_result_desc.fields[9].name = PyStructSequence_UnnamedField; - PyStructSequence_InitType(&StatResultType, &stat_result_desc); - structseq_new = StatResultType.tp_new; - StatResultType.tp_new = statresult_new; + if (!initialized) { + stat_result_desc.name = MODNAME ".stat_result"; + stat_result_desc.fields[7].name = PyStructSequence_UnnamedField; + stat_result_desc.fields[8].name = PyStructSequence_UnnamedField; + stat_result_desc.fields[9].name = PyStructSequence_UnnamedField; + PyStructSequence_InitType(&StatResultType, &stat_result_desc); + structseq_new = StatResultType.tp_new; + StatResultType.tp_new = statresult_new; + + statvfs_result_desc.name = MODNAME ".statvfs_result"; + PyStructSequence_InitType(&StatVFSResultType, &statvfs_result_desc); + } Py_INCREF((PyObject*) &StatResultType); PyModule_AddObject(m, "stat_result", (PyObject*) &StatResultType); - - statvfs_result_desc.name = MODNAME ".statvfs_result"; - PyStructSequence_InitType(&StatVFSResultType, &statvfs_result_desc); Py_INCREF((PyObject*) &StatVFSResultType); PyModule_AddObject(m, "statvfs_result", (PyObject*) &StatVFSResultType); + initialized = 1; } #ifdef __cplusplus Modified: python/trunk/Modules/pwdmodule.c ============================================================================== --- python/trunk/Modules/pwdmodule.c (original) +++ python/trunk/Modules/pwdmodule.c Sun Apr 16 20:55:50 2006 @@ -42,6 +42,7 @@ exception is raised if the entry asked for cannot be found."); +static int initialized; static PyTypeObject StructPwdType; static void @@ -186,9 +187,12 @@ if (m == NULL) return; - PyStructSequence_InitType(&StructPwdType, &struct_pwd_type_desc); + if (!initialized) + PyStructSequence_InitType(&StructPwdType, + &struct_pwd_type_desc); Py_INCREF((PyObject *) &StructPwdType); PyModule_AddObject(m, "struct_passwd", (PyObject *) &StructPwdType); /* And for b/w compatibility (this was defined by mistake): */ PyModule_AddObject(m, "struct_pwent", (PyObject *) &StructPwdType); + initialized = 1; } Modified: python/trunk/Modules/resource.c ============================================================================== --- python/trunk/Modules/resource.c (original) +++ python/trunk/Modules/resource.c Sun Apr 16 20:55:50 2006 @@ -55,6 +55,7 @@ 16 /* n_in_sequence */ }; +static int initialized; static PyTypeObject StructRUsageType; static PyObject * @@ -244,7 +245,10 @@ } Py_INCREF(ResourceError); PyModule_AddObject(m, "error", ResourceError); - PyStructSequence_InitType(&StructRUsageType, &struct_rusage_desc); + if (!initialized) + PyStructSequence_InitType(&StructRUsageType, + &struct_rusage_desc); + Py_INCREF(&StructRUsageType); PyModule_AddObject(m, "struct_rusage", (PyObject*) &StructRUsageType); @@ -320,4 +324,5 @@ if (v) { PyModule_AddObject(m, "RLIM_INFINITY", v); } + initialized = 1; } Modified: python/trunk/Modules/spwdmodule.c ============================================================================== --- python/trunk/Modules/spwdmodule.c (original) +++ python/trunk/Modules/spwdmodule.c Sun Apr 16 20:55:50 2006 @@ -52,6 +52,7 @@ 9, }; +static int initialized; static PyTypeObject StructSpwdType; @@ -173,7 +174,10 @@ m=Py_InitModule3("spwd", spwd_methods, spwd__doc__); if (m == NULL) return; - PyStructSequence_InitType(&StructSpwdType, &struct_spwd_type_desc); + if (!initialized) + PyStructSequence_InitType(&StructSpwdType, + &struct_spwd_type_desc); Py_INCREF((PyObject *) &StructSpwdType); PyModule_AddObject(m, "struct_spwd", (PyObject *) &StructSpwdType); + initialized = 1; } Modified: python/trunk/Modules/timemodule.c ============================================================================== --- python/trunk/Modules/timemodule.c (original) +++ python/trunk/Modules/timemodule.c Sun Apr 16 20:55:50 2006 @@ -228,6 +228,7 @@ 9, }; +static int initialized; static PyTypeObject StructTimeType; static PyObject * @@ -807,9 +808,13 @@ hInterruptEvent = CreateEvent(NULL, TRUE, FALSE, NULL); SetConsoleCtrlHandler( PyCtrlHandler, TRUE); #endif /* MS_WINDOWS */ - PyStructSequence_InitType(&StructTimeType, &struct_time_type_desc); + if (!initialized) { + PyStructSequence_InitType(&StructTimeType, + &struct_time_type_desc); + } Py_INCREF(&StructTimeType); PyModule_AddObject(m, "struct_time", (PyObject*) &StructTimeType); + initialized = 1; } From buildbot at python.org Sun Apr 16 21:05:34 2006 From: buildbot at python.org (buildbot at python.org) Date: Sun, 16 Apr 2006 19:05:34 +0000 Subject: [Python-checkins] buildbot warnings in alpha Tru64 5.1 trunk Message-ID: <20060416190534.DA7611E4007@bag.python.org> The Buildbot has detected a new failure of alpha Tru64 5.1 trunk. Full details are available at: http://www.python.org/dev/buildbot/all/alpha%2520Tru64%25205.1%2520trunk/builds/250 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: andrew.kuchling Build Had Warnings: warnings test sincerely, -The Buildbot From python-checkins at python.org Sun Apr 16 21:53:28 2006 From: python-checkins at python.org (andrew.kuchling) Date: Sun, 16 Apr 2006 21:53:28 +0200 (CEST) Subject: [Python-checkins] r45460 - python/trunk/Doc/whatsnew/whatsnew25.tex Message-ID: <20060416195328.246DD1E4007@bag.python.org> Author: andrew.kuchling Date: Sun Apr 16 21:53:27 2006 New Revision: 45460 Modified: python/trunk/Doc/whatsnew/whatsnew25.tex Log: Add item Modified: python/trunk/Doc/whatsnew/whatsnew25.tex ============================================================================== --- python/trunk/Doc/whatsnew/whatsnew25.tex (original) +++ python/trunk/Doc/whatsnew/whatsnew25.tex Sun Apr 16 21:53:27 2006 @@ -2,7 +2,6 @@ \usepackage{distutils} % $Id$ -% Writing context managers % The easy_install stuff % Stateful codec changes % Fix XXX comments @@ -1178,8 +1177,9 @@ and as a result sets will use a third less memory and are somewhat faster. (Implemented by Raymond Hettinger.) -\item The performance of some Unicode operations has been improved. -% XXX provide details? +\item The performance of some Unicode operations, such as +character map decoding, has been improved. +% Patch 1313939 \item The code generator's peephole optimizer now performs simple constant folding in expressions. If you write something like From theller at python.net Sun Apr 16 22:22:28 2006 From: theller at python.net (Thomas Heller) Date: Sun, 16 Apr 2006 22:22:28 +0200 Subject: [Python-checkins] Python Regression Test Failures refleak (1) In-Reply-To: References: <20060413210453.GA27083@python.psfb.org> <443EC10F.9040707@python.net> <1f7befae0604131542v3ef6d68buf902ef51af305d3b@mail.gmail.com> Message-ID: <4442A784.7050909@python.net> Neal Norwitz wrote: > On 4/13/06, Tim Peters wrote: >> [Thomas Heller] >>> Neal, does this mean that test_ctypes does not leak anymore, or is it >>> somehow excluded from this list? >> I'm guessing, but I believe Neal uses Misc/build.sh for this, and >> ctypes is now in its LEAKY_TESTS pattern of test names to skip: >> >> LEAKY_TESTS="test_(cmd_line|ctypes|filecmp|socket|threadedtempfile|threading|threading_local|urllib2)" > > Tim's right, the leak report comes from Misc/build.sh (same with > building the docs and other emails sent to python-checkins). ctypes > leaks inconsistently which is why it's in the LEAKY_TESTS. In this > case it didn't show any leaks. I don't know if the leaks will return. > When the leaks aren't consistent and report the same number, it > usually means there are references elsewhere that may or may not be > cleaned up. > > However, ctypes still leaks when reloaded. We probably ought to have > a test for each module that just does reload(module) and see if that > leaks. I wouldn't be surprised if several C modules leak. > I've added a tracker item for that. https://sourceforge.net/tracker/index.php?func=detail&aid=1471423&group_id=5470&atid=105470 Thomas From python-checkins at python.org Sun Apr 16 22:55:39 2006 From: python-checkins at python.org (martin.v.loewis) Date: Sun, 16 Apr 2006 22:55:39 +0200 (CEST) Subject: [Python-checkins] r45461 - in python/trunk: Lib/lib-tk/Tkinter.py Misc/NEWS Message-ID: <20060416205539.8E6751E4007@bag.python.org> Author: martin.v.loewis Date: Sun Apr 16 22:55:38 2006 New Revision: 45461 Modified: python/trunk/Lib/lib-tk/Tkinter.py python/trunk/Misc/NEWS Log: Patch #1063914: Add clipboard_get. Modified: python/trunk/Lib/lib-tk/Tkinter.py ============================================================================== --- python/trunk/Lib/lib-tk/Tkinter.py (original) +++ python/trunk/Lib/lib-tk/Tkinter.py Sun Apr 16 22:55:38 2006 @@ -483,7 +483,24 @@ def bell(self, displayof=0): """Ring a display's bell.""" self.tk.call(('bell',) + self._displayof(displayof)) + # Clipboard handling: + def clipboard_get(self, **kw): + """Retrieve data from the clipboard on window's display. + + The window keyword defaults to the root window of the Tkinter + application. + + The type keyword specifies the form in which the data is + to be returned and should be an atom name such as STRING + or FILE_NAME. Type defaults to STRING. + + This command is equivalent to: + + selection_get(CLIPBOARD) + """ + return self.tk.call(('clipboard', 'get') + self._options(kw)) + def clipboard_clear(self, **kw): """Clear the data in the Tk clipboard. Modified: python/trunk/Misc/NEWS ============================================================================== --- python/trunk/Misc/NEWS (original) +++ python/trunk/Misc/NEWS Sun Apr 16 22:55:38 2006 @@ -72,6 +72,8 @@ Library ------- +- Patch #1063914: Add Tkinter.Misc.clipboard_get(). + - Patch #1191700: Adjust column alignment in bdb breakpoint lists. - SimpleXMLRPCServer relied on the fcntl module, which is unavailable on From python-checkins at python.org Mon Apr 17 00:04:50 2006 From: python-checkins at python.org (thomas.wouters) Date: Mon, 17 Apr 2006 00:04:50 +0200 (CEST) Subject: [Python-checkins] r45462 - python/trunk/Objects/unicodeobject.c Message-ID: <20060416220450.27BC91E4007@bag.python.org> Author: thomas.wouters Date: Mon Apr 17 00:04:49 2006 New Revision: 45462 Modified: python/trunk/Objects/unicodeobject.c Log: Use %zd instead of %i as format character (in call to PyErr_Format) for Py_ssize_t argument. Modified: python/trunk/Objects/unicodeobject.c ============================================================================== --- python/trunk/Objects/unicodeobject.c (original) +++ python/trunk/Objects/unicodeobject.c Mon Apr 17 00:04:49 2006 @@ -4219,7 +4219,7 @@ /* Convert item to Unicode. */ if (! PyUnicode_Check(item) && ! PyString_Check(item)) { PyErr_Format(PyExc_TypeError, - "sequence item %i: expected string or Unicode," + "sequence item %zd: expected string or Unicode," " %.80s found", i, item->ob_type->tp_name); goto onError; From python-checkins at python.org Mon Apr 17 00:09:04 2006 From: python-checkins at python.org (tim.peters) Date: Mon, 17 Apr 2006 00:09:04 +0200 (CEST) Subject: [Python-checkins] r45463 - peps/trunk/pep-0358.txt peps/trunk/pep-3001.txt peps/trunk/pep-3099.txt peps/trunk/pep2pyramid.py Message-ID: <20060416220904.B36A81E4007@bag.python.org> Author: tim.peters Date: Mon Apr 17 00:09:04 2006 New Revision: 45463 Modified: peps/trunk/pep-0358.txt (props changed) peps/trunk/pep-3001.txt (props changed) peps/trunk/pep-3099.txt (props changed) peps/trunk/pep2pyramid.py (props changed) Log: Add missing SVN eol-style properties to text files. From python-checkins at python.org Mon Apr 17 00:09:59 2006 From: python-checkins at python.org (tim.peters) Date: Mon, 17 Apr 2006 00:09:59 +0200 (CEST) Subject: [Python-checkins] r45464 - peps/trunk/pep-3099.txt Message-ID: <20060416220959.95C071E4007@bag.python.org> Author: tim.peters Date: Mon Apr 17 00:09:59 2006 New Revision: 45464 Modified: peps/trunk/pep-3099.txt Log: Clarify that iterating over a dict already yields the keys. Modified: peps/trunk/pep-3099.txt ============================================================================== --- peps/trunk/pep-3099.txt (original) +++ peps/trunk/pep-3099.txt Mon Apr 17 00:09:59 2006 @@ -45,7 +45,7 @@ Standard types ============== -* Iterating over a dictionary will yield the keys. +* Iterating over a dictionary will continue to yield the keys. Thread: "Iterating over a dict", http://mail.python.org/pipermail/python-3000/2006-April/000283.html From python-checkins at python.org Mon Apr 17 00:11:30 2006 From: python-checkins at python.org (tim.peters) Date: Mon, 17 Apr 2006 00:11:30 +0200 (CEST) Subject: [Python-checkins] r45465 - in python/trunk: Lib/bsddb/test/test_pickle.py Lib/ctypes/test/test_unaligned_structures.py Lib/email/mime/__init__.py Lib/email/mime/application.py Lib/sqlite3/__init__.py Lib/sqlite3/dbapi2.py Lib/sqlite3/test/__init__.py Lib/sqlite3/test/dbapi.py Lib/sqlite3/test/factory.py Lib/sqlite3/test/hooks.py Lib/sqlite3/test/regression.py Lib/sqlite3/test/transactions.py Lib/sqlite3/test/types.py Lib/sqlite3/test/userfunctions.py Lib/test/leakers/test_selftype.py Lib/test/test_sqlite.py Lib/test/test_startfile.py Modules/_sqlite/adapters.c Modules/_sqlite/adapters.h Modules/_sqlite/cache.c Modules/_sqlite/cache.h Modules/_sqlite/connection.c Modules/_sqlite/connection.h Modules/_sqlite/converters.c Modules/_sqlite/converters.h Modules/_sqlite/cursor.c Modules/_sqlite/cursor.h Modules/_sqlite/microprotocols.c Modules/_sqlite/microprotocols.h Modules/_sqlite/module.c Modules/_sqlite/module.h Modules/_sqlite/prepare_protocol.c Modules/_sqlite/prepare_protocol.h Modules/_sqlite/row.c Modules/_sqlite/row.h Modules/_sqlite/sqlitecompat.h Modules/_sqlite/statement.c Modules/_sqlite/statement.h Modules/_sqlite/util.c Modules/_sqlite/util.h Modules/md5.c Tools/unicode/gencjkcodecs.py Message-ID: <20060416221130.1BFB31E4007@bag.python.org> Author: tim.peters Date: Mon Apr 17 00:11:28 2006 New Revision: 45465 Modified: python/trunk/Lib/bsddb/test/test_pickle.py (props changed) python/trunk/Lib/ctypes/test/test_unaligned_structures.py (contents, props changed) python/trunk/Lib/email/mime/__init__.py (props changed) python/trunk/Lib/email/mime/application.py (props changed) python/trunk/Lib/sqlite3/__init__.py (props changed) python/trunk/Lib/sqlite3/dbapi2.py (props changed) python/trunk/Lib/sqlite3/test/__init__.py (props changed) python/trunk/Lib/sqlite3/test/dbapi.py (props changed) python/trunk/Lib/sqlite3/test/factory.py (props changed) python/trunk/Lib/sqlite3/test/hooks.py (props changed) python/trunk/Lib/sqlite3/test/regression.py (props changed) python/trunk/Lib/sqlite3/test/transactions.py (props changed) python/trunk/Lib/sqlite3/test/types.py (props changed) python/trunk/Lib/sqlite3/test/userfunctions.py (props changed) python/trunk/Lib/test/leakers/test_selftype.py (props changed) python/trunk/Lib/test/test_sqlite.py (props changed) python/trunk/Lib/test/test_startfile.py (props changed) python/trunk/Modules/_sqlite/adapters.c (props changed) python/trunk/Modules/_sqlite/adapters.h (props changed) python/trunk/Modules/_sqlite/cache.c (props changed) python/trunk/Modules/_sqlite/cache.h (props changed) python/trunk/Modules/_sqlite/connection.c (props changed) python/trunk/Modules/_sqlite/connection.h (props changed) python/trunk/Modules/_sqlite/converters.c (props changed) python/trunk/Modules/_sqlite/converters.h (props changed) python/trunk/Modules/_sqlite/cursor.c (props changed) python/trunk/Modules/_sqlite/cursor.h (props changed) python/trunk/Modules/_sqlite/microprotocols.c (props changed) python/trunk/Modules/_sqlite/microprotocols.h (props changed) python/trunk/Modules/_sqlite/module.c (props changed) python/trunk/Modules/_sqlite/module.h (props changed) python/trunk/Modules/_sqlite/prepare_protocol.c (props changed) python/trunk/Modules/_sqlite/prepare_protocol.h (props changed) python/trunk/Modules/_sqlite/row.c (props changed) python/trunk/Modules/_sqlite/row.h (props changed) python/trunk/Modules/_sqlite/sqlitecompat.h (props changed) python/trunk/Modules/_sqlite/statement.c (props changed) python/trunk/Modules/_sqlite/statement.h (props changed) python/trunk/Modules/_sqlite/util.c (props changed) python/trunk/Modules/_sqlite/util.h (props changed) python/trunk/Modules/md5.c (props changed) python/trunk/Tools/unicode/gencjkcodecs.py (props changed) Log: Add missing SVN eol-style property to text files. Modified: python/trunk/Lib/ctypes/test/test_unaligned_structures.py ============================================================================== --- python/trunk/Lib/ctypes/test/test_unaligned_structures.py (original) +++ python/trunk/Lib/ctypes/test/test_unaligned_structures.py Mon Apr 17 00:11:28 2006 @@ -1,45 +1,45 @@ -import sys, unittest -from ctypes import * - -structures = [] -byteswapped_structures = [] - - -if sys.byteorder == "little": - SwappedStructure = BigEndianStructure -else: - SwappedStructure = LittleEndianStructure - -for typ in [c_short, c_int, c_long, c_longlong, - c_float, c_double, - c_ushort, c_uint, c_ulong, c_ulonglong]: - class X(Structure): - _pack_ = 1 - _fields_ = [("pad", c_byte), - ("value", typ)] - class Y(SwappedStructure): - _pack_ = 1 - _fields_ = [("pad", c_byte), - ("value", typ)] - structures.append(X) - byteswapped_structures.append(Y) - -class TestStructures(unittest.TestCase): - def test_native(self): - for typ in structures: -## print typ.value - self.failUnlessEqual(typ.value.offset, 1) - o = typ() - o.value = 4 - self.failUnlessEqual(o.value, 4) - - def test_swapped(self): - for typ in byteswapped_structures: -## print >> sys.stderr, typ.value - self.failUnlessEqual(typ.value.offset, 1) - o = typ() - o.value = 4 - self.failUnlessEqual(o.value, 4) - -if __name__ == '__main__': - unittest.main() +import sys, unittest +from ctypes import * + +structures = [] +byteswapped_structures = [] + + +if sys.byteorder == "little": + SwappedStructure = BigEndianStructure +else: + SwappedStructure = LittleEndianStructure + +for typ in [c_short, c_int, c_long, c_longlong, + c_float, c_double, + c_ushort, c_uint, c_ulong, c_ulonglong]: + class X(Structure): + _pack_ = 1 + _fields_ = [("pad", c_byte), + ("value", typ)] + class Y(SwappedStructure): + _pack_ = 1 + _fields_ = [("pad", c_byte), + ("value", typ)] + structures.append(X) + byteswapped_structures.append(Y) + +class TestStructures(unittest.TestCase): + def test_native(self): + for typ in structures: +## print typ.value + self.failUnlessEqual(typ.value.offset, 1) + o = typ() + o.value = 4 + self.failUnlessEqual(o.value, 4) + + def test_swapped(self): + for typ in byteswapped_structures: +## print >> sys.stderr, typ.value + self.failUnlessEqual(typ.value.offset, 1) + o = typ() + o.value = 4 + self.failUnlessEqual(o.value, 4) + +if __name__ == '__main__': + unittest.main() From python-checkins at python.org Mon Apr 17 00:13:12 2006 From: python-checkins at python.org (tim.peters) Date: Mon, 17 Apr 2006 00:13:12 +0200 (CEST) Subject: [Python-checkins] r45466 - in python/branches/release24-maint: Lib/bsddb/test/test_1413192.py Lib/bsddb/test/test_pickle.py Lib/email/test/data/msg_44.txt Tools/msi/uuids.py Message-ID: <20060416221312.B5FFF1E4007@bag.python.org> Author: tim.peters Date: Mon Apr 17 00:13:11 2006 New Revision: 45466 Modified: python/branches/release24-maint/Lib/bsddb/test/test_1413192.py (props changed) python/branches/release24-maint/Lib/bsddb/test/test_pickle.py (props changed) python/branches/release24-maint/Lib/email/test/data/msg_44.txt (props changed) python/branches/release24-maint/Tools/msi/uuids.py (props changed) Log: Add missing SVN eol-style property to text files. From python-checkins at python.org Mon Apr 17 00:22:37 2006 From: python-checkins at python.org (tim.peters) Date: Mon, 17 Apr 2006 00:22:37 +0200 (CEST) Subject: [Python-checkins] r45467 - python/trunk/Lib/lib-tk/Tkinter.py Message-ID: <20060416222237.6FAB81E4007@bag.python.org> Author: tim.peters Date: Mon Apr 17 00:22:36 2006 New Revision: 45467 Modified: python/trunk/Lib/lib-tk/Tkinter.py Log: Whitespace normalization. Modified: python/trunk/Lib/lib-tk/Tkinter.py ============================================================================== --- python/trunk/Lib/lib-tk/Tkinter.py (original) +++ python/trunk/Lib/lib-tk/Tkinter.py Mon Apr 17 00:22:36 2006 @@ -483,24 +483,24 @@ def bell(self, displayof=0): """Ring a display's bell.""" self.tk.call(('bell',) + self._displayof(displayof)) - + # Clipboard handling: def clipboard_get(self, **kw): """Retrieve data from the clipboard on window's display. - - The window keyword defaults to the root window of the Tkinter + + The window keyword defaults to the root window of the Tkinter application. - - The type keyword specifies the form in which the data is - to be returned and should be an atom name such as STRING - or FILE_NAME. Type defaults to STRING. - + + The type keyword specifies the form in which the data is + to be returned and should be an atom name such as STRING + or FILE_NAME. Type defaults to STRING. + This command is equivalent to: - + selection_get(CLIPBOARD) """ return self.tk.call(('clipboard', 'get') + self._options(kw)) - + def clipboard_clear(self, **kw): """Clear the data in the Tk clipboard. From python-checkins at python.org Mon Apr 17 02:33:24 2006 From: python-checkins at python.org (neal.norwitz) Date: Mon, 17 Apr 2006 02:33:24 +0200 (CEST) Subject: [Python-checkins] r45468 - python/trunk/Python/errors.c Message-ID: <20060417003324.370D81E4007@bag.python.org> Author: neal.norwitz Date: Mon Apr 17 02:33:23 2006 New Revision: 45468 Modified: python/trunk/Python/errors.c Log: moduleName can be NULL Modified: python/trunk/Python/errors.c ============================================================================== --- python/trunk/Python/errors.c (original) +++ python/trunk/Python/errors.c Mon Apr 17 02:33:23 2006 @@ -604,7 +604,7 @@ PyFile_WriteString(": ", f); PyFile_WriteObject(v, f, 0); } - Py_DECREF(moduleName); + Py_XDECREF(moduleName); } PyFile_WriteString(" in ", f); PyFile_WriteObject(obj, f, 0); From python-checkins at python.org Mon Apr 17 02:36:29 2006 From: python-checkins at python.org (neal.norwitz) Date: Mon, 17 Apr 2006 02:36:29 +0200 (CEST) Subject: [Python-checkins] r45469 - python/trunk/Modules/unicodedata.c Message-ID: <20060417003629.698F51E4007@bag.python.org> Author: neal.norwitz Date: Mon Apr 17 02:36:29 2006 New Revision: 45469 Modified: python/trunk/Modules/unicodedata.c Log: No reason to export get_decomp_record, make static Modified: python/trunk/Modules/unicodedata.c ============================================================================== --- python/trunk/Modules/unicodedata.c (original) +++ python/trunk/Modules/unicodedata.c Mon Apr 17 02:36:29 2006 @@ -446,7 +446,7 @@ return PyString_FromString(decomp); } -void +static void get_decomp_record(PyObject *self, Py_UCS4 code, int *index, int *prefix, int *count) { if (code >= 0x110000) { From python-checkins at python.org Mon Apr 17 03:48:07 2006 From: python-checkins at python.org (neal.norwitz) Date: Mon, 17 Apr 2006 03:48:07 +0200 (CEST) Subject: [Python-checkins] r45470 - python/trunk/Lib/test/test_cmd_line.py Message-ID: <20060417014807.4A0841E4007@bag.python.org> Author: neal.norwitz Date: Mon Apr 17 03:48:06 2006 New Revision: 45470 Modified: python/trunk/Lib/test/test_cmd_line.py Log: Get test to consistently show no leaks Modified: python/trunk/Lib/test/test_cmd_line.py ============================================================================== --- python/trunk/Lib/test/test_cmd_line.py (original) +++ python/trunk/Lib/test/test_cmd_line.py Mon Apr 17 03:48:06 2006 @@ -6,10 +6,12 @@ class CmdLineTest(unittest.TestCase): def start_python(self, cmd_line): - outfp, infp = popen2.popen4('%s %s' % (sys.executable, cmd_line)) + inst = popen2.Popen4('%s %s' % (sys.executable, cmd_line)) + outfp, infp = inst.fromchild, inst.tochild infp.close() data = outfp.read() outfp.close() + inst.wait() return data def exit_code(self, cmd_line): From python-checkins at python.org Mon Apr 17 03:48:41 2006 From: python-checkins at python.org (neal.norwitz) Date: Mon, 17 Apr 2006 03:48:41 +0200 (CEST) Subject: [Python-checkins] r45471 - python/trunk/Misc/build.sh Message-ID: <20060417014841.73D841E4007@bag.python.org> Author: neal.norwitz Date: Mon Apr 17 03:48:41 2006 New Revision: 45471 Modified: python/trunk/Misc/build.sh Log: test_cmd_line should not leak any more, ensure an empty reflog file exists if no leaks are found Modified: python/trunk/Misc/build.sh ============================================================================== --- python/trunk/Misc/build.sh (original) +++ python/trunk/Misc/build.sh Mon Apr 17 03:48:41 2006 @@ -60,7 +60,7 @@ # Note: test_XXX (none currently) really leak, but are disabled # so we don't send spam. Any test which really leaks should only # be listed here if there are also test cases under Lib/test/leakers. -LEAKY_TESTS="test_(cmd_line|ctypes|filecmp|socket|threadedtempfile|threading|threading_local|urllib2)" +LEAKY_TESTS="test_(ctypes|filecmp|socket|threadedtempfile|threading|threading_local|urllib2)" # Skip these tests altogether when looking for leaks. These tests # do not need to be stored above in LEAKY_TESTS too. @@ -167,6 +167,8 @@ ## run the tests looking for leaks F=make-test-refleak.out start=`current_time` + ## ensure that the reflog exists so the grep doesn't fail + touch $REFLOG ./python ./Lib/test/regrtest.py -R 4:3:$REFLOG -u network $LEAKY_SKIPS >& build/$F NUM_FAILURES=`egrep -vc "$LEAKY_TESTS" $REFLOG` update_status "Testing refleaks ($NUM_FAILURES failures)" "$F" $start From python-checkins at python.org Mon Apr 17 03:49:14 2006 From: python-checkins at python.org (neal.norwitz) Date: Mon, 17 Apr 2006 03:49:14 +0200 (CEST) Subject: [Python-checkins] r45472 - python/trunk/Misc/README.valgrind Message-ID: <20060417014914.47F7E1E4007@bag.python.org> Author: neal.norwitz Date: Mon Apr 17 03:49:14 2006 New Revision: 45472 Modified: python/trunk/Misc/README.valgrind Log: Add some notes about a couple of poorly behaved tests Modified: python/trunk/Misc/README.valgrind ============================================================================== --- python/trunk/Misc/README.valgrind (original) +++ python/trunk/Misc/README.valgrind Mon Apr 17 03:49:14 2006 @@ -24,6 +24,12 @@ the memory usage down to allow the tests to complete. It seems to take about 5 times longer to run --without-pymalloc. +Apr 15, 2006: + test_ctypes causes Valgrind 3.1.1 to fail (crash). + test_socket_ssl should be skipped when running valgrind. + The reason is that it purposely uses uninitialized memory. + This causes many spurious warnings, so it's easier to just skip it. + Details: -------- From python-checkins at python.org Mon Apr 17 03:49:28 2006 From: python-checkins at python.org (neal.norwitz) Date: Mon, 17 Apr 2006 03:49:28 +0200 (CEST) Subject: [Python-checkins] r45473 - python/trunk/Modules/_ssl.c Message-ID: <20060417014928.E3A261E400E@bag.python.org> Author: neal.norwitz Date: Mon Apr 17 03:49:28 2006 New Revision: 45473 Modified: python/trunk/Modules/_ssl.c Log: Remove unused field Modified: python/trunk/Modules/_ssl.c ============================================================================== --- python/trunk/Modules/_ssl.c (original) +++ python/trunk/Modules/_ssl.c Mon Apr 17 03:49:28 2006 @@ -55,7 +55,6 @@ SSL_CTX* ctx; SSL* ssl; X509* server_cert; - BIO* sbio; char server[X509_NAME_MAXLEN]; char issuer[X509_NAME_MAXLEN]; From buildbot at python.org Mon Apr 17 04:24:22 2006 From: buildbot at python.org (buildbot at python.org) Date: Mon, 17 Apr 2006 02:24:22 +0000 Subject: [Python-checkins] buildbot warnings in x86 XP trunk Message-ID: <20060417022422.7A99E1E4007@bag.python.org> The Buildbot has detected a new failure of x86 XP trunk. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520XP%2520trunk/builds/466 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: neal.norwitz Build Had Warnings: warnings test sincerely, -The Buildbot From python-checkins at python.org Mon Apr 17 04:39:38 2006 From: python-checkins at python.org (neal.norwitz) Date: Mon, 17 Apr 2006 04:39:38 +0200 (CEST) Subject: [Python-checkins] r45474 - python/trunk/Lib/test/test_cmd_line.py Message-ID: <20060417023938.5E91A1E4007@bag.python.org> Author: neal.norwitz Date: Mon Apr 17 04:39:37 2006 New Revision: 45474 Modified: python/trunk/Lib/test/test_cmd_line.py Log: Try to stop the test from leaking and yet still work on windows Modified: python/trunk/Lib/test/test_cmd_line.py ============================================================================== --- python/trunk/Lib/test/test_cmd_line.py (original) +++ python/trunk/Lib/test/test_cmd_line.py Mon Apr 17 04:39:37 2006 @@ -6,12 +6,11 @@ class CmdLineTest(unittest.TestCase): def start_python(self, cmd_line): - inst = popen2.Popen4('%s %s' % (sys.executable, cmd_line)) - outfp, infp = inst.fromchild, inst.tochild + outfp, infp = popen2.popen4('%s %s' % (sys.executable, cmd_line)) infp.close() data = outfp.read() outfp.close() - inst.wait() + popen2._cleanup() return data def exit_code(self, cmd_line): From python-checkins at python.org Mon Apr 17 04:41:25 2006 From: python-checkins at python.org (neal.norwitz) Date: Mon, 17 Apr 2006 04:41:25 +0200 (CEST) Subject: [Python-checkins] r45475 - python/trunk/Lib/test/test_cmd_line.py Message-ID: <20060417024125.00F8D1E4007@bag.python.org> Author: neal.norwitz Date: Mon Apr 17 04:41:25 2006 New Revision: 45475 Modified: python/trunk/Lib/test/test_cmd_line.py Log: Add a comment to explain why we are calling _cleanup() Modified: python/trunk/Lib/test/test_cmd_line.py ============================================================================== --- python/trunk/Lib/test/test_cmd_line.py (original) +++ python/trunk/Lib/test/test_cmd_line.py Mon Apr 17 04:41:25 2006 @@ -10,6 +10,8 @@ infp.close() data = outfp.read() outfp.close() + # try to cleanup the child so we don't appear to leak when running + # with regrtest -R. This should be a no-op on Windows. popen2._cleanup() return data From python-checkins at python.org Mon Apr 17 07:56:33 2006 From: python-checkins at python.org (neal.norwitz) Date: Mon, 17 Apr 2006 07:56:33 +0200 (CEST) Subject: [Python-checkins] r45476 - python/trunk/Objects/stringobject.c Message-ID: <20060417055633.1FC891E4007@bag.python.org> Author: neal.norwitz Date: Mon Apr 17 07:56:32 2006 New Revision: 45476 Modified: python/trunk/Objects/stringobject.c Log: No need to cast a Py_ssize_t, use %z in PyErr_Format Modified: python/trunk/Objects/stringobject.c ============================================================================== --- python/trunk/Objects/stringobject.c (original) +++ python/trunk/Objects/stringobject.c Mon Apr 17 07:56:32 2006 @@ -1677,9 +1677,9 @@ } #endif PyErr_Format(PyExc_TypeError, - "sequence item %i: expected string," + "sequence item %zd: expected string," " %.80s found", - /*XXX*/(int)i, item->ob_type->tp_name); + i, item->ob_type->tp_name); Py_DECREF(seq); return NULL; } From python-checkins at python.org Mon Apr 17 11:22:36 2006 From: python-checkins at python.org (armin.rigo) Date: Mon, 17 Apr 2006 11:22:36 +0200 (CEST) Subject: [Python-checkins] r45477 - python/trunk/Lib/distutils/sysconfig.py Message-ID: <20060417092236.516D01E4007@bag.python.org> Author: armin.rigo Date: Mon Apr 17 11:22:35 2006 New Revision: 45477 Modified: python/trunk/Lib/distutils/sysconfig.py Log: Fix for a bug exposed by r45232: /path/to/uninstalled/python setup.py build_ext now failed with pyconfig.h not found. Prior to r45232 the above command did not look for pyconfig.h, but the bug is really in the look-up code: expecting to find it in os.curdir is a rather fragile idea. Modified: python/trunk/Lib/distutils/sysconfig.py ============================================================================== --- python/trunk/Lib/distutils/sysconfig.py (original) +++ python/trunk/Lib/distutils/sysconfig.py Mon Apr 17 11:22:35 2006 @@ -31,7 +31,7 @@ python_build = os.path.isfile(landmark) -del argv0_path, landmark +del landmark def get_python_version(): @@ -185,7 +185,7 @@ def get_config_h_filename(): """Return full pathname of installed pyconfig.h file.""" if python_build: - inc_dir = os.curdir + inc_dir = argv0_path else: inc_dir = get_python_inc(plat_specific=1) if get_python_version() < '2.2': From python-checkins at python.org Mon Apr 17 11:46:48 2006 From: python-checkins at python.org (martin.v.loewis) Date: Mon, 17 Apr 2006 11:46:48 +0200 (CEST) Subject: [Python-checkins] r45478 - python/trunk/Tools/buildbot/build.bat python/trunk/Tools/buildbot/kill_python.bat python/trunk/Tools/buildbot/kill_python.c python/trunk/Tools/buildbot/kill_python.mak Message-ID: <20060417094648.795B81E4007@bag.python.org> Author: martin.v.loewis Date: Mon Apr 17 11:46:47 2006 New Revision: 45478 Added: python/trunk/Tools/buildbot/kill_python.bat (contents, props changed) python/trunk/Tools/buildbot/kill_python.c (contents, props changed) python/trunk/Tools/buildbot/kill_python.mak (contents, props changed) Modified: python/trunk/Tools/buildbot/build.bat Log: Add kill_python command. Modified: python/trunk/Tools/buildbot/build.bat ============================================================================== --- python/trunk/Tools/buildbot/build.bat (original) +++ python/trunk/Tools/buildbot/build.bat Mon Apr 17 11:46:47 2006 @@ -1,4 +1,5 @@ @rem Used by the buildbot "compile" step. cmd /c Tools\buildbot\external.bat call "%VS71COMNTOOLS%vsvars32.bat" +cmd /q/c Tools\buildbot\kill_python.bat devenv.com /useenv /build Debug PCbuild\pcbuild.sln Added: python/trunk/Tools/buildbot/kill_python.bat ============================================================================== --- (empty file) +++ python/trunk/Tools/buildbot/kill_python.bat Mon Apr 17 11:46:47 2006 @@ -0,0 +1,3 @@ +cd Tools\buildbot +nmake /C /S /f kill_python.mak +kill_python.exe Added: python/trunk/Tools/buildbot/kill_python.c ============================================================================== --- (empty file) +++ python/trunk/Tools/buildbot/kill_python.c Mon Apr 17 11:46:47 2006 @@ -0,0 +1,56 @@ +/* This program looks for processes which have build\PCbuild\python.exe + in their path and terminates them. */ +#include +#include +#include + +int main() +{ + DWORD pids[1024], cbNeeded; + int i, num_processes; + if (!EnumProcesses(pids, sizeof(pids), &cbNeeded)) { + printf("EnumProcesses failed\n"); + return 1; + } + num_processes = cbNeeded/sizeof(pids[0]); + for (i = 0; i < num_processes; i++) { + HANDLE hProcess; + char path[MAX_PATH]; + HMODULE mods[1024]; + int k, num_mods; + hProcess = OpenProcess(PROCESS_QUERY_INFORMATION + | PROCESS_VM_READ + | PROCESS_TERMINATE , + FALSE, pids[i]); + if (!hProcess) + /* process not accessible */ + continue; + if (!EnumProcessModules(hProcess, mods, sizeof(mods), &cbNeeded)) { + /* For unknown reasons, this sometimes returns ERROR_PARTIAL_COPY; + this apparently means we are not supposed to read the process. */ + if (GetLastError() == ERROR_PARTIAL_COPY) { + CloseHandle(hProcess); + continue; + } + printf("EnumProcessModules failed: %d\n", GetLastError()); + return 1; + } + if (!GetProcessImageFileName(hProcess, path, sizeof(path))) { + printf("GetProcessImageFileName failed\n"); + return 1; + } + + _strlwr(path); + /* printf("%s\n", path); */ + if (strstr(path, "build\\pcbuild\\python_d.exe") != NULL) { + printf("Terminating %s (pid %d)\n", path, pids[i]); + if (!TerminateProcess(hProcess, 1)) { + printf("Termination failed: %d\n", GetLastError()); + return 1; + } + return 0; + } + + CloseHandle(hProcess); + } +} \ No newline at end of file Added: python/trunk/Tools/buildbot/kill_python.mak ============================================================================== --- (empty file) +++ python/trunk/Tools/buildbot/kill_python.mak Mon Apr 17 11:46:47 2006 @@ -0,0 +1,2 @@ +kill_python.exe: kill_python.c + cl -nologo -o kill_python.exe kill_python.c psapi.lib From python-checkins at python.org Mon Apr 17 12:19:25 2006 From: python-checkins at python.org (martin.v.loewis) Date: Mon, 17 Apr 2006 12:19:25 +0200 (CEST) Subject: [Python-checkins] r45479 - python/trunk/Tools/buildbot/kill_python.c Message-ID: <20060417101925.F2CC21E4008@bag.python.org> Author: martin.v.loewis Date: Mon Apr 17 12:19:25 2006 New Revision: 45479 Modified: python/trunk/Tools/buildbot/kill_python.c Log: Try some tracing Modified: python/trunk/Tools/buildbot/kill_python.c ============================================================================== --- python/trunk/Tools/buildbot/kill_python.c (original) +++ python/trunk/Tools/buildbot/kill_python.c Mon Apr 17 12:19:25 2006 @@ -13,6 +13,7 @@ return 1; } num_processes = cbNeeded/sizeof(pids[0]); + printf("%d processes\n", num_processes); for (i = 0; i < num_processes; i++) { HANDLE hProcess; char path[MAX_PATH]; @@ -41,7 +42,8 @@ } _strlwr(path); - /* printf("%s\n", path); */ + printf("%s\n", path); + fflush(stdout); if (strstr(path, "build\\pcbuild\\python_d.exe") != NULL) { printf("Terminating %s (pid %d)\n", path, pids[i]); if (!TerminateProcess(hProcess, 1)) { @@ -53,4 +55,4 @@ CloseHandle(hProcess); } -} \ No newline at end of file +} From python-checkins at python.org Mon Apr 17 12:23:23 2006 From: python-checkins at python.org (martin.v.loewis) Date: Mon, 17 Apr 2006 12:23:23 +0200 (CEST) Subject: [Python-checkins] r45480 - python/trunk/Tools/buildbot/kill_python.bat python/trunk/Tools/buildbot/kill_python.mak Message-ID: <20060417102323.87C721E4007@bag.python.org> Author: martin.v.loewis Date: Mon Apr 17 12:23:23 2006 New Revision: 45480 Modified: python/trunk/Tools/buildbot/kill_python.bat python/trunk/Tools/buildbot/kill_python.mak Log: Rename binary to avoid conflicts with hanging processes on x86 w2k. Modified: python/trunk/Tools/buildbot/kill_python.bat ============================================================================== --- python/trunk/Tools/buildbot/kill_python.bat (original) +++ python/trunk/Tools/buildbot/kill_python.bat Mon Apr 17 12:23:23 2006 @@ -1,3 +1,3 @@ cd Tools\buildbot nmake /C /S /f kill_python.mak -kill_python.exe +kill_python2.exe Modified: python/trunk/Tools/buildbot/kill_python.mak ============================================================================== --- python/trunk/Tools/buildbot/kill_python.mak (original) +++ python/trunk/Tools/buildbot/kill_python.mak Mon Apr 17 12:23:23 2006 @@ -1,2 +1,2 @@ kill_python.exe: kill_python.c - cl -nologo -o kill_python.exe kill_python.c psapi.lib + cl -nologo -o kill_python2.exe kill_python.c psapi.lib From python-checkins at python.org Mon Apr 17 12:27:28 2006 From: python-checkins at python.org (martin.v.loewis) Date: Mon, 17 Apr 2006 12:27:28 +0200 (CEST) Subject: [Python-checkins] r45481 - python/trunk/Tools/buildbot/kill_python.bat python/trunk/Tools/buildbot/kill_python.mak Message-ID: <20060417102728.70F221E4007@bag.python.org> Author: martin.v.loewis Date: Mon Apr 17 12:27:28 2006 New Revision: 45481 Modified: python/trunk/Tools/buildbot/kill_python.bat python/trunk/Tools/buildbot/kill_python.mak Log: Rename binaries again; increase noise. Modified: python/trunk/Tools/buildbot/kill_python.bat ============================================================================== --- python/trunk/Tools/buildbot/kill_python.bat (original) +++ python/trunk/Tools/buildbot/kill_python.bat Mon Apr 17 12:27:28 2006 @@ -1,3 +1,3 @@ cd Tools\buildbot -nmake /C /S /f kill_python.mak -kill_python2.exe +nmake /f kill_python.mak +kill_python3.exe Modified: python/trunk/Tools/buildbot/kill_python.mak ============================================================================== --- python/trunk/Tools/buildbot/kill_python.mak (original) +++ python/trunk/Tools/buildbot/kill_python.mak Mon Apr 17 12:27:28 2006 @@ -1,2 +1,2 @@ kill_python.exe: kill_python.c - cl -nologo -o kill_python2.exe kill_python.c psapi.lib + cl -nologo -o kill_python3.exe kill_python.c psapi.lib From python-checkins at python.org Mon Apr 17 12:31:36 2006 From: python-checkins at python.org (martin.v.loewis) Date: Mon, 17 Apr 2006 12:31:36 +0200 (CEST) Subject: [Python-checkins] r45482 - python/trunk/Tools/buildbot/build.bat python/trunk/Tools/buildbot/kill_python.bat python/trunk/Tools/buildbot/kill_python.c python/trunk/Tools/buildbot/kill_python.mak Message-ID: <20060417103136.1E1741E4007@bag.python.org> Author: martin.v.loewis Date: Mon Apr 17 12:31:35 2006 New Revision: 45482 Modified: python/trunk/Tools/buildbot/build.bat python/trunk/Tools/buildbot/kill_python.bat python/trunk/Tools/buildbot/kill_python.c python/trunk/Tools/buildbot/kill_python.mak Log: Rename binary again; increase noise; stop trying to actually kill the process. Modified: python/trunk/Tools/buildbot/build.bat ============================================================================== --- python/trunk/Tools/buildbot/build.bat (original) +++ python/trunk/Tools/buildbot/build.bat Mon Apr 17 12:31:35 2006 @@ -1,5 +1,5 @@ @rem Used by the buildbot "compile" step. cmd /c Tools\buildbot\external.bat call "%VS71COMNTOOLS%vsvars32.bat" -cmd /q/c Tools\buildbot\kill_python.bat +cmd /c Tools\buildbot\kill_python.bat devenv.com /useenv /build Debug PCbuild\pcbuild.sln Modified: python/trunk/Tools/buildbot/kill_python.bat ============================================================================== --- python/trunk/Tools/buildbot/kill_python.bat (original) +++ python/trunk/Tools/buildbot/kill_python.bat Mon Apr 17 12:31:35 2006 @@ -1,3 +1,3 @@ cd Tools\buildbot nmake /f kill_python.mak -kill_python3.exe +kill_python4.exe Modified: python/trunk/Tools/buildbot/kill_python.c ============================================================================== --- python/trunk/Tools/buildbot/kill_python.c (original) +++ python/trunk/Tools/buildbot/kill_python.c Mon Apr 17 12:31:35 2006 @@ -44,6 +44,7 @@ _strlwr(path); printf("%s\n", path); fflush(stdout); + /* if (strstr(path, "build\\pcbuild\\python_d.exe") != NULL) { printf("Terminating %s (pid %d)\n", path, pids[i]); if (!TerminateProcess(hProcess, 1)) { @@ -52,6 +53,7 @@ } return 0; } + */ CloseHandle(hProcess); } Modified: python/trunk/Tools/buildbot/kill_python.mak ============================================================================== --- python/trunk/Tools/buildbot/kill_python.mak (original) +++ python/trunk/Tools/buildbot/kill_python.mak Mon Apr 17 12:31:35 2006 @@ -1,2 +1,2 @@ kill_python.exe: kill_python.c - cl -nologo -o kill_python3.exe kill_python.c psapi.lib + cl -nologo -o kill_python4.exe kill_python.c psapi.lib From python-checkins at python.org Mon Apr 17 12:36:18 2006 From: python-checkins at python.org (martin.v.loewis) Date: Mon, 17 Apr 2006 12:36:18 +0200 (CEST) Subject: [Python-checkins] r45483 - python/trunk/Tools/buildbot/kill_python.bat Message-ID: <20060417103618.A9E6D1E4007@bag.python.org> Author: martin.v.loewis Date: Mon Apr 17 12:36:18 2006 New Revision: 45483 Modified: python/trunk/Tools/buildbot/kill_python.bat Log: Check whether disk space is full. Modified: python/trunk/Tools/buildbot/kill_python.bat ============================================================================== --- python/trunk/Tools/buildbot/kill_python.bat (original) +++ python/trunk/Tools/buildbot/kill_python.bat Mon Apr 17 12:36:18 2006 @@ -1,3 +1,4 @@ cd Tools\buildbot +dir nmake /f kill_python.mak kill_python4.exe From python-checkins at python.org Mon Apr 17 12:39:39 2006 From: python-checkins at python.org (martin.v.loewis) Date: Mon, 17 Apr 2006 12:39:39 +0200 (CEST) Subject: [Python-checkins] r45484 - python/trunk/Tools/buildbot/build.bat python/trunk/Tools/buildbot/kill_python.bat python/trunk/Tools/buildbot/kill_python.c python/trunk/Tools/buildbot/kill_python.mak Message-ID: <20060417103939.E74931E4007@bag.python.org> Author: martin.v.loewis Date: Mon Apr 17 12:39:39 2006 New Revision: 45484 Modified: python/trunk/Tools/buildbot/build.bat python/trunk/Tools/buildbot/kill_python.bat python/trunk/Tools/buildbot/kill_python.c python/trunk/Tools/buildbot/kill_python.mak Log: Revert to 45478, disable kill_python command for now. Modified: python/trunk/Tools/buildbot/build.bat ============================================================================== --- python/trunk/Tools/buildbot/build.bat (original) +++ python/trunk/Tools/buildbot/build.bat Mon Apr 17 12:39:39 2006 @@ -1,5 +1,5 @@ @rem Used by the buildbot "compile" step. cmd /c Tools\buildbot\external.bat call "%VS71COMNTOOLS%vsvars32.bat" -cmd /c Tools\buildbot\kill_python.bat + at rem cmd /q/c Tools\buildbot\kill_python.bat devenv.com /useenv /build Debug PCbuild\pcbuild.sln Modified: python/trunk/Tools/buildbot/kill_python.bat ============================================================================== --- python/trunk/Tools/buildbot/kill_python.bat (original) +++ python/trunk/Tools/buildbot/kill_python.bat Mon Apr 17 12:39:39 2006 @@ -1,4 +1,3 @@ cd Tools\buildbot -dir -nmake /f kill_python.mak -kill_python4.exe +nmake /C /S /f kill_python.mak +kill_python.exe Modified: python/trunk/Tools/buildbot/kill_python.c ============================================================================== --- python/trunk/Tools/buildbot/kill_python.c (original) +++ python/trunk/Tools/buildbot/kill_python.c Mon Apr 17 12:39:39 2006 @@ -13,7 +13,6 @@ return 1; } num_processes = cbNeeded/sizeof(pids[0]); - printf("%d processes\n", num_processes); for (i = 0; i < num_processes; i++) { HANDLE hProcess; char path[MAX_PATH]; @@ -42,9 +41,7 @@ } _strlwr(path); - printf("%s\n", path); - fflush(stdout); - /* + /* printf("%s\n", path); */ if (strstr(path, "build\\pcbuild\\python_d.exe") != NULL) { printf("Terminating %s (pid %d)\n", path, pids[i]); if (!TerminateProcess(hProcess, 1)) { @@ -53,8 +50,7 @@ } return 0; } - */ CloseHandle(hProcess); } -} +} \ No newline at end of file Modified: python/trunk/Tools/buildbot/kill_python.mak ============================================================================== --- python/trunk/Tools/buildbot/kill_python.mak (original) +++ python/trunk/Tools/buildbot/kill_python.mak Mon Apr 17 12:39:39 2006 @@ -1,2 +1,2 @@ kill_python.exe: kill_python.c - cl -nologo -o kill_python4.exe kill_python.c psapi.lib + cl -nologo -o kill_python.exe kill_python.c psapi.lib From buildbot at python.org Mon Apr 17 12:45:09 2006 From: buildbot at python.org (buildbot at python.org) Date: Mon, 17 Apr 2006 10:45:09 +0000 Subject: [Python-checkins] buildbot failure in x86 W2k trunk Message-ID: <20060417104509.B95A61E4007@bag.python.org> The Buildbot has detected a new failure of x86 W2k trunk. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520W2k%2520trunk/builds/508 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: martin.v.loewis BUILD FAILED: failed compile sincerely, -The Buildbot From buildbot at python.org Mon Apr 17 13:02:11 2006 From: buildbot at python.org (buildbot at python.org) Date: Mon, 17 Apr 2006 11:02:11 +0000 Subject: [Python-checkins] buildbot warnings in x86 gentoo trunk Message-ID: <20060417110211.D7B821E4007@bag.python.org> The Buildbot has detected a new failure of x86 gentoo trunk. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520gentoo%2520trunk/builds/512 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: martin.v.loewis Build Had Warnings: warnings test sincerely, -The Buildbot From python-checkins at python.org Mon Apr 17 15:37:15 2006 From: python-checkins at python.org (ronald.oussoren) Date: Mon, 17 Apr 2006 15:37:15 +0200 (CEST) Subject: [Python-checkins] r45486 - python/trunk/Lib/platform.py Message-ID: <20060417133715.CA16D1E4007@bag.python.org> Author: ronald.oussoren Date: Mon Apr 17 15:37:15 2006 New Revision: 45486 Modified: python/trunk/Lib/platform.py Log: Teach platform about darwin/x86 Modified: python/trunk/Lib/platform.py ============================================================================== --- python/trunk/Lib/platform.py (original) +++ python/trunk/Lib/platform.py Mon Apr 17 15:37:15 2006 @@ -607,7 +607,8 @@ versioninfo = (version,stage,nonrel) if sysa: machine = {0x1: '68k', - 0x2: 'PowerPC'}.get(sysa,'') + 0x2: 'PowerPC', + 0xa: 'i386'}.get(sysa,'') return release,versioninfo,machine def _java_getprop(name,default): From python-checkins at python.org Mon Apr 17 15:40:08 2006 From: python-checkins at python.org (ronald.oussoren) Date: Mon, 17 Apr 2006 15:40:08 +0200 (CEST) Subject: [Python-checkins] r45487 - in python/trunk: Lib/plat-mac/applesingle.py Lib/test/test_applesingle.py Mac/Modules/gestaltmodule.c Python/mactoolboxglue.c Message-ID: <20060417134008.CADA21E400F@bag.python.org> Author: ronald.oussoren Date: Mon Apr 17 15:40:08 2006 New Revision: 45487 Modified: python/trunk/Lib/plat-mac/applesingle.py python/trunk/Lib/test/test_applesingle.py python/trunk/Mac/Modules/gestaltmodule.c python/trunk/Python/mactoolboxglue.c Log: This patches fixes a number of byteorder problems in MacOSX specific code. Modified: python/trunk/Lib/plat-mac/applesingle.py ============================================================================== --- python/trunk/Lib/plat-mac/applesingle.py (original) +++ python/trunk/Lib/plat-mac/applesingle.py Mon Apr 17 15:40:08 2006 @@ -25,7 +25,7 @@ pass # File header format: magic, version, unused, number of entries -AS_HEADER_FORMAT=">ll16sh" +AS_HEADER_FORMAT=">LL16sh" AS_HEADER_LENGTH=26 # The flag words for AppleSingle AS_MAGIC=0x00051600 Modified: python/trunk/Lib/test/test_applesingle.py ============================================================================== --- python/trunk/Lib/test/test_applesingle.py (original) +++ python/trunk/Lib/test/test_applesingle.py Mon Apr 17 15:40:08 2006 @@ -15,8 +15,8 @@ dataforkdata = 'hello\r\0world\n' resourceforkdata = 'goodbye\ncruel\0world\r' -applesingledata = struct.pack("ll16sh", AS_MAGIC, AS_VERSION, "foo", 2) + \ - struct.pack("llllll", 1, 50, len(dataforkdata), +applesingledata = struct.pack(">ll16sh", AS_MAGIC, AS_VERSION, "foo", 2) + \ + struct.pack(">llllll", 1, 50, len(dataforkdata), 2, 50+len(dataforkdata), len(resourceforkdata)) + \ dataforkdata + \ resourceforkdata Modified: python/trunk/Mac/Modules/gestaltmodule.c ============================================================================== --- python/trunk/Mac/Modules/gestaltmodule.c (original) +++ python/trunk/Mac/Modules/gestaltmodule.c Mon Apr 17 15:40:08 2006 @@ -33,17 +33,10 @@ gestalt_gestalt(PyObject *self, PyObject *args) { OSErr iErr; - char *str; - int size; OSType selector; long response; - if (!PyArg_Parse(args, "s#", &str, &size)) + if (!PyArg_Parse(args, "O&", PyMac_GetOSType, &selector)) return NULL; - if (size != 4) { - PyErr_SetString(PyExc_TypeError, "gestalt arg must be 4-char string"); - return NULL; - } - selector = *(OSType*)str; iErr = Gestalt ( selector, &response ); if (iErr != 0) return PyMac_Error(iErr); Modified: python/trunk/Python/mactoolboxglue.c ============================================================================== --- python/trunk/Python/mactoolboxglue.c (original) +++ python/trunk/Python/mactoolboxglue.c Mon Apr 17 15:40:08 2006 @@ -25,6 +25,7 @@ #include "Python.h" #include "pymactoolbox.h" +#include /* for ntohl, htonl */ /* Like strerror() but for Mac OS error numbers */ @@ -156,12 +157,14 @@ int PyMac_GetOSType(PyObject *v, OSType *pr) { + uint32_t tmp; if (!PyString_Check(v) || PyString_Size(v) != 4) { PyErr_SetString(PyExc_TypeError, "OSType arg must be string of 4 chars"); return 0; } - memcpy((char *)pr, PyString_AsString(v), 4); + memcpy((char *)&tmp, PyString_AsString(v), 4); + *pr = (OSType)ntohl(tmp); return 1; } @@ -169,7 +172,8 @@ PyObject * PyMac_BuildOSType(OSType t) { - return PyString_FromStringAndSize((char *)&t, 4); + uint32_t tmp = htonl((uint32_t)t); + return PyString_FromStringAndSize((char *)&tmp, 4); } /* Convert an NumVersion value to a 4-element tuple */ From python-checkins at python.org Mon Apr 17 16:00:31 2006 From: python-checkins at python.org (andrew.kuchling) Date: Mon, 17 Apr 2006 16:00:31 +0200 (CEST) Subject: [Python-checkins] r45488 - python/trunk/Doc/whatsnew/whatsnew25.tex Message-ID: <20060417140031.ACE721E4011@bag.python.org> Author: andrew.kuchling Date: Mon Apr 17 16:00:31 2006 New Revision: 45488 Modified: python/trunk/Doc/whatsnew/whatsnew25.tex Log: Remove translated code hidden inside a comment environment; latex2html gets confused and includes half of it anyway Modified: python/trunk/Doc/whatsnew/whatsnew25.tex ============================================================================== --- python/trunk/Doc/whatsnew/whatsnew25.tex (original) +++ python/trunk/Doc/whatsnew/whatsnew25.tex Mon Apr 17 16:00:31 2006 @@ -792,38 +792,6 @@ # return False \end{verbatim} -\begin{comment} -% XXX should I give the code, or is the above explanation sufficient? -\pep{343} shows the code generated for a \keyword{with} statement. A -statement such as: - -\begin{verbatim} -with EXPR as VAR: - BLOCK -\end{verbatim} - -is translated into: - -\begin{verbatim} -ctx = (EXPR).__context__() -exit = ctx.__exit__ # Not calling it yet -value = ctx.__enter__() -exc = True -try: - try: - VAR = value # Only if "as VAR" is present - BLOCK - except: - # The exceptional case is handled here - exc = False - if not exit(*sys.exc_info()): - raise -finally: - # The normal and non-local-goto cases are handled here - if exc: - exit(None, None, None) -\end{verbatim} -\end{comment} \subsection{The contextlib module\label{module-contextlib}} From python-checkins at python.org Mon Apr 17 16:01:36 2006 From: python-checkins at python.org (andrew.kuchling) Date: Mon, 17 Apr 2006 16:01:36 +0200 (CEST) Subject: [Python-checkins] r45489 - python/trunk/Doc/whatsnew/whatsnew25.tex Message-ID: <20060417140136.7AEC21E4019@bag.python.org> Author: andrew.kuchling Date: Mon Apr 17 16:01:36 2006 New Revision: 45489 Modified: python/trunk/Doc/whatsnew/whatsnew25.tex Log: Update status of document Modified: python/trunk/Doc/whatsnew/whatsnew25.tex ============================================================================== --- python/trunk/Doc/whatsnew/whatsnew25.tex (original) +++ python/trunk/Doc/whatsnew/whatsnew25.tex Mon Apr 17 16:01:36 2006 @@ -20,9 +20,8 @@ for Python 2.5 has been set; it will probably be released in the autumn of 2006. \pep{356} describes the planned release schedule. -(This is still an early draft, and some sections are still skeletal or -completely missing. Comments on the present material will still be -welcomed.) +Comments, suggestions, and error reports are welcome; please e-mail them +to the author or open a bug in the Python bug tracker. % XXX Compare with previous release in 2 - 3 sentences here. From buildbot at python.org Mon Apr 17 16:30:26 2006 From: buildbot at python.org (buildbot at python.org) Date: Mon, 17 Apr 2006 14:30:26 +0000 Subject: [Python-checkins] buildbot warnings in x86 XP trunk Message-ID: <20060417143026.97A0E1E4007@bag.python.org> The Buildbot has detected a new failure of x86 XP trunk. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520XP%2520trunk/builds/473 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: andrew.kuchling Build Had Warnings: warnings failed slave lost sincerely, -The Buildbot From python-checkins at python.org Mon Apr 17 16:43:31 2006 From: python-checkins at python.org (ronald.oussoren) Date: Mon, 17 Apr 2006 16:43:31 +0200 (CEST) Subject: [Python-checkins] r45490 - python/trunk/Lib/distutils/sysconfig.py Message-ID: <20060417144331.31BB01E4007@bag.python.org> Author: ronald.oussoren Date: Mon Apr 17 16:43:30 2006 New Revision: 45490 Modified: python/trunk/Lib/distutils/sysconfig.py Log: disutils checks if MACOSX_DEPLOYMENT_TARGET is consistent with the value at configure time. The current check is too strict and doesn't allow building extensions that can only run on newer versions of the OS than the version python was build for, that is python build for 10.3 or later and an extension for 10.4. This patch relaxes this check. This turned out to be a reimplementation of patch 1193190. Modified: python/trunk/Lib/distutils/sysconfig.py ============================================================================== --- python/trunk/Lib/distutils/sysconfig.py (original) +++ python/trunk/Lib/distutils/sysconfig.py Mon Apr 17 16:43:30 2006 @@ -372,7 +372,7 @@ if cur_target == '': cur_target = cfg_target os.putenv('MACOSX_DEPLOYMENT_TARGET', cfg_target) - if cfg_target != cur_target: + elif map(int, cfg_target.split('.')) > map(int, cur_target.split('.')): my_msg = ('$MACOSX_DEPLOYMENT_TARGET mismatch: now "%s" but "%s" during configure' % (cur_target, cfg_target)) raise DistutilsPlatformError(my_msg) From python-checkins at python.org Mon Apr 17 17:45:00 2006 From: python-checkins at python.org (george.yoshida) Date: Mon, 17 Apr 2006 17:45:00 +0200 (CEST) Subject: [Python-checkins] r45491 - python/trunk/Doc/whatsnew/whatsnew25.tex Message-ID: <20060417154500.AEE151E4007@bag.python.org> Author: george.yoshida Date: Mon Apr 17 17:44:59 2006 New Revision: 45491 Modified: python/trunk/Doc/whatsnew/whatsnew25.tex Log: fix long option markup Modified: python/trunk/Doc/whatsnew/whatsnew25.tex ============================================================================== --- python/trunk/Doc/whatsnew/whatsnew25.tex (original) +++ python/trunk/Doc/whatsnew/whatsnew25.tex Mon Apr 17 17:44:59 2006 @@ -46,8 +46,8 @@ distribution using the \command{sdist} Distutils command. Once that works, you can run \code{python setup.py upload} to add your package to the PyPI archive. Optionally you can GPG-sign the package by -supplying the \programopt{--sign} and -\programopt{--identity} options. +supplying the \longprogramopt{sign} and +\longprogramopt{identity} options. \begin{seealso} From python-checkins at python.org Mon Apr 17 19:08:39 2006 From: python-checkins at python.org (martin.v.loewis) Date: Mon, 17 Apr 2006 19:08:39 +0200 (CEST) Subject: [Python-checkins] r45492 - in python/trunk: Doc/lib/libpdb.tex Lib/pdb.py Misc/NEWS Message-ID: <20060417170839.02E141E4007@bag.python.org> Author: martin.v.loewis Date: Mon Apr 17 19:08:37 2006 New Revision: 45492 Modified: python/trunk/Doc/lib/libpdb.tex python/trunk/Lib/pdb.py python/trunk/Misc/NEWS Log: Patch #790710: Add breakpoint command lists in pdb. Modified: python/trunk/Doc/lib/libpdb.tex ============================================================================== --- python/trunk/Doc/lib/libpdb.tex (original) +++ python/trunk/Doc/lib/libpdb.tex Mon Apr 17 19:08:37 2006 @@ -240,6 +240,45 @@ the breakpoint is honored. If condition is absent, any existing condition is removed; i.e., the breakpoint is made unconditional. +\item[commands \optional{\var{bpnumber}}] + +Specify a list of commands for breakpoint number \var{bpnumber}. The +commands themselves appear on the following lines. Type a line +containing just 'end' to terminate the commands. An example: + +\begin{verbatim} +(Pdb) commands 1 +(com) print some_variable +(com) end +(Pdb) +\end{verbatim} + +To remove all commands from a breakpoint, type commands and +follow it immediately with end; that is, give no commands. + +With no \var{bpnumber} argument, commands refers to the last +breakpoint set. + +You can use breakpoint commands to start your program up again. +Simply use the continue command, or step, or any other +command that resumes execution. + +Specifying any command resuming execution (currently continue, +step, next, return, jump, quit and their abbreviations) terminates +the command list (as if that command was immediately followed by end). +This is because any time you resume execution +(even with a simple next or step), you may encounter· +another breakpoint--which could have its own command list, leading to +ambiguities about which list to execute. + + If you use the 'silent' command in the command list, the +usual message about stopping at a breakpoint is not printed. This may +be desirable for breakpoints that are to print a specific message and +then continue. If none of the other commands print anything, you +see no sign that the breakpoint was reached. + +\versionadded{2.5} + \item[s(tep)] Execute the current line, stop at the first possible occasion Modified: python/trunk/Lib/pdb.py ============================================================================== --- python/trunk/Lib/pdb.py (original) +++ python/trunk/Lib/pdb.py Mon Apr 17 19:08:37 2006 @@ -86,6 +86,12 @@ self.rcLines.append(line) rcFile.close() + self.commands = {} # associates a command list to breakpoint numbers + self.commands_doprompt = {} # for each bp num, tells if the prompt must be disp. after execing the cmd list + self.commands_silent = {} # for each bp num, tells if the stack trace must be disp. after execing the cmd list + self.commands_defining = False # True while in the process of defining a command list + self.commands_bnum = None # The breakpoint number for which we are defining a list + def reset(self): bdb.Bdb.reset(self) self.forget() @@ -132,7 +138,28 @@ or frame.f_lineno<= 0): return self._wait_for_mainpyfile = 0 - self.interaction(frame, None) + if self.bp_commands(frame): + self.interaction(frame, None) + + def bp_commands(self,frame): + """ Call every command that was set for the current active breakpoint (if there is one) + Returns True if the normal interaction function must be called, False otherwise """ + #self.currentbp is set in bdb.py in bdb.break_here if a breakpoint was hit + if getattr(self,"currentbp",False) and self.currentbp in self.commands: + currentbp = self.currentbp + self.currentbp = 0 + lastcmd_back = self.lastcmd + self.setup(frame, None) + for line in self.commands[currentbp]: + self.onecmd(line) + self.lastcmd = lastcmd_back + if not self.commands_silent[currentbp]: + self.print_stack_entry(self.stack[self.curindex]) + if self.commands_doprompt[currentbp]: + self.cmdloop() + self.forget() + return + return 1 def user_return(self, frame, return_value): """This function is called when a return trap is set here.""" @@ -197,12 +224,70 @@ line = line[:marker].rstrip() return line + def onecmd(self, line): + """Interpret the argument as though it had been typed in response + to the prompt. + + Checks wether this line is typed in the normal prompt or in a breakpoint command list definition + """ + if not self.commands_defining: + return cmd.Cmd.onecmd(self, line) + else: + return self.handle_command_def(line) + + def handle_command_def(self,line): + """ Handles one command line during command list definition. """ + cmd, arg, line = self.parseline(line) + if cmd == 'silent': + self.commands_silent[self.commands_bnum] = True + return # continue to handle other cmd def in the cmd list + elif cmd == 'end': + self.cmdqueue = [] + return 1 # end of cmd list + cmdlist = self.commands[self.commands_bnum] + if (arg): + cmdlist.append(cmd+' '+arg) + else: + cmdlist.append(cmd) + # Determine if we must stop + try: + func = getattr(self, 'do_' + cmd) + except AttributeError: + func = self.default + if func.func_name in self.commands_resuming : # one of the resuming commands. + self.commands_doprompt[self.commands_bnum] = False + self.cmdqueue = [] + return 1 + return + # Command definitions, called by cmdloop() # The argument is the remaining string on the command line # Return true to exit from the command loop do_h = cmd.Cmd.do_help + def do_commands(self, arg): + """Defines a list of commands associated to a breakpoint + Those commands will be executed whenever the breakpoint causes the program to stop execution.""" + if not arg: + bnum = len(bdb.Breakpoint.bpbynumber)-1 + else: + try: + bnum = int(arg) + except: + print "Usage : commands [bnum]\n ...\n end" + return + self.commands_bnum = bnum + self.commands[bnum] = [] + self.commands_doprompt[bnum] = True + self.commands_silent[bnum] = False + prompt_back = self.prompt + self.prompt = '(com) ' + self.commands_defining = True + self.cmdloop() + self.commands_defining = False + self.prompt = prompt_back + def do_break(self, arg, temporary = 0): # break [ ([filename:]lineno | function) [, "condition"] ] if not arg: @@ -686,6 +771,9 @@ if args[0] in self.aliases: del self.aliases[args[0]] + #list of all the commands making the program resume execution. + commands_resuming = ['do_continue', 'do_step', 'do_next', 'do_return', 'do_quit', 'do_jump'] + # Print a traceback starting at the top stack frame. # The most recently entered frame is printed last; # this is different from dbx and gdb, but consistent with @@ -939,6 +1027,41 @@ print """unalias name Deletes the specified alias.""" + def help_commands(self): + print """commands [bpnumber] +(com) ... +(com) end +(Pdb) + +Specify a list of commands for breakpoint number bpnumber. The +commands themselves appear on the following lines. Type a line +containing just 'end' to terminate the commands. + +To remove all commands from a breakpoint, type commands and +follow it immediately with end; that is, give no commands. + +With no bpnumber argument, commands refers to the last +breakpoint set. + +You can use breakpoint commands to start your program up again. +Simply use the continue command, or step, or any other +command that resumes execution. + +Specifying any command resuming execution (currently continue, +step, next, return, jump, quit and their abbreviations) terminates +the command list (as if that command was immediately followed by end). +This is because any time you resume execution +(even with a simple next or step), you may encounter· +another breakpoint--which could have its own command list, leading to +ambiguities about which list to execute. + + If you use the 'silent' command in the command list, the +usual message about stopping at a breakpoint is not printed. This may +be desirable for breakpoints that are to print a specific message and +then continue. If none of the other commands print anything, you +see no sign that the breakpoint was reached. +""" + def help_pdb(self): help() Modified: python/trunk/Misc/NEWS ============================================================================== --- python/trunk/Misc/NEWS (original) +++ python/trunk/Misc/NEWS Mon Apr 17 19:08:37 2006 @@ -72,6 +72,8 @@ Library ------- +- Patch #790710: Add breakpoint command lists in pdb. + - Patch #1063914: Add Tkinter.Misc.clipboard_get(). - Patch #1191700: Adjust column alignment in bdb breakpoint lists. From python-checkins at python.org Mon Apr 17 19:26:43 2006 From: python-checkins at python.org (martin.v.loewis) Date: Mon, 17 Apr 2006 19:26:43 +0200 (CEST) Subject: [Python-checkins] r45493 - python/trunk/Tools/buildbot/build.bat python/trunk/Tools/buildbot/kill_python.c Message-ID: <20060417172643.5E34C1E400A@bag.python.org> Author: martin.v.loewis Date: Mon Apr 17 19:26:42 2006 New Revision: 45493 Modified: python/trunk/Tools/buildbot/build.bat python/trunk/Tools/buildbot/kill_python.c Log: Use GetModuleFileNameEx instead of GetProcessImageFileName, as the latter is not available on Windows 2000. Modified: python/trunk/Tools/buildbot/build.bat ============================================================================== --- python/trunk/Tools/buildbot/build.bat (original) +++ python/trunk/Tools/buildbot/build.bat Mon Apr 17 19:26:42 2006 @@ -1,5 +1,5 @@ @rem Used by the buildbot "compile" step. cmd /c Tools\buildbot\external.bat call "%VS71COMNTOOLS%vsvars32.bat" - at rem cmd /q/c Tools\buildbot\kill_python.bat +cmd /q/c Tools\buildbot\kill_python.bat devenv.com /useenv /build Debug PCbuild\pcbuild.sln Modified: python/trunk/Tools/buildbot/kill_python.c ============================================================================== --- python/trunk/Tools/buildbot/kill_python.c (original) +++ python/trunk/Tools/buildbot/kill_python.c Mon Apr 17 19:26:42 2006 @@ -35,7 +35,7 @@ printf("EnumProcessModules failed: %d\n", GetLastError()); return 1; } - if (!GetProcessImageFileName(hProcess, path, sizeof(path))) { + if (!GetModuleFileNameEx(hProcess, NULL, path, sizeof(path))) { printf("GetProcessImageFileName failed\n"); return 1; } @@ -53,4 +53,4 @@ CloseHandle(hProcess); } -} \ No newline at end of file +} From buildbot at python.org Mon Apr 17 19:30:51 2006 From: buildbot at python.org (buildbot at python.org) Date: Mon, 17 Apr 2006 17:30:51 +0000 Subject: [Python-checkins] buildbot warnings in amd64 gentoo trunk Message-ID: <20060417173051.7D0CA1E4007@bag.python.org> The Buildbot has detected a new failure of amd64 gentoo trunk. Full details are available at: http://www.python.org/dev/buildbot/all/amd64%2520gentoo%2520trunk/builds/537 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: martin.v.loewis Build Had Warnings: warnings test sincerely, -The Buildbot From buildbot at python.org Mon Apr 17 19:30:53 2006 From: buildbot at python.org (buildbot at python.org) Date: Mon, 17 Apr 2006 17:30:53 +0000 Subject: [Python-checkins] buildbot warnings in x86 gentoo trunk Message-ID: <20060417173053.712E81E400F@bag.python.org> The Buildbot has detected a new failure of x86 gentoo trunk. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520gentoo%2520trunk/builds/517 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: martin.v.loewis Build Had Warnings: warnings test sincerely, -The Buildbot From buildbot at python.org Mon Apr 17 19:34:36 2006 From: buildbot at python.org (buildbot at python.org) Date: Mon, 17 Apr 2006 17:34:36 +0000 Subject: [Python-checkins] buildbot warnings in x86 XP-2 trunk Message-ID: <20060417173436.DDEE21E4007@bag.python.org> The Buildbot has detected a new failure of x86 XP-2 trunk. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520XP-2%2520trunk/builds/306 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: martin.v.loewis Build Had Warnings: warnings test sincerely, -The Buildbot From python-checkins at python.org Mon Apr 17 19:37:11 2006 From: python-checkins at python.org (martin.v.loewis) Date: Mon, 17 Apr 2006 19:37:11 +0200 (CEST) Subject: [Python-checkins] r45494 - python/trunk/Lib/pdb.py Message-ID: <20060417173711.36B4C1E400B@bag.python.org> Author: martin.v.loewis Date: Mon Apr 17 19:37:09 2006 New Revision: 45494 Modified: python/trunk/Lib/pdb.py Log: Remove bogus character. Modified: python/trunk/Lib/pdb.py ============================================================================== --- python/trunk/Lib/pdb.py (original) +++ python/trunk/Lib/pdb.py Mon Apr 17 19:37:09 2006 @@ -1051,7 +1051,7 @@ step, next, return, jump, quit and their abbreviations) terminates the command list (as if that command was immediately followed by end). This is because any time you resume execution -(even with a simple next or step), you may encounter· +(even with a simple next or step), you may encounter another breakpoint--which could have its own command list, leading to ambiguities about which list to execute. From buildbot at python.org Mon Apr 17 19:43:53 2006 From: buildbot at python.org (buildbot at python.org) Date: Mon, 17 Apr 2006 17:43:53 +0000 Subject: [Python-checkins] buildbot warnings in ppc Debian unstable trunk Message-ID: <20060417174353.C05FF1E4007@bag.python.org> The Buildbot has detected a new failure of ppc Debian unstable trunk. Full details are available at: http://www.python.org/dev/buildbot/all/ppc%2520Debian%2520unstable%2520trunk/builds/188 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: martin.v.loewis Build Had Warnings: warnings test sincerely, -The Buildbot From buildbot at python.org Mon Apr 17 19:46:09 2006 From: buildbot at python.org (buildbot at python.org) Date: Mon, 17 Apr 2006 17:46:09 +0000 Subject: [Python-checkins] buildbot warnings in x86 XP trunk Message-ID: <20060417174610.14CA51E4007@bag.python.org> The Buildbot has detected a new failure of x86 XP trunk. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520XP%2520trunk/builds/476 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: martin.v.loewis Build Had Warnings: warnings test sincerely, -The Buildbot From buildbot at python.org Mon Apr 17 19:48:34 2006 From: buildbot at python.org (buildbot at python.org) Date: Mon, 17 Apr 2006 17:48:34 +0000 Subject: [Python-checkins] buildbot warnings in x86 W2k trunk Message-ID: <20060417174834.A15B61E4007@bag.python.org> The Buildbot has detected a new failure of x86 W2k trunk. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520W2k%2520trunk/builds/514 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: The web-page 'force build' button was pressed by 'Martin von Loewis': Try new kill_python Build Source Stamp: [branch trunk] HEAD Blamelist: Build Had Warnings: warnings test sincerely, -The Buildbot From buildbot at python.org Mon Apr 17 19:49:30 2006 From: buildbot at python.org (buildbot at python.org) Date: Mon, 17 Apr 2006 17:49:30 +0000 Subject: [Python-checkins] buildbot warnings in g4 osx.4 trunk Message-ID: <20060417174931.1BD691E4007@bag.python.org> The Buildbot has detected a new failure of g4 osx.4 trunk. Full details are available at: http://www.python.org/dev/buildbot/all/g4%2520osx.4%2520trunk/builds/455 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: martin.v.loewis Build Had Warnings: warnings test sincerely, -The Buildbot From buildbot at python.org Mon Apr 17 19:50:38 2006 From: buildbot at python.org (buildbot at python.org) Date: Mon, 17 Apr 2006 17:50:38 +0000 Subject: [Python-checkins] buildbot warnings in sparc solaris10 gcc trunk Message-ID: <20060417175038.B363A1E402C@bag.python.org> The Buildbot has detected a new failure of sparc solaris10 gcc trunk. Full details are available at: http://www.python.org/dev/buildbot/all/sparc%2520solaris10%2520gcc%2520trunk/builds/462 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: martin.v.loewis Build Had Warnings: warnings test sincerely, -The Buildbot From buildbot at python.org Mon Apr 17 20:14:40 2006 From: buildbot at python.org (buildbot at python.org) Date: Mon, 17 Apr 2006 18:14:40 +0000 Subject: [Python-checkins] buildbot warnings in x86 OpenBSD trunk Message-ID: <20060417181441.113271E4007@bag.python.org> The Buildbot has detected a new failure of x86 OpenBSD trunk. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520OpenBSD%2520trunk/builds/352 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: martin.v.loewis Build Had Warnings: warnings test sincerely, -The Buildbot From buildbot at python.org Mon Apr 17 20:22:04 2006 From: buildbot at python.org (buildbot at python.org) Date: Mon, 17 Apr 2006 18:22:04 +0000 Subject: [Python-checkins] buildbot warnings in x86 Ubuntu dapper (icc) trunk Message-ID: <20060417182204.C4A461E4007@bag.python.org> The Buildbot has detected a new failure of x86 Ubuntu dapper (icc) trunk. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520Ubuntu%2520dapper%2520%2528icc%2529%2520trunk/builds/173 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: martin.v.loewis Build Had Warnings: warnings test sincerely, -The Buildbot From python-checkins at python.org Mon Apr 17 21:18:19 2006 From: python-checkins at python.org (martin.v.loewis) Date: Mon, 17 Apr 2006 21:18:19 +0200 (CEST) Subject: [Python-checkins] r45495 - python/trunk/Lib/pdb.py Message-ID: <20060417191819.25EA81E400A@bag.python.org> Author: martin.v.loewis Date: Mon Apr 17 21:18:18 2006 New Revision: 45495 Modified: python/trunk/Lib/pdb.py Log: Reindent. Modified: python/trunk/Lib/pdb.py ============================================================================== --- python/trunk/Lib/pdb.py (original) +++ python/trunk/Lib/pdb.py Mon Apr 17 21:18:18 2006 @@ -140,7 +140,7 @@ self._wait_for_mainpyfile = 0 if self.bp_commands(frame): self.interaction(frame, None) - + def bp_commands(self,frame): """ Call every command that was set for the current active breakpoint (if there is one) Returns True if the normal interaction function must be called, False otherwise """ @@ -158,7 +158,7 @@ if self.commands_doprompt[currentbp]: self.cmdloop() self.forget() - return + return return 1 def user_return(self, frame, return_value): @@ -226,8 +226,8 @@ def onecmd(self, line): """Interpret the argument as though it had been typed in response - to the prompt. - + to the prompt. + Checks wether this line is typed in the normal prompt or in a breakpoint command list definition """ if not self.commands_defining: @@ -235,7 +235,7 @@ else: return self.handle_command_def(line) - def handle_command_def(self,line): + def handle_command_def(self,line): """ Handles one command line during command list definition. """ cmd, arg, line = self.parseline(line) if cmd == 'silent': @@ -254,11 +254,11 @@ func = getattr(self, 'do_' + cmd) except AttributeError: func = self.default - if func.func_name in self.commands_resuming : # one of the resuming commands. + if func.func_name in self.commands_resuming : # one of the resuming commands. self.commands_doprompt[self.commands_bnum] = False self.cmdqueue = [] return 1 - return + return # Command definitions, called by cmdloop() # The argument is the remaining string on the command line From python-checkins at python.org Mon Apr 17 21:25:50 2006 From: python-checkins at python.org (martin.v.loewis) Date: Mon, 17 Apr 2006 21:25:50 +0200 (CEST) Subject: [Python-checkins] r45496 - in python/trunk: Makefile.pre.in Misc/NEWS Message-ID: <20060417192550.A9D081E4015@bag.python.org> Author: martin.v.loewis Date: Mon Apr 17 21:25:49 2006 New Revision: 45496 Modified: python/trunk/Makefile.pre.in python/trunk/Misc/NEWS Log: Add reindent target. Modified: python/trunk/Makefile.pre.in ============================================================================== --- python/trunk/Makefile.pre.in (original) +++ python/trunk/Makefile.pre.in Mon Apr 17 21:25:49 2006 @@ -956,6 +956,10 @@ .c.o: $(CC) -c $(PY_CFLAGS) -o $@ $< +# Run reindent on the library +reindent: + ./python$(EXEEXT) $(srcdir)/Tools/scripts/reindent.py -r $(srcdir)/Lib + # Rerun configure with the same options as it was run last time, # provided the config.status script exists recheck: Modified: python/trunk/Misc/NEWS ============================================================================== --- python/trunk/Misc/NEWS (original) +++ python/trunk/Misc/NEWS Mon Apr 17 21:25:49 2006 @@ -95,6 +95,9 @@ Build ----- +- The Makefile now has a reindent target, which runs reindent.py on + the library. + - Patch #1470875: Building Python with MS Free Compiler - Patch #1161914: Add a python-config script. From buildbot at python.org Mon Apr 17 22:09:34 2006 From: buildbot at python.org (buildbot at python.org) Date: Mon, 17 Apr 2006 20:09:34 +0000 Subject: [Python-checkins] buildbot warnings in x86 cygwin trunk Message-ID: <20060417200934.904071E4007@bag.python.org> The Buildbot has detected a new failure of x86 cygwin trunk. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520cygwin%2520trunk/builds/196 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: andrew.kuchling,armin.rigo,george.yoshida,martin.v.loewis,ronald.oussoren Build Had Warnings: warnings test sincerely, -The Buildbot From python-checkins at python.org Mon Apr 17 22:17:26 2006 From: python-checkins at python.org (phillip.eby) Date: Mon, 17 Apr 2006 22:17:26 +0200 (CEST) Subject: [Python-checkins] r45497 - python/trunk/Lib/pkgutil.py python/trunk/Lib/runpy.py Message-ID: <20060417201726.6BC4A1E4007@bag.python.org> Author: phillip.eby Date: Mon Apr 17 22:17:25 2006 New Revision: 45497 Modified: python/trunk/Lib/pkgutil.py python/trunk/Lib/runpy.py Log: First phase of refactoring for runpy, pkgutil, pydoc, and setuptools to share common PEP 302 support code, as described here: http://mail.python.org/pipermail/python-dev/2006-April/063724.html This revision strips all the PEP 302 emulation code from runpy, replacing it with published API classes and functions in pkgutil, mostly using setuptools' implementation of common functionality, but adding features from runpy, and doing some refactoring to make the layer pydoc needs easier to implement on top of this. One step down, four to go, although step #4 (adding C versions of the new APIs to 'imp') may not be able to make it in time for alpha 2. We'll see how that goes. Modified: python/trunk/Lib/pkgutil.py ============================================================================== --- python/trunk/Lib/pkgutil.py (original) +++ python/trunk/Lib/pkgutil.py Mon Apr 17 22:17:25 2006 @@ -1,7 +1,271 @@ """Utilities to support packages.""" +# NOTE: This module must remain compatible with Python 2.3, as it is shared +# by setuptools for distribution with Python 2.3 and up. + import os import sys +import imp +import os.path +from types import ModuleType + +__all__ = [ + 'get_importer', 'iter_importers', 'get_loader', 'find_loader', + 'ImpImporter', 'ImpLoader', 'read_code', 'extend_path', +] + +def read_code(stream): + # This helper is needed in order for the PEP 302 emulation to + # correctly handle compiled files + import marshal + + magic = stream.read(4) + if magic != imp.get_magic(): + return None + + stream.read(4) # Skip timestamp + return marshal.load(stream) + + +class ImpImporter: + """PEP 302 Importer that wraps Python's "classic" import algorithm + + ImpImporter(dirname) produces a PEP 302 importer that searches that + directory. ImpImporter(None) produces a PEP 302 importer that searches + the current sys.path, plus any modules that are frozen or built-in. + + Note that ImpImporter does not currently support being used by placement + on sys.meta_path. + """ + + def __init__(self, path=None): + self.path = path + + def find_module(self, fullname, path=None): + # Note: we ignore 'path' argument since it is only used via meta_path + subname = fullname.split(".")[-1] + if subname != fullname and self.path is None: + return None + if self.path is None: + path = None + else: + path = [self.path] + try: + file, filename, etc = imp.find_module(subname, path) + except ImportError: + return None + return ImpLoader(fullname, file, filename, etc) + + +class ImpLoader: + """PEP 302 Loader that wraps Python's "classic" import algorithm + """ + code = source = None + + def __init__(self, fullname, file, filename, etc): + self.file = file + self.filename = filename + self.fullname = fullname + self.etc = etc + + def load_module(self, fullname): + self._reopen() + try: + mod = imp.load_module(fullname, self.file, self.filename, self.etc) + finally: + if self.file: + self.file.close() + # Note: we don't set __loader__ because we want the module to look + # normal; i.e. this is just a wrapper for standard import machinery + return mod + + def get_data(self, pathname): + return open(pathname, "rb").read() + + def _reopen(self): + if self.file and self.file.closed: + if mod_type==imp.PY_SOURCE: + self.file = open(self.filename, 'rU') + elif mod_type in (imp.PY_COMPILED, imp.C_EXTENSION): + self.file = open(self.filename, 'rb') + + def _fix_name(self, fullname): + if fullname is None: + fullname = self.fullname + elif fullname != self.fullname: + raise ImportError("Loader for module %s cannot handle " + "module %s" % (self.fullname, fullname)) + return fullname + + def is_package(self): + return self.etc[2]==imp.PKG_DIRECTORY + + def get_code(self, fullname=None): + fullname = self._fix_name(fullname) + if self.code is None: + mod_type = self.etc[2] + if mod_type==imp.PY_SOURCE: + source = self.get_source(fullname) + self.code = compile(source, self.filename, 'exec') + elif mod_type==imp.PY_COMPILED: + self._reopen() + try: + self.code = read_code(self.file) + finally: + self.file.close() + elif mod_type==imp.PKG_DIRECTORY: + self.code = self._get_delegate().get_code() + return self.code + + def get_source(self, fullname=None): + fullname = self._fix_name(fullname) + if self.source is None: + mod_type = self.etc[2] + if mod_type==imp.PY_SOURCE: + self._reopen() + try: + self.source = self.file.read() + finally: + self.file.close() + elif mod_type==imp.PY_COMPILED: + if os.path.exists(self.filename[:-1]): + f = open(self.filename[:-1], 'rU') + self.source = f.read() + f.close() + elif mod_type==imp.PKG_DIRECTORY: + self.source = self._get_delegate().get_source() + return self.source + + def _get_delegate(self): + return ImpImporter(self.filename).find_module('__init__') + + def get_filename(self, fullname=None): + fullname = self._fix_name(fullname) + mod_type = self.etc[2] + if self.etc[2]==imp.PKG_DIRECTORY: + return self._get_delegate().get_filename() + elif self.etc[2] in (imp.PY_SOURCE, imp.PY_COMPILED, imp.C_EXTENSION): + return self.filename + return None + + +def get_importer(path_item): + """Retrieve a PEP 302 importer for the given path item + + The returned importer is cached in sys.path_importer_cache + if it was newly created by a path hook. + + If there is no importer, a wrapper around the basic import + machinery is returned. This wrapper is never inserted into + the importer cache (None is inserted instead). + + The cache (or part of it) can be cleared manually if a + rescan of sys.path_hooks is necessary. + """ + try: + importer = sys.path_importer_cache[path_item] + except KeyError: + for path_hook in sys.path_hooks: + try: + importer = path_hook(path_item) + break + except ImportError: + pass + else: + importer = None + sys.path_importer_cache.setdefault(path_item,importer) + + if importer is None: + try: + importer = ImpImporter(path_item) + except ImportError: + pass + return importer + + +def iter_importers(fullname): + """Yield PEP 302 importers for the given module name + + If fullname contains a '.', the importers will be for the package + containing fullname, otherwise they will be importers for sys.meta_path, + sys.path, and Python's "classic" import machinery, in that order. If + the named module is in a package, that package is imported as a side + effect of invoking this function. + + Non PEP 302 mechanisms (e.g. the Windows registry) used by the + standard import machinery to find files in alternative locations + are partially supported, but are searched AFTER sys.path. Normally, + these locations are searched BEFORE sys.path, preventing sys.path + entries from shadowing them. + + For this to cause a visible difference in behaviour, there must + be a module or package name that is accessible via both sys.path + and one of the non PEP 302 file system mechanisms. In this case, + the emulation will find the former version, while the builtin + import mechanism will find the latter. + + Items of the following types can be affected by this discrepancy: + imp.C_EXTENSION, imp.PY_SOURCE, imp.PY_COMPILED, imp.PKG_DIRECTORY + """ + if fullname.startswith('.'): + raise ImportError("Relative module names not supported") + if '.' in fullname: + # Get the containing package's __path__ + pkg = '.'.join(fullname.split('.')[:-1]) + if pkg not in sys.modules: + __import__(pkg) + path = getattr(sys.modules[pkg],'__path__',None) or [] + else: + for importer in sys.meta_path: + yield importer + path = sys.path + for item in path: + yield get_importer(item) + if '.' not in fullname: + yield ImpImporter() + + +def get_loader(module_or_name): + """Get a PEP 302 "loader" object for module_or_name + + If the module or package is accessible via the normal import + mechanism, a wrapper around the relevant part of that machinery + is returned. Returns None if the module cannot be found or imported. + If the named module is not already imported, its containing package + (if any) is imported, in order to establish the package __path__. + + This function uses iter_importers(), and is thus subject to the same + limitations regarding platform-specific special import locations such + as the Windows registry. + """ + if module_or_name in sys.modules: + module_or_name = sys.modules[module_or_name] + if isinstance(module_or_name, ModuleType): + module = module_or_name + loader = getattr(module,'__loader__',None) + if loader is not None: + return loader + fullname = module.__name__ + else: + fullname = module_or_name + return find_loader(fullname) + + +def find_loader(fullname): + """Find a PEP 302 "loader" object for fullname + + If fullname contains dots, path must be the containing package's __path__. + Returns None if the module cannot be found or imported. This function uses + iter_importers(), and is thus subject to the same limitations regarding + platform-specific special import locations such as the Windows registry. + """ + for importer in iter_importers(fullname): + loader = importer.find_module(fullname) + if loader is not None: + return loader + + return None + def extend_path(path, name): """Extend a package's path. Modified: python/trunk/Lib/runpy.py ============================================================================== --- python/trunk/Lib/runpy.py (original) +++ python/trunk/Lib/runpy.py Mon Apr 17 22:17:25 2006 @@ -11,349 +11,15 @@ import sys import imp +try: + from imp import get_loader +except ImportError: + from pkgutil import get_loader __all__ = [ "run_module", ] -try: - _get_loader = imp.get_loader -except AttributeError: - # get_loader() is not provided by the imp module, so emulate it - # as best we can using the PEP 302 import machinery exposed since - # Python 2.3. The emulation isn't perfect, but the differences - # in the way names are shadowed shouldn't matter in practice. - import os.path - import marshal # Handle compiled Python files - - # This helper is needed in order for the PEP 302 emulation to - # correctly handle compiled files - def _read_compiled_file(compiled_file): - magic = compiled_file.read(4) - if magic != imp.get_magic(): - return None - try: - compiled_file.read(4) # Skip timestamp - return marshal.load(compiled_file) - except Exception: - return None - - class _AbsoluteImporter(object): - """PEP 302 importer wrapper for top level import machinery""" - def find_module(self, mod_name, path=None): - if path is not None: - return None - try: - file, filename, mod_info = imp.find_module(mod_name) - except ImportError: - return None - suffix, mode, mod_type = mod_info - if mod_type == imp.PY_SOURCE: - loader = _SourceFileLoader(mod_name, file, - filename, mod_info) - elif mod_type == imp.PY_COMPILED: - loader = _CompiledFileLoader(mod_name, file, - filename, mod_info) - elif mod_type == imp.PKG_DIRECTORY: - loader = _PackageDirLoader(mod_name, file, - filename, mod_info) - elif mod_type == imp.C_EXTENSION: - loader = _FileSystemLoader(mod_name, file, - filename, mod_info) - else: - loader = _BasicLoader(mod_name, file, - filename, mod_info) - return loader - - - class _FileSystemImporter(object): - """PEP 302 importer wrapper for filesystem based imports""" - def __init__(self, path_item=None): - if path_item is not None: - if path_item != '' and not os.path.isdir(path_item): - raise ImportError("%s is not a directory" % path_item) - self.path_dir = path_item - else: - raise ImportError("Filesystem importer requires " - "a directory name") - - def find_module(self, mod_name, path=None): - if path is not None: - return None - path_dir = self.path_dir - if path_dir == '': - path_dir = os.getcwd() - sub_name = mod_name.rsplit(".", 1)[-1] - try: - file, filename, mod_info = imp.find_module(sub_name, - [path_dir]) - except ImportError: - return None - if not filename.startswith(path_dir): - return None - suffix, mode, mod_type = mod_info - if mod_type == imp.PY_SOURCE: - loader = _SourceFileLoader(mod_name, file, - filename, mod_info) - elif mod_type == imp.PY_COMPILED: - loader = _CompiledFileLoader(mod_name, file, - filename, mod_info) - elif mod_type == imp.PKG_DIRECTORY: - loader = _PackageDirLoader(mod_name, file, - filename, mod_info) - elif mod_type == imp.C_EXTENSION: - loader = _FileSystemLoader(mod_name, file, - filename, mod_info) - else: - loader = _BasicLoader(mod_name, file, - filename, mod_info) - return loader - - - class _BasicLoader(object): - """PEP 302 loader wrapper for top level import machinery""" - def __init__(self, mod_name, file, filename, mod_info): - self.mod_name = mod_name - self.file = file - self.filename = filename - self.mod_info = mod_info - - def _fix_name(self, mod_name): - if mod_name is None: - mod_name = self.mod_name - elif mod_name != self.mod_name: - raise ImportError("Loader for module %s cannot handle " - "module %s" % (self.mod_name, mod_name)) - return mod_name - - def load_module(self, mod_name=None): - mod_name = self._fix_name(mod_name) - mod = imp.load_module(mod_name, self.file, - self.filename, self.mod_info) - mod.__loader__ = self # for introspection - return mod - - def get_code(self, mod_name=None): - return None - - def get_source(self, mod_name=None): - return None - - def is_package(self, mod_name=None): - return False - - def close(self): - if self.file: - self.file.close() - - def __del__(self): - self.close() - - - class _FileSystemLoader(_BasicLoader): - """PEP 302 loader wrapper for filesystem based imports""" - def get_code(self, mod_name=None): - mod_name = self._fix_name(mod_name) - return self._get_code(mod_name) - - def get_data(self, pathname): - return open(pathname, "rb").read() - - def get_filename(self, mod_name=None): - mod_name = self._fix_name(mod_name) - return self._get_filename(mod_name) - - def get_source(self, mod_name=None): - mod_name = self._fix_name(mod_name) - return self._get_source(mod_name) - - def is_package(self, mod_name=None): - mod_name = self._fix_name(mod_name) - return self._is_package(mod_name) - - def _get_code(self, mod_name): - return None - - def _get_filename(self, mod_name): - return self.filename - - def _get_source(self, mod_name): - return None - - def _is_package(self, mod_name): - return False - - class _PackageDirLoader(_FileSystemLoader): - """PEP 302 loader wrapper for PKG_DIRECTORY directories""" - def _is_package(self, mod_name): - return True - - - class _SourceFileLoader(_FileSystemLoader): - """PEP 302 loader wrapper for PY_SOURCE modules""" - def _get_code(self, mod_name): - return compile(self._get_source(mod_name), - self.filename, 'exec') - - def _get_source(self, mod_name): - f = self.file - f.seek(0) - return f.read() - - - class _CompiledFileLoader(_FileSystemLoader): - """PEP 302 loader wrapper for PY_COMPILED modules""" - def _get_code(self, mod_name): - f = self.file - f.seek(0) - return _read_compiled_file(f) - - - def _get_importer(path_item): - """Retrieve a PEP 302 importer for the given path item - - The returned importer is cached in sys.path_importer_cache - if it was newly created by a path hook. - - If there is no importer, a wrapper around the basic import - machinery is returned. This wrapper is never inserted into - the importer cache (None is inserted instead). - - The cache (or part of it) can be cleared manually if a - rescan of sys.path_hooks is necessary. - """ - try: - importer = sys.path_importer_cache[path_item] - except KeyError: - for path_hook in sys.path_hooks: - try: - importer = path_hook(path_item) - break - except ImportError: - pass - else: - importer = None - sys.path_importer_cache[path_item] = importer - if importer is None: - try: - importer = _FileSystemImporter(path_item) - except ImportError: - pass - return importer - - - def _get_path_loader(mod_name, path=None): - """Retrieve a PEP 302 loader using a path importer""" - if path is None: - path = sys.path - absolute_loader = _AbsoluteImporter().find_module(mod_name) - if isinstance(absolute_loader, _FileSystemLoader): - # Found in filesystem, so scan path hooks - # before accepting this one as the right one - loader = None - else: - # Not found in filesystem, so use top-level loader - loader = absolute_loader - else: - loader = absolute_loader = None - if loader is None: - for path_item in path: - importer = _get_importer(path_item) - if importer is not None: - loader = importer.find_module(mod_name) - if loader is not None: - # Found a loader for our module - break - else: - # No path hook found, so accept the top level loader - loader = absolute_loader - return loader - - def _get_package(pkg_name): - """Retrieve a named package""" - pkg = __import__(pkg_name) - sub_pkg_names = pkg_name.split(".") - for sub_pkg in sub_pkg_names[1:]: - pkg = getattr(pkg, sub_pkg) - return pkg - - def _get_loader(mod_name, path=None): - """Retrieve a PEP 302 loader for the given module or package - - If the module or package is accessible via the normal import - mechanism, a wrapper around the relevant part of that machinery - is returned. - - Non PEP 302 mechanisms (e.g. the Windows registry) used by the - standard import machinery to find files in alternative locations - are partially supported, but are searched AFTER sys.path. Normally, - these locations are searched BEFORE sys.path, preventing sys.path - entries from shadowing them. - For this to cause a visible difference in behaviour, there must - be a module or package name that is accessible via both sys.path - and one of the non PEP 302 file system mechanisms. In this case, - the emulation will find the former version, while the builtin - import mechanism will find the latter. - Items of the following types can be affected by this discrepancy: - imp.C_EXTENSION - imp.PY_SOURCE - imp.PY_COMPILED - imp.PKG_DIRECTORY - """ - try: - loader = sys.modules[mod_name].__loader__ - except (KeyError, AttributeError): - loader = None - if loader is None: - imp.acquire_lock() - try: - # Module not in sys.modules, or uses an unhooked loader - parts = mod_name.rsplit(".", 1) - if len(parts) == 2: - # Sub package, so use parent package's path - pkg_name, sub_name = parts - if pkg_name and pkg_name[0] != '.': - if path is not None: - raise ImportError("Path argument must be None " - "for a dotted module name") - pkg = _get_package(pkg_name) - try: - path = pkg.__path__ - except AttributeError: - raise ImportError(pkg_name + - " is not a package") - else: - raise ImportError("Relative import syntax is not " - "supported by _get_loader()") - else: - # Top level module, so stick with default path - sub_name = mod_name - - for importer in sys.meta_path: - loader = importer.find_module(mod_name, path) - if loader is not None: - # Found a metahook to handle the module - break - else: - # Handling via the standard path mechanism - loader = _get_path_loader(mod_name, path) - finally: - imp.release_lock() - return loader - - -# This helper is needed due to a missing component in the PEP 302 -# loader protocol (specifically, "get_filename" is non-standard) -def _get_filename(loader, mod_name): - try: - get_filename = loader.get_filename - except AttributeError: - return None - else: - return get_filename(mod_name) - -# ------------------------------------------------------------ -# Done with the import machinery emulation, on with the code! def _run_code(code, run_globals, init_globals, mod_name, mod_fname, mod_loader): @@ -399,13 +65,24 @@ mod_name, mod_fname, mod_loader) +# This helper is needed due to a missing component in the PEP 302 +# loader protocol (specifically, "get_filename" is non-standard) +def _get_filename(loader, mod_name): + try: + get_filename = loader.get_filename + except AttributeError: + return None + else: + return get_filename(mod_name) + + def run_module(mod_name, init_globals=None, run_name=None, alter_sys=False): """Execute a module's code without importing it Returns the resulting top level namespace dictionary """ - loader = _get_loader(mod_name) + loader = get_loader(mod_name) if loader is None: raise ImportError("No module named " + mod_name) code = loader.get_code(mod_name) From buildbot at python.org Mon Apr 17 22:25:26 2006 From: buildbot at python.org (buildbot at python.org) Date: Mon, 17 Apr 2006 20:25:26 +0000 Subject: [Python-checkins] buildbot warnings in x86 Ubuntu dapper (icc) trunk Message-ID: <20060417202526.B5E821E4007@bag.python.org> The Buildbot has detected a new failure of x86 Ubuntu dapper (icc) trunk. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520Ubuntu%2520dapper%2520%2528icc%2529%2520trunk/builds/175 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: martin.v.loewis Build Had Warnings: warnings test sincerely, -The Buildbot From buildbot at python.org Mon Apr 17 22:57:39 2006 From: buildbot at python.org (buildbot at python.org) Date: Mon, 17 Apr 2006 20:57:39 +0000 Subject: [Python-checkins] buildbot failure in x86 cygwin trunk Message-ID: <20060417205739.707D71E400A@bag.python.org> The Buildbot has detected a new failure of x86 cygwin trunk. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520cygwin%2520trunk/builds/198 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: phillip.eby BUILD FAILED: failed compile sincerely, -The Buildbot From python-checkins at python.org Mon Apr 17 23:12:34 2006 From: python-checkins at python.org (tim.peters) Date: Mon, 17 Apr 2006 23:12:34 +0200 (CEST) Subject: [Python-checkins] r45498 - in python/trunk: Lib/_threading_local.py Misc/NEWS Message-ID: <20060417211234.42FC51E4007@bag.python.org> Author: tim.peters Date: Mon Apr 17 23:12:33 2006 New Revision: 45498 Modified: python/trunk/Lib/_threading_local.py python/trunk/Misc/NEWS Log: local.__del__(): This didn't actually do anything, because of too much convolution <0.5 wink>. Simplified to the point that it works, and test_threading_local no longer reports leaks under -R. Thanks to Thomas Wouters for initial analysis. Modified: python/trunk/Lib/_threading_local.py ============================================================================== --- python/trunk/Lib/_threading_local.py (original) +++ python/trunk/Lib/_threading_local.py Mon Apr 17 23:12:33 2006 @@ -133,7 +133,7 @@ >>> del mydata """ -# Threading import is at end +from threading import currentThread, RLock, enumerate class _localbase(object): __slots__ = '_local__key', '_local__args', '_local__lock' @@ -203,35 +203,30 @@ lock.release() - def __del__(): - threading_enumerate = enumerate - __getattribute__ = object.__getattribute__ + # The default argument is a hack, to give __del__ a local name for + # threading.enumerate (sidestepping problems with Python None'ing-out + # module globals at shutdown time). + def __del__(self, _threading_enumerate=enumerate): - def __del__(self): - key = __getattribute__(self, '_local__key') + key = object.__getattribute__(self, '_local__key') + try: + threads = list(_threading_enumerate()) + except: + # If enumerate fails, as it seems to do during + # shutdown, we'll skip cleanup under the assumption + # that there is nothing to clean up. + return + + for thread in threads: try: - threads = list(threading_enumerate()) - except: - # if enumerate fails, as it seems to do during - # shutdown, we'll skip cleanup under the assumption - # that there is nothing to clean up - return + __dict__ = thread.__dict__ + except AttributeError: + # Thread is dying, rest in peace. + continue - for thread in threads: + if key in __dict__: try: - __dict__ = thread.__dict__ - except AttributeError: - # Thread is dying, rest in peace - continue - - if key in __dict__: - try: - del __dict__[key] - except KeyError: - pass # didn't have anything in this thread - - return __del__ - __del__ = __del__() - -from threading import currentThread, enumerate, RLock + del __dict__[key] + except KeyError: + pass # didn't have anything in this thread Modified: python/trunk/Misc/NEWS ============================================================================== --- python/trunk/Misc/NEWS (original) +++ python/trunk/Misc/NEWS Mon Apr 17 23:12:33 2006 @@ -72,6 +72,9 @@ Library ------- +- The ``__del__`` method of class ``local`` in module ``_threading_local`` + returned before accomplishing any of its intended cleanup. + - Patch #790710: Add breakpoint command lists in pdb. - Patch #1063914: Add Tkinter.Misc.clipboard_get(). From python-checkins at python.org Mon Apr 17 23:15:54 2006 From: python-checkins at python.org (tim.peters) Date: Mon, 17 Apr 2006 23:15:54 +0200 (CEST) Subject: [Python-checkins] r45499 - in python/branches/release24-maint: Lib/_threading_local.py Misc/NEWS Message-ID: <20060417211554.7A7AA1E4007@bag.python.org> Author: tim.peters Date: Mon Apr 17 23:15:53 2006 New Revision: 45499 Modified: python/branches/release24-maint/Lib/_threading_local.py python/branches/release24-maint/Misc/NEWS Log: Merge rev 45498 from trunk. local.__del__(): This didn't actually do anything, because of too much convolution <0.5 wink>. Simplified to the point that it works, and test_threading_local no longer reports leaks under -R. Thanks to Thomas Wouters for initial analysis. Modified: python/branches/release24-maint/Lib/_threading_local.py ============================================================================== --- python/branches/release24-maint/Lib/_threading_local.py (original) +++ python/branches/release24-maint/Lib/_threading_local.py Mon Apr 17 23:15:53 2006 @@ -133,7 +133,7 @@ >>> del mydata """ -# Threading import is at end +from threading import currentThread, RLock, enumerate class _localbase(object): __slots__ = '_local__key', '_local__args', '_local__lock' @@ -203,35 +203,30 @@ lock.release() - def __del__(): - threading_enumerate = enumerate - __getattribute__ = object.__getattribute__ + # The default argument is a hack, to give __del__ a local name for + # threading.enumerate (sidestepping problems with Python None'ing-out + # module globals at shutdown time). + def __del__(self, _threading_enumerate=enumerate): - def __del__(self): - key = __getattribute__(self, '_local__key') + key = object.__getattribute__(self, '_local__key') + try: + threads = list(_threading_enumerate()) + except: + # If enumerate fails, as it seems to do during + # shutdown, we'll skip cleanup under the assumption + # that there is nothing to clean up. + return + + for thread in threads: try: - threads = list(threading_enumerate()) - except: - # if enumerate fails, as it seems to do during - # shutdown, we'll skip cleanup under the assumption - # that there is nothing to clean up - return + __dict__ = thread.__dict__ + except AttributeError: + # Thread is dying, rest in peace. + continue - for thread in threads: + if key in __dict__: try: - __dict__ = thread.__dict__ - except AttributeError: - # Thread is dying, rest in peace - continue - - if key in __dict__: - try: - del __dict__[key] - except KeyError: - pass # didn't have anything in this thread - - return __del__ - __del__ = __del__() - -from threading import currentThread, enumerate, RLock + del __dict__[key] + except KeyError: + pass # didn't have anything in this thread Modified: python/branches/release24-maint/Misc/NEWS ============================================================================== --- python/branches/release24-maint/Misc/NEWS (original) +++ python/branches/release24-maint/Misc/NEWS Mon Apr 17 23:15:53 2006 @@ -29,6 +29,9 @@ Library ------- +- The ``__del__`` method of class ``local`` in module ``_threading_local`` + returned before accomplishing any of its intended cleanup. + - Patch #1191700: Adjust column alignment in bdb breakpoint lists. - The email module's parsedate_tz function now sets the daylight savings From neal at metaslash.com Mon Apr 17 23:19:20 2006 From: neal at metaslash.com (Neal Norwitz) Date: Mon, 17 Apr 2006 17:19:20 -0400 Subject: [Python-checkins] Python Regression Test Failures refleak (1) Message-ID: <20060417211920.GA26354@python.psfb.org> test_cmd_line leaked [0, 0, 17] references test_threadedtempfile leaked [-84, 0, 0] references test_threading_local leaked [34, -69, 34] references test_urllib2 leaked [-121, 88, 99] references From python-checkins at python.org Tue Apr 18 02:27:47 2006 From: python-checkins at python.org (skip.montanaro) Date: Tue, 18 Apr 2006 02:27:47 +0200 (CEST) Subject: [Python-checkins] r45502 - in python/trunk: Include/abstract.h Objects/abstract.c Message-ID: <20060418002747.828191E4007@bag.python.org> Author: skip.montanaro Date: Tue Apr 18 02:27:46 2006 New Revision: 45502 Modified: python/trunk/Include/abstract.h python/trunk/Objects/abstract.c Log: C++ compilation cleanup: Migrate declaration of _PyObject_Call(Function|Method)_SizeT into Include/abstract.h. This gets them under the umbrella of the extern "C" { ... } block in that file. Modified: python/trunk/Include/abstract.h ============================================================================== --- python/trunk/Include/abstract.h (original) +++ python/trunk/Include/abstract.h Tue Apr 18 02:27:46 2006 @@ -348,6 +348,11 @@ Python expression: o.method(args). */ + PyAPI_FUNC(PyObject *) _PyObject_CallFunction_SizeT(PyObject *callable, + char *format, ...); + PyAPI_FUNC(PyObject *) _PyObject_CallMethod_SizeT(PyObject *o, + char *name, + char *format, ...); PyAPI_FUNC(PyObject *) PyObject_CallFunctionObjArgs(PyObject *callable, ...); Modified: python/trunk/Objects/abstract.c ============================================================================== --- python/trunk/Objects/abstract.c (original) +++ python/trunk/Objects/abstract.c Tue Apr 18 02:27:46 2006 @@ -10,13 +10,6 @@ #define HASINDEX(o) PyType_HasFeature((o)->ob_type, Py_TPFLAGS_HAVE_INDEX) -#ifdef HAVE_DECLSPEC_DLL -PyAPI_FUNC(PyObject *) _PyObject_CallFunction_SizeT(PyObject *callable_object, - char *format, ...); -PyAPI_FUNC(PyObject *) _PyObject_CallMethod_SizeT(PyObject *o, char *m, - char *format, ...); -#endif - /* Shorthands to return certain errors */ From python-checkins at python.org Tue Apr 18 02:29:29 2006 From: python-checkins at python.org (skip.montanaro) Date: Tue, 18 Apr 2006 02:29:29 +0200 (CEST) Subject: [Python-checkins] r45503 - python/trunk/Include/modsupport.h Message-ID: <20060418002929.997DC1E4007@bag.python.org> Author: skip.montanaro Date: Tue Apr 18 02:29:29 2006 New Revision: 45503 Modified: python/trunk/Include/modsupport.h Log: C++ compile cleanup: proper declaration of _Py_BuildValue_SizeT Modified: python/trunk/Include/modsupport.h ============================================================================== --- python/trunk/Include/modsupport.h (original) +++ python/trunk/Include/modsupport.h Tue Apr 18 02:29:29 2006 @@ -29,6 +29,7 @@ const char *, char **, ...); PyAPI_FUNC(int) PyArg_UnpackTuple(PyObject *, const char *, Py_ssize_t, Py_ssize_t, ...); PyAPI_FUNC(PyObject *) Py_BuildValue(const char *, ...); +PyAPI_FUNC(PyObject *) _Py_BuildValue_SizeT(const char *, ...); PyAPI_FUNC(int) _PyArg_NoKeywords(const char *funcname, PyObject *kw); PyAPI_FUNC(int) PyArg_VaParse(PyObject *, const char *, va_list); From python-checkins at python.org Tue Apr 18 02:35:44 2006 From: python-checkins at python.org (skip.montanaro) Date: Tue, 18 Apr 2006 02:35:44 +0200 (CEST) Subject: [Python-checkins] r45504 - python/trunk/Objects/complexobject.c python/trunk/Objects/floatobject.c python/trunk/Objects/intobject.c python/trunk/Objects/longobject.c python/trunk/Objects/stringobject.c python/trunk/Objects/typeobject.c Message-ID: <20060418003544.801A01E4007@bag.python.org> Author: skip.montanaro Date: Tue Apr 18 02:35:43 2006 New Revision: 45504 Modified: python/trunk/Objects/complexobject.c python/trunk/Objects/floatobject.c python/trunk/Objects/intobject.c python/trunk/Objects/longobject.c python/trunk/Objects/stringobject.c python/trunk/Objects/typeobject.c Log: C++ compiler cleanup: bunch-o-casts, plus use of unsigned loop index var in a couple places Modified: python/trunk/Objects/complexobject.c ============================================================================== --- python/trunk/Objects/complexobject.c (original) +++ python/trunk/Objects/complexobject.c Tue Apr 18 02:35:43 2006 @@ -688,7 +688,7 @@ } #ifdef Py_USING_UNICODE else if (PyUnicode_Check(v)) { - if (PyUnicode_GET_SIZE(v) >= sizeof(s_buffer)) { + if (PyUnicode_GET_SIZE(v) >= (Py_ssize_t)sizeof(s_buffer)) { PyErr_SetString(PyExc_ValueError, "complex() literal too large to convert"); return NULL; Modified: python/trunk/Objects/floatobject.c ============================================================================== --- python/trunk/Objects/floatobject.c (original) +++ python/trunk/Objects/floatobject.c Tue Apr 18 02:35:43 2006 @@ -97,7 +97,7 @@ } #ifdef Py_USING_UNICODE else if (PyUnicode_Check(v)) { - if (PyUnicode_GET_SIZE(v) >= sizeof(s_buffer)) { + if (PyUnicode_GET_SIZE(v) >= (Py_ssize_t)sizeof(s_buffer)) { PyErr_SetString(PyExc_ValueError, "Unicode float() literal too long to convert"); return NULL; Modified: python/trunk/Objects/intobject.c ============================================================================== --- python/trunk/Objects/intobject.c (original) +++ python/trunk/Objects/intobject.c Tue Apr 18 02:35:43 2006 @@ -255,18 +255,18 @@ if (op == NULL || (nb = op->ob_type->tp_as_number) == NULL || nb->nb_int == NULL) { PyErr_SetString(PyExc_TypeError, "an integer is required"); - return -1; + return (unsigned long)-1; } io = (PyIntObject*) (*nb->nb_int) (op); if (io == NULL) - return -1; + return (unsigned long)-1; if (!PyInt_Check(io)) { if (PyLong_Check(io)) { val = PyLong_AsUnsignedLongMask((PyObject *)io); Py_DECREF(io); if (PyErr_Occurred()) - return -1; + return (unsigned long)-1; return val; } else @@ -274,7 +274,7 @@ Py_DECREF(io); PyErr_SetString(PyExc_TypeError, "nb_int should return int object"); - return -1; + return (unsigned long)-1; } } @@ -300,18 +300,18 @@ if (op == NULL || (nb = op->ob_type->tp_as_number) == NULL || nb->nb_int == NULL) { PyErr_SetString(PyExc_TypeError, "an integer is required"); - return -1; + return (unsigned PY_LONG_LONG)-1; } io = (PyIntObject*) (*nb->nb_int) (op); if (io == NULL) - return -1; + return (unsigned PY_LONG_LONG)-1; if (!PyInt_Check(io)) { if (PyLong_Check(io)) { val = PyLong_AsUnsignedLongLongMask((PyObject *)io); Py_DECREF(io); if (PyErr_Occurred()) - return -1; + return (unsigned PY_LONG_LONG)-1; return val; } else @@ -319,7 +319,7 @@ Py_DECREF(io); PyErr_SetString(PyExc_TypeError, "nb_int should return int object"); - return -1; + return (unsigned PY_LONG_LONG)-1; } } @@ -1152,6 +1152,7 @@ PyIntObject *p; PyIntBlock *list, *next; int i; + unsigned int ctr; int bc, bf; /* block count, number of freed blocks */ int irem, isum; /* remaining unfreed ints per block, total */ @@ -1174,9 +1175,9 @@ while (list != NULL) { bc++; irem = 0; - for (i = 0, p = &list->objects[0]; - i < N_INTOBJECTS; - i++, p++) { + for (ctr = 0, p = &list->objects[0]; + ctr < N_INTOBJECTS; + ctr++, p++) { if (PyInt_CheckExact(p) && p->ob_refcnt != 0) irem++; } @@ -1184,9 +1185,9 @@ if (irem) { list->next = block_list; block_list = list; - for (i = 0, p = &list->objects[0]; - i < N_INTOBJECTS; - i++, p++) { + for (ctr = 0, p = &list->objects[0]; + ctr < N_INTOBJECTS; + ctr++, p++) { if (!PyInt_CheckExact(p) || p->ob_refcnt == 0) { p->ob_type = (struct _typeobject *) @@ -1227,9 +1228,9 @@ if (Py_VerboseFlag > 1) { list = block_list; while (list != NULL) { - for (i = 0, p = &list->objects[0]; - i < N_INTOBJECTS; - i++, p++) { + for (ctr = 0, p = &list->objects[0]; + ctr < N_INTOBJECTS; + ctr++, p++) { if (PyInt_CheckExact(p) && p->ob_refcnt != 0) /* XXX(twouters) cast refcount to long until %zd is universally Modified: python/trunk/Objects/longobject.c ============================================================================== --- python/trunk/Objects/longobject.c (original) +++ python/trunk/Objects/longobject.c Tue Apr 18 02:35:43 2006 @@ -419,7 +419,7 @@ digit msd = v->ob_digit[ndigits - 1]; result = (ndigits - 1) * SHIFT; - if (result / SHIFT != ndigits - 1) + if (result / SHIFT != (size_t)(ndigits - 1)) goto Overflow; do { ++result; @@ -953,7 +953,7 @@ if (vv == NULL || !PyLong_Check(vv)) { PyErr_BadInternalCall(); - return -1; + return (unsigned PY_LONG_LONG)-1; } res = _PyLong_AsByteArray( Modified: python/trunk/Objects/stringobject.c ============================================================================== --- python/trunk/Objects/stringobject.c (original) +++ python/trunk/Objects/stringobject.c Tue Apr 18 02:35:43 2006 @@ -746,7 +746,7 @@ *s = PyString_AS_STRING(obj); if (len != NULL) *len = PyString_GET_SIZE(obj); - else if (strlen(*s) != PyString_GET_SIZE(obj)) { + else if (strlen(*s) != (size_t)PyString_GET_SIZE(obj)) { PyErr_SetString(PyExc_TypeError, "expected string without null bytes"); return -1; Modified: python/trunk/Objects/typeobject.c ============================================================================== --- python/trunk/Objects/typeobject.c (original) +++ python/trunk/Objects/typeobject.c Tue Apr 18 02:35:43 2006 @@ -1118,12 +1118,12 @@ off = PyOS_snprintf(buf, sizeof(buf), "Cannot create a \ consistent method resolution\norder (MRO) for bases"); i = 0; - while (PyDict_Next(set, &i, &k, &v) && off < sizeof(buf)) { + while (PyDict_Next(set, &i, &k, &v) && (size_t)off < sizeof(buf)) { PyObject *name = class_name(k); off += PyOS_snprintf(buf + off, sizeof(buf) - off, " %s", name ? PyString_AS_STRING(name) : "?"); Py_XDECREF(name); - if (--n && off+1 < sizeof(buf)) { + if (--n && (size_t)(off+1) < sizeof(buf)) { buf[off++] = ','; buf[off] = '\0'; } @@ -5186,16 +5186,16 @@ /* Note: this depends on the order of the members of PyHeapTypeObject! */ assert(offset >= 0); - assert(offset < offsetof(PyHeapTypeObject, as_buffer)); - if (offset >= offsetof(PyHeapTypeObject, as_sequence)) { + assert((size_t)offset < offsetof(PyHeapTypeObject, as_buffer)); + if ((size_t)offset >= offsetof(PyHeapTypeObject, as_sequence)) { ptr = (char *)type->tp_as_sequence; offset -= offsetof(PyHeapTypeObject, as_sequence); } - else if (offset >= offsetof(PyHeapTypeObject, as_mapping)) { + else if ((size_t)offset >= offsetof(PyHeapTypeObject, as_mapping)) { ptr = (char *)type->tp_as_mapping; offset -= offsetof(PyHeapTypeObject, as_mapping); } - else if (offset >= offsetof(PyHeapTypeObject, as_number)) { + else if ((size_t)offset >= offsetof(PyHeapTypeObject, as_number)) { ptr = (char *)type->tp_as_number; offset -= offsetof(PyHeapTypeObject, as_number); } From python-checkins at python.org Tue Apr 18 02:49:50 2006 From: python-checkins at python.org (skip.montanaro) Date: Tue, 18 Apr 2006 02:49:50 +0200 (CEST) Subject: [Python-checkins] r45505 - python/trunk/Modules/posixmodule.c Message-ID: <20060418004950.454AC1E400A@bag.python.org> Author: skip.montanaro Date: Tue Apr 18 02:49:49 2006 New Revision: 45505 Modified: python/trunk/Modules/posixmodule.c Log: reset errno before calling confstr - use confstr() doc to simplify checks afterwards Modified: python/trunk/Modules/posixmodule.c ============================================================================== --- python/trunk/Modules/posixmodule.c (original) +++ python/trunk/Modules/posixmodule.c Tue Apr 18 02:49:49 2006 @@ -6812,17 +6812,19 @@ char buffer[64]; if (PyArg_ParseTuple(args, "O&:confstr", conv_confstr_confname, &name)) { - int len = confstr(name, buffer, sizeof(buffer)); + int len; errno = 0; - if (len == 0) { - if (errno != 0) - posix_error(); - else - result = PyString_FromString(""); + len = confstr(name, buffer, sizeof(buffer)); + + if (len == -1) { + posix_error(); + } + else if (len == 0) { + result = PyString_FromString(""); } else { - if (len >= sizeof(buffer)) { + if ((unsigned int)len >= sizeof(buffer)) { result = PyString_FromStringAndSize(NULL, len); if (result != NULL) confstr(name, PyString_AS_STRING(result), len+1); From python-checkins at python.org Tue Apr 18 02:53:07 2006 From: python-checkins at python.org (skip.montanaro) Date: Tue, 18 Apr 2006 02:53:07 +0200 (CEST) Subject: [Python-checkins] r45506 - python/trunk/Parser/tokenizer.c Message-ID: <20060418005307.1AC341E400A@bag.python.org> Author: skip.montanaro Date: Tue Apr 18 02:53:06 2006 New Revision: 45506 Modified: python/trunk/Parser/tokenizer.c Log: C++ compiler cleanup: cast signed to unsigned Modified: python/trunk/Parser/tokenizer.c ============================================================================== --- python/trunk/Parser/tokenizer.c (original) +++ python/trunk/Parser/tokenizer.c Tue Apr 18 02:53:06 2006 @@ -1230,7 +1230,7 @@ do { *tp++ = c = tok_nextc(tok); } while (c != EOF && c != '\n' && - tp - cbuf + 1 < sizeof(cbuf)); + (unsigned int)(tp - cbuf + 1) < sizeof(cbuf)); *tp = '\0'; for (cp = tabforms; cp < tabforms + sizeof(tabforms)/sizeof(tabforms[0]); From python-checkins at python.org Tue Apr 18 02:53:48 2006 From: python-checkins at python.org (skip.montanaro) Date: Tue, 18 Apr 2006 02:53:48 +0200 (CEST) Subject: [Python-checkins] r45507 - python/trunk/Python/modsupport.c Message-ID: <20060418005348.84EDE1E400A@bag.python.org> Author: skip.montanaro Date: Tue Apr 18 02:53:48 2006 New Revision: 45507 Modified: python/trunk/Python/modsupport.c Log: C++ compiler cleanup: migrate to modsupport.h Modified: python/trunk/Python/modsupport.c ============================================================================== --- python/trunk/Python/modsupport.c (original) +++ python/trunk/Python/modsupport.c Tue Apr 18 02:53:48 2006 @@ -7,9 +7,6 @@ typedef double va_double; static PyObject *va_build_value(const char *, va_list, int); -#ifdef HAVE_DECLSPEC_DLL -PyAPI_FUNC(PyObject *) _Py_BuildValue_SizeT(const char *, ...); -#endif /* Package context -- the full module name for package imports */ char *_Py_PackageContext = NULL; From python-checkins at python.org Tue Apr 18 02:55:46 2006 From: python-checkins at python.org (skip.montanaro) Date: Tue, 18 Apr 2006 02:55:46 +0200 (CEST) Subject: [Python-checkins] r45508 - python/trunk/Python/sysmodule.c Message-ID: <20060418005546.C7A3D1E401C@bag.python.org> Author: skip.montanaro Date: Tue Apr 18 02:55:46 2006 New Revision: 45508 Modified: python/trunk/Python/sysmodule.c Log: C++ compiler cleanup: extern "C" a couple declarations, cast int to size_t Modified: python/trunk/Python/sysmodule.c ============================================================================== --- python/trunk/Python/sysmodule.c (original) +++ python/trunk/Python/sysmodule.c Tue Apr 18 02:55:46 2006 @@ -700,6 +700,10 @@ 10. Number of stack pops performed by call_function()" ); +#ifdef __cplusplus +extern "C" { +#endif + #ifdef Py_TRACE_REFS /* Defined in objects.c because it uses static globals if that file */ extern PyObject *_Py_GetObjects(PyObject *, PyObject *); @@ -710,6 +714,10 @@ extern PyObject *_Py_GetDXProfile(PyObject *, PyObject *); #endif +#ifdef __cplusplus +} +#endif + static PyMethodDef sys_methods[] = { /* Might as well keep this in alphabetic order */ {"callstats", (PyCFunction)PyEval_GetCallStats, METH_NOARGS, @@ -1414,7 +1422,7 @@ PyErr_Clear(); fputs(buffer, fp); } - if (written < 0 || written >= sizeof(buffer)) { + if (written < 0 || (size_t)written >= sizeof(buffer)) { const char *truncated = "... truncated"; if (PyFile_WriteString(truncated, file) != 0) { PyErr_Clear(); From python-checkins at python.org Tue Apr 18 02:57:15 2006 From: python-checkins at python.org (skip.montanaro) Date: Tue, 18 Apr 2006 02:57:15 +0200 (CEST) Subject: [Python-checkins] r45509 - python/trunk/Python/getargs.c Message-ID: <20060418005715.A25A81E400A@bag.python.org> Author: skip.montanaro Date: Tue Apr 18 02:57:15 2006 New Revision: 45509 Modified: python/trunk/Python/getargs.c Log: C++ compiler cleanup: cast... Modified: python/trunk/Python/getargs.c ============================================================================== --- python/trunk/Python/getargs.c (original) +++ python/trunk/Python/getargs.c Tue Apr 18 02:57:15 2006 @@ -645,8 +645,8 @@ unsigned int ival; if (float_argument_error(arg)) return converterr("integer", arg, msgbuf, bufsize); - ival = PyInt_AsUnsignedLongMask(arg); - if (ival == -1 && PyErr_Occurred()) + ival = (unsigned int)PyInt_AsUnsignedLongMask(arg); + if (ival == (unsigned int)-1 && PyErr_Occurred()) return converterr("integer", arg, msgbuf, bufsize); else *p = ival; From python-checkins at python.org Tue Apr 18 02:59:56 2006 From: python-checkins at python.org (phillip.eby) Date: Tue, 18 Apr 2006 02:59:56 +0200 (CEST) Subject: [Python-checkins] r45510 - python/trunk/Lib/pkgutil.py python/trunk/Lib/pydoc.py Message-ID: <20060418005956.156301E400A@bag.python.org> Author: phillip.eby Date: Tue Apr 18 02:59:55 2006 New Revision: 45510 Modified: python/trunk/Lib/pkgutil.py python/trunk/Lib/pydoc.py Log: Second phase of refactoring for runpy, pkgutil, pydoc, and setuptools to share common PEP 302 support code, as described here: http://mail.python.org/pipermail/python-dev/2006-April/063724.html pydoc now supports PEP 302 importers, by way of utility functions in pkgutil, such as 'walk_packages()'. It will properly document modules that are in zip files, and is backward compatible to Python 2.3 (setuptools installs for Python <2.5 will bundle it so pydoc doesn't break when used with eggs.) What has not changed is that pydoc command line options do not support zip paths or other importer paths, and the webserver index does not support sys.meta_path. Those are probably okay as limitations. Tasks remaining: write docs and Misc/NEWS for pkgutil/pydoc changes, and update setuptools to use pkgutil wherever possible, then add it to the stdlib. Modified: python/trunk/Lib/pkgutil.py ============================================================================== --- python/trunk/Lib/pkgutil.py (original) +++ python/trunk/Lib/pkgutil.py Tue Apr 18 02:59:55 2006 @@ -11,6 +11,7 @@ __all__ = [ 'get_importer', 'iter_importers', 'get_loader', 'find_loader', + 'walk_packages', 'iter_modules', 'ImpImporter', 'ImpLoader', 'read_code', 'extend_path', ] @@ -27,6 +28,95 @@ return marshal.load(stream) +def simplegeneric(func): + """Make a trivial single-dispatch generic function""" + registry = {} + def wrapper(*args,**kw): + ob = args[0] + try: + cls = ob.__class__ + except AttributeError: + cls = type(ob) + try: + mro = cls.__mro__ + except AttributeError: + try: + class cls(cls,object): pass + mro = cls.__mro__[1:] + except TypeError: + mro = object, # must be an ExtensionClass or some such :( + for t in mro: + if t in registry: + return registry[t](*args,**kw) + else: + return func(*args,**kw) + try: + wrapper.__name__ = func.__name__ + except (TypeError,AttributeError): + pass # Python 2.3 doesn't allow functions to be renamed + + def register(typ, func=None): + if func is None: + return lambda f: register(typ, f) + registry[typ] = func + return func + + wrapper.__dict__ = func.__dict__ + wrapper.__doc__ = func.__doc__ + wrapper.register = register + return wrapper + + +def walk_packages(path=None, prefix='', onerror=None): + """Yield submodule names+loaders recursively, for path or sys.path""" + + def seen(p,m={}): + if p in m: return True + m[p] = True + + for importer, name, ispkg in iter_modules(path, prefix): + yield importer, name, ispkg + + if ispkg: + try: + __import__(name) + except ImportError: + if onerror is not None: + onerror() + else: + path = getattr(sys.modules[name], '__path__', None) or [] + + # don't traverse path items we've seen before + path = [p for p in path if not seen(p)] + + for item in walk_packages(path, name+'.'): + yield item + + +def iter_modules(path=None, prefix=''): + """Yield submodule names+loaders for path or sys.path""" + if path is None: + importers = iter_importers() + else: + importers = map(get_importer, path) + + yielded = {} + for i in importers: + for name, ispkg in iter_importer_modules(i, prefix): + if name not in yielded: + yielded[name] = 1 + yield i, name, ispkg + + +#@simplegeneric +def iter_importer_modules(importer, prefix=''): + if not hasattr(importer,'iter_modules'): + return [] + return importer.iter_modules(prefix) + +iter_importer_modules = simplegeneric(iter_importer_modules) + + class ImpImporter: """PEP 302 Importer that wraps Python's "classic" import algorithm @@ -49,13 +139,45 @@ if self.path is None: path = None else: - path = [self.path] + path = [os.path.realpath(self.path)] try: file, filename, etc = imp.find_module(subname, path) except ImportError: return None return ImpLoader(fullname, file, filename, etc) + def iter_modules(self, prefix=''): + if self.path is None or not os.path.isdir(self.path): + return + + yielded = {} + import inspect + + filenames = os.listdir(self.path) + filenames.sort() # handle packages before same-named modules + + for fn in filenames: + modname = inspect.getmodulename(fn) + if modname=='__init__' or modname in yielded: + continue + + path = os.path.join(self.path, fn) + ispkg = False + + if not modname and os.path.isdir(path) and '.' not in fn: + modname = fn + for fn in os.listdir(path): + subname = inspect.getmodulename(fn) + if subname=='__init__': + ispkg = True + break + else: + continue # not a package + + if modname and '.' not in modname: + yielded[modname] = 1 + yield prefix + modname, ispkg + class ImpLoader: """PEP 302 Loader that wraps Python's "classic" import algorithm @@ -97,7 +219,8 @@ "module %s" % (self.fullname, fullname)) return fullname - def is_package(self): + def is_package(self, fullname): + fullname = self._fix_name(fullname) return self.etc[2]==imp.PKG_DIRECTORY def get_code(self, fullname=None): @@ -136,6 +259,7 @@ self.source = self._get_delegate().get_source() return self.source + def _get_delegate(self): return ImpImporter(self.filename).find_module('__init__') @@ -149,6 +273,45 @@ return None +try: + import zipimport + from zipimport import zipimporter + + def iter_zipimport_modules(importer, prefix=''): + dirlist = zipimport._zip_directory_cache[importer.archive].keys() + dirlist.sort() + _prefix = importer.prefix + plen = len(_prefix) + yielded = {} + import inspect + for fn in dirlist: + if not fn.startswith(_prefix): + continue + + fn = fn[plen:].split(os.sep) + + if len(fn)==2 and fn[1].startswith('__init__.py'): + if fn[0] not in yielded: + yielded[fn[0]] = 1 + yield fn[0], True + + if len(fn)!=1: + continue + + modname = inspect.getmodulename(fn[0]) + if modname=='__init__': + continue + + if modname and '.' not in modname and modname not in yielded: + yielded[modname] = 1 + yield prefix + modname, False + + iter_importer_modules.register(zipimporter, iter_zipimport_modules) + +except ImportError: + pass + + def get_importer(path_item): """Retrieve a PEP 302 importer for the given path item @@ -183,7 +346,7 @@ return importer -def iter_importers(fullname): +def iter_importers(fullname=""): """Yield PEP 302 importers for the given module name If fullname contains a '.', the importers will be for the package @@ -224,7 +387,6 @@ if '.' not in fullname: yield ImpImporter() - def get_loader(module_or_name): """Get a PEP 302 "loader" object for module_or_name @@ -250,7 +412,6 @@ fullname = module_or_name return find_loader(fullname) - def find_loader(fullname): """Find a PEP 302 "loader" object for fullname Modified: python/trunk/Lib/pydoc.py ============================================================================== --- python/trunk/Lib/pydoc.py (original) +++ python/trunk/Lib/pydoc.py Tue Apr 18 02:59:55 2006 @@ -52,10 +52,16 @@ # the current directory is changed with os.chdir(), an incorrect # path will be displayed. -import sys, imp, os, re, types, inspect, __builtin__ +import sys, imp, os, re, types, inspect, __builtin__, pkgutil from repr import Repr from string import expandtabs, find, join, lower, split, strip, rfind, rstrip -from collections import deque +try: + from collections import deque +except ImportError: + # Python 2.3 compatibility + class deque(list): + def popleft(self): + return self.pop(0) # --------------------------------------------------------- common routines @@ -182,6 +188,23 @@ return True return False +def source_synopsis(file): + line = file.readline() + while line[:1] == '#' or not strip(line): + line = file.readline() + if not line: break + line = strip(line) + if line[:4] == 'r"""': line = line[1:] + if line[:3] == '"""': + line = line[3:] + if line[-1:] == '\\': line = line[:-1] + while not strip(line): + line = file.readline() + if not line: break + result = strip(split(line, '"""')[0]) + else: result = None + return result + def synopsis(filename, cache={}): """Get the one-line summary out of a module file.""" mtime = os.stat(filename).st_mtime @@ -196,24 +219,11 @@ if info and 'b' in info[2]: # binary modules have to be imported try: module = imp.load_module('__temp__', file, filename, info[1:]) except: return None - result = split(module.__doc__ or '', '\n')[0] + result = (module.__doc__ or '').splitlines()[0] del sys.modules['__temp__'] else: # text modules can be directly examined - line = file.readline() - while line[:1] == '#' or not strip(line): - line = file.readline() - if not line: break - line = strip(line) - if line[:4] == 'r"""': line = line[1:] - if line[:3] == '"""': - line = line[3:] - if line[-1:] == '\\': line = line[:-1] - while not strip(line): - line = file.readline() - if not line: break - result = strip(split(line, '"""')[0]) - else: result = None - file.close() + result = source_synopsis(file) + file.close() cache[filename] = (mtime, result) return result @@ -643,16 +653,8 @@ if hasattr(object, '__path__'): modpkgs = [] - modnames = [] - for file in os.listdir(object.__path__[0]): - path = os.path.join(object.__path__[0], file) - modname = inspect.getmodulename(file) - if modname != '__init__': - if modname and modname not in modnames: - modpkgs.append((modname, name, 0, 0)) - modnames.append(modname) - elif ispackage(path): - modpkgs.append((file, name, 1, 0)) + for importer, modname, ispkg in pkgutil.iter_modules(object.__path__): + modpkgs.append((modname, name, ispkg, 0)) modpkgs.sort() contents = self.multicolumn(modpkgs, self.modpkglink) result = result + self.bigsection( @@ -796,7 +798,10 @@ tag += ':
\n' # Sort attrs by name. - attrs.sort(key=lambda t: t[0]) + try: + attrs.sort(key=lambda t: t[0]) + except TypeError: + attrs.sort(lambda t1, t2: cmp(t1[0], t2[0])) # 2.3 compat # Pump out the attrs, segregated by kind. attrs = spill('Methods %s' % tag, attrs, @@ -914,25 +919,9 @@ """Generate an HTML index for a directory of modules.""" modpkgs = [] if shadowed is None: shadowed = {} - seen = {} - files = os.listdir(dir) - - def found(name, ispackage, - modpkgs=modpkgs, shadowed=shadowed, seen=seen): - if name not in seen: - modpkgs.append((name, '', ispackage, name in shadowed)) - seen[name] = 1 - shadowed[name] = 1 - - # Package spam/__init__.py takes precedence over module spam.py. - for file in files: - path = os.path.join(dir, file) - if ispackage(path): found(file, 1) - for file in files: - path = os.path.join(dir, file) - if os.path.isfile(path): - modname = inspect.getmodulename(file) - if modname: found(modname, 0) + for importer, name, ispkg in pkgutil.iter_modules([dir]): + modpkgs.append((name, '', ispkg, name in shadowed)) + shadowed[name] = 1 modpkgs.sort() contents = self.multicolumn(modpkgs, self.modpkglink) @@ -1059,14 +1048,12 @@ if hasattr(object, '__path__'): modpkgs = [] - for file in os.listdir(object.__path__[0]): - path = os.path.join(object.__path__[0], file) - modname = inspect.getmodulename(file) - if modname != '__init__': - if modname and modname not in modpkgs: - modpkgs.append(modname) - elif ispackage(path): - modpkgs.append(file + ' (package)') + for importer, modname, ispkg in pkgutil.iter_modules(object.__path__): + if ispkg: + modpkgs.append(modname + ' (package)') + else: + modpkgs.append(modname) + modpkgs.sort() result = result + self.section( 'PACKAGE CONTENTS', join(modpkgs, '\n')) @@ -1490,20 +1477,9 @@ def writedocs(dir, pkgpath='', done=None): """Write out HTML documentation for all modules in a directory tree.""" if done is None: done = {} - for file in os.listdir(dir): - path = os.path.join(dir, file) - if ispackage(path): - writedocs(path, pkgpath + file + '.', done) - elif os.path.isfile(path): - modname = inspect.getmodulename(path) - if modname: - if modname == '__init__': - modname = pkgpath[:-1] # remove trailing period - else: - modname = pkgpath + modname - if modname not in done: - done[modname] = 1 - writedoc(modname) + for importer, modname, ispkg in pkgutil.walk_packages([dir], pkgpath): + writedoc(modname) + return class Helper: keywords = { @@ -1830,30 +1806,9 @@ self.state.append((child, self.children(child))) return child -class ModuleScanner(Scanner): + +class ModuleScanner: """An interruptible scanner that searches module synopses.""" - def __init__(self): - roots = map(lambda dir: (dir, ''), pathdirs()) - Scanner.__init__(self, roots, self.submodules, self.isnewpackage) - self.inodes = map(lambda (dir, pkg): os.stat(dir).st_ino, roots) - - def submodules(self, (dir, package)): - children = [] - for file in os.listdir(dir): - path = os.path.join(dir, file) - if ispackage(path): - children.append((path, package + (package and '.') + file)) - else: - children.append((path, package)) - children.sort() # so that spam.py comes before spam.pyc or spam.pyo - return children - - def isnewpackage(self, (dir, package)): - inode = os.path.exists(dir) and os.stat(dir).st_ino - if not (os.path.islink(dir) and inode in self.inodes): - self.inodes.append(inode) # detect circular symbolic links - return ispackage(dir) - return False def run(self, callback, key=None, completer=None): if key: key = lower(key) @@ -1870,22 +1825,31 @@ if find(lower(modname + ' - ' + desc), key) >= 0: callback(None, modname, desc) - while not self.quit: - node = self.next() - if not node: break - path, package = node - modname = inspect.getmodulename(path) - if os.path.isfile(path) and modname: - modname = package + (package and '.') + modname - if not modname in seen: - seen[modname] = 1 # if we see spam.py, skip spam.pyc - if key is None: - callback(path, modname, '') + for importer, modname, ispkg in pkgutil.walk_packages(): + if self.quit: + break + if key is None: + callback(None, modname, '') + else: + loader = importer.find_module(modname) + if hasattr(loader,'get_source'): + import StringIO + desc = source_synopsis( + StringIO.StringIO(loader.get_source(modname)) + ) or '' + if hasattr(loader,'get_filename'): + path = loader.get_filename(modname) else: - desc = synopsis(path) or '' - if find(lower(modname + ' - ' + desc), key) >= 0: - callback(path, modname, desc) - if completer: completer() + path = None + else: + module = loader.load_module(modname) + desc = (module.__doc__ or '').splitlines()[0] + path = getattr(module,'__file__',None) + if find(lower(modname + ' - ' + desc), key) >= 0: + callback(path, modname, desc) + + if completer: + completer() def apropos(key): """Print all the one-line module summaries that contain a substring.""" @@ -1950,7 +1914,7 @@ 'Built-in Modules', '#ffffff', '#ee77aa', contents)] seen = {} - for dir in pathdirs(): + for dir in sys.path: indices.append(html.index(dir, seen)) contents = heading + join(indices) + '''

From python-checkins at python.org Tue Apr 18 03:01:41 2006 From: python-checkins at python.org (skip.montanaro) Date: Tue, 18 Apr 2006 03:01:41 +0200 (CEST) Subject: [Python-checkins] r45511 - python/trunk/Modules/_testcapimodule.c Message-ID: <20060418010141.B7A571E400A@bag.python.org> Author: skip.montanaro Date: Tue Apr 18 03:01:41 2006 New Revision: 45511 Modified: python/trunk/Modules/_testcapimodule.c Log: correct function signature Modified: python/trunk/Modules/_testcapimodule.c ============================================================================== --- python/trunk/Modules/_testcapimodule.c (original) +++ python/trunk/Modules/_testcapimodule.c Tue Apr 18 03:01:41 2006 @@ -234,7 +234,7 @@ #include "testcapi_long.h" static PyObject * -test_longlong_api(PyObject* self) +test_longlong_api(PyObject* self, PyObject *args) { return TESTNAME(raise_test_longlong_error); } From buildbot at python.org Tue Apr 18 03:23:11 2006 From: buildbot at python.org (buildbot at python.org) Date: Tue, 18 Apr 2006 01:23:11 +0000 Subject: [Python-checkins] buildbot warnings in amd64 gentoo trunk Message-ID: <20060418012311.B1E751E4003@bag.python.org> The Buildbot has detected a new failure of amd64 gentoo trunk. Full details are available at: http://www.python.org/dev/buildbot/all/amd64%2520gentoo%2520trunk/builds/546 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: phillip.eby,skip.montanaro Build Had Warnings: warnings test sincerely, -The Buildbot From buildbot at python.org Tue Apr 18 03:24:51 2006 From: buildbot at python.org (buildbot at python.org) Date: Tue, 18 Apr 2006 01:24:51 +0000 Subject: [Python-checkins] buildbot warnings in x86 gentoo trunk Message-ID: <20060418012452.1357B1E4003@bag.python.org> The Buildbot has detected a new failure of x86 gentoo trunk. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520gentoo%2520trunk/builds/526 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: phillip.eby,skip.montanaro Build Had Warnings: warnings test sincerely, -The Buildbot From buildbot at python.org Tue Apr 18 03:36:42 2006 From: buildbot at python.org (buildbot at python.org) Date: Tue, 18 Apr 2006 01:36:42 +0000 Subject: [Python-checkins] buildbot warnings in x86 XP-2 trunk Message-ID: <20060418013642.5187B1E4003@bag.python.org> The Buildbot has detected a new failure of x86 XP-2 trunk. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520XP-2%2520trunk/builds/315 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: phillip.eby,skip.montanaro Build Had Warnings: warnings test sincerely, -The Buildbot From buildbot at python.org Tue Apr 18 03:38:34 2006 From: buildbot at python.org (buildbot at python.org) Date: Tue, 18 Apr 2006 01:38:34 +0000 Subject: [Python-checkins] buildbot warnings in x86 W2k trunk Message-ID: <20060418013834.8BF911E4003@bag.python.org> The Buildbot has detected a new failure of x86 W2k trunk. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520W2k%2520trunk/builds/522 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: phillip.eby,skip.montanaro Build Had Warnings: warnings test sincerely, -The Buildbot From python-checkins at python.org Tue Apr 18 03:39:25 2006 From: python-checkins at python.org (phillip.eby) Date: Tue, 18 Apr 2006 03:39:25 +0200 (CEST) Subject: [Python-checkins] r45512 - python/trunk/Lib/test/test_pyclbr.py Message-ID: <20060418013925.9D3131E4003@bag.python.org> Author: phillip.eby Date: Tue Apr 18 03:39:25 2006 New Revision: 45512 Modified: python/trunk/Lib/test/test_pyclbr.py Log: test_pyclbr goes nuts when a module contains code to try importing a class and provide a substitute if the import fails, because pyclbr sees the class definition. Changed to ignore such cases' base classes and methods, since they will not match. Modified: python/trunk/Lib/test/test_pyclbr.py ============================================================================== --- python/trunk/Lib/test/test_pyclbr.py (original) +++ python/trunk/Lib/test/test_pyclbr.py Tue Apr 18 03:39:25 2006 @@ -95,6 +95,9 @@ self.assert_(isinstance(py_item, (FunctionType, BuiltinFunctionType))) else: self.failUnless(isinstance(py_item, (ClassType, type))) + if py_item.__module__!=moduleName: + continue # skip classes that came from somewhere else + real_bases = [base.__name__ for base in py_item.__bases__] pyclbr_bases = [ getattr(base, 'name', base) for base in value.super ] From buildbot at python.org Tue Apr 18 03:40:06 2006 From: buildbot at python.org (buildbot at python.org) Date: Tue, 18 Apr 2006 01:40:06 +0000 Subject: [Python-checkins] buildbot warnings in sparc solaris10 gcc trunk Message-ID: <20060418014006.A46E51E4003@bag.python.org> The Buildbot has detected a new failure of sparc solaris10 gcc trunk. Full details are available at: http://www.python.org/dev/buildbot/all/sparc%2520solaris10%2520gcc%2520trunk/builds/469 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: phillip.eby,skip.montanaro Build Had Warnings: warnings test sincerely, -The Buildbot From buildbot at python.org Tue Apr 18 03:42:18 2006 From: buildbot at python.org (buildbot at python.org) Date: Tue, 18 Apr 2006 01:42:18 +0000 Subject: [Python-checkins] buildbot warnings in g4 osx.4 trunk Message-ID: <20060418014218.7C18D1E4003@bag.python.org> The Buildbot has detected a new failure of g4 osx.4 trunk. Full details are available at: http://www.python.org/dev/buildbot/all/g4%2520osx.4%2520trunk/builds/462 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: phillip.eby,skip.montanaro Build Had Warnings: warnings test sincerely, -The Buildbot From buildbot at python.org Tue Apr 18 03:46:28 2006 From: buildbot at python.org (buildbot at python.org) Date: Tue, 18 Apr 2006 01:46:28 +0000 Subject: [Python-checkins] buildbot warnings in ppc Debian unstable trunk Message-ID: <20060418014628.55C341E4003@bag.python.org> The Buildbot has detected a new failure of ppc Debian unstable trunk. Full details are available at: http://www.python.org/dev/buildbot/all/ppc%2520Debian%2520unstable%2520trunk/builds/195 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: phillip.eby,skip.montanaro Build Had Warnings: warnings test sincerely, -The Buildbot From buildbot at python.org Tue Apr 18 04:05:41 2006 From: buildbot at python.org (buildbot at python.org) Date: Tue, 18 Apr 2006 02:05:41 +0000 Subject: [Python-checkins] buildbot warnings in x86 XP trunk Message-ID: <20060418020541.464731E4003@bag.python.org> The Buildbot has detected a new failure of x86 XP trunk. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520XP%2520trunk/builds/484 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: phillip.eby,skip.montanaro Build Had Warnings: warnings test sincerely, -The Buildbot From buildbot at python.org Tue Apr 18 04:25:43 2006 From: buildbot at python.org (buildbot at python.org) Date: Tue, 18 Apr 2006 02:25:43 +0000 Subject: [Python-checkins] buildbot warnings in x86 Ubuntu dapper (icc) trunk Message-ID: <20060418022544.138871E400E@bag.python.org> The Buildbot has detected a new failure of x86 Ubuntu dapper (icc) trunk. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520Ubuntu%2520dapper%2520%2528icc%2529%2520trunk/builds/179 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: phillip.eby,skip.montanaro Build Had Warnings: warnings test sincerely, -The Buildbot From python-checkins at python.org Tue Apr 18 05:02:12 2006 From: python-checkins at python.org (tim.peters) Date: Tue, 18 Apr 2006 05:02:12 +0200 (CEST) Subject: [Python-checkins] r45513 - python/trunk/Misc/build.sh Message-ID: <20060418030212.312051E403D@bag.python.org> Author: tim.peters Date: Tue Apr 18 05:02:10 2006 New Revision: 45513 Modified: python/trunk/Misc/build.sh Log: At least test_threading_local doesn't leak any more. Modified: python/trunk/Misc/build.sh ============================================================================== --- python/trunk/Misc/build.sh (original) +++ python/trunk/Misc/build.sh Tue Apr 18 05:02:10 2006 @@ -60,7 +60,7 @@ # Note: test_XXX (none currently) really leak, but are disabled # so we don't send spam. Any test which really leaks should only # be listed here if there are also test cases under Lib/test/leakers. -LEAKY_TESTS="test_(ctypes|filecmp|socket|threadedtempfile|threading|threading_local|urllib2)" +LEAKY_TESTS="test_(ctypes|filecmp|socket|threadedtempfile|threading|urllib2)" # Skip these tests altogether when looking for leaks. These tests # do not need to be stored above in LEAKY_TESTS too. From python-checkins at python.org Tue Apr 18 05:03:19 2006 From: python-checkins at python.org (phillip.eby) Date: Tue, 18 Apr 2006 05:03:19 +0200 (CEST) Subject: [Python-checkins] r45514 - in sandbox/trunk/setuptools: _pkgutil.py doctest.py pkg_resources.py pydoc.py setup.py setuptools/tests/doctest.py Message-ID: <20060418030319.EA5431E4003@bag.python.org> Author: phillip.eby Date: Tue Apr 18 05:03:16 2006 New Revision: 45514 Added: sandbox/trunk/setuptools/_pkgutil.py - copied unchanged from r45512, python/trunk/Lib/pkgutil.py sandbox/trunk/setuptools/doctest.py - copied, changed from r45512, python/trunk/Lib/doctest.py sandbox/trunk/setuptools/pydoc.py - copied, changed from r45512, python/trunk/Lib/pydoc.py Removed: sandbox/trunk/setuptools/setuptools/tests/doctest.py Modified: sandbox/trunk/setuptools/pkg_resources.py sandbox/trunk/setuptools/setup.py Log: Backport pkgutil, pydoc, and doctest from the 2.5 trunk to setuptools 0.7 trunk. (Sideport?) Setuptools 0.7 will install these in place of the 2.3/2.4 versions (at least of pydoc and doctest) to let them work properly with eggs. pkg_resources now depends on the 2.5 pkgutil, which is included here as _pkgutil, to work around the fact that some system packagers will install setuptools without overriding the stdlib modules. But users who install their own setuptools will get them, and the system packaged people probably don't need them. Copied: sandbox/trunk/setuptools/doctest.py (from r45512, python/trunk/Lib/doctest.py) ============================================================================== --- python/trunk/Lib/doctest.py (original) +++ sandbox/trunk/setuptools/doctest.py Tue Apr 18 05:03:16 2006 @@ -1328,13 +1328,13 @@ __LINECACHE_FILENAME_RE = re.compile(r'[\w\.]+)' r'\[(?P\d+)\]>$') - def __patched_linecache_getlines(self, filename, module_globals=None): + def __patched_linecache_getlines(self, filename): m = self.__LINECACHE_FILENAME_RE.match(filename) if m and m.group('name') == self.test.name: example = self.test.examples[int(m.group('examplenum'))] return example.source.splitlines(True) else: - return self.save_linecache_getlines(filename, module_globals) + return self.save_linecache_getlines(filename) def run(self, test, compileflags=None, out=None, clear_globs=True): """ Modified: sandbox/trunk/setuptools/pkg_resources.py ============================================================================== --- sandbox/trunk/setuptools/pkg_resources.py (original) +++ sandbox/trunk/setuptools/pkg_resources.py Tue Apr 18 05:03:16 2006 @@ -13,7 +13,7 @@ method. """ -import sys, os, zipimport, time, re, imp, new +import sys, os, zipimport, time, re, imp, new, pkgutil # XXX from sets import ImmutableSet from os import utime, rename, unlink # capture these to bypass sandboxing from os import open as os_open @@ -244,6 +244,15 @@ return get_distribution(dist).get_entry_info(group, name) +try: + from pkgutil import get_importer +except ImportError: + import _pkgutil as pkgutil + get_importer = pkgutil.get_importer +else: + import pkgutil + + class IMetadataProvider: def has_metadata(name): @@ -276,15 +285,6 @@ - - - - - - - - - class IResourceProvider(IMetadataProvider): """An object that provides access to package resources""" @@ -1411,7 +1411,6 @@ metadata = PathMetadata(egg_path, os.path.join(egg_path,'EGG-INFO')) dist = Distribution.from_filename(egg_path, metadata=metadata) """ - def __init__(self, path, egg_info): self.module_path = path self.egg_info = egg_info @@ -1433,87 +1432,6 @@ self._setup_prefix() -class ImpWrapper: - """PEP 302 Importer that wraps Python's "normal" import algorithm""" - - def __init__(self, path=None): - self.path = path - - def find_module(self, fullname, path=None): - subname = fullname.split(".")[-1] - if subname != fullname and self.path is None: - return None - if self.path is None: - path = None - else: - path = [self.path] - try: - file, filename, etc = imp.find_module(subname, path) - except ImportError: - return None - return ImpLoader(file, filename, etc) - - -class ImpLoader: - """PEP 302 Loader that wraps Python's "normal" import algorithm""" - - def __init__(self, file, filename, etc): - self.file = file - self.filename = filename - self.etc = etc - - def load_module(self, fullname): - try: - mod = imp.load_module(fullname, self.file, self.filename, self.etc) - finally: - if self.file: self.file.close() - # Note: we don't set __loader__ because we want the module to look - # normal; i.e. this is just a wrapper for standard import machinery - return mod - - - - -def get_importer(path_item): - """Retrieve a PEP 302 "importer" for the given path item - - If there is no importer, this returns a wrapper around the builtin import - machinery. The returned importer is only cached if it was created by a - path hook. - """ - try: - importer = sys.path_importer_cache[path_item] - except KeyError: - for hook in sys.path_hooks: - try: - importer = hook(path_item) - except ImportError: - pass - else: - break - else: - importer = None - - sys.path_importer_cache.setdefault(path_item,importer) - if importer is None: - try: - importer = ImpWrapper(path_item) - except ImportError: - pass - return importer - - - - - - - - - - - - - _distribution_finders = {} @@ -1595,7 +1513,7 @@ for item in find_distributions(line.rstrip()): yield item -register_finder(ImpWrapper,find_on_path) +register_finder(pkgutil.ImpImporter, find_on_path) _namespace_handlers = {} _namespace_packages = {} @@ -1691,8 +1609,8 @@ # Only return the path if it's not already there return subpath -register_namespace_handler(ImpWrapper,file_ns_handler) -register_namespace_handler(zipimport.zipimporter,file_ns_handler) +register_namespace_handler(pkgutil.ImpImporter, file_ns_handler) +register_namespace_handler(zipimport.zipimporter, file_ns_handler) def null_ns_handler(importer, path_item, packageName, module): Copied: sandbox/trunk/setuptools/pydoc.py (from r45512, python/trunk/Lib/pydoc.py) ============================================================================== --- python/trunk/Lib/pydoc.py (original) +++ sandbox/trunk/setuptools/pydoc.py Tue Apr 18 05:03:16 2006 @@ -53,6 +53,9 @@ # path will be displayed. import sys, imp, os, re, types, inspect, __builtin__, pkgutil +if not hasattr(pkgutil,'walk_packages'): + import _pkgutil as pkgutil + from repr import Repr from string import expandtabs, find, join, lower, split, strip, rfind, rstrip try: Modified: sandbox/trunk/setuptools/setup.py ============================================================================== --- sandbox/trunk/setuptools/setup.py (original) +++ sandbox/trunk/setuptools/setup.py Tue Apr 18 05:03:16 2006 @@ -18,12 +18,12 @@ from distutils.util import convert_path d = {}; execfile(convert_path('setuptools/command/__init__.py'), d) SETUP_COMMANDS = d['__all__'] - VERSION = "0.7a1" from setuptools import setup, find_packages import sys -scripts = [] - +modules = ['pkg_resources','easy_install'] +if sys.version<"2.5": + modules += ['_pkgutil', 'doctest', 'pydoc'] # fixups for older Pythons setup( name="setuptools", version=VERSION, @@ -39,7 +39,7 @@ packages = find_packages(), package_data = {'setuptools':['*.exe','site-patch.py']}, - py_modules = ['pkg_resources','easy_install'], + py_modules = modules, zip_safe = (sys.version>="2.5"), # <2.5 needs unzipped for -m to work entry_points = { "distutils.commands" : [ @@ -91,7 +91,7 @@ Topic :: System :: Archiving :: Packaging Topic :: System :: Systems Administration Topic :: Utilities""".splitlines() if f.strip()], - scripts = scripts, + scripts = [], # uncomment for testing # setup_requires = ['setuptools>=0.6a0'], Deleted: /sandbox/trunk/setuptools/setuptools/tests/doctest.py ============================================================================== --- /sandbox/trunk/setuptools/setuptools/tests/doctest.py Tue Apr 18 05:03:16 2006 +++ (empty file) @@ -1,2677 +0,0 @@ -# Module doctest. -# Released to the public domain 16-Jan-2001, by Tim Peters (tim at python.org). -# Major enhancements and refactoring by: -# Jim Fulton -# Edward Loper - -# Provided as-is; use at your own risk; no warranty; no promises; enjoy! - -try: - basestring -except NameError: - basestring = str,unicode - -try: - enumerate -except NameError: - def enumerate(seq): - return zip(range(len(seq)),seq) - -r"""Module doctest -- a framework for running examples in docstrings. - -In simplest use, end each module M to be tested with: - -def _test(): - import doctest - doctest.testmod() - -if __name__ == "__main__": - _test() - -Then running the module as a script will cause the examples in the -docstrings to get executed and verified: - -python M.py - -This won't display anything unless an example fails, in which case the -failing example(s) and the cause(s) of the failure(s) are printed to stdout -(why not stderr? because stderr is a lame hack <0.2 wink>), and the final -line of output is "Test failed.". - -Run it with the -v switch instead: - -python M.py -v - -and a detailed report of all examples tried is printed to stdout, along -with assorted summaries at the end. - -You can force verbose mode by passing "verbose=True" to testmod, or prohibit -it by passing "verbose=False". In either of those cases, sys.argv is not -examined by testmod. - -There are a variety of other ways to run doctests, including integration -with the unittest framework, and support for running non-Python text -files containing doctests. There are also many ways to override parts -of doctest's default behaviors. See the Library Reference Manual for -details. -""" - -__docformat__ = 'reStructuredText en' - -__all__ = [ - # 0, Option Flags - 'register_optionflag', - 'DONT_ACCEPT_TRUE_FOR_1', - 'DONT_ACCEPT_BLANKLINE', - 'NORMALIZE_WHITESPACE', - 'ELLIPSIS', - 'IGNORE_EXCEPTION_DETAIL', - 'COMPARISON_FLAGS', - 'REPORT_UDIFF', - 'REPORT_CDIFF', - 'REPORT_NDIFF', - 'REPORT_ONLY_FIRST_FAILURE', - 'REPORTING_FLAGS', - # 1. Utility Functions - 'is_private', - # 2. Example & DocTest - 'Example', - 'DocTest', - # 3. Doctest Parser - 'DocTestParser', - # 4. Doctest Finder - 'DocTestFinder', - # 5. Doctest Runner - 'DocTestRunner', - 'OutputChecker', - 'DocTestFailure', - 'UnexpectedException', - 'DebugRunner', - # 6. Test Functions - 'testmod', - 'testfile', - 'run_docstring_examples', - # 7. Tester - 'Tester', - # 8. Unittest Support - 'DocTestSuite', - 'DocFileSuite', - 'set_unittest_reportflags', - # 9. Debugging Support - 'script_from_examples', - 'testsource', - 'debug_src', - 'debug', -] - -import __future__ - -import sys, traceback, inspect, linecache, os, re, types -import unittest, difflib, pdb, tempfile -import warnings -from StringIO import StringIO - -# Don't whine about the deprecated is_private function in this -# module's tests. -warnings.filterwarnings("ignore", "is_private", DeprecationWarning, - __name__, 0) - -# There are 4 basic classes: -# - Example: a pair, plus an intra-docstring line number. -# - DocTest: a collection of examples, parsed from a docstring, plus -# info about where the docstring came from (name, filename, lineno). -# - DocTestFinder: extracts DocTests from a given object's docstring and -# its contained objects' docstrings. -# - DocTestRunner: runs DocTest cases, and accumulates statistics. -# -# So the basic picture is: -# -# list of: -# +------+ +---------+ +-------+ -# |object| --DocTestFinder-> | DocTest | --DocTestRunner-> |results| -# +------+ +---------+ +-------+ -# | Example | -# | ... | -# | Example | -# +---------+ - -# Option constants. - -OPTIONFLAGS_BY_NAME = {} -def register_optionflag(name): - flag = 1 << len(OPTIONFLAGS_BY_NAME) - OPTIONFLAGS_BY_NAME[name] = flag - return flag - -DONT_ACCEPT_TRUE_FOR_1 = register_optionflag('DONT_ACCEPT_TRUE_FOR_1') -DONT_ACCEPT_BLANKLINE = register_optionflag('DONT_ACCEPT_BLANKLINE') -NORMALIZE_WHITESPACE = register_optionflag('NORMALIZE_WHITESPACE') -ELLIPSIS = register_optionflag('ELLIPSIS') -IGNORE_EXCEPTION_DETAIL = register_optionflag('IGNORE_EXCEPTION_DETAIL') - -COMPARISON_FLAGS = (DONT_ACCEPT_TRUE_FOR_1 | - DONT_ACCEPT_BLANKLINE | - NORMALIZE_WHITESPACE | - ELLIPSIS | - IGNORE_EXCEPTION_DETAIL) - -REPORT_UDIFF = register_optionflag('REPORT_UDIFF') -REPORT_CDIFF = register_optionflag('REPORT_CDIFF') -REPORT_NDIFF = register_optionflag('REPORT_NDIFF') -REPORT_ONLY_FIRST_FAILURE = register_optionflag('REPORT_ONLY_FIRST_FAILURE') - -REPORTING_FLAGS = (REPORT_UDIFF | - REPORT_CDIFF | - REPORT_NDIFF | - REPORT_ONLY_FIRST_FAILURE) - -# Special string markers for use in `want` strings: -BLANKLINE_MARKER = '' -ELLIPSIS_MARKER = '...' - -###################################################################### -## Table of Contents -###################################################################### -# 1. Utility Functions -# 2. Example & DocTest -- store test cases -# 3. DocTest Parser -- extracts examples from strings -# 4. DocTest Finder -- extracts test cases from objects -# 5. DocTest Runner -- runs test cases -# 6. Test Functions -- convenient wrappers for testing -# 7. Tester Class -- for backwards compatibility -# 8. Unittest Support -# 9. Debugging Support -# 10. Example Usage - -###################################################################### -## 1. Utility Functions -###################################################################### - -def is_private(prefix, base): - """prefix, base -> true iff name prefix + "." + base is "private". - - Prefix may be an empty string, and base does not contain a period. - Prefix is ignored (although functions you write conforming to this - protocol may make use of it). - Return true iff base begins with an (at least one) underscore, but - does not both begin and end with (at least) two underscores. - - >>> is_private("a.b", "my_func") - False - >>> is_private("____", "_my_func") - True - >>> is_private("someclass", "__init__") - False - >>> is_private("sometypo", "__init_") - True - >>> is_private("x.y.z", "_") - True - >>> is_private("_x.y.z", "__") - False - >>> is_private("", "") # senseless but consistent - False - """ - warnings.warn("is_private is deprecated; it wasn't useful; " - "examine DocTestFinder.find() lists instead", - DeprecationWarning, stacklevel=2) - return base[:1] == "_" and not base[:2] == "__" == base[-2:] - -def _extract_future_flags(globs): - """ - Return the compiler-flags associated with the future features that - have been imported into the given namespace (globs). - """ - flags = 0 - for fname in __future__.all_feature_names: - feature = globs.get(fname, None) - if feature is getattr(__future__, fname): - flags |= feature.compiler_flag - return flags - -def _normalize_module(module, depth=2): - """ - Return the module specified by `module`. In particular: - - If `module` is a module, then return module. - - If `module` is a string, then import and return the - module with that name. - - If `module` is None, then return the calling module. - The calling module is assumed to be the module of - the stack frame at the given depth in the call stack. - """ - if inspect.ismodule(module): - return module - elif isinstance(module, (str, unicode)): - return __import__(module, globals(), locals(), ["*"]) - elif module is None: - return sys.modules[sys._getframe(depth).f_globals['__name__']] - else: - raise TypeError("Expected a module, string, or None") - -def _indent(s, indent=4): - """ - Add the given number of space characters to the beginning every - non-blank line in `s`, and return the result. - """ - # This regexp matches the start of non-blank lines: - return re.sub('(?m)^(?!$)', indent*' ', s) - -def _exception_traceback(exc_info): - """ - Return a string containing a traceback message for the given - exc_info tuple (as returned by sys.exc_info()). - """ - # Get a traceback message. - excout = StringIO() - exc_type, exc_val, exc_tb = exc_info - traceback.print_exception(exc_type, exc_val, exc_tb, file=excout) - return excout.getvalue() - -# Override some StringIO methods. -class _SpoofOut(StringIO): - def getvalue(self): - result = StringIO.getvalue(self) - # If anything at all was written, make sure there's a trailing - # newline. There's no way for the expected output to indicate - # that a trailing newline is missing. - if result and not result.endswith("\n"): - result += "\n" - # Prevent softspace from screwing up the next test case, in - # case they used print with a trailing comma in an example. - if hasattr(self, "softspace"): - del self.softspace - return result - - def truncate(self, size=None): - StringIO.truncate(self, size) - if hasattr(self, "softspace"): - del self.softspace - -# Worst-case linear-time ellipsis matching. -def _ellipsis_match(want, got): - """ - Essentially the only subtle case: - >>> _ellipsis_match('aa...aa', 'aaa') - False - """ - if want.find(ELLIPSIS_MARKER)==-1: - return want == got - - # Find "the real" strings. - ws = want.split(ELLIPSIS_MARKER) - assert len(ws) >= 2 - - # Deal with exact matches possibly needed at one or both ends. - startpos, endpos = 0, len(got) - w = ws[0] - if w: # starts with exact match - if got.startswith(w): - startpos = len(w) - del ws[0] - else: - return False - w = ws[-1] - if w: # ends with exact match - if got.endswith(w): - endpos -= len(w) - del ws[-1] - else: - return False - - if startpos > endpos: - # Exact end matches required more characters than we have, as in - # _ellipsis_match('aa...aa', 'aaa') - return False - - # For the rest, we only need to find the leftmost non-overlapping - # match for each piece. If there's no overall match that way alone, - # there's no overall match period. - for w in ws: - # w may be '' at times, if there are consecutive ellipses, or - # due to an ellipsis at the start or end of `want`. That's OK. - # Search for an empty string succeeds, and doesn't change startpos. - startpos = got.find(w, startpos, endpos) - if startpos < 0: - return False - startpos += len(w) - - return True - -def _comment_line(line): - "Return a commented form of the given line" - line = line.rstrip() - if line: - return '# '+line - else: - return '#' - -class _OutputRedirectingPdb(pdb.Pdb): - """ - A specialized version of the python debugger that redirects stdout - to a given stream when interacting with the user. Stdout is *not* - redirected when traced code is executed. - """ - def __init__(self, out): - self.__out = out - pdb.Pdb.__init__(self) - - def trace_dispatch(self, *args): - # Redirect stdout to the given stream. - save_stdout = sys.stdout - sys.stdout = self.__out - # Call Pdb's trace dispatch method. - try: - return pdb.Pdb.trace_dispatch(self, *args) - finally: - sys.stdout = save_stdout - -# [XX] Normalize with respect to os.path.pardir? -def _module_relative_path(module, path): - if not inspect.ismodule(module): - raise TypeError, 'Expected a module: %r' % module - if path.startswith('/'): - raise ValueError, 'Module-relative files may not have absolute paths' - - # Find the base directory for the path. - if hasattr(module, '__file__'): - # A normal module/package - basedir = os.path.split(module.__file__)[0] - elif module.__name__ == '__main__': - # An interactive session. - if len(sys.argv)>0 and sys.argv[0] != '': - basedir = os.path.split(sys.argv[0])[0] - else: - basedir = os.curdir - else: - # A module w/o __file__ (this includes builtins) - raise ValueError("Can't resolve paths relative to the module " + - module + " (it has no __file__)") - - # Combine the base directory and the path. - return os.path.join(basedir, *(path.split('/'))) - -###################################################################### -## 2. Example & DocTest -###################################################################### -## - An "example" is a pair, where "source" is a -## fragment of source code, and "want" is the expected output for -## "source." The Example class also includes information about -## where the example was extracted from. -## -## - A "doctest" is a collection of examples, typically extracted from -## a string (such as an object's docstring). The DocTest class also -## includes information about where the string was extracted from. - -class Example: - """ - A single doctest example, consisting of source code and expected - output. `Example` defines the following attributes: - - - source: A single Python statement, always ending with a newline. - The constructor adds a newline if needed. - - - want: The expected output from running the source code (either - from stdout, or a traceback in case of exception). `want` ends - with a newline unless it's empty, in which case it's an empty - string. The constructor adds a newline if needed. - - - exc_msg: The exception message generated by the example, if - the example is expected to generate an exception; or `None` if - it is not expected to generate an exception. This exception - message is compared against the return value of - `traceback.format_exception_only()`. `exc_msg` ends with a - newline unless it's `None`. The constructor adds a newline - if needed. - - - lineno: The line number within the DocTest string containing - this Example where the Example begins. This line number is - zero-based, with respect to the beginning of the DocTest. - - - indent: The example's indentation in the DocTest string. - I.e., the number of space characters that preceed the - example's first prompt. - - - options: A dictionary mapping from option flags to True or - False, which is used to override default options for this - example. Any option flags not contained in this dictionary - are left at their default value (as specified by the - DocTestRunner's optionflags). By default, no options are set. - """ - def __init__(self, source, want, exc_msg=None, lineno=0, indent=0, - options=None): - # Normalize inputs. - if not source.endswith('\n'): - source += '\n' - if want and not want.endswith('\n'): - want += '\n' - if exc_msg is not None and not exc_msg.endswith('\n'): - exc_msg += '\n' - # Store properties. - self.source = source - self.want = want - self.lineno = lineno - self.indent = indent - if options is None: options = {} - self.options = options - self.exc_msg = exc_msg - -class DocTest: - """ - A collection of doctest examples that should be run in a single - namespace. Each `DocTest` defines the following attributes: - - - examples: the list of examples. - - - globs: The namespace (aka globals) that the examples should - be run in. - - - name: A name identifying the DocTest (typically, the name of - the object whose docstring this DocTest was extracted from). - - - filename: The name of the file that this DocTest was extracted - from, or `None` if the filename is unknown. - - - lineno: The line number within filename where this DocTest - begins, or `None` if the line number is unavailable. This - line number is zero-based, with respect to the beginning of - the file. - - - docstring: The string that the examples were extracted from, - or `None` if the string is unavailable. - """ - def __init__(self, examples, globs, name, filename, lineno, docstring): - """ - Create a new DocTest containing the given examples. The - DocTest's globals are initialized with a copy of `globs`. - """ - assert not isinstance(examples, basestring), \ - "DocTest no longer accepts str; use DocTestParser instead" - self.examples = examples - self.docstring = docstring - self.globs = globs.copy() - self.name = name - self.filename = filename - self.lineno = lineno - - def __repr__(self): - if len(self.examples) == 0: - examples = 'no examples' - elif len(self.examples) == 1: - examples = '1 example' - else: - examples = '%d examples' % len(self.examples) - return ('' % - (self.name, self.filename, self.lineno, examples)) - - - # This lets us sort tests by name: - def __cmp__(self, other): - if not isinstance(other, DocTest): - return -1 - return cmp((self.name, self.filename, self.lineno, id(self)), - (other.name, other.filename, other.lineno, id(other))) - -###################################################################### -## 3. DocTestParser -###################################################################### - -class DocTestParser: - """ - A class used to parse strings containing doctest examples. - """ - # This regular expression is used to find doctest examples in a - # string. It defines three groups: `source` is the source code - # (including leading indentation and prompts); `indent` is the - # indentation of the first (PS1) line of the source code; and - # `want` is the expected output (including leading indentation). - _EXAMPLE_RE = re.compile(r''' - # Source consists of a PS1 line followed by zero or more PS2 lines. - (?P - (?:^(?P [ ]*) >>> .*) # PS1 line - (?:\n [ ]* \.\.\. .*)*) # PS2 lines - \n? - # Want consists of any non-blank lines that do not start with PS1. - (?P (?:(?![ ]*$) # Not a blank line - (?![ ]*>>>) # Not a line starting with PS1 - .*$\n? # But any other line - )*) - ''', re.MULTILINE | re.VERBOSE) - - # A regular expression for handling `want` strings that contain - # expected exceptions. It divides `want` into three pieces: - # - the traceback header line (`hdr`) - # - the traceback stack (`stack`) - # - the exception message (`msg`), as generated by - # traceback.format_exception_only() - # `msg` may have multiple lines. We assume/require that the - # exception message is the first non-indented line starting with a word - # character following the traceback header line. - _EXCEPTION_RE = re.compile(r""" - # Grab the traceback header. Different versions of Python have - # said different things on the first traceback line. - ^(?P Traceback\ \( - (?: most\ recent\ call\ last - | innermost\ last - ) \) : - ) - \s* $ # toss trailing whitespace on the header. - (?P .*?) # don't blink: absorb stuff until... - ^ (?P \w+ .*) # a line *starts* with alphanum. - """, re.VERBOSE | re.MULTILINE | re.DOTALL) - - # A callable returning a true value iff its argument is a blank line - # or contains a single comment. - _IS_BLANK_OR_COMMENT = re.compile(r'^[ ]*(#.*)?$').match - - def parse(self, string, name=''): - """ - Divide the given string into examples and intervening text, - and return them as a list of alternating Examples and strings. - Line numbers for the Examples are 0-based. The optional - argument `name` is a name identifying this string, and is only - used for error messages. - """ - string = string.expandtabs() - # If all lines begin with the same indentation, then strip it. - min_indent = self._min_indent(string) - if min_indent > 0: - string = '\n'.join([l[min_indent:] for l in string.split('\n')]) - - output = [] - charno, lineno = 0, 0 - # Find all doctest examples in the string: - for m in self._EXAMPLE_RE.finditer(string): - # Add the pre-example text to `output`. - output.append(string[charno:m.start()]) - # Update lineno (lines before this example) - lineno += string.count('\n', charno, m.start()) - # Extract info from the regexp match. - (source, options, want, exc_msg) = \ - self._parse_example(m, name, lineno) - # Create an Example, and add it to the list. - if not self._IS_BLANK_OR_COMMENT(source): - output.append( Example(source, want, exc_msg, - lineno=lineno, - indent=min_indent+len(m.group('indent')), - options=options) ) - # Update lineno (lines inside this example) - lineno += string.count('\n', m.start(), m.end()) - # Update charno. - charno = m.end() - # Add any remaining post-example text to `output`. - output.append(string[charno:]) - return output - - def get_doctest(self, string, globs, name, filename, lineno): - """ - Extract all doctest examples from the given string, and - collect them into a `DocTest` object. - - `globs`, `name`, `filename`, and `lineno` are attributes for - the new `DocTest` object. See the documentation for `DocTest` - for more information. - """ - return DocTest(self.get_examples(string, name), globs, - name, filename, lineno, string) - - def get_examples(self, string, name=''): - """ - Extract all doctest examples from the given string, and return - them as a list of `Example` objects. Line numbers are - 0-based, because it's most common in doctests that nothing - interesting appears on the same line as opening triple-quote, - and so the first interesting line is called \"line 1\" then. - - The optional argument `name` is a name identifying this - string, and is only used for error messages. - """ - return [x for x in self.parse(string, name) - if isinstance(x, Example)] - - def _parse_example(self, m, name, lineno): - """ - Given a regular expression match from `_EXAMPLE_RE` (`m`), - return a pair `(source, want)`, where `source` is the matched - example's source code (with prompts and indentation stripped); - and `want` is the example's expected output (with indentation - stripped). - - `name` is the string's name, and `lineno` is the line number - where the example starts; both are used for error messages. - """ - # Get the example's indentation level. - indent = len(m.group('indent')) - - # Divide source into lines; check that they're properly - # indented; and then strip their indentation & prompts. - source_lines = m.group('source').split('\n') - self._check_prompt_blank(source_lines, indent, name, lineno) - self._check_prefix(source_lines[1:], ' '*indent + '.', name, lineno) - source = '\n'.join([sl[indent+4:] for sl in source_lines]) - - # Divide want into lines; check that it's properly indented; and - # then strip the indentation. Spaces before the last newline should - # be preserved, so plain rstrip() isn't good enough. - want = m.group('want') - want_lines = want.split('\n') - if len(want_lines) > 1 and re.match(r' *$', want_lines[-1]): - del want_lines[-1] # forget final newline & spaces after it - self._check_prefix(want_lines, ' '*indent, name, - lineno + len(source_lines)) - want = '\n'.join([wl[indent:] for wl in want_lines]) - - # If `want` contains a traceback message, then extract it. - m = self._EXCEPTION_RE.match(want) - if m: - exc_msg = m.group('msg') - else: - exc_msg = None - - # Extract options from the source. - options = self._find_options(source, name, lineno) - - return source, options, want, exc_msg - - # This regular expression looks for option directives in the - # source code of an example. Option directives are comments - # starting with "doctest:". Warning: this may give false - # positives for string-literals that contain the string - # "#doctest:". Eliminating these false positives would require - # actually parsing the string; but we limit them by ignoring any - # line containing "#doctest:" that is *followed* by a quote mark. - _OPTION_DIRECTIVE_RE = re.compile(r'#\s*doctest:\s*([^\n\'"]*)$', - re.MULTILINE) - - def _find_options(self, source, name, lineno): - """ - Return a dictionary containing option overrides extracted from - option directives in the given source string. - - `name` is the string's name, and `lineno` is the line number - where the example starts; both are used for error messages. - """ - options = {} - # (note: with the current regexp, this will match at most once:) - for m in self._OPTION_DIRECTIVE_RE.finditer(source): - option_strings = m.group(1).replace(',', ' ').split() - for option in option_strings: - if (option[0] not in '+-' or - option[1:] not in OPTIONFLAGS_BY_NAME): - raise ValueError('line %r of the doctest for %s ' - 'has an invalid option: %r' % - (lineno+1, name, option)) - flag = OPTIONFLAGS_BY_NAME[option[1:]] - options[flag] = (option[0] == '+') - if options and self._IS_BLANK_OR_COMMENT(source): - raise ValueError('line %r of the doctest for %s has an option ' - 'directive on a line with no example: %r' % - (lineno, name, source)) - return options - - # This regular expression finds the indentation of every non-blank - # line in a string. - _INDENT_RE = re.compile('^([ ]*)(?=\S)', re.MULTILINE) - - def _min_indent(self, s): - "Return the minimum indentation of any non-blank line in `s`" - indents = [len(indent) for indent in self._INDENT_RE.findall(s)] - if len(indents) > 0: - return min(indents) - else: - return 0 - - def _check_prompt_blank(self, lines, indent, name, lineno): - """ - Given the lines of a source string (including prompts and - leading indentation), check to make sure that every prompt is - followed by a space character. If any line is not followed by - a space character, then raise ValueError. - """ - for i, line in enumerate(lines): - if len(line) >= indent+4 and line[indent+3] != ' ': - raise ValueError('line %r of the docstring for %s ' - 'lacks blank after %s: %r' % - (lineno+i+1, name, - line[indent:indent+3], line)) - - def _check_prefix(self, lines, prefix, name, lineno): - """ - Check that every line in the given list starts with the given - prefix; if any line does not, then raise a ValueError. - """ - for i, line in enumerate(lines): - if line and not line.startswith(prefix): - raise ValueError('line %r of the docstring for %s has ' - 'inconsistent leading whitespace: %r' % - (lineno+i+1, name, line)) - - -###################################################################### -## 4. DocTest Finder -###################################################################### - -class DocTestFinder: - """ - A class used to extract the DocTests that are relevant to a given - object, from its docstring and the docstrings of its contained - objects. Doctests can currently be extracted from the following - object types: modules, functions, classes, methods, staticmethods, - classmethods, and properties. - """ - - def __init__(self, verbose=False, parser=DocTestParser(), - recurse=True, _namefilter=None, exclude_empty=True): - """ - Create a new doctest finder. - - The optional argument `parser` specifies a class or - function that should be used to create new DocTest objects (or - objects that implement the same interface as DocTest). The - signature for this factory function should match the signature - of the DocTest constructor. - - If the optional argument `recurse` is false, then `find` will - only examine the given object, and not any contained objects. - - If the optional argument `exclude_empty` is false, then `find` - will include tests for objects with empty docstrings. - """ - self._parser = parser - self._verbose = verbose - self._recurse = recurse - self._exclude_empty = exclude_empty - # _namefilter is undocumented, and exists only for temporary backward- - # compatibility support of testmod's deprecated isprivate mess. - self._namefilter = _namefilter - - def find(self, obj, name=None, module=None, globs=None, - extraglobs=None): - """ - Return a list of the DocTests that are defined by the given - object's docstring, or by any of its contained objects' - docstrings. - - The optional parameter `module` is the module that contains - the given object. If the module is not specified or is None, then - the test finder will attempt to automatically determine the - correct module. The object's module is used: - - - As a default namespace, if `globs` is not specified. - - To prevent the DocTestFinder from extracting DocTests - from objects that are imported from other modules. - - To find the name of the file containing the object. - - To help find the line number of the object within its - file. - - Contained objects whose module does not match `module` are ignored. - - If `module` is False, no attempt to find the module will be made. - This is obscure, of use mostly in tests: if `module` is False, or - is None but cannot be found automatically, then all objects are - considered to belong to the (non-existent) module, so all contained - objects will (recursively) be searched for doctests. - - The globals for each DocTest is formed by combining `globs` - and `extraglobs` (bindings in `extraglobs` override bindings - in `globs`). A new copy of the globals dictionary is created - for each DocTest. If `globs` is not specified, then it - defaults to the module's `__dict__`, if specified, or {} - otherwise. If `extraglobs` is not specified, then it defaults - to {}. - - """ - # If name was not specified, then extract it from the object. - if name is None: - name = getattr(obj, '__name__', None) - if name is None: - raise ValueError("DocTestFinder.find: name must be given " - "when obj.__name__ doesn't exist: %r" % - (type(obj),)) - - # Find the module that contains the given object (if obj is - # a module, then module=obj.). Note: this may fail, in which - # case module will be None. - if module is False: - module = None - elif module is None: - module = inspect.getmodule(obj) - - # Read the module's source code. This is used by - # DocTestFinder._find_lineno to find the line number for a - # given object's docstring. - try: - file = inspect.getsourcefile(obj) or inspect.getfile(obj) - source_lines = linecache.getlines(file) - if not source_lines: - source_lines = None - except TypeError: - source_lines = None - - # Initialize globals, and merge in extraglobs. - if globs is None: - if module is None: - globs = {} - else: - globs = module.__dict__.copy() - else: - globs = globs.copy() - if extraglobs is not None: - globs.update(extraglobs) - - # Recursively expore `obj`, extracting DocTests. - tests = [] - self._find(tests, obj, name, module, source_lines, globs, {}) - return tests - - def _filter(self, obj, prefix, base): - """ - Return true if the given object should not be examined. - """ - return (self._namefilter is not None and - self._namefilter(prefix, base)) - - def _from_module(self, module, object): - """ - Return true if the given object is defined in the given - module. - """ - if module is None: - return True - elif inspect.isfunction(object): - return module.__dict__ is object.func_globals - elif inspect.isclass(object): - return module.__name__ == object.__module__ - elif inspect.getmodule(object) is not None: - return module is inspect.getmodule(object) - elif hasattr(object, '__module__'): - return module.__name__ == object.__module__ - elif isinstance(object, property): - return True # [XX] no way not be sure. - else: - raise ValueError("object must be a class or function") - - def _find(self, tests, obj, name, module, source_lines, globs, seen): - """ - Find tests for the given object and any contained objects, and - add them to `tests`. - """ - if self._verbose: - print 'Finding tests in %s' % name - - # If we've already processed this object, then ignore it. - if id(obj) in seen: - return - seen[id(obj)] = 1 - - # Find a test for this object, and add it to the list of tests. - test = self._get_test(obj, name, module, globs, source_lines) - if test is not None: - tests.append(test) - - # Look for tests in a module's contained objects. - if inspect.ismodule(obj) and self._recurse: - for valname, val in obj.__dict__.items(): - # Check if this contained object should be ignored. - if self._filter(val, name, valname): - continue - valname = '%s.%s' % (name, valname) - # Recurse to functions & classes. - if ((inspect.isfunction(val) or inspect.isclass(val)) and - self._from_module(module, val)): - self._find(tests, val, valname, module, source_lines, - globs, seen) - - # Look for tests in a module's __test__ dictionary. - if inspect.ismodule(obj) and self._recurse: - for valname, val in getattr(obj, '__test__', {}).items(): - if not isinstance(valname, basestring): - raise ValueError("DocTestFinder.find: __test__ keys " - "must be strings: %r" % - (type(valname),)) - if not (inspect.isfunction(val) or inspect.isclass(val) or - inspect.ismethod(val) or inspect.ismodule(val) or - isinstance(val, basestring)): - raise ValueError("DocTestFinder.find: __test__ values " - "must be strings, functions, methods, " - "classes, or modules: %r" % - (type(val),)) - valname = '%s.__test__.%s' % (name, valname) - self._find(tests, val, valname, module, source_lines, - globs, seen) - - # Look for tests in a class's contained objects. - if inspect.isclass(obj) and self._recurse: - for valname, val in obj.__dict__.items(): - # Check if this contained object should be ignored. - if self._filter(val, name, valname): - continue - # Special handling for staticmethod/classmethod. - if isinstance(val, staticmethod): - val = getattr(obj, valname) - if isinstance(val, classmethod): - val = getattr(obj, valname).im_func - - # Recurse to methods, properties, and nested classes. - if ((inspect.isfunction(val) or inspect.isclass(val) or - isinstance(val, property)) and - self._from_module(module, val)): - valname = '%s.%s' % (name, valname) - self._find(tests, val, valname, module, source_lines, - globs, seen) - - def _get_test(self, obj, name, module, globs, source_lines): - """ - Return a DocTest for the given object, if it defines a docstring; - otherwise, return None. - """ - # Extract the object's docstring. If it doesn't have one, - # then return None (no test for this object). - if isinstance(obj, basestring): - docstring = obj - else: - try: - if obj.__doc__ is None: - docstring = '' - else: - docstring = obj.__doc__ - if not isinstance(docstring, basestring): - docstring = str(docstring) - except (TypeError, AttributeError): - docstring = '' - - # Find the docstring's location in the file. - lineno = self._find_lineno(obj, source_lines) - - # Don't bother if the docstring is empty. - if self._exclude_empty and not docstring: - return None - - # Return a DocTest for this object. - if module is None: - filename = None - else: - filename = getattr(module, '__file__', module.__name__) - if filename[-4:] in (".pyc", ".pyo"): - filename = filename[:-1] - return self._parser.get_doctest(docstring, globs, name, - filename, lineno) - - def _find_lineno(self, obj, source_lines): - """ - Return a line number of the given object's docstring. Note: - this method assumes that the object has a docstring. - """ - lineno = None - - # Find the line number for modules. - if inspect.ismodule(obj): - lineno = 0 - - # Find the line number for classes. - # Note: this could be fooled if a class is defined multiple - # times in a single file. - if inspect.isclass(obj): - if source_lines is None: - return None - pat = re.compile(r'^\s*class\s*%s\b' % - getattr(obj, '__name__', '-')) - for i, line in enumerate(source_lines): - if pat.match(line): - lineno = i - break - - # Find the line number for functions & methods. - if inspect.ismethod(obj): obj = obj.im_func - if inspect.isfunction(obj): obj = obj.func_code - if inspect.istraceback(obj): obj = obj.tb_frame - if inspect.isframe(obj): obj = obj.f_code - if inspect.iscode(obj): - lineno = getattr(obj, 'co_firstlineno', None)-1 - - # Find the line number where the docstring starts. Assume - # that it's the first line that begins with a quote mark. - # Note: this could be fooled by a multiline function - # signature, where a continuation line begins with a quote - # mark. - if lineno is not None: - if source_lines is None: - return lineno+1 - pat = re.compile('(^|.*:)\s*\w*("|\')') - for lineno in range(lineno, len(source_lines)): - if pat.match(source_lines[lineno]): - return lineno - - # We couldn't find the line number. - return None - -###################################################################### -## 5. DocTest Runner -###################################################################### - -class DocTestRunner: - """ - A class used to run DocTest test cases, and accumulate statistics. - The `run` method is used to process a single DocTest case. It - returns a tuple `(f, t)`, where `t` is the number of test cases - tried, and `f` is the number of test cases that failed. - - >>> tests = DocTestFinder().find(_TestClass) - >>> runner = DocTestRunner(verbose=False) - >>> for test in tests: - ... print runner.run(test) - (0, 2) - (0, 1) - (0, 2) - (0, 2) - - The `summarize` method prints a summary of all the test cases that - have been run by the runner, and returns an aggregated `(f, t)` - tuple: - - >>> runner.summarize(verbose=1) - 4 items passed all tests: - 2 tests in _TestClass - 2 tests in _TestClass.__init__ - 2 tests in _TestClass.get - 1 tests in _TestClass.square - 7 tests in 4 items. - 7 passed and 0 failed. - Test passed. - (0, 7) - - The aggregated number of tried examples and failed examples is - also available via the `tries` and `failures` attributes: - - >>> runner.tries - 7 - >>> runner.failures - 0 - - The comparison between expected outputs and actual outputs is done - by an `OutputChecker`. This comparison may be customized with a - number of option flags; see the documentation for `testmod` for - more information. If the option flags are insufficient, then the - comparison may also be customized by passing a subclass of - `OutputChecker` to the constructor. - - The test runner's display output can be controlled in two ways. - First, an output function (`out) can be passed to - `TestRunner.run`; this function will be called with strings that - should be displayed. It defaults to `sys.stdout.write`. If - capturing the output is not sufficient, then the display output - can be also customized by subclassing DocTestRunner, and - overriding the methods `report_start`, `report_success`, - `report_unexpected_exception`, and `report_failure`. - """ - # This divider string is used to separate failure messages, and to - # separate sections of the summary. - DIVIDER = "*" * 70 - - def __init__(self, checker=None, verbose=None, optionflags=0): - """ - Create a new test runner. - - Optional keyword arg `checker` is the `OutputChecker` that - should be used to compare the expected outputs and actual - outputs of doctest examples. - - Optional keyword arg 'verbose' prints lots of stuff if true, - only failures if false; by default, it's true iff '-v' is in - sys.argv. - - Optional argument `optionflags` can be used to control how the - test runner compares expected output to actual output, and how - it displays failures. See the documentation for `testmod` for - more information. - """ - self._checker = checker or OutputChecker() - if verbose is None: - verbose = '-v' in sys.argv - self._verbose = verbose - self.optionflags = optionflags - self.original_optionflags = optionflags - - # Keep track of the examples we've run. - self.tries = 0 - self.failures = 0 - self._name2ft = {} - - # Create a fake output target for capturing doctest output. - self._fakeout = _SpoofOut() - - #///////////////////////////////////////////////////////////////// - # Reporting methods - #///////////////////////////////////////////////////////////////// - - def report_start(self, out, test, example): - """ - Report that the test runner is about to process the given - example. (Only displays a message if verbose=True) - """ - if self._verbose: - if example.want: - out('Trying:\n' + _indent(example.source) + - 'Expecting:\n' + _indent(example.want)) - else: - out('Trying:\n' + _indent(example.source) + - 'Expecting nothing\n') - - def report_success(self, out, test, example, got): - """ - Report that the given example ran successfully. (Only - displays a message if verbose=True) - """ - if self._verbose: - out("ok\n") - - def report_failure(self, out, test, example, got): - """ - Report that the given example failed. - """ - out(self._failure_header(test, example) + - self._checker.output_difference(example, got, self.optionflags)) - - def report_unexpected_exception(self, out, test, example, exc_info): - """ - Report that the given example raised an unexpected exception. - """ - out(self._failure_header(test, example) + - 'Exception raised:\n' + _indent(_exception_traceback(exc_info))) - - def _failure_header(self, test, example): - out = [self.DIVIDER] - if test.filename: - if test.lineno is not None and example.lineno is not None: - lineno = test.lineno + example.lineno + 1 - else: - lineno = '?' - out.append('File "%s", line %s, in %s' % - (test.filename, lineno, test.name)) - else: - out.append('Line %s, in %s' % (example.lineno+1, test.name)) - out.append('Failed example:') - source = example.source - out.append(_indent(source)) - return '\n'.join(out) - - #///////////////////////////////////////////////////////////////// - # DocTest Running - #///////////////////////////////////////////////////////////////// - - def __run(self, test, compileflags, out): - """ - Run the examples in `test`. Write the outcome of each example - with one of the `DocTestRunner.report_*` methods, using the - writer function `out`. `compileflags` is the set of compiler - flags that should be used to execute examples. Return a tuple - `(f, t)`, where `t` is the number of examples tried, and `f` - is the number of examples that failed. The examples are run - in the namespace `test.globs`. - """ - # Keep track of the number of failures and tries. - failures = tries = 0 - - # Save the option flags (since option directives can be used - # to modify them). - original_optionflags = self.optionflags - - SUCCESS, FAILURE, BOOM = range(3) # `outcome` state - - check = self._checker.check_output - - # Process each example. - for examplenum, example in enumerate(test.examples): - - # If REPORT_ONLY_FIRST_FAILURE is set, then supress - # reporting after the first failure. - quiet = (self.optionflags & REPORT_ONLY_FIRST_FAILURE and - failures > 0) - - # Merge in the example's options. - self.optionflags = original_optionflags - if example.options: - for (optionflag, val) in example.options.items(): - if val: - self.optionflags |= optionflag - else: - self.optionflags &= ~optionflag - - # Record that we started this example. - tries += 1 - if not quiet: - self.report_start(out, test, example) - - # Use a special filename for compile(), so we can retrieve - # the source code during interactive debugging (see - # __patched_linecache_getlines). - filename = '' % (test.name, examplenum) - - # Run the example in the given context (globs), and record - # any exception that gets raised. (But don't intercept - # keyboard interrupts.) - try: - # Don't blink! This is where the user's code gets run. - exec compile(example.source, filename, "single", - compileflags, 1) in test.globs - self.debugger.set_continue() # ==== Example Finished ==== - exception = None - except KeyboardInterrupt: - raise - except: - exception = sys.exc_info() - self.debugger.set_continue() # ==== Example Finished ==== - - got = self._fakeout.getvalue() # the actual output - self._fakeout.truncate(0) - outcome = FAILURE # guilty until proved innocent or insane - - # If the example executed without raising any exceptions, - # verify its output. - if exception is None: - if check(example.want, got, self.optionflags): - outcome = SUCCESS - - # The example raised an exception: check if it was expected. - else: - exc_info = sys.exc_info() - exc_msg = traceback.format_exception_only(*exc_info[:2])[-1] - if not quiet: - got += _exception_traceback(exc_info) - - # If `example.exc_msg` is None, then we weren't expecting - # an exception. - if example.exc_msg is None: - outcome = BOOM - - # We expected an exception: see whether it matches. - elif check(example.exc_msg, exc_msg, self.optionflags): - outcome = SUCCESS - - # Another chance if they didn't care about the detail. - elif self.optionflags & IGNORE_EXCEPTION_DETAIL: - m1 = re.match(r'[^:]*:', example.exc_msg) - m2 = re.match(r'[^:]*:', exc_msg) - if m1 and m2 and check(m1.group(0), m2.group(0), - self.optionflags): - outcome = SUCCESS - - # Report the outcome. - if outcome is SUCCESS: - if not quiet: - self.report_success(out, test, example, got) - elif outcome is FAILURE: - if not quiet: - self.report_failure(out, test, example, got) - failures += 1 - elif outcome is BOOM: - if not quiet: - self.report_unexpected_exception(out, test, example, - exc_info) - failures += 1 - else: - assert False, ("unknown outcome", outcome) - - # Restore the option flags (in case they were modified) - self.optionflags = original_optionflags - - # Record and return the number of failures and tries. - self.__record_outcome(test, failures, tries) - return failures, tries - - def __record_outcome(self, test, f, t): - """ - Record the fact that the given DocTest (`test`) generated `f` - failures out of `t` tried examples. - """ - f2, t2 = self._name2ft.get(test.name, (0,0)) - self._name2ft[test.name] = (f+f2, t+t2) - self.failures += f - self.tries += t - - __LINECACHE_FILENAME_RE = re.compile(r'[\w\.]+)' - r'\[(?P\d+)\]>$') - def __patched_linecache_getlines(self, filename): - m = self.__LINECACHE_FILENAME_RE.match(filename) - if m and m.group('name') == self.test.name: - example = self.test.examples[int(m.group('examplenum'))] - return example.source.splitlines(True) - else: - return self.save_linecache_getlines(filename) - - def run(self, test, compileflags=None, out=None, clear_globs=True): - """ - Run the examples in `test`, and display the results using the - writer function `out`. - - The examples are run in the namespace `test.globs`. If - `clear_globs` is true (the default), then this namespace will - be cleared after the test runs, to help with garbage - collection. If you would like to examine the namespace after - the test completes, then use `clear_globs=False`. - - `compileflags` gives the set of flags that should be used by - the Python compiler when running the examples. If not - specified, then it will default to the set of future-import - flags that apply to `globs`. - - The output of each example is checked using - `DocTestRunner.check_output`, and the results are formatted by - the `DocTestRunner.report_*` methods. - """ - self.test = test - - if compileflags is None: - compileflags = _extract_future_flags(test.globs) - - save_stdout = sys.stdout - if out is None: - out = save_stdout.write - sys.stdout = self._fakeout - - # Patch pdb.set_trace to restore sys.stdout during interactive - # debugging (so it's not still redirected to self._fakeout). - # Note that the interactive output will go to *our* - # save_stdout, even if that's not the real sys.stdout; this - # allows us to write test cases for the set_trace behavior. - save_set_trace = pdb.set_trace - self.debugger = _OutputRedirectingPdb(save_stdout) - self.debugger.reset() - pdb.set_trace = self.debugger.set_trace - - # Patch linecache.getlines, so we can see the example's source - # when we're inside the debugger. - self.save_linecache_getlines = linecache.getlines - linecache.getlines = self.__patched_linecache_getlines - - try: - return self.__run(test, compileflags, out) - finally: - sys.stdout = save_stdout - pdb.set_trace = save_set_trace - linecache.getlines = self.save_linecache_getlines - if clear_globs: - test.globs.clear() - - #///////////////////////////////////////////////////////////////// - # Summarization - #///////////////////////////////////////////////////////////////// - def summarize(self, verbose=None): - """ - Print a summary of all the test cases that have been run by - this DocTestRunner, and return a tuple `(f, t)`, where `f` is - the total number of failed examples, and `t` is the total - number of tried examples. - - The optional `verbose` argument controls how detailed the - summary is. If the verbosity is not specified, then the - DocTestRunner's verbosity is used. - """ - if verbose is None: - verbose = self._verbose - notests = [] - passed = [] - failed = [] - totalt = totalf = 0 - for x in self._name2ft.items(): - name, (f, t) = x - assert f <= t - totalt += t - totalf += f - if t == 0: - notests.append(name) - elif f == 0: - passed.append( (name, t) ) - else: - failed.append(x) - if verbose: - if notests: - print len(notests), "items had no tests:" - notests.sort() - for thing in notests: - print " ", thing - if passed: - print len(passed), "items passed all tests:" - passed.sort() - for thing, count in passed: - print " %3d tests in %s" % (count, thing) - if failed: - print self.DIVIDER - print len(failed), "items had failures:" - failed.sort() - for thing, (f, t) in failed: - print " %3d of %3d in %s" % (f, t, thing) - if verbose: - print totalt, "tests in", len(self._name2ft), "items." - print totalt - totalf, "passed and", totalf, "failed." - if totalf: - print "***Test Failed***", totalf, "failures." - elif verbose: - print "Test passed." - return totalf, totalt - - #///////////////////////////////////////////////////////////////// - # Backward compatibility cruft to maintain doctest.master. - #///////////////////////////////////////////////////////////////// - def merge(self, other): - d = self._name2ft - for name, (f, t) in other._name2ft.items(): - if name in d: - print "*** DocTestRunner.merge: '" + name + "' in both" \ - " testers; summing outcomes." - f2, t2 = d[name] - f = f + f2 - t = t + t2 - d[name] = f, t - -class OutputChecker: - """ - A class used to check the whether the actual output from a doctest - example matches the expected output. `OutputChecker` defines two - methods: `check_output`, which compares a given pair of outputs, - and returns true if they match; and `output_difference`, which - returns a string describing the differences between two outputs. - """ - def check_output(self, want, got, optionflags): - """ - Return True iff the actual output from an example (`got`) - matches the expected output (`want`). These strings are - always considered to match if they are identical; but - depending on what option flags the test runner is using, - several non-exact match types are also possible. See the - documentation for `TestRunner` for more information about - option flags. - """ - # Handle the common case first, for efficiency: - # if they're string-identical, always return true. - if got == want: - return True - - # The values True and False replaced 1 and 0 as the return - # value for boolean comparisons in Python 2.3. - if not (optionflags & DONT_ACCEPT_TRUE_FOR_1): - if (got,want) == ("True\n", "1\n"): - return True - if (got,want) == ("False\n", "0\n"): - return True - - # can be used as a special sequence to signify a - # blank line, unless the DONT_ACCEPT_BLANKLINE flag is used. - if not (optionflags & DONT_ACCEPT_BLANKLINE): - # Replace in want with a blank line. - want = re.sub('(?m)^%s\s*?$' % re.escape(BLANKLINE_MARKER), - '', want) - # If a line in got contains only spaces, then remove the - # spaces. - got = re.sub('(?m)^\s*?$', '', got) - if got == want: - return True - - # This flag causes doctest to ignore any differences in the - # contents of whitespace strings. Note that this can be used - # in conjunction with the ELLIPSIS flag. - if optionflags & NORMALIZE_WHITESPACE: - got = ' '.join(got.split()) - want = ' '.join(want.split()) - if got == want: - return True - - # The ELLIPSIS flag says to let the sequence "..." in `want` - # match any substring in `got`. - if optionflags & ELLIPSIS: - if _ellipsis_match(want, got): - return True - - # We didn't find any match; return false. - return False - - # Should we do a fancy diff? - def _do_a_fancy_diff(self, want, got, optionflags): - # Not unless they asked for a fancy diff. - if not optionflags & (REPORT_UDIFF | - REPORT_CDIFF | - REPORT_NDIFF): - return False - - # If expected output uses ellipsis, a meaningful fancy diff is - # too hard ... or maybe not. In two real-life failures Tim saw, - # a diff was a major help anyway, so this is commented out. - # [todo] _ellipsis_match() knows which pieces do and don't match, - # and could be the basis for a kick-ass diff in this case. - ##if optionflags & ELLIPSIS and ELLIPSIS_MARKER in want: - ## return False - - # ndiff does intraline difference marking, so can be useful even - # for 1-line differences. - if optionflags & REPORT_NDIFF: - return True - - # The other diff types need at least a few lines to be helpful. - return want.count('\n') > 2 and got.count('\n') > 2 - - def output_difference(self, example, got, optionflags): - """ - Return a string describing the differences between the - expected output for a given example (`example`) and the actual - output (`got`). `optionflags` is the set of option flags used - to compare `want` and `got`. - """ - want = example.want - # If s are being used, then replace blank lines - # with in the actual output string. - if not (optionflags & DONT_ACCEPT_BLANKLINE): - got = re.sub('(?m)^[ ]*(?=\n)', BLANKLINE_MARKER, got) - - # Check if we should use diff. - if self._do_a_fancy_diff(want, got, optionflags): - # Split want & got into lines. - want_lines = want.splitlines(True) # True == keep line ends - got_lines = got.splitlines(True) - # Use difflib to find their differences. - if optionflags & REPORT_UDIFF: - diff = difflib.unified_diff(want_lines, got_lines, n=2) - diff = list(diff)[2:] # strip the diff header - kind = 'unified diff with -expected +actual' - elif optionflags & REPORT_CDIFF: - diff = difflib.context_diff(want_lines, got_lines, n=2) - diff = list(diff)[2:] # strip the diff header - kind = 'context diff with expected followed by actual' - elif optionflags & REPORT_NDIFF: - engine = difflib.Differ(charjunk=difflib.IS_CHARACTER_JUNK) - diff = list(engine.compare(want_lines, got_lines)) - kind = 'ndiff with -expected +actual' - else: - assert 0, 'Bad diff option' - # Remove trailing whitespace on diff output. - diff = [line.rstrip() + '\n' for line in diff] - return 'Differences (%s):\n' % kind + _indent(''.join(diff)) - - # If we're not using diff, then simply list the expected - # output followed by the actual output. - if want and got: - return 'Expected:\n%sGot:\n%s' % (_indent(want), _indent(got)) - elif want: - return 'Expected:\n%sGot nothing\n' % _indent(want) - elif got: - return 'Expected nothing\nGot:\n%s' % _indent(got) - else: - return 'Expected nothing\nGot nothing\n' - -class DocTestFailure(Exception): - """A DocTest example has failed in debugging mode. - - The exception instance has variables: - - - test: the DocTest object being run - - - excample: the Example object that failed - - - got: the actual output - """ - def __init__(self, test, example, got): - self.test = test - self.example = example - self.got = got - - def __str__(self): - return str(self.test) - -class UnexpectedException(Exception): - """A DocTest example has encountered an unexpected exception - - The exception instance has variables: - - - test: the DocTest object being run - - - excample: the Example object that failed - - - exc_info: the exception info - """ - def __init__(self, test, example, exc_info): - self.test = test - self.example = example - self.exc_info = exc_info - - def __str__(self): - return str(self.test) - -class DebugRunner(DocTestRunner): - r"""Run doc tests but raise an exception as soon as there is a failure. - - If an unexpected exception occurs, an UnexpectedException is raised. - It contains the test, the example, and the original exception: - - >>> runner = DebugRunner(verbose=False) - >>> test = DocTestParser().get_doctest('>>> raise KeyError\n42', - ... {}, 'foo', 'foo.py', 0) - >>> try: - ... runner.run(test) - ... except UnexpectedException, failure: - ... pass - - >>> failure.test is test - True - - >>> failure.example.want - '42\n' - - >>> exc_info = failure.exc_info - >>> raise exc_info[0], exc_info[1], exc_info[2] - Traceback (most recent call last): - ... - KeyError - - We wrap the original exception to give the calling application - access to the test and example information. - - If the output doesn't match, then a DocTestFailure is raised: - - >>> test = DocTestParser().get_doctest(''' - ... >>> x = 1 - ... >>> x - ... 2 - ... ''', {}, 'foo', 'foo.py', 0) - - >>> try: - ... runner.run(test) - ... except DocTestFailure, failure: - ... pass - - DocTestFailure objects provide access to the test: - - >>> failure.test is test - True - - As well as to the example: - - >>> failure.example.want - '2\n' - - and the actual output: - - >>> failure.got - '1\n' - - If a failure or error occurs, the globals are left intact: - - >>> del test.globs['__builtins__'] - >>> test.globs - {'x': 1} - - >>> test = DocTestParser().get_doctest(''' - ... >>> x = 2 - ... >>> raise KeyError - ... ''', {}, 'foo', 'foo.py', 0) - - >>> runner.run(test) - Traceback (most recent call last): - ... - UnexpectedException: - - >>> del test.globs['__builtins__'] - >>> test.globs - {'x': 2} - - But the globals are cleared if there is no error: - - >>> test = DocTestParser().get_doctest(''' - ... >>> x = 2 - ... ''', {}, 'foo', 'foo.py', 0) - - >>> runner.run(test) - (0, 1) - - >>> test.globs - {} - - """ - - def run(self, test, compileflags=None, out=None, clear_globs=True): - r = DocTestRunner.run(self, test, compileflags, out, False) - if clear_globs: - test.globs.clear() - return r - - def report_unexpected_exception(self, out, test, example, exc_info): - raise UnexpectedException(test, example, exc_info) - - def report_failure(self, out, test, example, got): - raise DocTestFailure(test, example, got) - -###################################################################### -## 6. Test Functions -###################################################################### -# These should be backwards compatible. - -# For backward compatibility, a global instance of a DocTestRunner -# class, updated by testmod. -master = None - -def testmod(m=None, name=None, globs=None, verbose=None, isprivate=None, - report=True, optionflags=0, extraglobs=None, - raise_on_error=False, exclude_empty=False): - """m=None, name=None, globs=None, verbose=None, isprivate=None, - report=True, optionflags=0, extraglobs=None, raise_on_error=False, - exclude_empty=False - - Test examples in docstrings in functions and classes reachable - from module m (or the current module if m is not supplied), starting - with m.__doc__. Unless isprivate is specified, private names - are not skipped. - - Also test examples reachable from dict m.__test__ if it exists and is - not None. m.__test__ maps names to functions, classes and strings; - function and class docstrings are tested even if the name is private; - strings are tested directly, as if they were docstrings. - - Return (#failures, #tests). - - See doctest.__doc__ for an overview. - - Optional keyword arg "name" gives the name of the module; by default - use m.__name__. - - Optional keyword arg "globs" gives a dict to be used as the globals - when executing examples; by default, use m.__dict__. A copy of this - dict is actually used for each docstring, so that each docstring's - examples start with a clean slate. - - Optional keyword arg "extraglobs" gives a dictionary that should be - merged into the globals that are used to execute examples. By - default, no extra globals are used. This is new in 2.4. - - Optional keyword arg "verbose" prints lots of stuff if true, prints - only failures if false; by default, it's true iff "-v" is in sys.argv. - - Optional keyword arg "report" prints a summary at the end when true, - else prints nothing at the end. In verbose mode, the summary is - detailed, else very brief (in fact, empty if all tests passed). - - Optional keyword arg "optionflags" or's together module constants, - and defaults to 0. This is new in 2.3. Possible values (see the - docs for details): - - DONT_ACCEPT_TRUE_FOR_1 - DONT_ACCEPT_BLANKLINE - NORMALIZE_WHITESPACE - ELLIPSIS - IGNORE_EXCEPTION_DETAIL - REPORT_UDIFF - REPORT_CDIFF - REPORT_NDIFF - REPORT_ONLY_FIRST_FAILURE - - Optional keyword arg "raise_on_error" raises an exception on the - first unexpected exception or failure. This allows failures to be - post-mortem debugged. - - Deprecated in Python 2.4: - Optional keyword arg "isprivate" specifies a function used to - determine whether a name is private. The default function is - treat all functions as public. Optionally, "isprivate" can be - set to doctest.is_private to skip over functions marked as private - using the underscore naming convention; see its docs for details. - - Advanced tomfoolery: testmod runs methods of a local instance of - class doctest.Tester, then merges the results into (or creates) - global Tester instance doctest.master. Methods of doctest.master - can be called directly too, if you want to do something unusual. - Passing report=0 to testmod is especially useful then, to delay - displaying a summary. Invoke doctest.master.summarize(verbose) - when you're done fiddling. - """ - global master - - if isprivate is not None: - warnings.warn("the isprivate argument is deprecated; " - "examine DocTestFinder.find() lists instead", - DeprecationWarning) - - # If no module was given, then use __main__. - if m is None: - # DWA - m will still be None if this wasn't invoked from the command - # line, in which case the following TypeError is about as good an error - # as we should expect - m = sys.modules.get('__main__') - - # Check that we were actually given a module. - if not inspect.ismodule(m): - raise TypeError("testmod: module required; %r" % (m,)) - - # If no name was given, then use the module's name. - if name is None: - name = m.__name__ - - # Find, parse, and run all tests in the given module. - finder = DocTestFinder(_namefilter=isprivate, exclude_empty=exclude_empty) - - if raise_on_error: - runner = DebugRunner(verbose=verbose, optionflags=optionflags) - else: - runner = DocTestRunner(verbose=verbose, optionflags=optionflags) - - for test in finder.find(m, name, globs=globs, extraglobs=extraglobs): - runner.run(test) - - if report: - runner.summarize() - - if master is None: - master = runner - else: - master.merge(runner) - - return runner.failures, runner.tries - -def testfile(filename, module_relative=True, name=None, package=None, - globs=None, verbose=None, report=True, optionflags=0, - extraglobs=None, raise_on_error=False, parser=DocTestParser()): - """ - Test examples in the given file. Return (#failures, #tests). - - Optional keyword arg "module_relative" specifies how filenames - should be interpreted: - - - If "module_relative" is True (the default), then "filename" - specifies a module-relative path. By default, this path is - relative to the calling module's directory; but if the - "package" argument is specified, then it is relative to that - package. To ensure os-independence, "filename" should use - "/" characters to separate path segments, and should not - be an absolute path (i.e., it may not begin with "/"). - - - If "module_relative" is False, then "filename" specifies an - os-specific path. The path may be absolute or relative (to - the current working directory). - - Optional keyword arg "name" gives the name of the test; by default - use the file's basename. - - Optional keyword argument "package" is a Python package or the - name of a Python package whose directory should be used as the - base directory for a module relative filename. If no package is - specified, then the calling module's directory is used as the base - directory for module relative filenames. It is an error to - specify "package" if "module_relative" is False. - - Optional keyword arg "globs" gives a dict to be used as the globals - when executing examples; by default, use {}. A copy of this dict - is actually used for each docstring, so that each docstring's - examples start with a clean slate. - - Optional keyword arg "extraglobs" gives a dictionary that should be - merged into the globals that are used to execute examples. By - default, no extra globals are used. - - Optional keyword arg "verbose" prints lots of stuff if true, prints - only failures if false; by default, it's true iff "-v" is in sys.argv. - - Optional keyword arg "report" prints a summary at the end when true, - else prints nothing at the end. In verbose mode, the summary is - detailed, else very brief (in fact, empty if all tests passed). - - Optional keyword arg "optionflags" or's together module constants, - and defaults to 0. Possible values (see the docs for details): - - DONT_ACCEPT_TRUE_FOR_1 - DONT_ACCEPT_BLANKLINE - NORMALIZE_WHITESPACE - ELLIPSIS - IGNORE_EXCEPTION_DETAIL - REPORT_UDIFF - REPORT_CDIFF - REPORT_NDIFF - REPORT_ONLY_FIRST_FAILURE - - Optional keyword arg "raise_on_error" raises an exception on the - first unexpected exception or failure. This allows failures to be - post-mortem debugged. - - Optional keyword arg "parser" specifies a DocTestParser (or - subclass) that should be used to extract tests from the files. - - Advanced tomfoolery: testmod runs methods of a local instance of - class doctest.Tester, then merges the results into (or creates) - global Tester instance doctest.master. Methods of doctest.master - can be called directly too, if you want to do something unusual. - Passing report=0 to testmod is especially useful then, to delay - displaying a summary. Invoke doctest.master.summarize(verbose) - when you're done fiddling. - """ - global master - - if package and not module_relative: - raise ValueError("Package may only be specified for module-" - "relative paths.") - - # Relativize the path - if module_relative: - package = _normalize_module(package) - filename = _module_relative_path(package, filename) - - # If no name was given, then use the file's name. - if name is None: - name = os.path.basename(filename) - - # Assemble the globals. - if globs is None: - globs = {} - else: - globs = globs.copy() - if extraglobs is not None: - globs.update(extraglobs) - - if raise_on_error: - runner = DebugRunner(verbose=verbose, optionflags=optionflags) - else: - runner = DocTestRunner(verbose=verbose, optionflags=optionflags) - - # Read the file, convert it to a test, and run it. - s = open(filename).read() - test = parser.get_doctest(s, globs, name, filename, 0) - runner.run(test) - - if report: - runner.summarize() - - if master is None: - master = runner - else: - master.merge(runner) - - return runner.failures, runner.tries - -def run_docstring_examples(f, globs, verbose=False, name="NoName", - compileflags=None, optionflags=0): - """ - Test examples in the given object's docstring (`f`), using `globs` - as globals. Optional argument `name` is used in failure messages. - If the optional argument `verbose` is true, then generate output - even if there are no failures. - - `compileflags` gives the set of flags that should be used by the - Python compiler when running the examples. If not specified, then - it will default to the set of future-import flags that apply to - `globs`. - - Optional keyword arg `optionflags` specifies options for the - testing and output. See the documentation for `testmod` for more - information. - """ - # Find, parse, and run all tests in the given module. - finder = DocTestFinder(verbose=verbose, recurse=False) - runner = DocTestRunner(verbose=verbose, optionflags=optionflags) - for test in finder.find(f, name, globs=globs): - runner.run(test, compileflags=compileflags) - -###################################################################### -## 7. Tester -###################################################################### -# This is provided only for backwards compatibility. It's not -# actually used in any way. - -class Tester: - def __init__(self, mod=None, globs=None, verbose=None, - isprivate=None, optionflags=0): - - warnings.warn("class Tester is deprecated; " - "use class doctest.DocTestRunner instead", - DeprecationWarning, stacklevel=2) - if mod is None and globs is None: - raise TypeError("Tester.__init__: must specify mod or globs") - if mod is not None and not inspect.ismodule(mod): - raise TypeError("Tester.__init__: mod must be a module; %r" % - (mod,)) - if globs is None: - globs = mod.__dict__ - self.globs = globs - - self.verbose = verbose - self.isprivate = isprivate - self.optionflags = optionflags - self.testfinder = DocTestFinder(_namefilter=isprivate) - self.testrunner = DocTestRunner(verbose=verbose, - optionflags=optionflags) - - def runstring(self, s, name): - test = DocTestParser().get_doctest(s, self.globs, name, None, None) - if self.verbose: - print "Running string", name - (f,t) = self.testrunner.run(test) - if self.verbose: - print f, "of", t, "examples failed in string", name - return (f,t) - - def rundoc(self, object, name=None, module=None): - f = t = 0 - tests = self.testfinder.find(object, name, module=module, - globs=self.globs) - for test in tests: - (f2, t2) = self.testrunner.run(test) - (f,t) = (f+f2, t+t2) - return (f,t) - - def rundict(self, d, name, module=None): - import new - m = new.module(name) - m.__dict__.update(d) - if module is None: - module = False - return self.rundoc(m, name, module) - - def run__test__(self, d, name): - import new - m = new.module(name) - m.__test__ = d - return self.rundoc(m, name) - - def summarize(self, verbose=None): - return self.testrunner.summarize(verbose) - - def merge(self, other): - self.testrunner.merge(other.testrunner) - -###################################################################### -## 8. Unittest Support -###################################################################### - -_unittest_reportflags = 0 - -def set_unittest_reportflags(flags): - """Sets the unittest option flags. - - The old flag is returned so that a runner could restore the old - value if it wished to: - - >>> old = _unittest_reportflags - >>> set_unittest_reportflags(REPORT_NDIFF | - ... REPORT_ONLY_FIRST_FAILURE) == old - True - - >>> import doctest - >>> doctest._unittest_reportflags == (REPORT_NDIFF | - ... REPORT_ONLY_FIRST_FAILURE) - True - - Only reporting flags can be set: - - >>> set_unittest_reportflags(ELLIPSIS) - Traceback (most recent call last): - ... - ValueError: ('Only reporting flags allowed', 8) - - >>> set_unittest_reportflags(old) == (REPORT_NDIFF | - ... REPORT_ONLY_FIRST_FAILURE) - True - """ - global _unittest_reportflags - - if (flags & REPORTING_FLAGS) != flags: - raise ValueError("Only reporting flags allowed", flags) - old = _unittest_reportflags - _unittest_reportflags = flags - return old - - -class DocTestCase(unittest.TestCase): - - def __init__(self, test, optionflags=0, setUp=None, tearDown=None, - checker=None): - - unittest.TestCase.__init__(self) - self._dt_optionflags = optionflags - self._dt_checker = checker - self._dt_test = test - self._dt_setUp = setUp - self._dt_tearDown = tearDown - - def setUp(self): - test = self._dt_test - - if self._dt_setUp is not None: - self._dt_setUp(test) - - def tearDown(self): - test = self._dt_test - - if self._dt_tearDown is not None: - self._dt_tearDown(test) - - test.globs.clear() - - def runTest(self): - test = self._dt_test - old = sys.stdout - new = StringIO() - optionflags = self._dt_optionflags - - if not (optionflags & REPORTING_FLAGS): - # The option flags don't include any reporting flags, - # so add the default reporting flags - optionflags |= _unittest_reportflags - - runner = DocTestRunner(optionflags=optionflags, - checker=self._dt_checker, verbose=False) - - try: - runner.DIVIDER = "-"*70 - failures, tries = runner.run( - test, out=new.write, clear_globs=False) - finally: - sys.stdout = old - - if failures: - raise self.failureException(self.format_failure(new.getvalue())) - - def format_failure(self, err): - test = self._dt_test - if test.lineno is None: - lineno = 'unknown line number' - else: - lineno = '%s' % test.lineno - lname = '.'.join(test.name.split('.')[-1:]) - return ('Failed doctest test for %s\n' - ' File "%s", line %s, in %s\n\n%s' - % (test.name, test.filename, lineno, lname, err) - ) - - def debug(self): - r"""Run the test case without results and without catching exceptions - - The unit test framework includes a debug method on test cases - and test suites to support post-mortem debugging. The test code - is run in such a way that errors are not caught. This way a - caller can catch the errors and initiate post-mortem debugging. - - The DocTestCase provides a debug method that raises - UnexpectedException errors if there is an unexepcted - exception: - - >>> test = DocTestParser().get_doctest('>>> raise KeyError\n42', - ... {}, 'foo', 'foo.py', 0) - >>> case = DocTestCase(test) - >>> try: - ... case.debug() - ... except UnexpectedException, failure: - ... pass - - The UnexpectedException contains the test, the example, and - the original exception: - - >>> failure.test is test - True - - >>> failure.example.want - '42\n' - - >>> exc_info = failure.exc_info - >>> raise exc_info[0], exc_info[1], exc_info[2] - Traceback (most recent call last): - ... - KeyError - - If the output doesn't match, then a DocTestFailure is raised: - - >>> test = DocTestParser().get_doctest(''' - ... >>> x = 1 - ... >>> x - ... 2 - ... ''', {}, 'foo', 'foo.py', 0) - >>> case = DocTestCase(test) - - >>> try: - ... case.debug() - ... except DocTestFailure, failure: - ... pass - - DocTestFailure objects provide access to the test: - - >>> failure.test is test - True - - As well as to the example: - - >>> failure.example.want - '2\n' - - and the actual output: - - >>> failure.got - '1\n' - - """ - - self.setUp() - runner = DebugRunner(optionflags=self._dt_optionflags, - checker=self._dt_checker, verbose=False) - runner.run(self._dt_test) - self.tearDown() - - def id(self): - return self._dt_test.name - - def __repr__(self): - name = self._dt_test.name.split('.') - return "%s (%s)" % (name[-1], '.'.join(name[:-1])) - - __str__ = __repr__ - - def shortDescription(self): - return "Doctest: " + self._dt_test.name - -def DocTestSuite(module=None, globs=None, extraglobs=None, test_finder=None, - **options): - """ - Convert doctest tests for a module to a unittest test suite. - - This converts each documentation string in a module that - contains doctest tests to a unittest test case. If any of the - tests in a doc string fail, then the test case fails. An exception - is raised showing the name of the file containing the test and a - (sometimes approximate) line number. - - The `module` argument provides the module to be tested. The argument - can be either a module or a module name. - - If no argument is given, the calling module is used. - - A number of options may be provided as keyword arguments: - - setUp - A set-up function. This is called before running the - tests in each file. The setUp function will be passed a DocTest - object. The setUp function can access the test globals as the - globs attribute of the test passed. - - tearDown - A tear-down function. This is called after running the - tests in each file. The tearDown function will be passed a DocTest - object. The tearDown function can access the test globals as the - globs attribute of the test passed. - - globs - A dictionary containing initial global variables for the tests. - - optionflags - A set of doctest option flags expressed as an integer. - """ - - if test_finder is None: - test_finder = DocTestFinder() - - module = _normalize_module(module) - tests = test_finder.find(module, globs=globs, extraglobs=extraglobs) - if globs is None: - globs = module.__dict__ - if not tests: - # Why do we want to do this? Because it reveals a bug that might - # otherwise be hidden. - raise ValueError(module, "has no tests") - - tests.sort() - suite = unittest.TestSuite() - for test in tests: - if len(test.examples) == 0: - continue - if not test.filename: - filename = module.__file__ - if filename[-4:] in (".pyc", ".pyo"): - filename = filename[:-1] - test.filename = filename - suite.addTest(DocTestCase(test, **options)) - - return suite - -class DocFileCase(DocTestCase): - - def id(self): - return '_'.join(self._dt_test.name.split('.')) - - def __repr__(self): - return self._dt_test.filename - __str__ = __repr__ - - def format_failure(self, err): - return ('Failed doctest test for %s\n File "%s", line 0\n\n%s' - % (self._dt_test.name, self._dt_test.filename, err) - ) - -def DocFileTest(path, module_relative=True, package=None, - globs=None, parser=DocTestParser(), **options): - if globs is None: - globs = {} - - if package and not module_relative: - raise ValueError("Package may only be specified for module-" - "relative paths.") - - # Relativize the path. - if module_relative: - package = _normalize_module(package) - path = _module_relative_path(package, path) - - # Find the file and read it. - name = os.path.basename(path) - doc = open(path).read() - - # Convert it to a test, and wrap it in a DocFileCase. - test = parser.get_doctest(doc, globs, name, path, 0) - return DocFileCase(test, **options) - -def DocFileSuite(*paths, **kw): - """A unittest suite for one or more doctest files. - - The path to each doctest file is given as a string; the - interpretation of that string depends on the keyword argument - "module_relative". - - A number of options may be provided as keyword arguments: - - module_relative - If "module_relative" is True, then the given file paths are - interpreted as os-independent module-relative paths. By - default, these paths are relative to the calling module's - directory; but if the "package" argument is specified, then - they are relative to that package. To ensure os-independence, - "filename" should use "/" characters to separate path - segments, and may not be an absolute path (i.e., it may not - begin with "/"). - - If "module_relative" is False, then the given file paths are - interpreted as os-specific paths. These paths may be absolute - or relative (to the current working directory). - - package - A Python package or the name of a Python package whose directory - should be used as the base directory for module relative paths. - If "package" is not specified, then the calling module's - directory is used as the base directory for module relative - filenames. It is an error to specify "package" if - "module_relative" is False. - - setUp - A set-up function. This is called before running the - tests in each file. The setUp function will be passed a DocTest - object. The setUp function can access the test globals as the - globs attribute of the test passed. - - tearDown - A tear-down function. This is called after running the - tests in each file. The tearDown function will be passed a DocTest - object. The tearDown function can access the test globals as the - globs attribute of the test passed. - - globs - A dictionary containing initial global variables for the tests. - - optionflags - A set of doctest option flags expressed as an integer. - - parser - A DocTestParser (or subclass) that should be used to extract - tests from the files. - """ - suite = unittest.TestSuite() - - # We do this here so that _normalize_module is called at the right - # level. If it were called in DocFileTest, then this function - # would be the caller and we might guess the package incorrectly. - if kw.get('module_relative', True): - kw['package'] = _normalize_module(kw.get('package')) - - for path in paths: - suite.addTest(DocFileTest(path, **kw)) - - return suite - -###################################################################### -## 9. Debugging Support -###################################################################### - -def script_from_examples(s): - r"""Extract script from text with examples. - - Converts text with examples to a Python script. Example input is - converted to regular code. Example output and all other words - are converted to comments: - - >>> text = ''' - ... Here are examples of simple math. - ... - ... Python has super accurate integer addition - ... - ... >>> 2 + 2 - ... 5 - ... - ... And very friendly error messages: - ... - ... >>> 1/0 - ... To Infinity - ... And - ... Beyond - ... - ... You can use logic if you want: - ... - ... >>> if 0: - ... ... blah - ... ... blah - ... ... - ... - ... Ho hum - ... ''' - - >>> print script_from_examples(text) - # Here are examples of simple math. - # - # Python has super accurate integer addition - # - 2 + 2 - # Expected: - ## 5 - # - # And very friendly error messages: - # - 1/0 - # Expected: - ## To Infinity - ## And - ## Beyond - # - # You can use logic if you want: - # - if 0: - blah - blah - # - # Ho hum - """ - output = [] - for piece in DocTestParser().parse(s): - if isinstance(piece, Example): - # Add the example's source code (strip trailing NL) - output.append(piece.source[:-1]) - # Add the expected output: - want = piece.want - if want: - output.append('# Expected:') - output += ['## '+l for l in want.split('\n')[:-1]] - else: - # Add non-example text. - output += [_comment_line(l) - for l in piece.split('\n')[:-1]] - - # Trim junk on both ends. - while output and output[-1] == '#': - output.pop() - while output and output[0] == '#': - output.pop(0) - # Combine the output, and return it. - return '\n'.join(output) - -def testsource(module, name): - """Extract the test sources from a doctest docstring as a script. - - Provide the module (or dotted name of the module) containing the - test to be debugged and the name (within the module) of the object - with the doc string with tests to be debugged. - """ - module = _normalize_module(module) - tests = DocTestFinder().find(module) - test = [t for t in tests if t.name == name] - if not test: - raise ValueError(name, "not found in tests") - test = test[0] - testsrc = script_from_examples(test.docstring) - return testsrc - -def debug_src(src, pm=False, globs=None): - """Debug a single doctest docstring, in argument `src`'""" - testsrc = script_from_examples(src) - debug_script(testsrc, pm, globs) - -def debug_script(src, pm=False, globs=None): - "Debug a test script. `src` is the script, as a string." - import pdb - - # Note that tempfile.NameTemporaryFile() cannot be used. As the - # docs say, a file so created cannot be opened by name a second time - # on modern Windows boxes, and execfile() needs to open it. - srcfilename = tempfile.mktemp(".py", "doctestdebug") - f = open(srcfilename, 'w') - f.write(src) - f.close() - - try: - if globs: - globs = globs.copy() - else: - globs = {} - - if pm: - try: - execfile(srcfilename, globs, globs) - except: - print sys.exc_info()[1] - pdb.post_mortem(sys.exc_info()[2]) - else: - # Note that %r is vital here. '%s' instead can, e.g., cause - # backslashes to get treated as metacharacters on Windows. - pdb.run("execfile(%r)" % srcfilename, globs, globs) - - finally: - os.remove(srcfilename) - -def debug(module, name, pm=False): - """Debug a single doctest docstring. - - Provide the module (or dotted name of the module) containing the - test to be debugged and the name (within the module) of the object - with the docstring with tests to be debugged. - """ - module = _normalize_module(module) - testsrc = testsource(module, name) - debug_script(testsrc, pm, module.__dict__) - -###################################################################### -## 10. Example Usage -###################################################################### -class _TestClass: - """ - A pointless class, for sanity-checking of docstring testing. - - Methods: - square() - get() - - >>> _TestClass(13).get() + _TestClass(-12).get() - 1 - >>> hex(_TestClass(13).square().get()) - '0xa9' - """ - - def __init__(self, val): - """val -> _TestClass object with associated value val. - - >>> t = _TestClass(123) - >>> print t.get() - 123 - """ - - self.val = val - - def square(self): - """square() -> square TestClass's associated value - - >>> _TestClass(13).square().get() - 169 - """ - - self.val = self.val ** 2 - return self - - def get(self): - """get() -> return TestClass's associated value. - - >>> x = _TestClass(-42) - >>> print x.get() - -42 - """ - - return self.val - -__test__ = {"_TestClass": _TestClass, - "string": r""" - Example of a string object, searched as-is. - >>> x = 1; y = 2 - >>> x + y, x * y - (3, 2) - """, - - "bool-int equivalence": r""" - In 2.2, boolean expressions displayed - 0 or 1. By default, we still accept - them. This can be disabled by passing - DONT_ACCEPT_TRUE_FOR_1 to the new - optionflags argument. - >>> 4 == 4 - 1 - >>> 4 == 4 - True - >>> 4 > 4 - 0 - >>> 4 > 4 - False - """, - - "blank lines": r""" - Blank lines can be marked with : - >>> print 'foo\n\nbar\n' - foo - - bar - - """, - - "ellipsis": r""" - If the ellipsis flag is used, then '...' can be used to - elide substrings in the desired output: - >>> print range(1000) #doctest: +ELLIPSIS - [0, 1, 2, ..., 999] - """, - - "whitespace normalization": r""" - If the whitespace normalization flag is used, then - differences in whitespace are ignored. - >>> print range(30) #doctest: +NORMALIZE_WHITESPACE - [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, - 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, - 27, 28, 29] - """, - } - -def _test(): - r = unittest.TextTestRunner() - r.run(DocTestSuite()) - -if __name__ == "__main__": - _test() - From python-checkins at python.org Tue Apr 18 05:08:40 2006 From: python-checkins at python.org (phillip.eby) Date: Tue, 18 Apr 2006 05:08:40 +0200 (CEST) Subject: [Python-checkins] r45515 - in sandbox/trunk/setuptools: api_tests.txt setuptools/tests/__init__.py setuptools/tests/api_tests.txt Message-ID: <20060418030840.6FB0E1E4003@bag.python.org> Author: phillip.eby Date: Tue Apr 18 05:08:38 2006 New Revision: 45515 Added: sandbox/trunk/setuptools/setuptools/tests/api_tests.txt - copied unchanged from r45281, sandbox/trunk/setuptools/api_tests.txt Removed: sandbox/trunk/setuptools/api_tests.txt Modified: sandbox/trunk/setuptools/setuptools/tests/__init__.py Log: Move api_tests.txt file to tests subpackage for easier integration w/stdlib test suites. Deleted: /sandbox/trunk/setuptools/api_tests.txt ============================================================================== --- /sandbox/trunk/setuptools/api_tests.txt Tue Apr 18 05:08:38 2006 +++ (empty file) @@ -1,330 +0,0 @@ -Pluggable Distributions of Python Software -========================================== - -Distributions -------------- - -A "Distribution" is a collection of files that represent a "Release" of a -"Project" as of a particular point in time, denoted by a -"Version":: - - >>> import sys, pkg_resources - >>> from pkg_resources import Distribution - >>> Distribution(project_name="Foo", version="1.2") - Foo 1.2 - -Distributions have a location, which can be a filename, URL, or really anything -else you care to use:: - - >>> dist = Distribution( - ... location="http://example.com/something", - ... project_name="Bar", version="0.9" - ... ) - - >>> dist - Bar 0.9 (http://example.com/something) - - -Distributions have various introspectable attributes:: - - >>> dist.location - 'http://example.com/something' - - >>> dist.project_name - 'Bar' - - >>> dist.version - '0.9' - - >>> dist.py_version == sys.version[:3] - True - - >>> print dist.platform - None - -Including various computed attributes:: - - >>> from pkg_resources import parse_version - >>> dist.parsed_version == parse_version(dist.version) - True - - >>> dist.key # case-insensitive form of the project name - 'bar' - -Distributions are compared (and hashed) by version first:: - - >>> Distribution(version='1.0') == Distribution(version='1.0') - True - >>> Distribution(version='1.0') == Distribution(version='1.1') - False - >>> Distribution(version='1.0') < Distribution(version='1.1') - True - -but also by project name (case-insensitive), platform, Python version, -location, etc.:: - - >>> Distribution(project_name="Foo",version="1.0") == \ - ... Distribution(project_name="Foo",version="1.0") - True - - >>> Distribution(project_name="Foo",version="1.0") == \ - ... Distribution(project_name="foo",version="1.0") - True - - >>> Distribution(project_name="Foo",version="1.0") == \ - ... Distribution(project_name="Foo",version="1.1") - False - - >>> Distribution(project_name="Foo",py_version="2.3",version="1.0") == \ - ... Distribution(project_name="Foo",py_version="2.4",version="1.0") - False - - >>> Distribution(location="spam",version="1.0") == \ - ... Distribution(location="spam",version="1.0") - True - - >>> Distribution(location="spam",version="1.0") == \ - ... Distribution(location="baz",version="1.0") - False - - - -Hash and compare distribution by prio/plat - -Get version from metadata -provider capabilities -egg_name() -as_requirement() -from_location, from_filename (w/path normalization) - -Releases may have zero or more "Requirements", which indicate -what releases of another project the release requires in order to -function. A Requirement names the other project, expresses some criteria -as to what releases of that project are acceptable, and lists any "Extras" -that the requiring release may need from that project. (An Extra is an -optional feature of a Release, that can only be used if its additional -Requirements are satisfied.) - - - -The Working Set ---------------- - -A collection of active distributions is called a Working Set. Note that a -Working Set can contain any importable distribution, not just pluggable ones. -For example, the Python standard library is an importable distribution that -will usually be part of the Working Set, even though it is not pluggable. -Similarly, when you are doing development work on a project, the files you are -editing are also a Distribution. (And, with a little attention to the -directory names used, and including some additional metadata, such a -"development distribution" can be made pluggable as well.) - - >>> from pkg_resources import WorkingSet - -A working set's entries are the sys.path entries that correspond to the active -distributions. By default, the working set's entries are the items on -``sys.path``:: - - >>> ws = WorkingSet() - >>> ws.entries == sys.path - True - -But you can also create an empty working set explicitly, and add distributions -to it:: - - >>> ws = WorkingSet([]) - >>> ws.add(dist) - >>> ws.entries - ['http://example.com/something'] - >>> dist in ws - True - >>> Distribution('foo',version="") in ws - False - -And you can iterate over its distributions:: - - >>> list(ws) - [Bar 0.9 (http://example.com/something)] - -Adding the same distribution more than once is a no-op:: - - >>> ws.add(dist) - >>> list(ws) - [Bar 0.9 (http://example.com/something)] - -For that matter, adding multiple distributions for the same project also does -nothing, because a working set can only hold one active distribution per -project -- the first one added to it:: - - >>> ws.add( - ... Distribution( - ... 'http://example.com/something', project_name="Bar", - ... version="7.2" - ... ) - ... ) - >>> list(ws) - [Bar 0.9 (http://example.com/something)] - -You can append a path entry to a working set using ``add_entry()``:: - - >>> ws.entries - ['http://example.com/something'] - >>> ws.add_entry(pkg_resources.__file__) - >>> ws.entries - ['http://example.com/something', '...pkg_resources.py...'] - -Multiple additions result in multiple entries, even if the entry is already in -the working set (because ``sys.path`` can contain the same entry more than -once):: - - >>> ws.add_entry(pkg_resources.__file__) - >>> ws.entries - ['...example.com...', '...pkg_resources...', '...pkg_resources...'] - -And you can specify the path entry a distribution was found under, using the -optional second parameter to ``add()``:: - - >>> ws = WorkingSet([]) - >>> ws.add(dist,"foo") - >>> ws.entries - ['foo'] - -But even if a distribution is found under multiple path entries, it still only -shows up once when iterating the working set: - - >>> ws.add_entry(ws.entries[0]) - >>> list(ws) - [Bar 0.9 (http://example.com/something)] - -You can ask a WorkingSet to ``find()`` a distribution matching a requirement:: - - >>> from pkg_resources import Requirement - >>> print ws.find(Requirement.parse("Foo==1.0")) # no match, return None - None - - >>> ws.find(Requirement.parse("Bar==0.9")) # match, return distribution - Bar 0.9 (http://example.com/something) - -Note that asking for a conflicting version of a distribution already in a -working set triggers a ``pkg_resources.VersionConflict`` error: - - >>> ws.find(Requirement.parse("Bar==1.0")) # doctest: +NORMALIZE_WHITESPACE - Traceback (most recent call last): - ... - VersionConflict: (Bar 0.9 (http://example.com/something), - Requirement.parse('Bar==1.0')) - -You can subscribe a callback function to receive notifications whenever a new -distribution is added to a working set. The callback is immediately invoked -once for each existing distribution in the working set, and then is called -again for new distributions added thereafter:: - - >>> def added(dist): print "Added", dist - >>> ws.subscribe(added) - Added Bar 0.9 - >>> foo12 = Distribution(project_name="Foo", version="1.2", location="f12") - >>> ws.add(foo12) - Added Foo 1.2 - -Note, however, that only the first distribution added for a given project name -will trigger a callback, even during the initial ``subscribe()`` callback:: - - >>> foo14 = Distribution(project_name="Foo", version="1.4", location="f14") - >>> ws.add(foo14) # no callback, because Foo 1.2 is already active - - >>> ws = WorkingSet([]) - >>> ws.add(foo12) - >>> ws.add(foo14) - >>> ws.subscribe(added) - Added Foo 1.2 - -And adding a callback more than once has no effect, either:: - - >>> ws.subscribe(added) # no callbacks - - # and no double-callbacks on subsequent additions, either - >>> just_a_test = Distribution(project_name="JustATest", version="0.99") - >>> ws.add(just_a_test) - Added JustATest 0.99 - - -Finding Plugins ---------------- - -``WorkingSet`` objects can be used to figure out what plugins in an -``Environment`` can be loaded without any resolution errors:: - - >>> from pkg_resources import Environment - - >>> plugins = Environment([]) # normally, a list of plugin directories - >>> plugins.add(foo12) - >>> plugins.add(foo14) - >>> plugins.add(just_a_test) - -In the simplest case, we just get the newest version of each distribution in -the plugin environment:: - - >>> ws = WorkingSet([]) - >>> ws.find_plugins(plugins) - ([JustATest 0.99, Foo 1.4 (f14)], {}) - -But if there's a problem with a version conflict or missing requirements, the -method falls back to older versions, and the error info dict will contain an -exception instance for each unloadable plugin:: - - >>> ws.add(foo12) # this will conflict with Foo 1.4 - >>> ws.find_plugins(plugins) - ([JustATest 0.99, Foo 1.2 (f12)], {Foo 1.4 (f14): <...VersionConflict...>}) - -But if you disallow fallbacks, the failed plugin will be skipped instead of -trying older versions:: - - >>> ws.find_plugins(plugins, fallback=False) - ([JustATest 0.99], {Foo 1.4 (f14): <...VersionConflict...>}) - - - -Platform Compatibility Rules ----------------------------- - -On the Mac, there are potential compatibility issues for modules compiled -on newer versions of Mac OS X than what the user is running. Additionally, -Mac OS X will soon have two platforms to contend with: Intel and PowerPC. - -Basic equality works as on other platforms:: - - >>> from pkg_resources import compatible_platforms as cp - >>> reqd = 'macosx-10.4-ppc' - >>> cp(reqd, reqd) - True - >>> cp("win32", reqd) - False - -Distributions made on other machine types are not compatible:: - - >>> cp("macosx-10.4-i386", reqd) - False - -Distributions made on earlier versions of the OS are compatible, as -long as they are from the same top-level version. The patchlevel version -number does not matter:: - - >>> cp("macosx-10.4-ppc", reqd) - True - >>> cp("macosx-10.3-ppc", reqd) - True - >>> cp("macosx-10.5-ppc", reqd) - False - >>> cp("macosx-9.5-ppc", reqd) - False - -Backwards compatibility for packages made via earlier versions of -setuptools is provided as well:: - - >>> cp("darwin-8.2.0-Power_Macintosh", reqd) - True - >>> cp("darwin-7.2.0-Power_Macintosh", reqd) - True - >>> cp("darwin-8.2.0-Power_Macintosh", "macosx-10.3-ppc") - False - Modified: sandbox/trunk/setuptools/setuptools/tests/__init__.py ============================================================================== --- sandbox/trunk/setuptools/setuptools/tests/__init__.py (original) +++ sandbox/trunk/setuptools/setuptools/tests/__init__.py Tue Apr 18 05:08:38 2006 @@ -16,7 +16,7 @@ def additional_tests(): import doctest return doctest.DocFileSuite( - 'api_tests.txt', optionflags=doctest.ELLIPSIS, package='pkg_resources', + 'api_tests.txt', optionflags=doctest.ELLIPSIS, package=__name__, ) From python-checkins at python.org Tue Apr 18 05:28:33 2006 From: python-checkins at python.org (tim.peters) Date: Tue, 18 Apr 2006 05:28:33 +0200 (CEST) Subject: [Python-checkins] r45516 - python/trunk/Lib/_threading_local.py Message-ID: <20060418032833.9BB061E4003@bag.python.org> Author: tim.peters Date: Tue Apr 18 05:28:32 2006 New Revision: 45516 Modified: python/trunk/Lib/_threading_local.py Log: Finally figured out why this module did its imports at the bottom of the file. Restored that, and added a comment explaining why this is necessary. Hint: on my box, and yours, it's not :-( Also added an __all__ list. Modified: python/trunk/Lib/_threading_local.py ============================================================================== --- python/trunk/Lib/_threading_local.py (original) +++ python/trunk/Lib/_threading_local.py Tue Apr 18 05:28:32 2006 @@ -1,9 +1,9 @@ -"""Thread-local objects +"""Thread-local objects. -(Note that this module provides a Python version of thread - threading.local class. Depending on the version of Python you're - using, there may be a faster one available. You should always import - the local class from threading.) +(Note that this module provides a Python version of the threading.local + class. Depending on the version of Python you're using, there may be a + faster one available. You should always import the `local` class from + `threading`.) Thread-local objects support the management of thread-local data. If you have data that you want to be local to a thread, simply create @@ -133,7 +133,17 @@ >>> del mydata """ -from threading import currentThread, RLock, enumerate +__all__ = ["local"] + +# We need to use objects from the threading module, but the threading +# module may also want to use our `local` class, if support for locals +# isn't compiled in to the `thread` module. This creates potential problems +# with circular imports. For that reason, we don't import `threading` +# until the bottom of this file (a hack sufficient to worm around the +# potential problems). Note that almost all platforms do have support for +# locals in the `thread` module, and there is no circular import problem +# then, so problems introduced by fiddling the order of imports here won't +# manifest on most boxes. class _localbase(object): __slots__ = '_local__key', '_local__args', '_local__lock' @@ -202,16 +212,13 @@ finally: lock.release() - - # The default argument is a hack, to give __del__ a local name for - # threading.enumerate (sidestepping problems with Python None'ing-out - # module globals at shutdown time). - def __del__(self, _threading_enumerate=enumerate): + def __del__(self): + import threading key = object.__getattribute__(self, '_local__key') try: - threads = list(_threading_enumerate()) + threads = list(threading.enumerate()) except: # If enumerate fails, as it seems to do during # shutdown, we'll skip cleanup under the assumption @@ -230,3 +237,5 @@ del __dict__[key] except KeyError: pass # didn't have anything in this thread + +from threading import currentThread, RLock From python-checkins at python.org Tue Apr 18 05:29:11 2006 From: python-checkins at python.org (phillip.eby) Date: Tue, 18 Apr 2006 05:29:11 +0200 (CEST) Subject: [Python-checkins] r45517 - in sandbox/trunk/setuptools: pkg_resources.py setuptools/tests/api_tests.txt Message-ID: <20060418032911.264711E4003@bag.python.org> Author: phillip.eby Date: Tue Apr 18 05:29:10 2006 New Revision: 45517 Modified: sandbox/trunk/setuptools/pkg_resources.py sandbox/trunk/setuptools/setuptools/tests/api_tests.txt Log: Make errors __repr__ the same as the default exception repr in Python 2.5, so doctests will work right. :( Modified: sandbox/trunk/setuptools/pkg_resources.py ============================================================================== --- sandbox/trunk/setuptools/pkg_resources.py (original) +++ sandbox/trunk/setuptools/pkg_resources.py Tue Apr 18 05:29:10 2006 @@ -82,6 +82,8 @@ ] class ResolutionError(Exception): """Abstract base for dependency resolution errors""" + def __repr__(self): + return self.__class__.__name__+repr(self.args) class VersionConflict(ResolutionError): """An already-installed version conflicts with the requested version""" @@ -160,8 +162,6 @@ darwinVersionString = re.compile(r"darwin-(\d+)\.(\d+)\.(\d+)-(.*)") - - def compatible_platforms(provided,required): """Can code for the `provided` platform run on the `required` platform? Modified: sandbox/trunk/setuptools/setuptools/tests/api_tests.txt ============================================================================== --- sandbox/trunk/setuptools/setuptools/tests/api_tests.txt (original) +++ sandbox/trunk/setuptools/setuptools/tests/api_tests.txt Tue Apr 18 05:29:10 2006 @@ -274,13 +274,13 @@ >>> ws.add(foo12) # this will conflict with Foo 1.4 >>> ws.find_plugins(plugins) - ([JustATest 0.99, Foo 1.2 (f12)], {Foo 1.4 (f14): <...VersionConflict...>}) + ([JustATest 0.99, Foo 1.2 (f12)], {Foo 1.4 (f14): VersionConflict(...)}) But if you disallow fallbacks, the failed plugin will be skipped instead of trying older versions:: >>> ws.find_plugins(plugins, fallback=False) - ([JustATest 0.99], {Foo 1.4 (f14): <...VersionConflict...>}) + ([JustATest 0.99], {Foo 1.4 (f14): VersionConflict(...)}) From python-checkins at python.org Tue Apr 18 05:30:43 2006 From: python-checkins at python.org (tim.peters) Date: Tue, 18 Apr 2006 05:30:43 +0200 (CEST) Subject: [Python-checkins] r45518 - python/branches/release24-maint/Lib/_threading_local.py Message-ID: <20060418033043.F10D91E4017@bag.python.org> Author: tim.peters Date: Tue Apr 18 05:30:43 2006 New Revision: 45518 Modified: python/branches/release24-maint/Lib/_threading_local.py Log: Merge rev 45516 from trunk. Finally figured out why this module did its imports at the bottom of the file. Restored that, and added a comment explaining why this is necessary. Hint: on my box, and yours, it's not :-( Also added an __all__ list. Modified: python/branches/release24-maint/Lib/_threading_local.py ============================================================================== --- python/branches/release24-maint/Lib/_threading_local.py (original) +++ python/branches/release24-maint/Lib/_threading_local.py Tue Apr 18 05:30:43 2006 @@ -1,9 +1,9 @@ -"""Thread-local objects +"""Thread-local objects. -(Note that this module provides a Python version of thread - threading.local class. Depending on the version of Python you're - using, there may be a faster one available. You should always import - the local class from threading.) +(Note that this module provides a Python version of the threading.local + class. Depending on the version of Python you're using, there may be a + faster one available. You should always import the `local` class from + `threading`.) Thread-local objects support the management of thread-local data. If you have data that you want to be local to a thread, simply create @@ -133,7 +133,17 @@ >>> del mydata """ -from threading import currentThread, RLock, enumerate +__all__ = ["local"] + +# We need to use objects from the threading module, but the threading +# module may also want to use our `local` class, if support for locals +# isn't compiled in to the `thread` module. This creates potential problems +# with circular imports. For that reason, we don't import `threading` +# until the bottom of this file (a hack sufficient to worm around the +# potential problems). Note that almost all platforms do have support for +# locals in the `thread` module, and there is no circular import problem +# then, so problems introduced by fiddling the order of imports here won't +# manifest on most boxes. class _localbase(object): __slots__ = '_local__key', '_local__args', '_local__lock' @@ -202,16 +212,13 @@ finally: lock.release() - - # The default argument is a hack, to give __del__ a local name for - # threading.enumerate (sidestepping problems with Python None'ing-out - # module globals at shutdown time). - def __del__(self, _threading_enumerate=enumerate): + def __del__(self): + import threading key = object.__getattribute__(self, '_local__key') try: - threads = list(_threading_enumerate()) + threads = list(threading.enumerate()) except: # If enumerate fails, as it seems to do during # shutdown, we'll skip cleanup under the assumption @@ -230,3 +237,5 @@ del __dict__[key] except KeyError: pass # didn't have anything in this thread + +from threading import currentThread, RLock From python-checkins at python.org Tue Apr 18 05:40:52 2006 From: python-checkins at python.org (phillip.eby) Date: Tue, 18 Apr 2006 05:40:52 +0200 (CEST) Subject: [Python-checkins] r45519 - in sandbox/trunk/setuptools/setuptools: dist.py tests/__init__.py Message-ID: <20060418034052.1AB291E4003@bag.python.org> Author: phillip.eby Date: Tue Apr 18 05:40:51 2006 New Revision: 45519 Modified: sandbox/trunk/setuptools/setuptools/dist.py sandbox/trunk/setuptools/setuptools/tests/__init__.py Log: Changes to the experimental 'Feature' feature to be compatible with Python 2.5 distutils, so the tests don't bomb under 2.5 Modified: sandbox/trunk/setuptools/setuptools/dist.py ============================================================================== --- sandbox/trunk/setuptools/setuptools/dist.py (original) +++ sandbox/trunk/setuptools/setuptools/dist.py Tue Apr 18 05:40:51 2006 @@ -1,4 +1,4 @@ -__all__ = ['Distribution', 'Feature'] +__all__ = ['Distribution'] from distutils.core import Distribution as _Distribution from setuptools.depends import Require @@ -207,7 +207,7 @@ have_package_data = hasattr(self, "package_data") if not have_package_data: self.package_data = {} - self.requires = [] # XXX + self.require_features = [] self.features = {} self.dist_files = [] self.patch_missing_pkg_info(attrs) @@ -676,10 +676,10 @@ based on 'availabile', 'standard', and whether any other feature requires it. The default setting is 'True'. - 'requires' -- a string or sequence of strings naming features that should - also be included if this feature is included. Defaults to empty list. - May also contain 'Require' objects that should be added/removed from - the distribution. + 'require_features' -- a string or sequence of strings naming features + that should also be included if this feature is included. Defaults to + empty list. May also contain 'Require' objects that should be + added/removed from the distribution. 'remove' -- a string or list of strings naming packages to be removed from the distribution if this feature is *not* included. If the @@ -704,34 +704,34 @@ Aside from the methods, the only feature attributes that distributions look at are 'description' and 'optional'. """ - def __init__(self, description, standard=False, available=True, - optional=True, requires=(), remove=(), **extras + optional=True, require_features=(), remove=(), **extras ): self.description = description self.standard = standard self.available = available self.optional = optional - if isinstance(requires,(str,Require)): - requires = requires, + if isinstance(require_features,(str,Require)): + require_features = require_features, - self.requires = [r for r in requires if isinstance(r,str)] - er = [r for r in requires if not isinstance(r,str)] - if er: extras['requires'] = er + self.require_features = [ + r for r in require_features if isinstance(r,str) + ] + er = [r for r in require_features if not isinstance(r,str)] + if er: extras['require_features'] = er if isinstance(remove,str): remove = remove, self.remove = remove self.extras = extras - if not remove and not requires and not extras: + if not remove and not require_features and not extras: raise DistutilsSetupError( - "Feature %s: must define 'requires', 'remove', or at least one" + "Feature %s: must define 'require_features', 'remove', or at least one" " of 'packages', 'py_modules', etc." ) - def include_by_default(self): """Should this feature be included by default?""" return self.available and self.standard @@ -754,7 +754,7 @@ dist.include(**self.extras) - for f in self.requires: + for f in self.require_features: dist.include_feature(f) Modified: sandbox/trunk/setuptools/setuptools/tests/__init__.py ============================================================================== --- sandbox/trunk/setuptools/setuptools/tests/__init__.py (original) +++ sandbox/trunk/setuptools/setuptools/tests/__init__.py Tue Apr 18 05:40:51 2006 @@ -250,7 +250,7 @@ self.req = Require('Distutils','1.0.3','distutils') self.dist = makeSetup( features={ - 'foo': Feature("foo",standard=True,requires=['baz',self.req]), + 'foo': Feature("foo",standard=True,require_features=['baz',self.req]), 'bar': Feature("bar", standard=True, packages=['pkg.bar'], py_modules=['bar_et'], remove=['bar.ext'], ), @@ -276,7 +276,7 @@ self.failUnless( Feature("test",standard=True,remove='x').include_by_default() ) - # Feature must have either kwargs, removes, or requires + # Feature must have either kwargs, removes, or require_features self.assertRaises(DistutilsSetupError, Feature, "test") def testAvailability(self): @@ -315,7 +315,7 @@ self.failUnless('scripts/baz_it' in dist.scripts) self.failUnless(('libfoo','foo/foofoo.c') in dist.libraries) self.assertEqual(dist.ext_modules,[]) - self.assertEqual(dist.requires, [self.req]) + self.assertEqual(dist.require_features, [self.req]) # If we ask for bar, it should fail because we explicitly disabled # it on the command line From buildbot at python.org Tue Apr 18 05:46:47 2006 From: buildbot at python.org (buildbot at python.org) Date: Tue, 18 Apr 2006 03:46:47 +0000 Subject: [Python-checkins] buildbot warnings in sparc solaris10 gcc trunk Message-ID: <20060418034647.A6E881E4003@bag.python.org> The Buildbot has detected a new failure of sparc solaris10 gcc trunk. Full details are available at: http://www.python.org/dev/buildbot/all/sparc%2520solaris10%2520gcc%2520trunk/builds/471 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: tim.peters Build Had Warnings: warnings test sincerely, -The Buildbot From buildbot at python.org Tue Apr 18 05:51:37 2006 From: buildbot at python.org (buildbot at python.org) Date: Tue, 18 Apr 2006 03:51:37 +0000 Subject: [Python-checkins] buildbot warnings in x86 OpenBSD trunk Message-ID: <20060418035138.1638C1E4003@bag.python.org> The Buildbot has detected a new failure of x86 OpenBSD trunk. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520OpenBSD%2520trunk/builds/361 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: tim.peters Build Had Warnings: warnings test sincerely, -The Buildbot From buildbot at python.org Tue Apr 18 05:54:14 2006 From: buildbot at python.org (buildbot at python.org) Date: Tue, 18 Apr 2006 03:54:14 +0000 Subject: [Python-checkins] buildbot warnings in x86 gentoo trunk Message-ID: <20060418035414.3D02B1E4003@bag.python.org> The Buildbot has detected a new failure of x86 gentoo trunk. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520gentoo%2520trunk/builds/529 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: tim.peters Build Had Warnings: warnings test sincerely, -The Buildbot From buildbot at python.org Tue Apr 18 06:04:31 2006 From: buildbot at python.org (buildbot at python.org) Date: Tue, 18 Apr 2006 04:04:31 +0000 Subject: [Python-checkins] buildbot warnings in x86 XP 2.4 Message-ID: <20060418040431.4B0381E400A@bag.python.org> The Buildbot has detected a new failure of x86 XP 2.4. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520XP%25202.4/builds/89 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch branches/release24-maint] HEAD Blamelist: tim.peters Build Had Warnings: warnings test sincerely, -The Buildbot From python-checkins at python.org Tue Apr 18 06:05:35 2006 From: python-checkins at python.org (phillip.eby) Date: Tue, 18 Apr 2006 06:05:35 +0200 (CEST) Subject: [Python-checkins] r45520 - in python/trunk/Lib: easy_install.py pkg_resources.py setuptools setuptools.egg-info setuptools.egg-info/PKG-INFO setuptools.egg-info/entry_points.txt setuptools.egg-info/top_level.txt setuptools.egg-info/zip-safe test/test_setuptools.py Message-ID: <20060418040535.6168A1E4003@bag.python.org> Author: phillip.eby Date: Tue Apr 18 06:05:34 2006 New Revision: 45520 Added: python/trunk/Lib/easy_install.py - copied unchanged from r45519, sandbox/trunk/setuptools/easy_install.py python/trunk/Lib/pkg_resources.py - copied unchanged from r45517, sandbox/trunk/setuptools/pkg_resources.py python/trunk/Lib/setuptools/ - copied from r45519, sandbox/trunk/setuptools/setuptools/ python/trunk/Lib/setuptools.egg-info/ - copied from r45515, sandbox/trunk/setuptools/setuptools.egg-info/ python/trunk/Lib/setuptools.egg-info/PKG-INFO python/trunk/Lib/setuptools.egg-info/top_level.txt (contents, props changed) python/trunk/Lib/setuptools.egg-info/zip-safe python/trunk/Lib/test/test_setuptools.py (contents, props changed) Modified: python/trunk/Lib/setuptools.egg-info/entry_points.txt Log: Initial import of setuptools, with integrated tests. Added: python/trunk/Lib/setuptools.egg-info/PKG-INFO ============================================================================== --- (empty file) +++ python/trunk/Lib/setuptools.egg-info/PKG-INFO Tue Apr 18 06:05:34 2006 @@ -0,0 +1,89 @@ +Metadata-Version: 1.0 +Name: setuptools +Version: 0.7a1dev-r45519 +Summary: Download, build, install, upgrade, and uninstall Python packages -- easily! +Home-page: http://peak.telecommunity.com/DevCenter/setuptools +Author: Phillip J. Eby +Author-email: peak at eby-sarna.com +License: PSF or ZPL +Description: ``setuptools`` is a collection of enhancements to the Python ``distutils`` + (for Python 2.3.5 and up on most platforms; 64-bit platforms require a minimum + of Python 2.4) that allow you to more easily build and distribute Python + packages, especially ones that have dependencies on other packages. + + Packages built and distributed using ``setuptools`` look to the user like + ordinary Python packages based on the ``distutils``. Your users don't need to + install or even know about setuptools in order to use them, and you don't + have to include the entire setuptools package in your distributions. By + including just a single `bootstrap module`_ (an 8K .py file), your package will + automatically download and install ``setuptools`` if the user is building your + package from source and doesn't have a suitable version already installed. + + .. _bootstrap module: http://peak.telecommunity.com/dist/ez_setup.py + + Feature Highlights: + + * Automatically find/download/install/upgrade dependencies at build time using + the `EasyInstall tool `_, + which supports downloading via HTTP, FTP, Subversion, and SourceForge, and + automatically scans web pages linked from PyPI to find download links. (It's + the closest thing to CPAN currently available for Python.) + + * Create `Python Eggs `_ - + a single-file importable distribution format + + * Include data files inside your package directories, where your code can + actually use them. (Python 2.4 distutils also supports this feature, but + setuptools provides the feature for Python 2.3 packages also, and supports + accessing data files in zipped packages too.) + + * Automatically include all packages in your source tree, without listing them + individually in setup.py + + * Automatically include all relevant files in your source distributions, + without needing to create a ``MANIFEST.in`` file, and without having to force + regeneration of the ``MANIFEST`` file when your source tree changes. + + * Automatically generate wrapper scripts or Windows (console and GUI) .exe + files for any number of "main" functions in your project. (Note: this is not + a py2exe replacement; the .exe files rely on the local Python installation.) + + * Transparent Pyrex support, so that your setup.py can list ``.pyx`` files and + still work even when the end-user doesn't have Pyrex installed (as long as + you include the Pyrex-generated C in your source distribution) + + * Command aliases - create project-specific, per-user, or site-wide shortcut + names for commonly used commands and options + + * PyPI upload support - upload your source distributions and eggs to PyPI + + * Deploy your project in "development mode", such that it's available on + ``sys.path``, yet can still be edited directly from its source checkout. + + * Easily extend the distutils with new commands or ``setup()`` arguments, and + distribute/reuse your extensions for multiple projects, without copying code. + + * Create extensible applications and frameworks that automatically discover + extensions, using simple "entry points" declared in a project's setup script. + + In addition to the PyPI downloads, the development version of ``setuptools`` + is available from the `Python SVN sandbox`_, and in-development versions of the + `0.6 branch`_ are available as well. + + .. _0.6 branch: http://svn.python.org/projects/sandbox/branches/setuptools-0.6/#egg=setuptools-dev06 + + .. _Python SVN sandbox: http://svn.python.org/projects/sandbox/trunk/setuptools/#egg=setuptools-dev + + +Keywords: CPAN PyPI distutils eggs package management +Platform: UNKNOWN +Classifier: Development Status :: 3 - Alpha +Classifier: Intended Audience :: Developers +Classifier: License :: OSI Approved :: Python Software Foundation License +Classifier: License :: OSI Approved :: Zope Public License +Classifier: Operating System :: OS Independent +Classifier: Programming Language :: Python +Classifier: Topic :: Software Development :: Libraries :: Python Modules +Classifier: Topic :: System :: Archiving :: Packaging +Classifier: Topic :: System :: Systems Administration +Classifier: Topic :: Utilities Modified: python/trunk/Lib/setuptools.egg-info/entry_points.txt ============================================================================== --- sandbox/trunk/setuptools/setuptools.egg-info/entry_points.txt (original) +++ python/trunk/Lib/setuptools.egg-info/entry_points.txt Tue Apr 18 06:05:34 2006 @@ -28,7 +28,7 @@ [console_scripts] easy_install = setuptools.command.easy_install:main -easy_install-2.3 = setuptools.command.easy_install:main +easy_install-2.5 = setuptools.command.easy_install:main [distutils.commands] bdist_rpm = setuptools.command.bdist_rpm:bdist_rpm @@ -38,7 +38,6 @@ build_py = setuptools.command.build_py:build_py saveopts = setuptools.command.saveopts:saveopts egg_info = setuptools.command.egg_info:egg_info -upload = setuptools.command.upload:upload install_egg_info = setuptools.command.install_egg_info:install_egg_info alias = setuptools.command.alias:alias easy_install = setuptools.command.easy_install:easy_install Added: python/trunk/Lib/setuptools.egg-info/top_level.txt ============================================================================== --- (empty file) +++ python/trunk/Lib/setuptools.egg-info/top_level.txt Tue Apr 18 06:05:34 2006 @@ -0,0 +1,3 @@ +easy_install +pkg_resources +setuptools Added: python/trunk/Lib/setuptools.egg-info/zip-safe ============================================================================== Added: python/trunk/Lib/test/test_setuptools.py ============================================================================== --- (empty file) +++ python/trunk/Lib/test/test_setuptools.py Tue Apr 18 06:05:34 2006 @@ -0,0 +1,16 @@ +"""Tests for setuptools. + +The tests for setuptools are defined in the setuptools.tests package; +this runs them from there. +""" + +import test.test_support +from setuptools.command.test import ScanningLoader + +def test_main(): + test.test_support.run_suite( + ScanningLoader().loadTestsFromName('setuptools.tests') + ) + +if __name__ == "__main__": + test_main() From buildbot at python.org Tue Apr 18 06:08:46 2006 From: buildbot at python.org (buildbot at python.org) Date: Tue, 18 Apr 2006 04:08:46 +0000 Subject: [Python-checkins] buildbot warnings in x86 OpenBSD 2.4 Message-ID: <20060418040847.135B71E4003@bag.python.org> The Buildbot has detected a new failure of x86 OpenBSD 2.4. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520OpenBSD%25202.4/builds/63 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch branches/release24-maint] HEAD Blamelist: tim.peters Build Had Warnings: warnings test sincerely, -The Buildbot From buildbot at python.org Tue Apr 18 06:22:01 2006 From: buildbot at python.org (buildbot at python.org) Date: Tue, 18 Apr 2006 04:22:01 +0000 Subject: [Python-checkins] buildbot warnings in x86 Ubuntu dapper (icc) trunk Message-ID: <20060418042201.4F7A11E400C@bag.python.org> The Buildbot has detected a new failure of x86 Ubuntu dapper (icc) trunk. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520Ubuntu%2520dapper%2520%2528icc%2529%2520trunk/builds/181 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: tim.peters Build Had Warnings: warnings test sincerely, -The Buildbot From python-checkins at python.org Tue Apr 18 06:25:23 2006 From: python-checkins at python.org (phillip.eby) Date: Tue, 18 Apr 2006 06:25:23 +0200 (CEST) Subject: [Python-checkins] r45521 - sandbox/trunk/setuptools/setuptools/command/easy_install.py Message-ID: <20060418042523.74FBD1E4015@bag.python.org> Author: phillip.eby Date: Tue Apr 18 06:25:18 2006 New Revision: 45521 Modified: sandbox/trunk/setuptools/setuptools/command/easy_install.py Log: Handle being run via -m with no __file__ if done from a zipfile in Python 2.5. Modified: sandbox/trunk/setuptools/setuptools/command/easy_install.py ============================================================================== --- sandbox/trunk/setuptools/setuptools/command/easy_install.py (original) +++ sandbox/trunk/setuptools/setuptools/command/easy_install.py Tue Apr 18 06:25:18 2006 @@ -1549,6 +1549,7 @@ with_ei_usage(lambda: setup( script_args = ['-q','easy_install', '-v']+argv, + script_name = sys.argv[0] or 'easy_install', distclass=DistributionWithoutHelpCommands, **kw ) ) @@ -1557,4 +1558,3 @@ - From python-checkins at python.org Tue Apr 18 06:27:01 2006 From: python-checkins at python.org (neal.norwitz) Date: Tue, 18 Apr 2006 06:27:01 +0200 (CEST) Subject: [Python-checkins] r45522 - peps/trunk/pep-0291.txt Message-ID: <20060418042701.316141E4016@bag.python.org> Author: neal.norwitz Date: Tue Apr 18 06:26:58 2006 New Revision: 45522 Modified: peps/trunk/pep-0291.txt Log: pkgutil now requires b/w compat with 2.3 per checkin comment by phillip eby Modified: peps/trunk/pep-0291.txt ============================================================================== --- peps/trunk/pep-0291.txt (original) +++ peps/trunk/pep-0291.txt Tue Apr 18 06:26:58 2006 @@ -88,6 +88,7 @@ decimal Raymond Hettinger 2.3 [2] email Barry Warsaw 2.1 / 2.3 [1] logging Vinay Sajip 1.5.2 + pkgutil Phillip Eby 2.3 sre Fredrik Lundh 2.1 subprocess Peter Astrand 2.2 xml (PyXML) Martin v. Loewis 2.0 From python-checkins at python.org Tue Apr 18 06:31:47 2006 From: python-checkins at python.org (phillip.eby) Date: Tue, 18 Apr 2006 06:31:47 +0200 (CEST) Subject: [Python-checkins] r45523 - in python/trunk/Lib: setuptools.egg-info/PKG-INFO setuptools/command/easy_install.py Message-ID: <20060418043147.3FDB01E4003@bag.python.org> Author: phillip.eby Date: Tue Apr 18 06:31:46 2006 New Revision: 45523 Modified: python/trunk/Lib/setuptools.egg-info/PKG-INFO python/trunk/Lib/setuptools/command/easy_install.py Log: Handle easy_install being run via -m with no __file__ if done from a zipfile. Modified: python/trunk/Lib/setuptools.egg-info/PKG-INFO ============================================================================== --- python/trunk/Lib/setuptools.egg-info/PKG-INFO (original) +++ python/trunk/Lib/setuptools.egg-info/PKG-INFO Tue Apr 18 06:31:46 2006 @@ -1,6 +1,6 @@ Metadata-Version: 1.0 Name: setuptools -Version: 0.7a1dev-r45519 +Version: 0.7a1dev-r45521 Summary: Download, build, install, upgrade, and uninstall Python packages -- easily! Home-page: http://peak.telecommunity.com/DevCenter/setuptools Author: Phillip J. Eby Modified: python/trunk/Lib/setuptools/command/easy_install.py ============================================================================== --- python/trunk/Lib/setuptools/command/easy_install.py (original) +++ python/trunk/Lib/setuptools/command/easy_install.py Tue Apr 18 06:31:46 2006 @@ -1549,6 +1549,7 @@ with_ei_usage(lambda: setup( script_args = ['-q','easy_install', '-v']+argv, + script_name = sys.argv[0] or 'easy_install', distclass=DistributionWithoutHelpCommands, **kw ) ) @@ -1557,4 +1558,3 @@ - From python-checkins at python.org Tue Apr 18 06:32:36 2006 From: python-checkins at python.org (neal.norwitz) Date: Tue, 18 Apr 2006 06:32:36 +0200 (CEST) Subject: [Python-checkins] r45524 - peps/trunk/pep-0291.txt Message-ID: <20060418043236.C01161E4003@bag.python.org> Author: neal.norwitz Date: Tue Apr 18 06:32:36 2006 New Revision: 45524 Modified: peps/trunk/pep-0291.txt Log: Alphabetize list Modified: peps/trunk/pep-0291.txt ============================================================================== --- peps/trunk/pep-0291.txt (original) +++ peps/trunk/pep-0291.txt Tue Apr 18 06:32:36 2006 @@ -85,18 +85,18 @@ bsddb Greg Smith 2.1 Barry Warsaw compiler Jeremy Hylton 2.1 + ctypes Thomas Heller 2.3 decimal Raymond Hettinger 2.3 [2] email Barry Warsaw 2.1 / 2.3 [1] logging Vinay Sajip 1.5.2 + modulefinder Thomas Heller 2.2 + Just van Rossum pkgutil Phillip Eby 2.3 + platform Marc-Andre Lemburg 1.5.2 sre Fredrik Lundh 2.1 subprocess Peter Astrand 2.2 xml (PyXML) Martin v. Loewis 2.0 xmlrpclib Fredrik Lundh 2.1 - modulefinder Thomas Heller 2.2 - Just van Rossum - platform Marc-Andre Lemburg 1.5.2 - ctypes Thomas Heller 2.3 Tool Maintainer(s) Python Version ---- ------------- -------------- From python-checkins at python.org Tue Apr 18 06:34:51 2006 From: python-checkins at python.org (phillip.eby) Date: Tue, 18 Apr 2006 06:34:51 +0200 (CEST) Subject: [Python-checkins] r45525 - python/trunk/Makefile.pre.in Message-ID: <20060418043451.445761E4003@bag.python.org> Author: phillip.eby Date: Tue Apr 18 06:34:50 2006 New Revision: 45525 Modified: python/trunk/Makefile.pre.in Log: It's probably a good idea to actually *install* setuptools, too. ;) Modified: python/trunk/Makefile.pre.in ============================================================================== --- python/trunk/Makefile.pre.in (original) +++ python/trunk/Makefile.pre.in Tue Apr 18 06:34:50 2006 @@ -691,6 +691,7 @@ logging bsddb bsddb/test csv \ ctypes ctypes/test ctypes/macholib idlelib idlelib/Icons \ distutils distutils/command distutils/tests $(XMLLIBSUBDIRS) \ + setuptools setuptools/command setuptools/tests setuptools.egg-info \ curses $(MACHDEPS) libinstall: $(BUILDPYTHON) $(srcdir)/Lib/$(PLATDIR) @for i in $(SCRIPTDIR) $(LIBDEST); \ From python-checkins at python.org Tue Apr 18 06:53:30 2006 From: python-checkins at python.org (neal.norwitz) Date: Tue, 18 Apr 2006 06:53:30 +0200 (CEST) Subject: [Python-checkins] r45526 - python/trunk/Lib/test/test_pyclbr.py Message-ID: <20060418045330.463031E4003@bag.python.org> Author: neal.norwitz Date: Tue Apr 18 06:53:28 2006 New Revision: 45526 Modified: python/trunk/Lib/test/test_pyclbr.py Log: Whitespace normalization Modified: python/trunk/Lib/test/test_pyclbr.py ============================================================================== --- python/trunk/Lib/test/test_pyclbr.py (original) +++ python/trunk/Lib/test/test_pyclbr.py Tue Apr 18 06:53:28 2006 @@ -95,7 +95,7 @@ self.assert_(isinstance(py_item, (FunctionType, BuiltinFunctionType))) else: self.failUnless(isinstance(py_item, (ClassType, type))) - if py_item.__module__!=moduleName: + if py_item.__module__ != moduleName: continue # skip classes that came from somewhere else real_bases = [base.__name__ for base in py_item.__bases__] From buildbot at python.org Tue Apr 18 07:16:45 2006 From: buildbot at python.org (buildbot at python.org) Date: Tue, 18 Apr 2006 05:16:45 +0000 Subject: [Python-checkins] buildbot warnings in g4 osx.4 trunk Message-ID: <20060418051645.899AC1E4003@bag.python.org> The Buildbot has detected a new failure of g4 osx.4 trunk. Full details are available at: http://www.python.org/dev/buildbot/all/g4%2520osx.4%2520trunk/builds/466 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: phillip.eby Build Had Warnings: warnings test sincerely, -The Buildbot From nnorwitz at gmail.com Tue Apr 18 07:53:50 2006 From: nnorwitz at gmail.com (Neal Norwitz) Date: Mon, 17 Apr 2006 22:53:50 -0700 Subject: [Python-checkins] r45505 - python/trunk/Modules/posixmodule.c In-Reply-To: <20060418004950.454AC1E400A@bag.python.org> References: <20060418004950.454AC1E400A@bag.python.org> Message-ID: > Modified: python/trunk/Modules/posixmodule.c > ============================================================================== > --- python/trunk/Modules/posixmodule.c (original) > +++ python/trunk/Modules/posixmodule.c Tue Apr 18 02:49:49 2006 > @@ -6812,17 +6812,19 @@ > char buffer[64]; > > if (PyArg_ParseTuple(args, "O&:confstr", conv_confstr_confname, &name)) { > - int len = confstr(name, buffer, sizeof(buffer)); > + int len; > > errno = 0; > - if (len == 0) { > - if (errno != 0) > - posix_error(); > - else > - result = PyString_FromString(""); > + len = confstr(name, buffer, sizeof(buffer)); > + > + if (len == -1) { > + posix_error(); > + } > + else if (len == 0) { > + result = PyString_FromString(""); > } > else { > - if (len >= sizeof(buffer)) { > + if ((unsigned int)len >= sizeof(buffer)) { > result = PyString_FromStringAndSize(NULL, len); > if (result != NULL) > confstr(name, PyString_AS_STRING(result), len+1); This code seems broken (both before and after this patch) based on: http://www.opengroup.org/onlinepubs/009695399/functions/confstr.html The return type is size_t and cannot return -1. Do we know if the previous code ever worked? Is there any system that does return -1? Also, since sizeof(size_t) != sizeof(int), this code isn't right. If size_t is 8 bytes, but int is 4 bytes and len is (1<<32) + 5, we will try to copy 4+GB. If len is a size_t this problem goes away, since the (unsigned int) cast is not necessary. n From python-checkins at python.org Tue Apr 18 08:24:10 2006 From: python-checkins at python.org (martin.v.loewis) Date: Tue, 18 Apr 2006 08:24:10 +0200 (CEST) Subject: [Python-checkins] r45527 - in python/trunk: Include/object.h Misc/NEWS Objects/object.c Python/pythonrun.c Message-ID: <20060418062410.78FE31E4003@bag.python.org> Author: martin.v.loewis Date: Tue Apr 18 08:24:08 2006 New Revision: 45527 Modified: python/trunk/Include/object.h python/trunk/Misc/NEWS python/trunk/Objects/object.c python/trunk/Python/pythonrun.c Log: Remove types from type_list if they have no objects and unlist_types_without_objects is set. Give dump_counts a FILE* argument. Modified: python/trunk/Include/object.h ============================================================================== --- python/trunk/Include/object.h (original) +++ python/trunk/Include/object.h Tue Apr 18 08:24:08 2006 @@ -339,6 +339,7 @@ Py_ssize_t tp_allocs; Py_ssize_t tp_frees; Py_ssize_t tp_maxalloc; + struct _typeobject *tp_prev; struct _typeobject *tp_next; #endif } PyTypeObject; @@ -598,8 +599,9 @@ #ifdef COUNT_ALLOCS PyAPI_FUNC(void) inc_count(PyTypeObject *); +PyAPI_FUNC(void) dec_count(PyTypeObject *); #define _Py_INC_TPALLOCS(OP) inc_count((OP)->ob_type) -#define _Py_INC_TPFREES(OP) (OP)->ob_type->tp_frees++ +#define _Py_INC_TPFREES(OP) dec_count((OP)->ob_type) #define _Py_DEC_TPFREES(OP) (OP)->ob_type->tp_frees-- #define _Py_COUNT_ALLOCS_COMMA , #else Modified: python/trunk/Misc/NEWS ============================================================================== --- python/trunk/Misc/NEWS (original) +++ python/trunk/Misc/NEWS Tue Apr 18 08:24:08 2006 @@ -12,6 +12,8 @@ Core and builtins ----------------- +- Under COUNT_ALLOCS, types are not necessarily immortal anymore. + - All uses of PyStructSequence_InitType have been changed to initialize the type objects only once, even if the interpreter is initialized multiple times. Modified: python/trunk/Objects/object.c ============================================================================== --- python/trunk/Objects/object.c (original) +++ python/trunk/Objects/object.c Tue Apr 18 08:24:08 2006 @@ -74,23 +74,30 @@ #ifdef COUNT_ALLOCS static PyTypeObject *type_list; +/* All types are added to type_list, atleast when + they get one object created. That makes them + immortal, which unfortunately contributes to + garbage itself. If unlist_types_without_objects + is set, they will be removed from the type_list + once the last object is deallocated. */ +int unlist_types_without_objects; extern int tuple_zero_allocs, fast_tuple_allocs; extern int quick_int_allocs, quick_neg_int_allocs; extern int null_strings, one_strings; void -dump_counts(void) +dump_counts(FILE* f) { PyTypeObject *tp; for (tp = type_list; tp; tp = tp->tp_next) - fprintf(stderr, "%s alloc'd: %d, freed: %d, max in use: %d\n", + fprintf(f, "%s alloc'd: %d, freed: %d, max in use: %d\n", tp->tp_name, tp->tp_allocs, tp->tp_frees, tp->tp_maxalloc); - fprintf(stderr, "fast tuple allocs: %d, empty: %d\n", + fprintf(f, "fast tuple allocs: %d, empty: %d\n", fast_tuple_allocs, tuple_zero_allocs); - fprintf(stderr, "fast int allocs: pos: %d, neg: %d\n", + fprintf(f, "fast int allocs: pos: %d, neg: %d\n", quick_int_allocs, quick_neg_int_allocs); - fprintf(stderr, "null strings: %d, 1-strings: %d\n", + fprintf(f, "null strings: %d, 1-strings: %d\n", null_strings, one_strings); } @@ -124,10 +131,12 @@ void inc_count(PyTypeObject *tp) { - if (tp->tp_allocs == 0) { + if (tp->tp_next == NULL && tp->tp_prev == NULL) { /* first time; insert in linked list */ if (tp->tp_next != NULL) /* sanity check */ Py_FatalError("XXX inc_count sanity check"); + if (type_list) + type_list->tp_prev = tp; tp->tp_next = type_list; /* Note that as of Python 2.2, heap-allocated type objects * can go away, but this code requires that they stay alive @@ -150,6 +159,24 @@ if (tp->tp_allocs - tp->tp_frees > tp->tp_maxalloc) tp->tp_maxalloc = tp->tp_allocs - tp->tp_frees; } + +void dec_count(PyTypeObject *tp) +{ + tp->tp_frees++; + if (unlist_types_without_objects && + tp->tp_allocs == tp->tp_frees) { + /* unlink the type from type_list */ + if (tp->tp_prev) + tp->tp_prev->tp_next = tp->tp_next; + else + type_list = tp->tp_next; + if (tp->tp_next) + tp->tp_next->tp_prev = tp->tp_prev; + tp->tp_next = tp->tp_prev = NULL; + Py_DECREF(tp); + } +} + #endif #ifdef Py_REF_DEBUG Modified: python/trunk/Python/pythonrun.c ============================================================================== --- python/trunk/Python/pythonrun.c (original) +++ python/trunk/Python/pythonrun.c Tue Apr 18 08:24:08 2006 @@ -311,7 +311,7 @@ #ifdef COUNT_ALLOCS -extern void dump_counts(void); +extern void dump_counts(FILE*); #endif /* Undo the effect of Py_Initialize(). @@ -373,6 +373,13 @@ * XXX I haven't seen a real-life report of either of these. */ PyGC_Collect(); +#ifdef COUNT_ALLOCS + /* With COUNT_ALLOCS, it helps to run GC multiple times: + each collection might release some types from the type + list, so they become garbage. */ + while (PyGC_Collect() > 0) + /* nothing */; +#endif /* Destroy all modules */ PyImport_Cleanup(); @@ -401,7 +408,7 @@ /* Debugging stuff */ #ifdef COUNT_ALLOCS - dump_counts(); + dump_counts(stdout); #endif PRINT_TOTAL_REFS(); From martin at v.loewis.de Tue Apr 18 08:34:50 2006 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 18 Apr 2006 08:34:50 +0200 Subject: [Python-checkins] r45505 - python/trunk/Modules/posixmodule.c In-Reply-To: References: <20060418004950.454AC1E400A@bag.python.org> Message-ID: <4444888A.60603@v.loewis.de> Neal Norwitz wrote: >> - if (len == 0) { >> + if (len == -1) { > This code seems broken (both before and after this patch) based on: > http://www.opengroup.org/onlinepubs/009695399/functions/confstr.html > > The return type is size_t and cannot return -1. Do we know if the > previous code ever worked? Is there any system that does return -1? It's now broken in a different way. Previously, it was broken because it reset errno after the call. Now it is broken because it treats -1 as an error indication; the correct error indication is 0 (but 0 also indicates that there is no configuration-defined value, hence errno must be set to 0 before the call). Also, I suggest to use None as the return value for "no value available"; it might be that the configured value is an empty string (in which case confstr returns 1). Also, the PyString_FromStringAndSize is wrong: the confstr result includes the terminating \0, whereas PyString_FromStringAndSize doesn't. Also, for symmetry, it would be better if PyString_FromStringAndSize is also used in the case without resizing. This would also have the advantage of allowing for results that include \0 (although I doubt they are actually used). Regards, Martin From mal at egenix.com Tue Apr 18 10:55:25 2006 From: mal at egenix.com (M.-A. Lemburg) Date: Tue, 18 Apr 2006 10:55:25 +0200 Subject: [Python-checkins] r45510 - python/trunk/Lib/pkgutil.py python/trunk/Lib/pydoc.py In-Reply-To: <20060418005956.156301E400A@bag.python.org> References: <20060418005956.156301E400A@bag.python.org> Message-ID: <4444A97D.40406@egenix.com> Phillip.eby wrote: > Author: phillip.eby > Date: Tue Apr 18 02:59:55 2006 > New Revision: 45510 > > Modified: > python/trunk/Lib/pkgutil.py > python/trunk/Lib/pydoc.py > Log: > Second phase of refactoring for runpy, pkgutil, pydoc, and setuptools > to share common PEP 302 support code, as described here: > > http://mail.python.org/pipermail/python-dev/2006-April/063724.html Shouldn't this new module be named "pkglib" to be in line with the naming scheme used for all the other utility modules, e.g. httplib, imaplib, poplib, etc. ? > pydoc now supports PEP 302 importers, by way of utility functions in > pkgutil, such as 'walk_packages()'. It will properly document > modules that are in zip files, and is backward compatible to Python > 2.3 (setuptools installs for Python <2.5 will bundle it so pydoc > doesn't break when used with eggs.) Are you saying that the installation of setuptools in Python 2.3 and 2.4 will then overwrite the standard pydoc included with those versions ? I think that's the wrong way to go if not made an explicit option in the installation process or a separate installation altogether. I bothered by the fact that installing setuptools actually changes the standard Python installation by either overriding stdlib modules or monkey-patching them at setuptools import time. > What has not changed is that pydoc command line options do not support > zip paths or other importer paths, and the webserver index does not > support sys.meta_path. Those are probably okay as limitations. > > Tasks remaining: write docs and Misc/NEWS for pkgutil/pydoc changes, > and update setuptools to use pkgutil wherever possible, then add it > to the stdlib. Add setuptools to the stdlib ? I'm still missing the PEP for this along with the needed discussion touching among other things, the change of the distutils standard "python setup.py install" to install an egg instead of a site package. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Apr 18 2006) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From neal at metaslash.com Tue Apr 18 11:06:37 2006 From: neal at metaslash.com (Neal Norwitz) Date: Tue, 18 Apr 2006 05:06:37 -0400 Subject: [Python-checkins] Python Regression Test Failures refleak (1) Message-ID: <20060418090637.GA10956@python.psfb.org> test_cmd_line leaked [-17, 0, 0] references test_socket leaked [217, -217, 0] references test_threading_local leaked [-93, 0, 0] references test_urllib2 leaked [-121, 88, 99] references From python-checkins at python.org Tue Apr 18 13:49:54 2006 From: python-checkins at python.org (andrew.kuchling) Date: Tue, 18 Apr 2006 13:49:54 +0200 (CEST) Subject: [Python-checkins] r45528 - python/trunk/Objects/object.c Message-ID: <20060418114954.598461E400A@bag.python.org> Author: andrew.kuchling Date: Tue Apr 18 13:49:53 2006 New Revision: 45528 Modified: python/trunk/Objects/object.c Log: Comment typo fix Modified: python/trunk/Objects/object.c ============================================================================== --- python/trunk/Objects/object.c (original) +++ python/trunk/Objects/object.c Tue Apr 18 13:49:53 2006 @@ -74,7 +74,7 @@ #ifdef COUNT_ALLOCS static PyTypeObject *type_list; -/* All types are added to type_list, atleast when +/* All types are added to type_list, at least when they get one object created. That makes them immortal, which unfortunately contributes to garbage itself. If unlist_types_without_objects From python-checkins at python.org Tue Apr 18 13:53:09 2006 From: python-checkins at python.org (skip.montanaro) Date: Tue, 18 Apr 2006 13:53:09 +0200 (CEST) Subject: [Python-checkins] r45529 - python/trunk/Modules/_sre.c Message-ID: <20060418115309.B72171E400A@bag.python.org> Author: skip.montanaro Date: Tue Apr 18 13:53:09 2006 New Revision: 45529 Modified: python/trunk/Modules/_sre.c Log: C++ compiler cleanup: proper casts Modified: python/trunk/Modules/_sre.c ============================================================================== --- python/trunk/Modules/_sre.c (original) +++ python/trunk/Modules/_sre.c Tue Apr 18 13:53:09 2006 @@ -2284,10 +2284,10 @@ ptr = getstring(ptemplate, &n, &b); if (ptr) { if (b == 1) { - literal = sre_literal_template(ptr, n); + literal = sre_literal_template((unsigned char *)ptr, n); } else { #if defined(HAVE_UNICODE) - literal = sre_uliteral_template(ptr, n); + literal = sre_uliteral_template((Py_UNICODE *)ptr, n); #endif } } else { From buildbot at python.org Tue Apr 18 14:20:02 2006 From: buildbot at python.org (buildbot at python.org) Date: Tue, 18 Apr 2006 12:20:02 +0000 Subject: [Python-checkins] buildbot warnings in x86 W2k trunk Message-ID: <20060418122002.B7E3E1E400A@bag.python.org> The Buildbot has detected a new failure of x86 W2k trunk. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520W2k%2520trunk/builds/530 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: andrew.kuchling,skip.montanaro Build Had Warnings: warnings test sincerely, -The Buildbot From python-checkins at python.org Tue Apr 18 14:38:20 2006 From: python-checkins at python.org (andrew.kuchling) Date: Tue, 18 Apr 2006 14:38:20 +0200 (CEST) Subject: [Python-checkins] r45530 - python/trunk/Doc/whatsnew/whatsnew25.tex Message-ID: <20060418123820.137121E400A@bag.python.org> Author: andrew.kuchling Date: Tue Apr 18 14:38:19 2006 New Revision: 45530 Modified: python/trunk/Doc/whatsnew/whatsnew25.tex Log: Add item Modified: python/trunk/Doc/whatsnew/whatsnew25.tex ============================================================================== --- python/trunk/Doc/whatsnew/whatsnew25.tex (original) +++ python/trunk/Doc/whatsnew/whatsnew25.tex Tue Apr 18 14:38:19 2006 @@ -3,6 +3,7 @@ % $Id$ % The easy_install stuff +% Describe the pkgutil module % Stateful codec changes % Fix XXX comments % Count up the patches and bugs @@ -1173,8 +1174,6 @@ % the cPickle module no longer accepts the deprecated None option in the % args tuple returned by __reduce__(). -% XXX csv module improvements - % XXX datetime.datetime() now has a strptime class method which can be used to % create datetime object using a string and format. @@ -1240,6 +1239,17 @@ module's interface, will continue to be maintained in future versions of Python. (Contributed by Armin Rigo.) +\item The \module{csv} module, which parses files in +comma-separated value format, received several enhancements and a +number of bugfixes. You can now set the maximum size in bytes of a +field by calling the \method{csv.field_size_limit(\var{new_limit})} +function; omitting the \var{new_limit} argument will return the +currently-set limit. The \class{reader} class now has a +\member{line_num} attribute that counts the number of physical lines +read from the source; records can span multiple physical lines, so +\member{line_num} is not the same as the number of records read. +(Contributed by Skip Montanaro and Andrew McNamara.) + \item In the \module{gc} module, the new \function{get_count()} function returns a 3-tuple containing the current collection counts for the three GC generations. This is accounting information for the garbage From python-checkins at python.org Tue Apr 18 15:52:33 2006 From: python-checkins at python.org (michael.hudson) Date: Tue, 18 Apr 2006 15:52:33 +0200 (CEST) Subject: [Python-checkins] r45531 - python/trunk/Lib/test/crashers/nasty_eq_vs_dict.py Message-ID: <20060418135233.207D01E400A@bag.python.org> Author: michael.hudson Date: Tue Apr 18 15:52:32 2006 New Revision: 45531 Added: python/trunk/Lib/test/crashers/nasty_eq_vs_dict.py (contents, props changed) Log: add a very old crasher from the 2.1 -> 2.2 round of dictionary fixes. Added: python/trunk/Lib/test/crashers/nasty_eq_vs_dict.py ============================================================================== --- (empty file) +++ python/trunk/Lib/test/crashers/nasty_eq_vs_dict.py Tue Apr 18 15:52:32 2006 @@ -0,0 +1,47 @@ +# from http://mail.python.org/pipermail/python-dev/2001-June/015239.html + +# if you keep changing a dictionary while looking up a key, you can +# provoke an infinite recursion in C + +# At the time neither Tim nor Michael could be bothered to think of a +# way to fix it. + +class Yuck: + def __init__(self): + self.i = 0 + + def make_dangerous(self): + self.i = 1 + + def __hash__(self): + # direct to slot 4 in table of size 8; slot 12 when size 16 + return 4 + 8 + + def __eq__(self, other): + if self.i == 0: + # leave dict alone + pass + elif self.i == 1: + # fiddle to 16 slots + self.__fill_dict(6) + self.i = 2 + else: + # fiddle to 8 slots + self.__fill_dict(4) + self.i = 1 + + return 1 + + def __fill_dict(self, n): + self.i = 0 + dict.clear() + for i in range(n): + dict[i] = i + dict[self] = "OK!" + +y = Yuck() +dict = {y: "OK!"} + +z = Yuck() +y.make_dangerous() +print dict[z] From python-checkins at python.org Tue Apr 18 16:00:02 2006 From: python-checkins at python.org (armin.rigo) Date: Tue, 18 Apr 2006 16:00:02 +0200 (CEST) Subject: [Python-checkins] r45532 - python/trunk/Lib/test/crashers/dictresize_attack.py Message-ID: <20060418140002.102421E400A@bag.python.org> Author: armin.rigo Date: Tue Apr 18 16:00:01 2006 New Revision: 45532 Added: python/trunk/Lib/test/crashers/dictresize_attack.py (contents, props changed) Log: A dictresize() attack. If oldtable == mp->ma_smalltable then pure Python code can mangle with mp->ma_smalltable while it is being walked over. Added: python/trunk/Lib/test/crashers/dictresize_attack.py ============================================================================== --- (empty file) +++ python/trunk/Lib/test/crashers/dictresize_attack.py Tue Apr 18 16:00:01 2006 @@ -0,0 +1,32 @@ +# http://www.python.org/sf/1456209 + +# A dictresize() attack. If oldtable == mp->ma_smalltable then pure +# Python code can mangle with mp->ma_smalltable while it is being walked +# over. + +class X(object): + + def __hash__(self): + return 5 + + def __eq__(self, other): + if resizing: + d.clear() + return False + + +d = {} + +resizing = False + +d[X()] = 1 +d[X()] = 2 +d[X()] = 3 +d[X()] = 4 +d[X()] = 5 + +# now trigger a resize +resizing = True +d[9] = 6 + +# ^^^ I get Segmentation fault or Illegal instruction here. From python-checkins at python.org Tue Apr 18 16:04:57 2006 From: python-checkins at python.org (andrew.kuchling) Date: Tue, 18 Apr 2006 16:04:57 +0200 (CEST) Subject: [Python-checkins] r45533 - python/trunk/Misc/NEWS Message-ID: <20060418140457.EBCC21E400A@bag.python.org> Author: andrew.kuchling Date: Tue Apr 18 16:04:57 2006 New Revision: 45533 Modified: python/trunk/Misc/NEWS Log: Typo fix Modified: python/trunk/Misc/NEWS ============================================================================== --- python/trunk/Misc/NEWS (original) +++ python/trunk/Misc/NEWS Tue Apr 18 16:04:57 2006 @@ -954,7 +954,7 @@ dictates. + the parser now removes the escapechar prefix from escaped characters. + when quoting=QUOTE_NONNUMERIC, the writer now tests for numeric - types, rather than any object than can be represented as a numeric. + types, rather than any object that can be represented as a numeric. + when quoting=QUOTE_NONNUMERIC, the reader now casts unquoted fields to floats. + reader now allows \r characters to be quoted (previously it only allowed From buildbot at python.org Tue Apr 18 16:29:50 2006 From: buildbot at python.org (buildbot at python.org) Date: Tue, 18 Apr 2006 14:29:50 +0000 Subject: [Python-checkins] buildbot warnings in amd64 gentoo trunk Message-ID: <20060418142950.CFFFD1E400A@bag.python.org> The Buildbot has detected a new failure of amd64 gentoo trunk. Full details are available at: http://www.python.org/dev/buildbot/all/amd64%2520gentoo%2520trunk/builds/557 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: andrew.kuchling,armin.rigo Build Had Warnings: warnings test sincerely, -The Buildbot From buildbot at python.org Tue Apr 18 16:30:14 2006 From: buildbot at python.org (buildbot at python.org) Date: Tue, 18 Apr 2006 14:30:14 +0000 Subject: [Python-checkins] buildbot warnings in x86 gentoo trunk Message-ID: <20060418143014.3BD941E400A@bag.python.org> The Buildbot has detected a new failure of x86 gentoo trunk. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520gentoo%2520trunk/builds/537 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: andrew.kuchling,armin.rigo Build Had Warnings: warnings test sincerely, -The Buildbot From buildbot at python.org Tue Apr 18 16:30:53 2006 From: buildbot at python.org (buildbot at python.org) Date: Tue, 18 Apr 2006 14:30:53 +0000 Subject: [Python-checkins] buildbot failure in x86 XP trunk Message-ID: <20060418143053.3F1731E400A@bag.python.org> The Buildbot has detected a new failure of x86 XP trunk. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520XP%2520trunk/builds/494 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: andrew.kuchling,armin.rigo BUILD FAILED: failed failed slave lost sincerely, -The Buildbot From python-checkins at python.org Tue Apr 18 16:47:01 2006 From: python-checkins at python.org (jeremy.hylton) Date: Tue, 18 Apr 2006 16:47:01 +0200 (CEST) Subject: [Python-checkins] r45534 - in python/trunk: Include/code.h Objects/codeobject.c Python/ceval.c Message-ID: <20060418144701.C286F1E4002@bag.python.org> Author: jeremy.hylton Date: Tue Apr 18 16:47:00 2006 New Revision: 45534 Modified: python/trunk/Include/code.h python/trunk/Objects/codeobject.c python/trunk/Python/ceval.c Log: Refactor: Move code that uses co_lnotab from ceval to codeobject Modified: python/trunk/Include/code.h ============================================================================== --- python/trunk/Include/code.h (original) +++ python/trunk/Include/code.h Tue Apr 18 16:47:00 2006 @@ -72,6 +72,21 @@ ((*(co)->co_code->ob_type->tp_as_buffer->bf_getreadbuffer) \ ((co)->co_code, 0, (void **)(pp))) +typedef struct _addr_pair { + int ap_lower; + int ap_upper; +} PyAddrPair; + +/* Check whether lasti (an instruction offset) falls outside bounds + and whether it is a line number that should be traced. Returns + a line number if it should be traced or -1 if the line should not. + + If lasti is not within bounds, updates bounds. +*/ + +PyAPI_FUNC(int) PyCode_CheckLineNumber(PyCodeObject* co, + int lasti, PyAddrPair *bounds); + #ifdef __cplusplus } #endif Modified: python/trunk/Objects/codeobject.c ============================================================================== --- python/trunk/Objects/codeobject.c (original) +++ python/trunk/Objects/codeobject.c Tue Apr 18 16:47:00 2006 @@ -451,3 +451,136 @@ } return line; } + +/* + Check whether the current instruction is at the start of a line. + + */ + + /* The theory of SET_LINENO-less tracing. + + In a nutshell, we use the co_lnotab field of the code object + to tell when execution has moved onto a different line. + + As mentioned above, the basic idea is so set things up so + that + + *instr_lb <= frame->f_lasti < *instr_ub + + is true so long as execution does not change lines. + + This is all fairly simple. Digging the information out of + co_lnotab takes some work, but is conceptually clear. + + Somewhat harder to explain is why we don't *always* call the + line trace function when the above test fails. + + Consider this code: + + 1: def f(a): + 2: if a: + 3: print 1 + 4: else: + 5: print 2 + + which compiles to this: + + 2 0 LOAD_FAST 0 (a) + 3 JUMP_IF_FALSE 9 (to 15) + 6 POP_TOP + + 3 7 LOAD_CONST 1 (1) + 10 PRINT_ITEM + 11 PRINT_NEWLINE + 12 JUMP_FORWARD 6 (to 21) + >> 15 POP_TOP + + 5 16 LOAD_CONST 2 (2) + 19 PRINT_ITEM + 20 PRINT_NEWLINE + >> 21 LOAD_CONST 0 (None) + 24 RETURN_VALUE + + If 'a' is false, execution will jump to instruction at offset + 15 and the co_lnotab will claim that execution has moved to + line 3. This is at best misleading. In this case we could + associate the POP_TOP with line 4, but that doesn't make + sense in all cases (I think). + + What we do is only call the line trace function if the co_lnotab + indicates we have jumped to the *start* of a line, i.e. if the + current instruction offset matches the offset given for the + start of a line by the co_lnotab. + + This also takes care of the situation where 'a' is true. + Execution will jump from instruction offset 12 to offset 21. + Then the co_lnotab would imply that execution has moved to line + 5, which is again misleading. + + Why do we set f_lineno when tracing? Well, consider the code + above when 'a' is true. If stepping through this with 'n' in + pdb, you would stop at line 1 with a "call" type event, then + line events on lines 2 and 3, then a "return" type event -- but + you would be shown line 5 during this event. This is a change + from the behaviour in 2.2 and before, and I've found it + confusing in practice. By setting and using f_lineno when + tracing, one can report a line number different from that + suggested by f_lasti on this one occasion where it's desirable. + */ + + +int +PyCode_CheckLineNumber(PyCodeObject* co, int lasti, PyAddrPair *bounds) +{ + int size, addr, line; + unsigned char* p; + + p = (unsigned char*)PyString_AS_STRING(co->co_lnotab); + size = PyString_GET_SIZE(co->co_lnotab) / 2; + + addr = 0; + line = co->co_firstlineno; + assert(line > 0); + + /* possible optimization: if f->f_lasti == instr_ub + (likely to be a common case) then we already know + instr_lb -- if we stored the matching value of p + somwhere we could skip the first while loop. */ + + /* see comments in compile.c for the description of + co_lnotab. A point to remember: increments to p + should come in pairs -- although we don't care about + the line increments here, treating them as byte + increments gets confusing, to say the least. */ + + while (size > 0) { + if (addr + *p > lasti) + break; + addr += *p++; + if (*p) + bounds->ap_lower = addr; + line += *p++; + --size; + } + + /* If lasti and addr don't match exactly, we don't want to + change the lineno slot on the frame or execute a trace + function. Return -1 instead. + */ + if (addr != lasti) + line = -1; + + if (size > 0) { + while (--size >= 0) { + addr += *p++; + if (*p++) + break; + } + bounds->ap_upper = addr; + } + else { + bounds->ap_upper = INT_MAX; + } + + return line; +} Modified: python/trunk/Python/ceval.c ============================================================================== --- python/trunk/Python/ceval.c (original) +++ python/trunk/Python/ceval.c Tue Apr 18 16:47:00 2006 @@ -3219,132 +3219,29 @@ PyFrameObject *frame, int *instr_lb, int *instr_ub, int *instr_prev) { - /* The theory of SET_LINENO-less tracing. - - In a nutshell, we use the co_lnotab field of the code object - to tell when execution has moved onto a different line. - - As mentioned above, the basic idea is so set things up so - that - - *instr_lb <= frame->f_lasti < *instr_ub - - is true so long as execution does not change lines. - - This is all fairly simple. Digging the information out of - co_lnotab takes some work, but is conceptually clear. - - Somewhat harder to explain is why we don't *always* call the - line trace function when the above test fails. - - Consider this code: - - 1: def f(a): - 2: if a: - 3: print 1 - 4: else: - 5: print 2 - - which compiles to this: - - 2 0 LOAD_FAST 0 (a) - 3 JUMP_IF_FALSE 9 (to 15) - 6 POP_TOP - - 3 7 LOAD_CONST 1 (1) - 10 PRINT_ITEM - 11 PRINT_NEWLINE - 12 JUMP_FORWARD 6 (to 21) - >> 15 POP_TOP - - 5 16 LOAD_CONST 2 (2) - 19 PRINT_ITEM - 20 PRINT_NEWLINE - >> 21 LOAD_CONST 0 (None) - 24 RETURN_VALUE - - If 'a' is false, execution will jump to instruction at offset - 15 and the co_lnotab will claim that execution has moved to - line 3. This is at best misleading. In this case we could - associate the POP_TOP with line 4, but that doesn't make - sense in all cases (I think). - - What we do is only call the line trace function if the co_lnotab - indicates we have jumped to the *start* of a line, i.e. if the - current instruction offset matches the offset given for the - start of a line by the co_lnotab. - - This also takes care of the situation where 'a' is true. - Execution will jump from instruction offset 12 to offset 21. - Then the co_lnotab would imply that execution has moved to line - 5, which is again misleading. - - Why do we set f_lineno when tracing? Well, consider the code - above when 'a' is true. If stepping through this with 'n' in - pdb, you would stop at line 1 with a "call" type event, then - line events on lines 2 and 3, then a "return" type event -- but - you would be shown line 5 during this event. This is a change - from the behaviour in 2.2 and before, and I've found it - confusing in practice. By setting and using f_lineno when - tracing, one can report a line number different from that - suggested by f_lasti on this one occasion where it's desirable. - */ - int result = 0; + /* If the last instruction executed isn't in the current + instruction window, reset the window. If the last + instruction happens to fall at the start of a line or if it + represents a jump backwards, call the trace function. + */ if ((frame->f_lasti < *instr_lb || frame->f_lasti >= *instr_ub)) { - PyCodeObject* co = frame->f_code; - int size, addr, line; - unsigned char* p; - - size = PyString_GET_SIZE(co->co_lnotab) / 2; - p = (unsigned char*)PyString_AS_STRING(co->co_lnotab); - - addr = 0; - line = co->co_firstlineno; - - /* possible optimization: if f->f_lasti == instr_ub - (likely to be a common case) then we already know - instr_lb -- if we stored the matching value of p - somwhere we could skip the first while loop. */ - - /* see comments in compile.c for the description of - co_lnotab. A point to remember: increments to p - should come in pairs -- although we don't care about - the line increments here, treating them as byte - increments gets confusing, to say the least. */ - - while (size > 0) { - if (addr + *p > frame->f_lasti) - break; - addr += *p++; - if (*p) *instr_lb = addr; - line += *p++; - --size; - } + int line; + PyAddrPair bounds; - if (addr == frame->f_lasti) { + line = PyCode_CheckLineNumber(frame->f_code, frame->f_lasti, + &bounds); + if (line >= 0) { frame->f_lineno = line; result = call_trace(func, obj, frame, PyTrace_LINE, Py_None); - } - - if (size > 0) { - while (--size >= 0) { - addr += *p++; - if (*p++) - break; - } - *instr_ub = addr; - } - else { - *instr_ub = INT_MAX; - } + } + *instr_lb = bounds.ap_lower; + *instr_ub = bounds.ap_upper; } else if (frame->f_lasti <= *instr_prev) { - /* jumping back in the same line forces a trace event */ - result = call_trace(func, obj, frame, - PyTrace_LINE, Py_None); + result = call_trace(func, obj, frame, PyTrace_LINE, Py_None); } *instr_prev = frame->f_lasti; return result; From python-checkins at python.org Tue Apr 18 16:57:40 2006 From: python-checkins at python.org (thomas.heller) Date: Tue, 18 Apr 2006 16:57:40 +0200 (CEST) Subject: [Python-checkins] r45535 - python/trunk/Modules/_ctypes/_ctypes.c Message-ID: <20060418145740.620091E4002@bag.python.org> Author: thomas.heller Date: Tue Apr 18 16:57:39 2006 New Revision: 45535 Modified: python/trunk/Modules/_ctypes/_ctypes.c Log: Fix refcounting. This makes 'import ctypes; reload(ctypes)' no longer leak reference counts. Modified: python/trunk/Modules/_ctypes/_ctypes.c ============================================================================== --- python/trunk/Modules/_ctypes/_ctypes.c (original) +++ python/trunk/Modules/_ctypes/_ctypes.c Tue Apr 18 16:57:39 2006 @@ -1283,6 +1283,7 @@ suffix = PyString_FromString("_be"); #endif + Py_INCREF(name); PyString_Concat(&name, suffix); if (name == NULL) return NULL; @@ -1459,6 +1460,7 @@ PyObject_SetAttrString(swapped, "__ctype_le__", (PyObject *)result); PyObject_SetAttrString(swapped, "__ctype_be__", swapped); #endif + Py_DECREF(swapped); }; return (PyObject *)result; From buildbot at python.org Tue Apr 18 17:09:34 2006 From: buildbot at python.org (buildbot at python.org) Date: Tue, 18 Apr 2006 15:09:34 +0000 Subject: [Python-checkins] buildbot warnings in x86 OpenBSD trunk Message-ID: <20060418150934.AB2EB1E4002@bag.python.org> The Buildbot has detected a new failure of x86 OpenBSD trunk. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520OpenBSD%2520trunk/builds/371 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: jeremy.hylton Build Had Warnings: warnings test sincerely, -The Buildbot From python-checkins at python.org Tue Apr 18 17:27:07 2006 From: python-checkins at python.org (phillip.eby) Date: Tue, 18 Apr 2006 17:27:07 +0200 (CEST) Subject: [Python-checkins] r45536 - in sandbox/trunk/setuptools: pkg_resources.py pkg_resources.txt setuptools/command/bdist_egg.py Message-ID: <20060418152707.8F8141E4002@bag.python.org> Author: phillip.eby Date: Tue Apr 18 17:27:06 2006 New Revision: 45536 Modified: sandbox/trunk/setuptools/pkg_resources.py sandbox/trunk/setuptools/pkg_resources.txt sandbox/trunk/setuptools/setuptools/command/bdist_egg.py Log: Split ``get_platform()`` into ``get_supported_platform()`` and ``get_build_platform()`` to work around a Mac versioning problem that caused the behavior of ``compatible_platforms()`` to be platform specific. Modified: sandbox/trunk/setuptools/pkg_resources.py ============================================================================== --- sandbox/trunk/setuptools/pkg_resources.py (original) +++ sandbox/trunk/setuptools/pkg_resources.py Tue Apr 18 17:27:06 2006 @@ -18,7 +18,7 @@ from os import utime, rename, unlink # capture these to bypass sandboxing from os import open as os_open -def _get_max_platform(plat): +def get_supported_platform(): """Return this platform's maximum compatible version. distutils.util.get_platform() normally reports the minimum version @@ -31,7 +31,7 @@ If this condition occurs for any other platform with a version in its platform strings, this function should be extended accordingly. """ - m = macosVersionString.match(plat) + plat = get_build_platform(); m = macosVersionString.match(plat) if m is not None and sys.platform == "darwin": try: plat = 'macosx-%s-%s' % ('.'.join(_macosx_vers()[:2]), m.group(3)) @@ -138,7 +138,7 @@ def _macosx_arch(machine): return {'PowerPC':'ppc', 'Power_Macintosh':'ppc'}.get(machine,machine) -def get_platform(): +def get_build_platform(): """Return this platform's string for platform-specific distributions XXX Currently this is the same as ``distutils.util.get_platform()``, but it @@ -160,7 +160,7 @@ macosVersionString = re.compile(r"macosx-(\d+)\.(\d+)-(.*)") darwinVersionString = re.compile(r"darwin-(\d+)\.(\d+)\.(\d+)-(.*)") - +get_platform = get_build_platform # XXX backward compat def compatible_platforms(provided,required): """Can code for the `provided` platform run on the `required` platform? @@ -171,8 +171,6 @@ """ if provided is None or required is None or provided==required: return True # easy case - provided = _get_max_platform(provided) - if provided==required: return True # Mac OS X special cases reqMac = macosVersionString.match(required) @@ -203,6 +201,8 @@ provMac.group(3) != reqMac.group(3): return False + + # is the required OS major update >= the provided one? if int(provMac.group(2)) > int(reqMac.group(2)): return False @@ -616,7 +616,7 @@ class Environment(object): """Searchable snapshot of distributions on a search path""" - def __init__(self,search_path=None,platform=get_platform(),python=PY_MAJOR): + def __init__(self, search_path=None, platform=get_supported_platform(), python=PY_MAJOR): """Snapshot distributions available on a search path Any distributions found on `search_path` are added to the environment. Modified: sandbox/trunk/setuptools/pkg_resources.txt ============================================================================== --- sandbox/trunk/setuptools/pkg_resources.txt (original) +++ sandbox/trunk/setuptools/pkg_resources.txt Tue Apr 18 17:27:06 2006 @@ -447,7 +447,7 @@ ``Environment`` objects are used by ``pkg_resources`` to index available distributions during dependency resolution. -``Environment(search_path=None, platform=get_platform(), python=PY_MAJOR)`` +``Environment(search_path=None, platform=get_supported_platform(), python=PY_MAJOR)`` Create an environment snapshot by scanning `search_path` for distributions compatible with `platform` and `python`. `search_path` should be a sequence of strings such as might be used on ``sys.path``. If a @@ -1596,11 +1596,21 @@ Platform Utilities ------------------ -``get_platform()`` +``get_build_platform()`` Return this platform's identifier string. For Windows, the return value is ``"win32"``, and for Mac OS X it is a string of the form ``"macosx-10.4-ppc"``. All other platforms return the same uname-based string that the ``distutils.util.get_platform()`` function returns. + This string is the minimum platform version required by distributions built + on the local machine. (Backward compatibility note: setuptools versions + prior to 0.6b1 called this function ``get_platform()``, and the function is + still available under that name for backward compatibility reasons.) + +``get_supported_platform()`` (New in 0.6b1) + This is the similar to ``get_build_platform()``, but is the maximum + platform version that the local machine supports. You will usually want + to use this value as the ``provided`` argument to the + ``compatible_platforms()`` function. ``compatible_platforms(provided, required)`` Return true if a distribution built on the `provided` platform may be used @@ -1665,3 +1675,7 @@ of a namespace package's ``__init__.py`` files must include a ``declare_namespace()`` call. + * Split ``get_platform()`` into ``get_supported_platform()`` and + ``get_build_platform()`` to work around a Mac versioning problem that caused + the behavior of ``compatible_platforms()`` to be platform specific. + Modified: sandbox/trunk/setuptools/setuptools/command/bdist_egg.py ============================================================================== --- sandbox/trunk/setuptools/setuptools/command/bdist_egg.py (original) +++ sandbox/trunk/setuptools/setuptools/command/bdist_egg.py Tue Apr 18 17:27:06 2006 @@ -8,7 +8,7 @@ from distutils.dir_util import remove_tree, mkpath from distutils.sysconfig import get_python_version, get_python_lib from distutils import log -from pkg_resources import get_platform, Distribution +from pkg_resources import get_build_platform, Distribution from types import CodeType from setuptools.extension import Library @@ -48,7 +48,7 @@ "temporary directory for creating the distribution"), ('plat-name=', 'p', "platform name to embed in generated filenames " - "(default: %s)" % get_platform()), + "(default: %s)" % get_build_platform()), ('exclude-source-files', None, "remove all .py files from the generated egg"), ('keep-temp', 'k', @@ -99,7 +99,7 @@ self.bdist_dir = os.path.join(bdist_base, 'egg') if self.plat_name is None: - self.plat_name = get_platform() + self.plat_name = get_build_platform() self.set_undefined_options('bdist',('dist_dir', 'dist_dir')) From python-checkins at python.org Tue Apr 18 17:30:06 2006 From: python-checkins at python.org (phillip.eby) Date: Tue, 18 Apr 2006 17:30:06 +0200 (CEST) Subject: [Python-checkins] r45537 - in python/trunk/Lib: pkg_resources.py setuptools.egg-info/PKG-INFO setuptools/command/bdist_egg.py Message-ID: <20060418153006.0A4D11E4015@bag.python.org> Author: phillip.eby Date: Tue Apr 18 17:30:05 2006 New Revision: 45537 Modified: python/trunk/Lib/pkg_resources.py python/trunk/Lib/setuptools.egg-info/PKG-INFO python/trunk/Lib/setuptools/command/bdist_egg.py Log: Split ``get_platform()`` into ``get_supported_platform()`` and ``get_build_platform()`` to work around a Mac versioning problem that caused the behavior of ``compatible_platforms()`` to be platform specific. Modified: python/trunk/Lib/pkg_resources.py ============================================================================== --- python/trunk/Lib/pkg_resources.py (original) +++ python/trunk/Lib/pkg_resources.py Tue Apr 18 17:30:05 2006 @@ -18,7 +18,7 @@ from os import utime, rename, unlink # capture these to bypass sandboxing from os import open as os_open -def _get_max_platform(plat): +def get_supported_platform(): """Return this platform's maximum compatible version. distutils.util.get_platform() normally reports the minimum version @@ -31,7 +31,7 @@ If this condition occurs for any other platform with a version in its platform strings, this function should be extended accordingly. """ - m = macosVersionString.match(plat) + plat = get_build_platform(); m = macosVersionString.match(plat) if m is not None and sys.platform == "darwin": try: plat = 'macosx-%s-%s' % ('.'.join(_macosx_vers()[:2]), m.group(3)) @@ -138,7 +138,7 @@ def _macosx_arch(machine): return {'PowerPC':'ppc', 'Power_Macintosh':'ppc'}.get(machine,machine) -def get_platform(): +def get_build_platform(): """Return this platform's string for platform-specific distributions XXX Currently this is the same as ``distutils.util.get_platform()``, but it @@ -160,7 +160,7 @@ macosVersionString = re.compile(r"macosx-(\d+)\.(\d+)-(.*)") darwinVersionString = re.compile(r"darwin-(\d+)\.(\d+)\.(\d+)-(.*)") - +get_platform = get_build_platform # XXX backward compat def compatible_platforms(provided,required): """Can code for the `provided` platform run on the `required` platform? @@ -171,8 +171,6 @@ """ if provided is None or required is None or provided==required: return True # easy case - provided = _get_max_platform(provided) - if provided==required: return True # Mac OS X special cases reqMac = macosVersionString.match(required) @@ -203,6 +201,8 @@ provMac.group(3) != reqMac.group(3): return False + + # is the required OS major update >= the provided one? if int(provMac.group(2)) > int(reqMac.group(2)): return False @@ -616,7 +616,7 @@ class Environment(object): """Searchable snapshot of distributions on a search path""" - def __init__(self,search_path=None,platform=get_platform(),python=PY_MAJOR): + def __init__(self, search_path=None, platform=get_supported_platform(), python=PY_MAJOR): """Snapshot distributions available on a search path Any distributions found on `search_path` are added to the environment. Modified: python/trunk/Lib/setuptools.egg-info/PKG-INFO ============================================================================== --- python/trunk/Lib/setuptools.egg-info/PKG-INFO (original) +++ python/trunk/Lib/setuptools.egg-info/PKG-INFO Tue Apr 18 17:30:05 2006 @@ -1,6 +1,6 @@ Metadata-Version: 1.0 Name: setuptools -Version: 0.7a1dev-r45521 +Version: 0.7a1dev-r45536 Summary: Download, build, install, upgrade, and uninstall Python packages -- easily! Home-page: http://peak.telecommunity.com/DevCenter/setuptools Author: Phillip J. Eby Modified: python/trunk/Lib/setuptools/command/bdist_egg.py ============================================================================== --- python/trunk/Lib/setuptools/command/bdist_egg.py (original) +++ python/trunk/Lib/setuptools/command/bdist_egg.py Tue Apr 18 17:30:05 2006 @@ -8,7 +8,7 @@ from distutils.dir_util import remove_tree, mkpath from distutils.sysconfig import get_python_version, get_python_lib from distutils import log -from pkg_resources import get_platform, Distribution +from pkg_resources import get_build_platform, Distribution from types import CodeType from setuptools.extension import Library @@ -48,7 +48,7 @@ "temporary directory for creating the distribution"), ('plat-name=', 'p', "platform name to embed in generated filenames " - "(default: %s)" % get_platform()), + "(default: %s)" % get_build_platform()), ('exclude-source-files', None, "remove all .py files from the generated egg"), ('keep-temp', 'k', @@ -99,7 +99,7 @@ self.bdist_dir = os.path.join(bdist_base, 'egg') if self.plat_name is None: - self.plat_name = get_platform() + self.plat_name = get_build_platform() self.set_undefined_options('bdist',('dist_dir', 'dist_dir')) From pje at telecommunity.com Tue Apr 18 17:52:57 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 18 Apr 2006 11:52:57 -0400 Subject: [Python-checkins] [Python-Dev] r45510 - python/trunk/Lib/pkgutil.py python/trunk/Lib/pydoc.py In-Reply-To: <4444A97D.40406@egenix.com> References: <20060418005956.156301E400A@bag.python.org> <20060418005956.156301E400A@bag.python.org> Message-ID: <5.1.1.6.0.20060418113808.0217a4a0@mail.telecommunity.com> At 10:55 AM 4/18/2006 +0200, M.-A. Lemburg wrote: >Phillip.eby wrote: > > Author: phillip.eby > > Date: Tue Apr 18 02:59:55 2006 > > New Revision: 45510 > > > > Modified: > > python/trunk/Lib/pkgutil.py > > python/trunk/Lib/pydoc.py > > Log: > > Second phase of refactoring for runpy, pkgutil, pydoc, and setuptools > > to share common PEP 302 support code, as described here: > > > > http://mail.python.org/pipermail/python-dev/2006-April/063724.html > >Shouldn't this new module be named "pkglib" to be in line with >the naming scheme used for all the other utility modules, e.g. httplib, >imaplib, poplib, etc. ? It's not a new module; it was previously a module with only one function in it, introduced in Python 2.3. If it were a new module, I'd have inquired about a name for it first. > > pydoc now supports PEP 302 importers, by way of utility functions in > > pkgutil, such as 'walk_packages()'. It will properly document > > modules that are in zip files, and is backward compatible to Python > > 2.3 (setuptools installs for Python <2.5 will bundle it so pydoc > > doesn't break when used with eggs.) > >Are you saying that the installation of setuptools in Python 2.3 >and 2.4 will then overwrite the standard pydoc included with >those versions ? Yes. As far as I can tell, there were no API changes to pydoc during this time, so this is effectively a "hot fix". This hot-fixing doesn't apply to setuptools system packages built with --root or --single-version-externally-managed, however, so OS vendors who build packages that wrap setuptools will have to make an explicit decision whether to also apply any fixes. If they do not, an end-user can of course install setuptools in their local PYTHONPATH and the hotfix will take effect. >I bothered by the fact that installing setuptools actually changes >the standard Python installation by either overriding stdlib modules >or monkey-patching them at setuptools import time. Please feel free to propose alternative solutions that will still ensure that setuptools "just works" for end-users. Both this and the pydoc hotfix are "practicality beats purity" issues. >Add setuptools to the stdlib ? I'm still missing the PEP for this >along with the needed discussion touching among other things, >the change of the distutils standard "python setup.py install" >to install an egg instead of a site package. Setuptools in the stdlib simply means that people wanting to use it can import it. It does not affect programs that do not import it. It also means that "python -m easy_install" is available without having to first download ez_setup.py. As for discussion, Guido originally brought up the question here a few months ago, and it's been listed in PEP 356 for a while. I've also posted things related to the inclusion both here and in distutils-sig. From python-checkins at python.org Tue Apr 18 18:16:35 2006 From: python-checkins at python.org (phillip.eby) Date: Tue, 18 Apr 2006 18:16:35 +0200 (CEST) Subject: [Python-checkins] r45538 - in sandbox/branches/setuptools-0.6: pkg_resources.py pkg_resources.txt setuptools/command/bdist_egg.py Message-ID: <20060418161635.4771F1E4002@bag.python.org> Author: phillip.eby Date: Tue Apr 18 18:16:33 2006 New Revision: 45538 Modified: sandbox/branches/setuptools-0.6/pkg_resources.py sandbox/branches/setuptools-0.6/pkg_resources.txt sandbox/branches/setuptools-0.6/setuptools/command/bdist_egg.py Log: Split ``get_platform()`` into ``get_supported_platform()`` and ``get_build_platform()`` to work around a Mac versioning problem that caused the behavior of ``compatible_platforms()`` to be platform specific. (Backport from 0.7 trunk, rev 45536) Modified: sandbox/branches/setuptools-0.6/pkg_resources.py ============================================================================== --- sandbox/branches/setuptools-0.6/pkg_resources.py (original) +++ sandbox/branches/setuptools-0.6/pkg_resources.py Tue Apr 18 18:16:33 2006 @@ -18,7 +18,7 @@ from os import utime, rename, unlink # capture these to bypass sandboxing from os import open as os_open -def _get_max_platform(plat): +def get_supported_platform(): """Return this platform's maximum compatible version. distutils.util.get_platform() normally reports the minimum version @@ -31,7 +31,7 @@ If this condition occurs for any other platform with a version in its platform strings, this function should be extended accordingly. """ - m = macosVersionString.match(plat) + plat = get_build_platform(); m = macosVersionString.match(plat) if m is not None and sys.platform == "darwin": try: plat = 'macosx-%s-%s' % ('.'.join(_macosx_vers()[:2]), m.group(3)) @@ -136,7 +136,7 @@ def _macosx_arch(machine): return {'PowerPC':'ppc', 'Power_Macintosh':'ppc'}.get(machine,machine) -def get_platform(): +def get_build_platform(): """Return this platform's string for platform-specific distributions XXX Currently this is the same as ``distutils.util.get_platform()``, but it @@ -158,7 +158,7 @@ macosVersionString = re.compile(r"macosx-(\d+)\.(\d+)-(.*)") darwinVersionString = re.compile(r"darwin-(\d+)\.(\d+)\.(\d+)-(.*)") - +get_platform = get_build_platform # XXX backward compat @@ -171,8 +171,6 @@ """ if provided is None or required is None or provided==required: return True # easy case - provided = _get_max_platform(provided) - if provided==required: return True # Mac OS X special cases reqMac = macosVersionString.match(required) @@ -203,6 +201,8 @@ provMac.group(3) != reqMac.group(3): return False + + # is the required OS major update >= the provided one? if int(provMac.group(2)) > int(reqMac.group(2)): return False @@ -616,7 +616,7 @@ class Environment(object): """Searchable snapshot of distributions on a search path""" - def __init__(self,search_path=None,platform=get_platform(),python=PY_MAJOR): + def __init__(self, search_path=None, platform=get_supported_platform(), python=PY_MAJOR): """Snapshot distributions available on a search path Any distributions found on `search_path` are added to the environment. Modified: sandbox/branches/setuptools-0.6/pkg_resources.txt ============================================================================== --- sandbox/branches/setuptools-0.6/pkg_resources.txt (original) +++ sandbox/branches/setuptools-0.6/pkg_resources.txt Tue Apr 18 18:16:33 2006 @@ -441,7 +441,7 @@ ``Environment`` objects are used by ``pkg_resources`` to index available distributions during dependency resolution. -``Environment(search_path=None, platform=get_platform(), python=PY_MAJOR)`` +``Environment(search_path=None, platform=get_supported_platform(), python=PY_MAJOR)`` Create an environment snapshot by scanning `search_path` for distributions compatible with `platform` and `python`. `search_path` should be a sequence of strings such as might be used on ``sys.path``. If a @@ -1590,11 +1590,21 @@ Platform Utilities ------------------ -``get_platform()`` +``get_build_platform()`` Return this platform's identifier string. For Windows, the return value is ``"win32"``, and for Mac OS X it is a string of the form ``"macosx-10.4-ppc"``. All other platforms return the same uname-based string that the ``distutils.util.get_platform()`` function returns. + This string is the minimum platform version required by distributions built + on the local machine. (Backward compatibility note: setuptools versions + prior to 0.6b1 called this function ``get_platform()``, and the function is + still available under that name for backward compatibility reasons.) + +``get_supported_platform()`` (New in 0.6b1) + This is the similar to ``get_build_platform()``, but is the maximum + platform version that the local machine supports. You will usually want + to use this value as the ``provided`` argument to the + ``compatible_platforms()`` function. ``compatible_platforms(provided, required)`` Return true if a distribution built on the `provided` platform may be used @@ -1652,6 +1662,11 @@ Release Notes/Change History ---------------------------- +0.6b1 + * Split ``get_platform()`` into ``get_supported_platform()`` and + ``get_build_platform()`` to work around a Mac versioning problem that caused + the behavior of ``compatible_platforms()`` to be platform specific. + 0.6a11 * Added ``ExtractionError`` and ``ResourceManager.extraction_error()`` so that cache permission problems get a more user-friendly explanation of the Modified: sandbox/branches/setuptools-0.6/setuptools/command/bdist_egg.py ============================================================================== --- sandbox/branches/setuptools-0.6/setuptools/command/bdist_egg.py (original) +++ sandbox/branches/setuptools-0.6/setuptools/command/bdist_egg.py Tue Apr 18 18:16:33 2006 @@ -8,7 +8,7 @@ from distutils.dir_util import remove_tree, mkpath from distutils.sysconfig import get_python_version, get_python_lib from distutils import log -from pkg_resources import get_platform, Distribution +from pkg_resources import get_build_platform, Distribution from types import CodeType from setuptools.extension import Library @@ -48,7 +48,7 @@ "temporary directory for creating the distribution"), ('plat-name=', 'p', "platform name to embed in generated filenames " - "(default: %s)" % get_platform()), + "(default: %s)" % get_build_platform()), ('exclude-source-files', None, "remove all .py files from the generated egg"), ('keep-temp', 'k', @@ -99,7 +99,7 @@ self.bdist_dir = os.path.join(bdist_base, 'egg') if self.plat_name is None: - self.plat_name = get_platform() + self.plat_name = get_build_platform() self.set_undefined_options('bdist',('dist_dir', 'dist_dir')) From python-checkins at python.org Tue Apr 18 18:18:15 2006 From: python-checkins at python.org (george.yoshida) Date: Tue, 18 Apr 2006 18:18:15 +0200 (CEST) Subject: [Python-checkins] r45539 - python/trunk/Doc/lib/libxmlrpclib.tex Message-ID: <20060418161815.D7E461E4002@bag.python.org> Author: george.yoshida Date: Tue Apr 18 18:18:15 2006 New Revision: 45539 Modified: python/trunk/Doc/lib/libxmlrpclib.tex Log: fix typo Modified: python/trunk/Doc/lib/libxmlrpclib.tex ============================================================================== --- python/trunk/Doc/lib/libxmlrpclib.tex (original) +++ python/trunk/Doc/lib/libxmlrpclib.tex Tue Apr 18 18:18:15 2006 @@ -203,7 +203,7 @@ \subsection{Binary Objects \label{binary-objects}} -This class may initialized from string data (which may include NULs). +This class may be initialized from string data (which may include NULs). The primary access to the content of a \class{Binary} object is provided by an attribute: From python-checkins at python.org Tue Apr 18 18:20:10 2006 From: python-checkins at python.org (george.yoshida) Date: Tue, 18 Apr 2006 18:20:10 +0200 (CEST) Subject: [Python-checkins] r45540 - python/branches/release24-maint/Doc/lib/libxmlrpclib.tex Message-ID: <20060418162010.2FD1E1E4018@bag.python.org> Author: george.yoshida Date: Tue Apr 18 18:20:09 2006 New Revision: 45540 Modified: python/branches/release24-maint/Doc/lib/libxmlrpclib.tex Log: fix typo Modified: python/branches/release24-maint/Doc/lib/libxmlrpclib.tex ============================================================================== --- python/branches/release24-maint/Doc/lib/libxmlrpclib.tex (original) +++ python/branches/release24-maint/Doc/lib/libxmlrpclib.tex Tue Apr 18 18:20:09 2006 @@ -189,7 +189,7 @@ \subsection{Binary Objects \label{binary-objects}} -This class may initialized from string data (which may include NULs). +This class may be initialized from string data (which may include NULs). The primary access to the content of a \class{Binary} object is provided by an attribute: From python-checkins at python.org Tue Apr 18 18:45:15 2006 From: python-checkins at python.org (phillip.eby) Date: Tue, 18 Apr 2006 18:45:15 +0200 (CEST) Subject: [Python-checkins] r45541 - python/trunk/Misc/NEWS Message-ID: <20060418164515.3E6B41E4016@bag.python.org> Author: phillip.eby Date: Tue Apr 18 18:45:14 2006 New Revision: 45541 Modified: python/trunk/Misc/NEWS Log: add info re: pydoc, pkgutil, and setuptools additions Modified: python/trunk/Misc/NEWS ============================================================================== --- python/trunk/Misc/NEWS (original) +++ python/trunk/Misc/NEWS Tue Apr 18 18:45:14 2006 @@ -74,6 +74,17 @@ Library ------- +- New modules: setuptools, easy_install, and pkg_resources, to support + building, installing, and using Python eggs, respectively. + +- The pydoc module now supports documenting packages contained in + .zip or .egg files. + +- The pkgutil module now has several new utility functions, such + as ``walk_packages()`` to support working with packages that are either + in the filesystem or zip files. + + - The ``__del__`` method of class ``local`` in module ``_threading_local`` returned before accomplishing any of its intended cleanup. From mal at egenix.com Tue Apr 18 19:15:19 2006 From: mal at egenix.com (M.-A. Lemburg) Date: Tue, 18 Apr 2006 19:15:19 +0200 Subject: [Python-checkins] [Python-Dev] setuptools in the stdlib ( r45510 - python/trunk/Lib/pkgutil.py python/trunk/Lib/pydoc.py) In-Reply-To: <5.1.1.6.0.20060418113808.0217a4a0@mail.telecommunity.com> References: <20060418005956.156301E400A@bag.python.org> <20060418005956.156301E400A@bag.python.org> <5.1.1.6.0.20060418113808.0217a4a0@mail.telecommunity.com> Message-ID: <44451EA7.4020907@egenix.com> Phillip J. Eby wrote: > At 10:55 AM 4/18/2006 +0200, M.-A. Lemburg wrote: >> Phillip.eby wrote: >> > Author: phillip.eby >> > Date: Tue Apr 18 02:59:55 2006 >> > New Revision: 45510 >> > >> > Modified: >> > python/trunk/Lib/pkgutil.py >> > python/trunk/Lib/pydoc.py >> > Log: >> > Second phase of refactoring for runpy, pkgutil, pydoc, and setuptools >> > to share common PEP 302 support code, as described here: >> > >> > http://mail.python.org/pipermail/python-dev/2006-April/063724.html >> >> Shouldn't this new module be named "pkglib" to be in line with >> the naming scheme used for all the other utility modules, e.g. httplib, >> imaplib, poplib, etc. ? > > It's not a new module; it was previously a module with only one function > in it, introduced in Python 2.3. If it were a new module, I'd have > inquired about a name for it first. Didn't realize that. Too bad the time machine didn't work on this one :-/ >> > pydoc now supports PEP 302 importers, by way of utility functions in >> > pkgutil, such as 'walk_packages()'. It will properly document >> > modules that are in zip files, and is backward compatible to Python >> > 2.3 (setuptools installs for Python <2.5 will bundle it so pydoc >> > doesn't break when used with eggs.) >> >> Are you saying that the installation of setuptools in Python 2.3 >> and 2.4 will then overwrite the standard pydoc included with >> those versions ? > > Yes. As far as I can tell, there were no API changes to pydoc during > this time, so this is effectively a "hot fix". Why should a 3rd party extension be hot-fixing the standard Python distribution ? > This hot-fixing doesn't apply to setuptools system packages built with > --root or --single-version-externally-managed, however, so OS vendors > who build packages that wrap setuptools will have to make an explicit > decision whether to also apply any fixes. If they do not, an end-user > can of course install setuptools in their local PYTHONPATH and the > hotfix will take effect. What does setuptools have to do with pydoc ? Why should a user installing setuptools assume that some arbitrary stdlib modules get (effectively) replaced by installing setuptools ? If you want to provide a hot fix for Python 2.3 and 2.4, you should make this a separate install, so that users are aware that their Python distribution is about to get modified in ways that have nothing to do with setuptools. >> I bothered by the fact that installing setuptools actually changes >> the standard Python installation by either overriding stdlib modules >> or monkey-patching them at setuptools import time. > > Please feel free to propose alternative solutions that will still ensure > that setuptools "just works" for end-users. Both this and the pydoc > hotfix are "practicality beats purity" issues. Not really. I'd consider them design flaws. distutils is built to be extended without having to monkey-patch it, e.g. you can easily override commands with your own variants by supplying them via the cmdclass and distclass keyword arguments to setup(). By monkey patching distutils during import of setuptools, you effectively *change* distutils at run-time and not only for the application space that you implement in setuptools, but for all the rest of the application. If an application wants to use setuptools for e.g. plugin management, then standard distutils features will get replaced by setuptools implementations which are not compatible to the standard distutils commands, effectively making it impossible to access the original versions. Monkey patching is only a last resort in case nothing else works. In this case, it's not needed, since distutils provides the interfaces needed to extend its command classes via the setup() call. See e.g. mxSetup.py in the eGenix extensions for an example of how effective the distutils design can be put to use without having to introduce lots of unwanted side-effects. >> Add setuptools to the stdlib ? I'm still missing the PEP for this >> along with the needed discussion touching among other things, >> the change of the distutils standard "python setup.py install" >> to install an egg instead of a site package. > > Setuptools in the stdlib simply means that people wanting to use it can > import it. It does not affect programs that do not import it. It also > means that "python -m easy_install" is available without having to first > download ez_setup.py. Doesn't really seem like much of an argument for the addition... the above is true for any 3rd party module. > As for discussion, Guido originally brought up the question here a few > months ago, and it's been listed in PEP 356 for a while. I've also > posted things related to the inclusion both here and in distutils-sig. I know, but the discussions haven't really helped much in getting the setuptools design compatible with standard distutils. Unless that's being put in place, I'm -1 on the addition, due to the invasive nature of setuptools and its various side-effects on systems management. Note that it only takes one single module in an application doing "import setuptools" for whatever reason, to have it modify distutils at run-time. The PEP is needed to document these effects, discuss their merits/drawbacks and find solutions to the problems prior to creating a status quo that's hard to modify after the fact due to the large install base of the Python distribution. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Apr 18 2006) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From python-checkins at python.org Tue Apr 18 19:32:19 2006 From: python-checkins at python.org (tim.peters) Date: Tue, 18 Apr 2006 19:32:19 +0200 (CEST) Subject: [Python-checkins] r45542 - in python/trunk/Lib: easy_install.py pkg_resources.py pkgutil.py setuptools/__init__.py setuptools/archive_util.py setuptools/command/__init__.py setuptools/command/alias.py setuptools/command/bdist_egg.py setuptools/command/bdist_rpm.py setuptools/command/build_ext.py setuptools/command/build_py.py setuptools/command/develop.py setuptools/command/easy_install.py setuptools/command/egg_info.py setuptools/command/install.py setuptools/command/install_egg_info.py setuptools/command/install_lib.py setuptools/command/install_scripts.py setuptools/command/rotate.py setuptools/command/saveopts.py setuptools/command/sdist.py setuptools/command/setopt.py setuptools/command/test.py setuptools/depends.py setuptools/dist.py setuptools/extension.py setuptools/package_index.py setuptools/sandbox.py setuptools/site-patch.py setuptools/tests/__init__.py setuptools/tests/test_resources.py Message-ID: <20060418173219.68C301E4002@bag.python.org> Author: tim.peters Date: Tue Apr 18 19:32:12 2006 New Revision: 45542 Modified: python/trunk/Lib/easy_install.py python/trunk/Lib/pkg_resources.py python/trunk/Lib/pkgutil.py python/trunk/Lib/setuptools/__init__.py python/trunk/Lib/setuptools/archive_util.py python/trunk/Lib/setuptools/command/__init__.py python/trunk/Lib/setuptools/command/alias.py python/trunk/Lib/setuptools/command/bdist_egg.py python/trunk/Lib/setuptools/command/bdist_rpm.py python/trunk/Lib/setuptools/command/build_ext.py python/trunk/Lib/setuptools/command/build_py.py python/trunk/Lib/setuptools/command/develop.py python/trunk/Lib/setuptools/command/easy_install.py python/trunk/Lib/setuptools/command/egg_info.py python/trunk/Lib/setuptools/command/install.py python/trunk/Lib/setuptools/command/install_egg_info.py python/trunk/Lib/setuptools/command/install_lib.py python/trunk/Lib/setuptools/command/install_scripts.py python/trunk/Lib/setuptools/command/rotate.py python/trunk/Lib/setuptools/command/saveopts.py python/trunk/Lib/setuptools/command/sdist.py python/trunk/Lib/setuptools/command/setopt.py python/trunk/Lib/setuptools/command/test.py python/trunk/Lib/setuptools/depends.py python/trunk/Lib/setuptools/dist.py python/trunk/Lib/setuptools/extension.py python/trunk/Lib/setuptools/package_index.py python/trunk/Lib/setuptools/sandbox.py python/trunk/Lib/setuptools/site-patch.py python/trunk/Lib/setuptools/tests/__init__.py python/trunk/Lib/setuptools/tests/test_resources.py Log: Whilespace normalization (reindint.py). Modified: python/trunk/Lib/easy_install.py ============================================================================== --- python/trunk/Lib/easy_install.py (original) +++ python/trunk/Lib/easy_install.py Tue Apr 18 19:32:12 2006 @@ -3,4 +3,3 @@ if __name__ == '__main__': from setuptools.command.easy_install import main main() - Modified: python/trunk/Lib/pkg_resources.py ============================================================================== --- python/trunk/Lib/pkg_resources.py (original) +++ python/trunk/Lib/pkg_resources.py Tue Apr 18 19:32:12 2006 @@ -58,7 +58,7 @@ # Exceptions 'ResolutionError','VersionConflict','DistributionNotFound','UnknownExtra', 'ExtractionError', - + # Parsing functions and string utilities 'parse_requirements', 'parse_version', 'safe_name', 'safe_version', 'get_platform', 'compatible_platforms', 'yield_lines', 'split_sections', @@ -823,7 +823,7 @@ old_exc = sys.exc_info()[1] cache_path = self.extraction_path or get_default_cache() - + err = ExtractionError("""Can't extract file(s) to egg cache The following error occurred while trying to extract file(s) to the Python egg @@ -878,7 +878,7 @@ ensure_directory(target_path) except: self.extraction_error() - + self.cached_files[target_path] = 1 return target_path @@ -1264,11 +1264,11 @@ try: rename(tmpnam, real_path) - - except os.error: + + except os.error: if os.path.isfile(real_path): stat = os.stat(real_path) - + if stat.st_size==size and stat.st_mtime==timestamp: # size and stamp match, somebody did it just ahead of # us, so we're done @@ -2375,4 +2375,3 @@ # calling ``require()``) will get activated as well. add_activation_listener(lambda dist: dist.activate()) working_set.entries=[]; map(working_set.add_entry,sys.path) # match order - Modified: python/trunk/Lib/pkgutil.py ============================================================================== --- python/trunk/Lib/pkgutil.py (original) +++ python/trunk/Lib/pkgutil.py Tue Apr 18 19:32:12 2006 @@ -44,7 +44,7 @@ class cls(cls,object): pass mro = cls.__mro__[1:] except TypeError: - mro = object, # must be an ExtensionClass or some such :( + mro = object, # must be an ExtensionClass or some such :( for t in mro: if t in registry: return registry[t](*args,**kw) @@ -64,7 +64,7 @@ wrapper.__dict__ = func.__dict__ wrapper.__doc__ = func.__doc__ wrapper.register = register - return wrapper + return wrapper def walk_packages(path=None, prefix='', onerror=None): @@ -160,7 +160,7 @@ modname = inspect.getmodulename(fn) if modname=='__init__' or modname in yielded: continue - + path = os.path.join(self.path, fn) ispkg = False @@ -276,7 +276,7 @@ try: import zipimport from zipimport import zipimporter - + def iter_zipimport_modules(importer, prefix=''): dirlist = zipimport._zip_directory_cache[importer.archive].keys() dirlist.sort() Modified: python/trunk/Lib/setuptools/__init__.py ============================================================================== --- python/trunk/Lib/setuptools/__init__.py (original) +++ python/trunk/Lib/setuptools/__init__.py Tue Apr 18 19:32:12 2006 @@ -40,7 +40,7 @@ return out setup = distutils.core.setup - + _Command = _get_unpatched(_Command) class Command(_Command): @@ -53,7 +53,7 @@ _Command.__init__(self,dist) for k,v in kw.items(): setattr(self,k,v) - + def reinitialize_command(self, command, reinit_subcommands=0, **kw): cmd = _Command.reinitialize_command(self, command, reinit_subcommands) for k,v in kw.items(): @@ -62,21 +62,3 @@ import distutils.core distutils.core.Command = Command # we can't patch distutils.cmd, alas - - - - - - - - - - - - - - - - - - Modified: python/trunk/Lib/setuptools/archive_util.py ============================================================================== --- python/trunk/Lib/setuptools/archive_util.py (original) +++ python/trunk/Lib/setuptools/archive_util.py Tue Apr 18 19:32:12 2006 @@ -14,7 +14,7 @@ """Couldn't recognize the archive type""" def default_filter(src,dst): - """The default progress/filter callback; returns True for all files""" + """The default progress/filter callback; returns True for all files""" return dst @@ -184,7 +184,7 @@ name = member.name # don't extract absolute paths or ones with .. in them if not name.startswith('/') and '..' not in name: - dst = os.path.join(extract_dir, *name.split('/')) + dst = os.path.join(extract_dir, *name.split('/')) dst = progress_filter(name, dst) if dst: if dst.endswith(os.sep): @@ -198,8 +198,3 @@ extraction_drivers = unpack_directory, unpack_zipfile, unpack_tarfile - - - - - Modified: python/trunk/Lib/setuptools/command/__init__.py ============================================================================== --- python/trunk/Lib/setuptools/command/__init__.py (original) +++ python/trunk/Lib/setuptools/command/__init__.py Tue Apr 18 19:32:12 2006 @@ -8,7 +8,7 @@ if sys.version>='2.5': # In Python 2.5 and above, distutils includes its own upload command __all__.remove('upload') - + from distutils.command.bdist import bdist Modified: python/trunk/Lib/setuptools/command/alias.py ============================================================================== --- python/trunk/Lib/setuptools/command/alias.py (original) +++ python/trunk/Lib/setuptools/command/alias.py Tue Apr 18 19:32:12 2006 @@ -11,17 +11,17 @@ if c in arg: return repr(arg) if arg.split()<>[arg]: return repr(arg) - return arg + return arg class alias(option_base): """Define a shortcut that invokes one or more commands""" - + description = "define a shortcut to invoke one or more commands" command_consumes_arguments = True user_options = [ - ('remove', 'r', 'remove (unset) the alias'), + ('remove', 'r', 'remove (unset) the alias'), ] + option_base.user_options boolean_options = option_base.boolean_options + ['remove'] @@ -77,6 +77,3 @@ else: source = '--filename=%r' % source return source+name+' '+command - - - Modified: python/trunk/Lib/setuptools/command/bdist_egg.py ============================================================================== --- python/trunk/Lib/setuptools/command/bdist_egg.py (original) +++ python/trunk/Lib/setuptools/command/bdist_egg.py Tue Apr 18 19:32:12 2006 @@ -233,7 +233,7 @@ if self.exclude_source_files: self.zap_pyfiles() - + # Make the archive make_zipfile(self.egg_output, archive_root, verbose=self.verbose, dry_run=self.dry_run) @@ -262,7 +262,7 @@ def make_init_files(self): """Create missing package __init__ files""" - init_files = [] + init_files = [] for base,dirs,files in walk_egg(self.bdist_dir): if base==self.bdist_dir: # don't put an __init__ in the root @@ -276,7 +276,7 @@ filename = os.path.join(base,'__init__.py') if not self.dry_run: f = open(filename,'w'); f.write(NS_PKG_STUB) - f.close() + f.close() init_files.append(filename) break else: @@ -329,7 +329,7 @@ def walk_egg(egg_dir): """Walk an unpacked egg's contents, skipping the metadata directory""" walker = os.walk(egg_dir) - base,dirs,files = walker.next() + base,dirs,files = walker.next() if 'EGG-INFO' in dirs: dirs.remove('EGG-INFO') yield base,dirs,files @@ -447,5 +447,3 @@ os.path.walk(base_dir, visit, None) return zip_filename - - Modified: python/trunk/Lib/setuptools/command/bdist_rpm.py ============================================================================== --- python/trunk/Lib/setuptools/command/bdist_rpm.py (original) +++ python/trunk/Lib/setuptools/command/bdist_rpm.py Tue Apr 18 19:32:12 2006 @@ -35,34 +35,3 @@ ] spec.insert(spec.index(line24)+1, "%define unmangled_version "+version) return spec - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Modified: python/trunk/Lib/setuptools/command/build_ext.py ============================================================================== --- python/trunk/Lib/setuptools/command/build_ext.py (original) +++ python/trunk/Lib/setuptools/command/build_ext.py Tue Apr 18 19:32:12 2006 @@ -283,5 +283,3 @@ self.create_static_lib( objects, basename, output_dir, debug, target_lang ) - - Modified: python/trunk/Lib/setuptools/command/build_py.py ============================================================================== --- python/trunk/Lib/setuptools/command/build_py.py (original) +++ python/trunk/Lib/setuptools/command/build_py.py Tue Apr 18 19:32:12 2006 @@ -93,7 +93,7 @@ ei_cmd = self.get_finalized_command('egg_info') for path in ei_cmd.filelist.files: if path.endswith('.py'): - continue + continue d,f = os.path.split(assert_relative(path)) prev = None while d and d!=prev and d not in src_dirs: @@ -142,7 +142,7 @@ f = open(init_py,'rU') if 'declare_namespace' not in f.read(): - from distutils.errors import DistutilsError + from distutils.errors import DistutilsError raise DistutilsError( "Namespace package problem: %s is a namespace package, but its\n" "__init__.py does not call declare_namespace()! Please fix it.\n" @@ -167,7 +167,7 @@ globs = (self.exclude_package_data.get('', []) + self.exclude_package_data.get(package, [])) bad = [] - for pattern in globs: + for pattern in globs: bad.extend( fnmatch.filter( files, os.path.join(src_dir, convert_path(pattern)) @@ -190,16 +190,3 @@ setup.py directory, *never* absolute paths. """ % path ) - - - - - - - - - - - - - Modified: python/trunk/Lib/setuptools/command/develop.py ============================================================================== --- python/trunk/Lib/setuptools/command/develop.py (original) +++ python/trunk/Lib/setuptools/command/develop.py Tue Apr 18 19:32:12 2006 @@ -46,7 +46,7 @@ "Please rename %r to %r before using 'develop'" % (ei.egg_info, ei.broken_egg_info) ) - self.args = [ei.egg_name] + self.args = [ei.egg_name] easy_install.finalize_options(self) self.egg_link = os.path.join(self.install_dir, ei.egg_name+'.egg-link') self.egg_base = ei.egg_base @@ -104,7 +104,7 @@ # create wrapper scripts in the script dir, pointing to dist.scripts # new-style... - self.install_wrapper_scripts(dist) + self.install_wrapper_scripts(dist) # ...and old-style for script_name in self.distribution.scripts or []: @@ -114,10 +114,3 @@ script_text = f.read() f.close() self.install_script(dist, script_name, script_text, script_path) - - - - - - - Modified: python/trunk/Lib/setuptools/command/easy_install.py ============================================================================== --- python/trunk/Lib/setuptools/command/easy_install.py (original) +++ python/trunk/Lib/setuptools/command/easy_install.py Tue Apr 18 19:32:12 2006 @@ -1357,7 +1357,7 @@ """Write changed .pth file back to disk""" if not self.dirty: return - + data = '\n'.join(map(self.make_relative,self.paths)) if data: log.debug("Saving %s", self.filename) @@ -1434,7 +1434,7 @@ del zdc[p] return - + def get_script_args(dist, executable=sys_executable): """Yield write_script() argument tuples for a distribution's entrypoints""" spec = str(dist.as_requirement()) @@ -1553,8 +1553,3 @@ distclass=DistributionWithoutHelpCommands, **kw ) ) - - - - - Modified: python/trunk/Lib/setuptools/command/egg_info.py ============================================================================== --- python/trunk/Lib/setuptools/command/egg_info.py (original) +++ python/trunk/Lib/setuptools/command/egg_info.py Tue Apr 18 19:32:12 2006 @@ -363,7 +363,3 @@ if match: return int(match.group(1)) return 0 - - - - Modified: python/trunk/Lib/setuptools/command/install.py ============================================================================== --- python/trunk/Lib/setuptools/command/install.py (original) +++ python/trunk/Lib/setuptools/command/install.py Tue Apr 18 19:32:12 2006 @@ -60,7 +60,7 @@ caller = sys._getframe(2) caller_module = caller.f_globals.get('__name__','') caller_name = caller.f_code.co_name - + if caller_module != 'distutils.dist' or caller_name!='run_commands': # We weren't called from the command line or setup(), so we # should run in backward-compatibility mode to support bdist_* @@ -68,7 +68,7 @@ _install.run(self) else: self.do_egg_install() - + @@ -99,25 +99,3 @@ cmd.args = args cmd.run() setuptools.bootstrap_install_from = None - - - - - - - - - - - - - - - - - - - - - - Modified: python/trunk/Lib/setuptools/command/install_egg_info.py ============================================================================== --- python/trunk/Lib/setuptools/command/install_egg_info.py (original) +++ python/trunk/Lib/setuptools/command/install_egg_info.py Tue Apr 18 19:32:12 2006 @@ -22,7 +22,7 @@ None, None, ei_cmd.egg_name, ei_cmd.egg_version ).egg_name()+'.egg-info' self.source = ei_cmd.egg_info - self.target = os.path.join(self.install_dir, basename) + self.target = os.path.join(self.install_dir, basename) self.outputs = [self.target] def run(self): @@ -43,7 +43,7 @@ return self.outputs def copytree(self): - # Copy the .egg-info tree to site-packages + # Copy the .egg-info tree to site-packages def skimmer(src,dst): # filter out source-control directories; note that 'src' is always # a '/'-separated path, regardless of platform. 'dst' is a @@ -78,5 +78,4 @@ "(p not in mp) and mp.append(p)\n" % locals() ) - f.close() - + f.close() Modified: python/trunk/Lib/setuptools/command/install_lib.py ============================================================================== --- python/trunk/Lib/setuptools/command/install_lib.py (original) +++ python/trunk/Lib/setuptools/command/install_lib.py Tue Apr 18 19:32:12 2006 @@ -74,9 +74,3 @@ if exclude: return [f for f in outputs if f not in exclude] return outputs - - - - - - Modified: python/trunk/Lib/setuptools/command/install_scripts.py ============================================================================== --- python/trunk/Lib/setuptools/command/install_scripts.py (original) +++ python/trunk/Lib/setuptools/command/install_scripts.py Tue Apr 18 19:32:12 2006 @@ -11,7 +11,7 @@ def initialize_options(self): _install_scripts.initialize_options(self) self.no_ep = False - + def run(self): self.run_command("egg_info") if self.distribution.scripts: @@ -20,9 +20,9 @@ self.outfiles = [] if self.no_ep: # don't install entry point scripts into .egg file! - return + return - ei_cmd = self.get_finalized_command("egg_info") + ei_cmd = self.get_finalized_command("egg_info") dist = Distribution( ei_cmd.egg_base, PathMetadata(ei_cmd.egg_base, ei_cmd.egg_info), ei_cmd.egg_name, ei_cmd.egg_version, @@ -54,29 +54,3 @@ os.chmod(target,0755) except (AttributeError, os.error): pass - - - - - - - - - - - - - - - - - - - - - - - - - - Modified: python/trunk/Lib/setuptools/command/rotate.py ============================================================================== --- python/trunk/Lib/setuptools/command/rotate.py (original) +++ python/trunk/Lib/setuptools/command/rotate.py Tue Apr 18 19:32:12 2006 @@ -28,7 +28,7 @@ "(e.g. '.zip' or '.egg')" ) if self.keep is None: - raise DistutilsOptionError("Must specify number of files to keep") + raise DistutilsOptionError("Must specify number of files to keep") try: self.keep = int(self.keep) except ValueError: @@ -55,28 +55,3 @@ log.info("Deleting %s", f) if not self.dry_run: os.unlink(f) - - - - - - - - - - - - - - - - - - - - - - - - - Modified: python/trunk/Lib/setuptools/command/saveopts.py ============================================================================== --- python/trunk/Lib/setuptools/command/saveopts.py (original) +++ python/trunk/Lib/setuptools/command/saveopts.py Tue Apr 18 19:32:12 2006 @@ -22,4 +22,3 @@ settings.setdefault(cmd,{})[opt] = val edit_config(self.filename, settings, self.dry_run) - Modified: python/trunk/Lib/setuptools/command/sdist.py ============================================================================== --- python/trunk/Lib/setuptools/command/sdist.py (original) +++ python/trunk/Lib/setuptools/command/sdist.py Tue Apr 18 19:32:12 2006 @@ -144,7 +144,7 @@ self.filelist.append(os.path.join(ei_cmd.egg_info,'SOURCES.txt')) self.check_metadata() - self.make_distribution() + self.make_distribution() dist_files = getattr(self.distribution,'dist_files',[]) for file in self.archive_files: @@ -161,4 +161,3 @@ # dying and thus masking the real error sys.exc_info()[2].tb_next.tb_frame.f_locals['template'].close() raise - Modified: python/trunk/Lib/setuptools/command/setopt.py ============================================================================== --- python/trunk/Lib/setuptools/command/setopt.py (original) +++ python/trunk/Lib/setuptools/command/setopt.py Tue Apr 18 19:32:12 2006 @@ -82,7 +82,7 @@ class option_base(Command): """Abstract base class for commands that mess with config files""" - + user_options = [ ('global-config', 'g', "save options to the site-wide distutils.cfg file"), @@ -94,7 +94,7 @@ boolean_options = [ 'global-config', 'user-config', - ] + ] def initialize_options(self): self.global_config = None @@ -116,7 +116,7 @@ "Must specify only one configuration file option", filenames ) - self.filename, = filenames + self.filename, = filenames @@ -130,7 +130,7 @@ ('command=', 'c', 'command to set an option for'), ('option=', 'o', 'option to set'), ('set-value=', 's', 'value of the option'), - ('remove', 'r', 'remove (unset) the value'), + ('remove', 'r', 'remove (unset) the value'), ] + option_base.user_options boolean_options = option_base.boolean_options + ['remove'] @@ -156,9 +156,3 @@ }, self.dry_run ) - - - - - - Modified: python/trunk/Lib/setuptools/command/test.py ============================================================================== --- python/trunk/Lib/setuptools/command/test.py (original) +++ python/trunk/Lib/setuptools/command/test.py Tue Apr 18 19:32:12 2006 @@ -88,7 +88,7 @@ self.reinitialize_command('build_ext', inplace=1) self.run_command('build_ext') - if self.distribution.tests_require: + if self.distribution.tests_require: self.distribution.fetch_build_eggs(self.distribution.tests_require) if self.test_suite: @@ -117,7 +117,3 @@ None, None, [unittest.__file__]+self.test_args, testLoader = loader_class() ) - - - - Modified: python/trunk/Lib/setuptools/depends.py ============================================================================== --- python/trunk/Lib/setuptools/depends.py (original) +++ python/trunk/Lib/setuptools/depends.py Tue Apr 18 19:32:12 2006 @@ -237,10 +237,3 @@ return const else: const = default - - - - - - - Modified: python/trunk/Lib/setuptools/dist.py ============================================================================== --- python/trunk/Lib/setuptools/dist.py (original) +++ python/trunk/Lib/setuptools/dist.py Tue Apr 18 19:32:12 2006 @@ -796,25 +796,3 @@ " doesn't contain any packages or modules under %s" % (self.description, item, item) ) - - - - - - - - - - - - - - - - - - - - - - Modified: python/trunk/Lib/setuptools/extension.py ============================================================================== --- python/trunk/Lib/setuptools/extension.py (original) +++ python/trunk/Lib/setuptools/extension.py Tue Apr 18 19:32:12 2006 @@ -14,7 +14,7 @@ """Extension that uses '.c' files in place of '.pyx' files""" if not have_pyrex: - # convert .pyx extensions to .c + # convert .pyx extensions to .c def __init__(self,*args,**kw): _Extension.__init__(self,*args,**kw) sources = [] @@ -33,4 +33,3 @@ distutils.extension.Extension = Extension if 'distutils.command.build_ext' in sys.modules: sys.modules['distutils.command.build_ext'].Extension = Extension - Modified: python/trunk/Lib/setuptools/package_index.py ============================================================================== --- python/trunk/Lib/setuptools/package_index.py (original) +++ python/trunk/Lib/setuptools/package_index.py Tue Apr 18 19:32:12 2006 @@ -260,7 +260,7 @@ def find_packages(self, requirement): self.scan_url(self.index_url + requirement.unsafe_name+'/') - + if not self.package_pages.get(requirement.key): # Fall back to safe version of the name self.scan_url(self.index_url + requirement.project_name+'/') @@ -489,7 +489,7 @@ "Can't process plain .py files without an '#egg=name-version'" " suffix to enable automatic setup script generation." ) - + dl_blocksize = 8192 def _download_to(self, url, filename): self.url_ok(url,True) # raises error if not allowed @@ -672,26 +672,3 @@ # DNS-bl0ck1n9 f1r3w4llz sUx0rs! _sf_mirrors[:] = ['dl.sourceforge.net'] return random.choice(_sf_mirrors) - - - - - - - - - - - - - - - - - - - - - - - Modified: python/trunk/Lib/setuptools/sandbox.py ============================================================================== --- python/trunk/Lib/setuptools/sandbox.py (original) +++ python/trunk/Lib/setuptools/sandbox.py Tue Apr 18 19:32:12 2006 @@ -201,5 +201,3 @@ support alternate installation locations even if you run its setup script by hand. Please inform the package's author and the EasyInstall maintainers to find out if a fix or workaround is available.""" % self.args - - Modified: python/trunk/Lib/setuptools/site-patch.py ============================================================================== --- python/trunk/Lib/setuptools/site-patch.py (original) +++ python/trunk/Lib/setuptools/site-patch.py Tue Apr 18 19:32:12 2006 @@ -1,5 +1,5 @@ def __boot(): - import sys, imp, os, os.path + import sys, imp, os, os.path PYTHONPATH = os.environ.get('PYTHONPATH') if PYTHONPATH is None or (sys.platform=='win32' and not PYTHONPATH): PYTHONPATH = [] @@ -48,7 +48,7 @@ addsitedir(item) sys.__egginsert += oldpos # restore effective old position - + d,nd = makepath(stdpath[0]) insert_at = None new_path = [] @@ -66,17 +66,9 @@ # new path after the insert point, back-insert it new_path.insert(insert_at, item) insert_at += 1 - + sys.path[:] = new_path -if __name__=='site': +if __name__=='site': __boot() del __boot - - - - - - - - Modified: python/trunk/Lib/setuptools/tests/__init__.py ============================================================================== --- python/trunk/Lib/setuptools/tests/__init__.py (original) +++ python/trunk/Lib/setuptools/tests/__init__.py Tue Apr 18 19:32:12 2006 @@ -362,8 +362,3 @@ ts5 = makeSetup().get_command_obj('test') ts5.ensure_finalized() self.assertEqual(ts5.test_suite, None) - - - - - Modified: python/trunk/Lib/setuptools/tests/test_resources.py ============================================================================== --- python/trunk/Lib/setuptools/tests/test_resources.py (original) +++ python/trunk/Lib/setuptools/tests/test_resources.py Tue Apr 18 19:32:12 2006 @@ -143,7 +143,7 @@ self.assertRaises(VersionConflict, ws.resolve, parse_requirements("Foo==0.9"), ad) ws = WorkingSet([]) # reset - + # Request an extra that causes an unresolved dependency for "Baz" self.assertRaises( DistributionNotFound, ws.resolve,parse_requirements("Foo[bar]"), ad @@ -161,7 +161,7 @@ self.assertRaises( VersionConflict, ws.resolve, parse_requirements("Foo==1.2\nFoo!=1.2"), ad ) - + def testDistroDependsOptions(self): d = self.distRequires(""" Twisted>=1.5 @@ -481,12 +481,3 @@ for p,v1 in enumerate(torture): for v2 in torture[p+1:]: c(v2,v1) - - - - - - - - - From pje at telecommunity.com Tue Apr 18 19:55:08 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 18 Apr 2006 13:55:08 -0400 Subject: [Python-checkins] [Python-Dev] setuptools in the stdlib ( r45510 - python/trunk/Lib/pkgutil.py python/trunk/Lib/pydoc.py) In-Reply-To: <44451EA7.4020907@egenix.com> References: <5.1.1.6.0.20060418113808.0217a4a0@mail.telecommunity.com> <20060418005956.156301E400A@bag.python.org> <20060418005956.156301E400A@bag.python.org> <5.1.1.6.0.20060418113808.0217a4a0@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20060418133427.0211a268@mail.telecommunity.com> At 07:15 PM 4/18/2006 +0200, M.-A. Lemburg wrote: >Why should a 3rd party extension be hot-fixing the standard >Python distribution ? Because setuptools installs things in zip files, and older versions of pydoc don't work for packages zip files. >If you want to provide a hot fix for Python 2.3 and 2.4, you >should make this a separate install, so that users are aware >that their Python distribution is about to get modified in ways >that have nothing to do with setuptools. Their Python distribution is not "modified" -- new modules are merely placed on sys.path ahead of the stdlib. (Which, I might add, is a perfectly normal process in Python -- nothing stops users from installing their own version of pydoc or any other module via PYTHONPATH.) Note also that uninstalling setuptools by removing the .egg file or directory will effectively remove the hot fix, since the modules live in the .egg, not in the stdlib. >If an application wants to use setuptools for e.g. plugin >management, then standard distutils features will get >replaced by setuptools implementations which are not compatible >to the standard distutils commands, effectively making it >impossible to access the original versions. Please do a little research before you spread FUD. The 'pkg_resources' module is used for runtime plugin management, and it does not monkeypatch anything. >Monkey patching is only a last resort in case nothing >else works. In this case, it's not needed, since distutils >provides the interfaces needed to extend its command classes >via the setup() call. The monkeypatching is there so that the easy_install command can build eggs for packages that use the distutils. It's also there so that other distutils extensions that monkeypatch distutils (and there are a few of them out there) will have a better chance of working with setuptools. I originally took a minimally-invasive approach to setuptools-distutils interaction, but it was simply not possible to provide a high-degree of backward and "sideward" compatibility without it. In fact, I seem to recall finding some behaviors in some versions of distutils that can't be modified without monkeypatching, although the details are escaping me at this moment. > > As for discussion, Guido originally brought up the question here a few > > months ago, and it's been listed in PEP 356 for a while. I've also > > posted things related to the inclusion both here and in distutils-sig. > >I know, but the discussions haven't really helped much in >getting the setuptools design compatible with standard >distutils. That's because the job was already done. :) The setuptools design bends over backwards to be compatible with Python 2.3 and 2.4 versions of distutils, not to mention py2exe, Pyrex, and other distutils extensions, along with the quirky uses of distutils that exist in dozens of distributed Python packages. However, I think you and I may perhaps have different definitions of "compatibility". Mine is that things "just work" and users don't have to do anything special. For that definition, setuptools is extremely compatible with the standard distutils. In many situations it's more compatible than the distutils themselves, in that more things "just work". ;) From python-checkins at python.org Tue Apr 18 20:51:07 2006 From: python-checkins at python.org (thomas.heller) Date: Tue, 18 Apr 2006 20:51:07 +0200 (CEST) Subject: [Python-checkins] r45543 - in python/trunk: Misc/NEWS Python/pythonrun.c Message-ID: <20060418185107.8FFC21E4002@bag.python.org> Author: thomas.heller Date: Tue Apr 18 20:51:06 2006 New Revision: 45543 Modified: python/trunk/Misc/NEWS python/trunk/Python/pythonrun.c Log: Change those parts of the Python-api that were functions in 2.4, and are now macros to exported functions again. Fixes [ 1465834 ] bdist_wininst preinstall script support is broken in 2.5a1. Modified: python/trunk/Misc/NEWS ============================================================================== --- python/trunk/Misc/NEWS (original) +++ python/trunk/Misc/NEWS Tue Apr 18 20:51:06 2006 @@ -12,6 +12,15 @@ Core and builtins ----------------- +- Bug #1465834: 'bdist_wininst preinstall script support' was fixed + by converting these apis from macros into exported functions again: + + PyParser_SimpleParseFile PyParser_SimpleParseString PyRun_AnyFile + PyRun_AnyFileEx PyRun_AnyFileFlags PyRun_File PyRun_FileEx + PyRun_FileFlags PyRun_InteractiveLoop PyRun_InteractiveOne + PyRun_SimpleFile PyRun_SimpleFileEx PyRun_SimpleString + PyRun_String Py_CompileString + - Under COUNT_ALLOCS, types are not necessarily immortal anymore. - All uses of PyStructSequence_InitType have been changed to initialize Modified: python/trunk/Python/pythonrun.c ============================================================================== --- python/trunk/Python/pythonrun.c (original) +++ python/trunk/Python/pythonrun.c Tue Apr 18 20:51:06 2006 @@ -1690,20 +1690,112 @@ /* Deprecated C API functions still provided for binary compatiblity */ #undef PyParser_SimpleParseFile -#undef PyParser_SimpleParseString - -node * +PyAPI_FUNC(node *) PyParser_SimpleParseFile(FILE *fp, const char *filename, int start) { return PyParser_SimpleParseFileFlags(fp, filename, start, 0); } -node * +#undef PyParser_SimpleParseString +PyAPI_FUNC(node *) PyParser_SimpleParseString(const char *str, int start) { return PyParser_SimpleParseStringFlags(str, start, 0); } +#undef PyRun_AnyFile +PyAPI_FUNC(int) +PyRun_AnyFile(FILE *fp, const char *name) +{ + return PyRun_AnyFileExFlags(fp, name, 0, NULL); +} + +#undef PyRun_AnyFileEx +PyAPI_FUNC(int) +PyRun_AnyFileEx(FILE *fp, const char *name, int closeit) +{ + return PyRun_AnyFileExFlags(fp, name, closeit, NULL); +} + +#undef PyRun_AnyFileFlags +PyAPI_FUNC(int) +PyRun_AnyFileFlags(FILE *fp, const char *name, PyCompilerFlags *flags) +{ + return PyRun_AnyFileExFlags(fp, name, 0, flags); +} + +#undef PyRun_File +PyAPI_FUNC(PyObject *) +PyRun_File(FILE *fp, const char *p, int s, PyObject *g, PyObject *l) +{ + return PyRun_FileExFlags(fp, p, s, g, l, 0, NULL); +} + +#undef PyRun_FileEx +PyAPI_FUNC(PyObject *) +PyRun_FileEx(FILE *fp, const char *p, int s, PyObject *g, PyObject *l, int c) +{ + return PyRun_FileExFlags(fp, p, s, g, l, c, NULL); +} + +#undef PyRun_FileFlags +PyAPI_FUNC(PyObject *) +PyRun_FileFlags(FILE *fp, const char *p, int s, PyObject *g, PyObject *l, + PyCompilerFlags *flags) +{ + return PyRun_FileExFlags(fp, p, s, g, l, 0, flags); +} + +#undef PyRun_SimpleFile +PyAPI_FUNC(int) +PyRun_SimpleFile(FILE *f, const char *p) +{ + return PyRun_SimpleFileExFlags(f, p, 0, NULL); +} + +#undef PyRun_SimpleFileEx +PyAPI_FUNC(int) +PyRun_SimpleFileEx(FILE *f, const char *p, int c) +{ + return PyRun_SimpleFileExFlags(f, p, c, NULL); +} + + +#undef PyRun_String +PyAPI_FUNC(PyObject *) +PyRun_String(const char *str, int s, PyObject *g, PyObject *l) +{ + return PyRun_StringFlags(str, s, g, l, NULL); +} + +#undef PyRun_SimpleString +PyAPI_FUNC(int) +PyRun_SimpleString(const char *s) +{ + return PyRun_SimpleStringFlags(s, NULL); +} + +#undef Py_CompileString +PyAPI_FUNC(PyObject *) +Py_CompileString(const char *str, const char *p, int s) +{ + return Py_CompileStringFlags(str, p, s, NULL); +} + +#undef PyRun_InteractiveOne +PyAPI_FUNC(int) +PyRun_InteractiveOne(FILE *f, const char *p) +{ + return PyRun_InteractiveOneFlags(f, p, NULL); +} + +#undef PyRun_InteractiveLoop +PyAPI_FUNC(int) +PyRun_InteractiveLoop(FILE *f, const char *p) +{ + return PyRun_InteractiveLoopFlags(f, p, NULL); +} + #ifdef __cplusplus } #endif From python-checkins at python.org Tue Apr 18 20:55:18 2006 From: python-checkins at python.org (phillip.eby) Date: Tue, 18 Apr 2006 20:55:18 +0200 (CEST) Subject: [Python-checkins] r45544 - peps/trunk/pep-0343.txt Message-ID: <20060418185518.0359E1E407C@bag.python.org> Author: phillip.eby Date: Tue Apr 18 20:55:17 2006 New Revision: 45544 Modified: peps/trunk/pep-0343.txt Log: Make "context" vs. "context manager" terminology consistent. Originally, there were only context managers, so a lot of text referred to context managers that now should refer to contexts since the __context__ method was added. All contexts are context managers, so these references were technically correct in some sense, but are easier to understand if the more-specific terms are used. Modified: peps/trunk/pep-0343.txt ============================================================================== --- peps/trunk/pep-0343.txt (original) +++ peps/trunk/pep-0343.txt Tue Apr 18 20:55:17 2006 @@ -265,8 +265,8 @@ threading.RLock) may provide its own __enter__() and __exit__() methods, and simply return 'self' from its __context__ method. On the other hand, an object with more complex state requirements - (such as decimal.Context) may return a distinct context manager - object each time its __context__ method is invoked. + (such as decimal.Context) may return a distinct context object + each time its __context__ method is invoked. If the "as VAR" part of the syntax is omitted, the "VAR =" part of the translation is omitted (but ctx.__enter__() is still called). @@ -473,9 +473,9 @@ This PEP proposes that the protocol used by the with statement be known as the "context management protocol", and that objects that implement that protocol be known as "context managers". The term - "context" then encompasses all objects with a __context__() method - that returns a context manager (this means that all context managers - are contexts, but not all contexts are context managers). + "context manager" then encompasses all objects with a __context__() + method that returns a context object. (This means that all contexts + are context managers, but not all context managers are contexts). The term "context" is based on the concept that the context object defines a context of execution for the code that forms the body @@ -529,10 +529,11 @@ on python-dev [4] settled on the term "context manager" for objects which provide __enter__ and __exit__ methods, and "context management protocol" for the protocol itself. With the - addition of the __context__ method to the protocol, a natural - extension is to call all objects which provide a __context__ - method "contexts" (or "manageable contexts" in situations where - the general term "context" would be ambiguous). + addition of the __context__ method to the protocol, the natural + adjustment is to call all objects which provide a __context__ + method "context managers", and the objects with __enter__ and + __exit__ methods "contexts" (or "manageable contexts" in + situations where the general term "context" would be ambiguous). This is now documented in the "Standard Terminology" section. 4. The originally approved version of this PEP did not include a @@ -541,9 +542,9 @@ appropriate __enter__ and __exit__ methods for decimal.Context [5]. This approach allows a class to define a native context manager using generator syntax. It also allows a class to use an - existing independent context manager as its native context - manager by applying the independent context manager to 'self' in - its __context__ method. It even allows a class written in C to + existing independent context as its native context object by + applying the independent context to 'self' in its __context__ + method. It even allows a class written in C to use a generator context manager written in Python. The __context__ method parallels the __iter__ method which forms part of the iterator protocol. @@ -558,10 +559,10 @@ 342 [6]. Guido rejected this idea [7]. The following are some of benefits of keeping the __enter__ and __exit__ methods: - - it makes it easy to implement a simple context manager in C + - it makes it easy to implement a simple context in C without having to rely on a separate coroutine builder - it makes it easy to provide a low-overhead implementation - for context managers which don't need to maintain any + for contexts that don't need to maintain any special state between the __enter__ and __exit__ methods (having to use a generator for these would impose unnecessary overhead without any compensating benefit) From mal at egenix.com Tue Apr 18 21:02:20 2006 From: mal at egenix.com (M.-A. Lemburg) Date: Tue, 18 Apr 2006 21:02:20 +0200 Subject: [Python-checkins] [Python-Dev] setuptools in the stdlib ( r45510 - python/trunk/Lib/pkgutil.py python/trunk/Lib/pydoc.py) In-Reply-To: <5.1.1.6.0.20060418133427.0211a268@mail.telecommunity.com> References: <5.1.1.6.0.20060418113808.0217a4a0@mail.telecommunity.com> <20060418005956.156301E400A@bag.python.org> <20060418005956.156301E400A@bag.python.org> <5.1.1.6.0.20060418113808.0217a4a0@mail.telecommunity.com> <5.1.1.6.0.20060418133427.0211a268@mail.telecommunity.com> Message-ID: <444537BC.7030702@egenix.com> Phillip J. Eby wrote: > At 07:15 PM 4/18/2006 +0200, M.-A. Lemburg wrote: >> Why should a 3rd party extension be hot-fixing the standard >> Python distribution ? > > Because setuptools installs things in zip files, and older versions of > pydoc don't work for packages zip files. That doesn't answer my question. >> If you want to provide a hot fix for Python 2.3 and 2.4, you >> should make this a separate install, so that users are aware >> that their Python distribution is about to get modified in ways >> that have nothing to do with setuptools. > > Their Python distribution is not "modified" -- new modules are merely > placed on sys.path ahead of the stdlib. (Which, I might add, is a > perfectly normal process in Python -- nothing stops users from installing > their own version of pydoc or any other module via PYTHONPATH.) > > Note also that uninstalling setuptools by removing the .egg file or > directory will effectively remove the hot fix, since the modules live in > the .egg, not in the stdlib. Whether you place a module with the same name in front of the stdlib path in PYTHONPATH (e.g. copy it into site-packages) or replace the file in the Python installation is really the same thing to the user. Third-party extension *should not do this* ! It's OK to have private modified copies of a module inside a package or used inside an application, but "python setup.py install" should never (effectively) replace a Python stdlib module with some modified copy without explicit user interaction. >> If an application wants to use setuptools for e.g. plugin >> management, then standard distutils features will get >> replaced by setuptools implementations which are not compatible >> to the standard distutils commands, effectively making it >> impossible to access the original versions. > > Please do a little research before you spread FUD. The 'pkg_resources' > module is used for runtime plugin management, and it does not monkeypatch > anything. I'm talking about the setuptools package which does apply monkey patching and is needed to manage the download and installation of plugin eggs, AFAIK. >> Monkey patching is only a last resort in case nothing >> else works. In this case, it's not needed, since distutils >> provides the interfaces needed to extend its command classes >> via the setup() call. > > The monkeypatching is there so that the easy_install command can build eggs > for packages that use the distutils. It's also there so that other > distutils extensions that monkeypatch distutils (and there are a few of > them out there) will have a better chance of working with setuptools. > > I originally took a minimally-invasive approach to setuptools-distutils > interaction, but it was simply not possible to provide a high-degree of > backward and "sideward" compatibility without it. In fact, I seem to > recall finding some behaviors in some versions of distutils that can't be > modified without monkeypatching, although the details are escaping me at > this moment. That's a very vague comment. The distutils mechanism for providing your own command classes lets you take complete control over distutils if needed. What's good about it, is that this approach doesn't modify anything inside distutils at run-time, but does these modifications on a per-setup()-call basis. As for setuptools, you import the package and suddenly distutils isn't what's documented on python.org anymore. >>> As for discussion, Guido originally brought up the question here a few >>> months ago, and it's been listed in PEP 356 for a while. I've also >>> posted things related to the inclusion both here and in distutils-sig. >> >> I know, but the discussions haven't really helped much in >> getting the setuptools design compatible with standard >> distutils. > > That's because the job was already done. :) Not much of an argument, if you ask me. Some of the design decisions you made in setuptools are simply wrong IMHO and these need to be discussed in a PEP process. > The setuptools design bends > over backwards to be compatible with Python 2.3 and 2.4 versions of > distutils, not to mention py2exe, Pyrex, and other distutils extensions, > along with the quirky uses of distutils that exist in dozens of distributed > Python packages. > > However, I think you and I may perhaps have different definitions of > "compatibility". Mine is that things "just work" and users don't have to > do anything special. For that definition, setuptools is extremely > compatible with the standard distutils. In many situations it's more > compatible than the distutils themselves, in that more things "just work". ;) You've implemented your own view of "just works". This is fine, but please remember that Python is a collaborative work, so design decisions have to be worked out in collaboration as well. There aren't all that many things that are wrong in setuptools, but some of them are essential: * setuptools should not monkey patch distutils on import * the standard "python setup.py install" should continue to work as documented in the Python documentation; any new install command should use a different name, e.g. "install_egg" * placing too many ZIP files on PYTHONPATH is a bad idea since it slows down import searches for all Python applications, not just ones relying on eggs; a solution to this would be to have a PYTHONEGGPATH which is then only scanned by egg-aware modules/applications * the user should have freedom of choice in whether to have her Python installation rely on eggs or not (and not only --by-using-some-complicated-options) -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Apr 18 2006) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From buildbot at python.org Tue Apr 18 21:26:32 2006 From: buildbot at python.org (buildbot at python.org) Date: Tue, 18 Apr 2006 19:26:32 +0000 Subject: [Python-checkins] buildbot warnings in x86 XP trunk Message-ID: <20060418192632.6C19A1E401A@bag.python.org> The Buildbot has detected a new failure of x86 XP trunk. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520XP%2520trunk/builds/501 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: thomas.heller Build Had Warnings: warnings test sincerely, -The Buildbot From python-checkins at python.org Tue Apr 18 21:35:05 2006 From: python-checkins at python.org (skip.montanaro) Date: Tue, 18 Apr 2006 21:35:05 +0200 (CEST) Subject: [Python-checkins] r45545 - python/trunk/Modules/datetimemodule.c Message-ID: <20060418193505.A67531E4002@bag.python.org> Author: skip.montanaro Date: Tue Apr 18 21:35:04 2006 New Revision: 45545 Modified: python/trunk/Modules/datetimemodule.c Log: C++ compiler cleanup: "typename" is a C++ keyword Modified: python/trunk/Modules/datetimemodule.c ============================================================================== --- python/trunk/Modules/datetimemodule.c (original) +++ python/trunk/Modules/datetimemodule.c Tue Apr 18 21:35:04 2006 @@ -2411,11 +2411,11 @@ date_repr(PyDateTime_Date *self) { char buffer[1028]; - const char *typename; + const char *type_name; - typename = self->ob_type->tp_name; + type_name = self->ob_type->tp_name; PyOS_snprintf(buffer, sizeof(buffer), "%s(%d, %d, %d)", - typename, + type_name, GET_YEAR(self), GET_MONTH(self), GET_DAY(self)); return PyString_FromString(buffer); @@ -3138,7 +3138,7 @@ time_repr(PyDateTime_Time *self) { char buffer[100]; - const char *typename = self->ob_type->tp_name; + const char *type_name = self->ob_type->tp_name; int h = TIME_GET_HOUR(self); int m = TIME_GET_MINUTE(self); int s = TIME_GET_SECOND(self); @@ -3147,13 +3147,13 @@ if (us) PyOS_snprintf(buffer, sizeof(buffer), - "%s(%d, %d, %d, %d)", typename, h, m, s, us); + "%s(%d, %d, %d, %d)", type_name, h, m, s, us); else if (s) PyOS_snprintf(buffer, sizeof(buffer), - "%s(%d, %d, %d)", typename, h, m, s); + "%s(%d, %d, %d)", type_name, h, m, s); else PyOS_snprintf(buffer, sizeof(buffer), - "%s(%d, %d)", typename, h, m); + "%s(%d, %d)", type_name, h, m); result = PyString_FromString(buffer); if (result != NULL && HASTZINFO(self)) result = append_keyword_tzinfo(result, self->tzinfo); @@ -4036,13 +4036,13 @@ datetime_repr(PyDateTime_DateTime *self) { char buffer[1000]; - const char *typename = self->ob_type->tp_name; + const char *type_name = self->ob_type->tp_name; PyObject *baserepr; if (DATE_GET_MICROSECOND(self)) { PyOS_snprintf(buffer, sizeof(buffer), "%s(%d, %d, %d, %d, %d, %d, %d)", - typename, + type_name, GET_YEAR(self), GET_MONTH(self), GET_DAY(self), DATE_GET_HOUR(self), DATE_GET_MINUTE(self), DATE_GET_SECOND(self), @@ -4051,7 +4051,7 @@ else if (DATE_GET_SECOND(self)) { PyOS_snprintf(buffer, sizeof(buffer), "%s(%d, %d, %d, %d, %d, %d)", - typename, + type_name, GET_YEAR(self), GET_MONTH(self), GET_DAY(self), DATE_GET_HOUR(self), DATE_GET_MINUTE(self), DATE_GET_SECOND(self)); @@ -4059,7 +4059,7 @@ else { PyOS_snprintf(buffer, sizeof(buffer), "%s(%d, %d, %d, %d, %d)", - typename, + type_name, GET_YEAR(self), GET_MONTH(self), GET_DAY(self), DATE_GET_HOUR(self), DATE_GET_MINUTE(self)); } From python-checkins at python.org Tue Apr 18 21:39:48 2006 From: python-checkins at python.org (skip.montanaro) Date: Tue, 18 Apr 2006 21:39:48 +0200 (CEST) Subject: [Python-checkins] r45546 - python/trunk/Modules/arraymodule.c Message-ID: <20060418193948.0156E1E400D@bag.python.org> Author: skip.montanaro Date: Tue Apr 18 21:39:48 2006 New Revision: 45546 Modified: python/trunk/Modules/arraymodule.c Log: C++ compiler cleanup: a cast here, a cast there... still does not compile under C++ though... Modified: python/trunk/Modules/arraymodule.c ============================================================================== --- python/trunk/Modules/arraymodule.c (original) +++ python/trunk/Modules/arraymodule.c Tue Apr 18 21:39:48 2006 @@ -1164,7 +1164,7 @@ register char *p, *q; /* little buffer to hold items while swapping */ char tmp[256]; /* 8 is probably enough -- but why skimp */ - assert(itemsize <= sizeof(tmp)); + assert((size_t)itemsize <= sizeof(tmp)); if (self->ob_size > 1) { for (p = self->ob_item, @@ -1674,7 +1674,8 @@ } self->ob_size -= slicelength; - self->ob_item = PyMem_REALLOC(self->ob_item, itemsize*self->ob_size); + self->ob_item = (char *)PyMem_REALLOC(self->ob_item, + itemsize*self->ob_size); self->allocated = self->ob_size; return 0; @@ -1866,7 +1867,7 @@ if (n > 0) { arrayobject *self = (arrayobject *)a; char *item = self->ob_item; - item = PyMem_Realloc(item, n); + item = (char *)PyMem_Realloc(item, n); if (item == NULL) { PyErr_NoMemory(); Py_DECREF(a); From python-checkins at python.org Tue Apr 18 21:45:19 2006 From: python-checkins at python.org (skip.montanaro) Date: Tue, 18 Apr 2006 21:45:19 +0200 (CEST) Subject: [Python-checkins] r45547 - python/trunk/Modules/_ctypes/_ctypes.c python/trunk/Modules/_ctypes/_ctypes_test.c python/trunk/Modules/_ctypes/callbacks.c python/trunk/Modules/_ctypes/callproc.c python/trunk/Modules/_ctypes/ctypes.h python/trunk/Modules/_ctypes/stgdict.c Message-ID: <20060418194519.D25731E401A@bag.python.org> Author: skip.montanaro Date: Tue Apr 18 21:45:17 2006 New Revision: 45547 Modified: python/trunk/Modules/_ctypes/_ctypes.c python/trunk/Modules/_ctypes/_ctypes_test.c python/trunk/Modules/_ctypes/callbacks.c python/trunk/Modules/_ctypes/callproc.c python/trunk/Modules/_ctypes/ctypes.h python/trunk/Modules/_ctypes/stgdict.c Log: C++ compiler cleanup: the typical few casts, and ... C++ didn't like that the StgDictObject's ffi_type member had the same name as its type. I changed that to ffi_type_pointer. Feel free to change it to something else more meaningful, just not ffi_type. Modified: python/trunk/Modules/_ctypes/_ctypes.c ============================================================================== --- python/trunk/Modules/_ctypes/_ctypes.c (original) +++ python/trunk/Modules/_ctypes/_ctypes.c Tue Apr 18 21:45:17 2006 @@ -549,7 +549,7 @@ stgdict->size = sizeof(void *); stgdict->align = getentry("P")->pffi_type->alignment; stgdict->length = 1; - stgdict->ffi_type = ffi_type_pointer; + stgdict->ffi_type_pointer = ffi_type_pointer; proto = PyDict_GetItemString(typedict, "_type_"); /* Borrowed ref */ if (proto && -1 == PointerType_SetProto(stgdict, proto)) { @@ -949,7 +949,7 @@ stgdict->proto = proto; /* Arrays are passed as pointers to function calls. */ - stgdict->ffi_type = ffi_type_pointer; + stgdict->ffi_type_pointer = ffi_type_pointer; /* create the new instance (which is a class, since we are a metatype!) */ @@ -1307,7 +1307,7 @@ if (!stgdict) /* XXX leaks result! */ return NULL; - stgdict->ffi_type = *fmt->pffi_type; + stgdict->ffi_type_pointer = *fmt->pffi_type; stgdict->align = fmt->pffi_type->alignment; stgdict->length = 0; stgdict->size = fmt->pffi_type->size; @@ -1365,7 +1365,7 @@ fmt = getentry(PyString_AS_STRING(proto)); - stgdict->ffi_type = *fmt->pffi_type; + stgdict->ffi_type_pointer = *fmt->pffi_type; stgdict->align = fmt->pffi_type->alignment; stgdict->length = 0; stgdict->size = fmt->pffi_type->size; @@ -1635,7 +1635,7 @@ stgdict->size = sizeof(void *); stgdict->setfunc = NULL; stgdict->getfunc = NULL; - stgdict->ffi_type = ffi_type_pointer; + stgdict->ffi_type_pointer = ffi_type_pointer; ob = PyDict_GetItemString((PyObject *)stgdict, "_flags_"); if (!ob || !PyInt_Check(ob)) { @@ -1857,7 +1857,7 @@ StgDictObject *dict = PyObject_stgdict((PyObject *)self); Py_CLEAR(self->b_objects); if ((self->b_needsfree) - && (dict->size > sizeof(self->b_value))) + && ((size_t)dict->size > sizeof(self->b_value))) PyMem_Free(self->b_ptr); self->b_ptr = NULL; Py_CLEAR(self->b_base); @@ -1979,7 +1979,7 @@ static void CData_MallocBuffer(CDataObject *obj, StgDictObject *dict) { - if (dict->size <= sizeof(obj->b_value)) { + if ((size_t)dict->size <= sizeof(obj->b_value)) { /* No need to call malloc, can use the default buffer */ obj->b_ptr = (char *)&obj->b_value; obj->b_needsfree = 1; @@ -1987,7 +1987,7 @@ /* In python 2.4, and ctypes 0.9.6, the malloc call took about 33% of the creation time for c_int(). */ - obj->b_ptr = PyMem_Malloc(dict->size); + obj->b_ptr = (char *)PyMem_Malloc(dict->size); obj->b_needsfree = 1; memset(obj->b_ptr, 0, dict->size); } @@ -2052,7 +2052,7 @@ if (!pd) return NULL; assert(CDataObject_Check(pd)); - pd->b_ptr = buf; + pd->b_ptr = (char *)buf; pd->b_length = dict->length; pd->b_size = dict->size; return (PyObject *)pd; @@ -3295,7 +3295,7 @@ parg->tag = 'V'; stgdict = PyObject_stgdict((PyObject *)self); - parg->pffi_type = &stgdict->ffi_type; + parg->pffi_type = &stgdict->ffi_type_pointer; /* For structure parameters (by value), parg->value doesn't contain the structure data itself, instead parg->value.p *points* to the structure's data See also _ctypes.c, function _call_function_pointer(). Modified: python/trunk/Modules/_ctypes/_ctypes_test.c ============================================================================== --- python/trunk/Modules/_ctypes/_ctypes_test.c (original) +++ python/trunk/Modules/_ctypes/_ctypes_test.c Tue Apr 18 21:45:17 2006 @@ -74,7 +74,7 @@ EXPORT(char *) _testfunc_p_p(void *s) { - return s; + return (char *)s; } EXPORT(void *) _testfunc_c_p_p(int *argcp, char **argv) @@ -89,7 +89,7 @@ EXPORT(char *) my_strdup(char *src) { - char *dst = malloc(strlen(src)+1); + char *dst = (char *)malloc(strlen(src)+1); if (!dst) return NULL; strcpy(dst, src); @@ -100,7 +100,7 @@ EXPORT(wchar_t *) my_wcsdup(wchar_t *src) { size_t len = wcslen(src); - wchar_t *ptr = malloc((len + 1) * sizeof(wchar_t)); + wchar_t *ptr = (wchar_t *)malloc((len + 1) * sizeof(wchar_t)); if (ptr == NULL) return NULL; memcpy(ptr, src, (len+1) * sizeof(wchar_t)); @@ -191,7 +191,7 @@ { static char message[] = "Hello, World"; if (p) { - *p = malloc(sizeof(char *)); + *p = (char **)malloc(sizeof(char *)); printf("malloc returned %p\n", *p); **p = message; return 1; Modified: python/trunk/Modules/_ctypes/callbacks.c ============================================================================== --- python/trunk/Modules/_ctypes/callbacks.c (original) +++ python/trunk/Modules/_ctypes/callbacks.c Tue Apr 18 21:45:17 2006 @@ -318,7 +318,7 @@ if (dict == NULL) goto error; p->setfunc = dict->setfunc; - p->restype = &dict->ffi_type; + p->restype = &dict->ffi_type_pointer; } cc = FFI_DEFAULT_ABI; Modified: python/trunk/Modules/_ctypes/callproc.c ============================================================================== --- python/trunk/Modules/_ctypes/callproc.c (original) +++ python/trunk/Modules/_ctypes/callproc.c Tue Apr 18 21:45:17 2006 @@ -588,7 +588,7 @@ return &ffi_type_sint64; } #endif - return &dict->ffi_type; + return &dict->ffi_type_pointer; } Modified: python/trunk/Modules/_ctypes/ctypes.h ============================================================================== --- python/trunk/Modules/_ctypes/ctypes.h (original) +++ python/trunk/Modules/_ctypes/ctypes.h Tue Apr 18 21:45:17 2006 @@ -198,7 +198,7 @@ Py_ssize_t size; /* number of bytes */ Py_ssize_t align; /* alignment requirements */ Py_ssize_t length; /* number of fields */ - ffi_type ffi_type; + ffi_type ffi_type_pointer; PyObject *proto; /* Only for Pointer/ArrayObject */ SETFUNC setfunc; /* Only for simple objects */ GETFUNC getfunc; /* Only for simple objects */ Modified: python/trunk/Modules/_ctypes/stgdict.c ============================================================================== --- python/trunk/Modules/_ctypes/stgdict.c (original) +++ python/trunk/Modules/_ctypes/stgdict.c Tue Apr 18 21:45:17 2006 @@ -38,7 +38,7 @@ StgDict_dealloc(StgDictObject *self) { StgDict_clear(self); - PyMem_Free(self->ffi_type.elements); + PyMem_Free(self->ffi_type_pointer.elements); PyDict_Type.tp_dealloc((PyObject *)self); } @@ -49,8 +49,8 @@ int size; StgDict_clear(dst); - PyMem_Free(dst->ffi_type.elements); - dst->ffi_type.elements = NULL; + PyMem_Free(dst->ffi_type_pointer.elements); + dst->ffi_type_pointer.elements = NULL; d = (char *)dst; s = (char *)src; @@ -64,13 +64,15 @@ Py_XINCREF(dst->restype); Py_XINCREF(dst->checker); - if (src->ffi_type.elements == NULL) + if (src->ffi_type_pointer.elements == NULL) return 0; size = sizeof(ffi_type *) * (src->length + 1); - dst->ffi_type.elements = PyMem_Malloc(size); - if (dst->ffi_type.elements == NULL) + dst->ffi_type_pointer.elements = PyMem_Malloc(size); + if (dst->ffi_type_pointer.elements == NULL) return -1; - memcpy(dst->ffi_type.elements, src->ffi_type.elements, size); + memcpy(dst->ffi_type_pointer.elements, + src->ffi_type_pointer.elements, + size); return 0; } @@ -234,8 +236,8 @@ stuff is sucessfully finished. */ stgdict->flags |= DICTFLAG_FINAL; /* set final */ - if (stgdict->ffi_type.elements) - PyMem_Free(stgdict->ffi_type.elements); + if (stgdict->ffi_type_pointer.elements) + PyMem_Free(stgdict->ffi_type_pointer.elements); basedict = PyType_stgdict((PyObject *)((PyTypeObject *)type)->tp_base); if (basedict && !use_broken_old_ctypes_semantics) { @@ -243,10 +245,12 @@ align = basedict->align; union_size = 0; total_align = align ? align : 1; - stgdict->ffi_type.type = FFI_TYPE_STRUCT; - stgdict->ffi_type.elements = PyMem_Malloc(sizeof(ffi_type *) * (basedict->length + len + 1)); - memset(stgdict->ffi_type.elements, 0, sizeof(ffi_type *) * (basedict->length + len + 1)); - memcpy(stgdict->ffi_type.elements, basedict->ffi_type.elements, + stgdict->ffi_type_pointer.type = FFI_TYPE_STRUCT; + stgdict->ffi_type_pointer.elements = PyMem_Malloc(sizeof(ffi_type *) * (basedict->length + len + 1)); + memset(stgdict->ffi_type_pointer.elements, 0, + sizeof(ffi_type *) * (basedict->length + len + 1)); + memcpy(stgdict->ffi_type_pointer.elements, + basedict->ffi_type_pointer.elements, sizeof(ffi_type *) * (basedict->length)); ffi_ofs = basedict->length; } else { @@ -255,9 +259,10 @@ align = 0; union_size = 0; total_align = 1; - stgdict->ffi_type.type = FFI_TYPE_STRUCT; - stgdict->ffi_type.elements = PyMem_Malloc(sizeof(ffi_type *) * (len + 1)); - memset(stgdict->ffi_type.elements, 0, sizeof(ffi_type *) * (len + 1)); + stgdict->ffi_type_pointer.type = FFI_TYPE_STRUCT; + stgdict->ffi_type_pointer.elements = PyMem_Malloc(sizeof(ffi_type *) * (len + 1)); + memset(stgdict->ffi_type_pointer.elements, 0, + sizeof(ffi_type *) * (len + 1)); ffi_ofs = 0; } @@ -283,10 +288,10 @@ i); return -1; } - stgdict->ffi_type.elements[ffi_ofs + i] = &dict->ffi_type; + stgdict->ffi_type_pointer.elements[ffi_ofs + i] = &dict->ffi_type_pointer; dict->flags |= DICTFLAG_FINAL; /* mark field type final */ if (PyTuple_Size(pair) == 3) { /* bits specified */ - switch(dict->ffi_type.type) { + switch(dict->ffi_type_pointer.type) { case FFI_TYPE_UINT8: case FFI_TYPE_UINT16: case FFI_TYPE_UINT32: @@ -357,8 +362,8 @@ /* Adjust the size according to the alignment requirements */ size = ((size + total_align - 1) / total_align) * total_align; - stgdict->ffi_type.alignment = total_align; - stgdict->ffi_type.size = size; + stgdict->ffi_type_pointer.alignment = total_align; + stgdict->ffi_type_pointer.size = size; stgdict->size = size; stgdict->align = total_align; From theller at python.net Tue Apr 18 21:55:30 2006 From: theller at python.net (Thomas Heller) Date: Tue, 18 Apr 2006 21:55:30 +0200 Subject: [Python-checkins] r45547 - python/trunk/Modules/_ctypes/_ctypes.c python/trunk/Modules/_ctypes/_ctypes_test.c python/trunk/Modules/_ctypes/callbacks.c python/trunk/Modules/_ctypes/callproc.c python/trunk/Modules/_ctypes/ctypes.h python/trunk/Modules/_ctypes/stgdict.c In-Reply-To: <20060418194519.D25731E401A@bag.python.org> References: <20060418194519.D25731E401A@bag.python.org> Message-ID: <44454432.6090106@python.net> skip.montanaro wrote: > Author: skip.montanaro > Date: Tue Apr 18 21:45:17 2006 > New Revision: 45547 > > Modified: > python/trunk/Modules/_ctypes/_ctypes.c > python/trunk/Modules/_ctypes/_ctypes_test.c > python/trunk/Modules/_ctypes/callbacks.c > python/trunk/Modules/_ctypes/callproc.c > python/trunk/Modules/_ctypes/ctypes.h > python/trunk/Modules/_ctypes/stgdict.c > Log: > C++ compiler cleanup: the typical few casts, and ... C++ didn't like that > the StgDictObject's ffi_type member had the same name as its type. I > changed that to ffi_type_pointer. Feel free to change it to something else > more meaningful, just not ffi_type. That makes sense. Thanks you. Thomas From theller at python.net Tue Apr 18 21:55:30 2006 From: theller at python.net (Thomas Heller) Date: Tue, 18 Apr 2006 21:55:30 +0200 Subject: [Python-checkins] r45547 - python/trunk/Modules/_ctypes/_ctypes.c python/trunk/Modules/_ctypes/_ctypes_test.c python/trunk/Modules/_ctypes/callbacks.c python/trunk/Modules/_ctypes/callproc.c python/trunk/Modules/_ctypes/ctypes.h python/trunk/Modules/_ctypes/stgdict.c In-Reply-To: <20060418194519.D25731E401A@bag.python.org> References: <20060418194519.D25731E401A@bag.python.org> Message-ID: <44454432.6090106@python.net> skip.montanaro wrote: > Author: skip.montanaro > Date: Tue Apr 18 21:45:17 2006 > New Revision: 45547 > > Modified: > python/trunk/Modules/_ctypes/_ctypes.c > python/trunk/Modules/_ctypes/_ctypes_test.c > python/trunk/Modules/_ctypes/callbacks.c > python/trunk/Modules/_ctypes/callproc.c > python/trunk/Modules/_ctypes/ctypes.h > python/trunk/Modules/_ctypes/stgdict.c > Log: > C++ compiler cleanup: the typical few casts, and ... C++ didn't like that > the StgDictObject's ffi_type member had the same name as its type. I > changed that to ffi_type_pointer. Feel free to change it to something else > more meaningful, just not ffi_type. That makes sense. Thanks you. Thomas From buildbot at python.org Tue Apr 18 22:00:27 2006 From: buildbot at python.org (buildbot at python.org) Date: Tue, 18 Apr 2006 20:00:27 +0000 Subject: [Python-checkins] buildbot failure in x86 XP-2 trunk Message-ID: <20060418200027.D43411E400F@bag.python.org> The Buildbot has detected a new failure of x86 XP-2 trunk. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520XP-2%2520trunk/builds/329 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: skip.montanaro BUILD FAILED: failed compile sincerely, -The Buildbot From buildbot at python.org Tue Apr 18 22:03:36 2006 From: buildbot at python.org (buildbot at python.org) Date: Tue, 18 Apr 2006 20:03:36 +0000 Subject: [Python-checkins] buildbot warnings in x86 cygwin trunk Message-ID: <20060418200336.84FB91E4002@bag.python.org> The Buildbot has detected a new failure of x86 cygwin trunk. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520cygwin%2520trunk/builds/218 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: phillip.eby,thomas.heller,tim.peters Build Had Warnings: warnings test sincerely, -The Buildbot From buildbot at python.org Tue Apr 18 22:07:50 2006 From: buildbot at python.org (buildbot at python.org) Date: Tue, 18 Apr 2006 20:07:50 +0000 Subject: [Python-checkins] buildbot failure in x86 W2k trunk Message-ID: <20060418200750.D7E511E4002@bag.python.org> The Buildbot has detected a new failure of x86 W2k trunk. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520W2k%2520trunk/builds/542 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: skip.montanaro BUILD FAILED: failed compile sincerely, -The Buildbot From python-checkins at python.org Tue Apr 18 22:09:28 2006 From: python-checkins at python.org (thomas.heller) Date: Tue, 18 Apr 2006 22:09:28 +0200 (CEST) Subject: [Python-checkins] r45548 - python/trunk/Modules/_ctypes/callproc.c Message-ID: <20060418200928.2A3651E4002@bag.python.org> Author: thomas.heller Date: Tue Apr 18 22:09:27 2006 New Revision: 45548 Modified: python/trunk/Modules/_ctypes/callproc.c Log: Rename remaining StgDictObject's ffi_type fields to ffi_type_pointer. Modified: python/trunk/Modules/_ctypes/callproc.c ============================================================================== --- python/trunk/Modules/_ctypes/callproc.c (original) +++ python/trunk/Modules/_ctypes/callproc.c Tue Apr 18 22:09:27 2006 @@ -581,10 +581,10 @@ /* This little trick works correctly with MSVC. It returns small structures in registers */ - if (dict->ffi_type.type == FFI_TYPE_STRUCT) { - if (dict->ffi_type.size <= 4) + if (dict->ffi_type_pointer.type == FFI_TYPE_STRUCT) { + if (dict->ffi_type_pointer.size <= 4) return &ffi_type_sint32; - else if (dict->ffi_type.size <= 8) + else if (dict->ffi_type_pointer.size <= 8) return &ffi_type_sint64; } #endif From buildbot at python.org Tue Apr 18 22:34:27 2006 From: buildbot at python.org (buildbot at python.org) Date: Tue, 18 Apr 2006 20:34:27 +0000 Subject: [Python-checkins] buildbot warnings in x86 OpenBSD trunk Message-ID: <20060418203427.80A8C1E4009@bag.python.org> The Buildbot has detected a new failure of x86 OpenBSD trunk. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520OpenBSD%2520trunk/builds/378 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: skip.montanaro Build Had Warnings: warnings test sincerely, -The Buildbot From python-checkins at python.org Tue Apr 18 22:52:06 2006 From: python-checkins at python.org (david.goodger) Date: Tue, 18 Apr 2006 22:52:06 +0200 (CEST) Subject: [Python-checkins] r45549 - peps/trunk/pep-0001.txt Message-ID: <20060418205206.974001E4002@bag.python.org> Author: david.goodger Date: Tue Apr 18 22:52:06 2006 New Revision: 45549 Modified: peps/trunk/pep-0001.txt Log: changed CVS references to SVN; updated SVN viewing URL Modified: peps/trunk/pep-0001.txt ============================================================================== --- peps/trunk/pep-0001.txt (original) +++ peps/trunk/pep-0001.txt Tue Apr 18 22:52:06 2006 @@ -104,7 +104,7 @@ The author of the PEP is then responsible for posting the PEP to the community forums, and marshaling community support for it. As updates are necessary, the PEP author can check in new versions if they have -CVS commit permissions, or can email new PEP versions to the PEP +SVN commit permissions, or can email new PEP versions to the PEP editor for committing. Standards Track PEPs consist of two parts, a design document and a @@ -255,8 +255,8 @@ PEP: Title: - Version: - Last-Modified: + Version: + Last-Modified: Author: * Discussions-To: Status: References: <5.1.1.6.0.20060418133427.0211a268@mail.telecommunity.com> <5.1.1.6.0.20060418113808.0217a4a0@mail.telecommunity.com> <20060418005956.156301E400A@bag.python.org> <20060418005956.156301E400A@bag.python.org> <5.1.1.6.0.20060418113808.0217a4a0@mail.telecommunity.com> <5.1.1.6.0.20060418133427.0211a268@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20060418151010.01e06940@mail.telecommunity.com> At 09:02 PM 4/18/2006 +0200, M.-A. Lemburg wrote: >Phillip J. Eby wrote: > > At 07:15 PM 4/18/2006 +0200, M.-A. Lemburg wrote: > >> Why should a 3rd party extension be hot-fixing the standard > >> Python distribution ? > > > > Because setuptools installs things in zip files, and older versions of > > pydoc don't work for packages zip files. > >That doesn't answer my question. That is the answer to the question you asked: "why hot-fix?" Because setuptools uses zip files, and older pydocs crash when trying to display the documentation for a package (not module; modules were fine) that is in a zip file. >Third-party extension *should not do this* ! If you install setuptools, you presumably would like for things to work, and the hot fix eliminates a bug that interferes with them working. I'm not sure, however, what you believe any of that has to do with python-checkins or python-dev. The version of setuptools that will do this is not yet released, and the hotfix aspect will be documented. If it causes actual problems for actual setuptools users, it will be actually fixed or actually removed. :) (Meanwhile, the separately-distributed setuptools package is not a part of Python, any more than the Optik is, despite Python including the 'optparse' package spun off from Optik.) >I'm talking about the setuptools package which does apply >monkey patching and is needed to manage the download and >installation of plugin eggs, AFAIK. In which case the monkeypatching is needed, to handle situations like building eggs for third-party non-setuptools libraries listed as dependencies. You can't have it both ways; either you need setuptools or you don't. If you do, you will almost certainly need the monkeypatching, and you'll definitely need it to build eggs for non-setuptools-based packages. >What's good about it, is that this approach doesn't modify anything >inside distutils at run-time, but does these modifications >on a per-setup()-call basis. What's bad about it, is that it provides no options for virtualizing the execution of other packages' setup scripts. The 'run_setup' mechanism was horribly broken in the versions of the distutils that I wrestled with. (I also don't see anybody stepping up to provide alternative implementations or patches to third-party setup scripts or distutils extensions to deal with this issue.) >There aren't all that many things that are wrong in setuptools, >but some of them are essential: > >* setuptools should not monkey patch distutils on import Please propose an alternative, or better still, produce a patch. Be sure that it still allows distutils extensions like py2exe to work. The only real alternative to monkeypatching that I'm aware of for that issue is to actually merge setuptools with the distutils, but my guess is that you'll like that even less. :) >* the standard "python setup.py install" should continue to > work as documented in the Python documentation; any new install > command should use a different name, e.g. "install_egg" > >* placing too many ZIP files on PYTHONPATH is a bad idea > since it slows down import searches for all Python > applications, not just ones relying on eggs; a solution > to this would be to have a PYTHONEGGPATH which is then > only scanned by egg-aware modules/applications > >* the user should have freedom of choice in whether to > have her Python installation rely on eggs or not (and > not only --by-using-some-complicated-options) These questions have been hashed to death on the distutils-sig already, but I will try to summarize my responses here as briefly as practical while still covering the full spectrum of issues. I would begin by saying that the tradeoffs I've made favor inexperienced users at the expense of experts (who I assume are capable of learning to use --root or --single-version-externally managed in order to have finer-grained control). Your proposals, however, generally favor experts at the expense of the average user, and treat it as axiomatic that the benefits provided by setuptools are not worth having, no matter how small the cost. By contrast, users who *want* to use setuptools for either distribution or installation of packages, would rather it "just work", no matter how much complexity is required behind-the-scenes. They don't care about my cost to implement, they just care that they type "easy_install foo" to get a package from PyPI or "setup.py sdist bdist_egg upload" to put it there. Therefore, it makes no sense to apply your design approach to setuptools, since by your criteria it wouldn't exist in the first place! After all, expert users can munge distutils extensions to their hearts' content (and you certainly have done so already). Next, your own measurements posted to the distutils-sig debunked the PYTHONPATH performance question, IMO: if you installed *every Python package listed on PyPI at that time* as a zip file (assuming that the average zipfile directory size was the same as in your tests, around 1100 total packages on PyPI, and linear degradation), Python startup time would have increased by little more than half a second. To me, that says the performance is pretty darn good, since the average user isn't going to install anywhere near that many packages. Note also that installing eggs with -m (or --multi-version) doesn't put them on sys.path to begin with unless they're needed at runtime, so if you care about performance tweaking, you can just use -m for everything, or put it in your distutils.cfg or ~/.pydistutils.cfg file so it's automatic. Since scripts that depend on eggs can automatically find and add the eggs to sys.path at runtime, they don't need the eggs to be on sys.path. The reason they are put on sys.path by default is to make the default case of installation "just work". Specifically, it's so that "setup.py install" works the same way as it does with the distutils (in terms of what the users can *do* with a package after installation, not in terms of how the files are laid out). As far as I can tell, your remaining arguments can be reduced to this: Users who care about how stuff works inside (i.e. experts) will have to use command-line options to get the old behaviors or to tweak the new ones, when working with packages that *explicitly chose* to use setuptools. And that is a perfectly valid concern -- for an audience that's quite capable of learning to do something new, however. Also note that nothing stops people from creating packages that warp distutils into installing things in weird places, so arguing that "setup.py install" should always do the same thing is moot anyway; you would not get your wish even if setuptools had never existed. May I suggest that perhaps a better solution to the actual practical problem here is to make the distutils "install" command accept the options that setuptools supports? Then, people who want it can just use --single-version-externally-managed with any package under Python 2.5 and be guaranteed to get the old-fashioned behavior (that can't be cleanly upgraded or uninstalled without a package manager). As far as I know, however, most tools needing the old installation layout use "install --root" anyway (including bdist_wininst, bdist_msi, bdist_rpm, etc.), and setuptools automatically switches to compatibility mode in that case. So setuptools "just works" with these tools already. From skip at pobox.com Tue Apr 18 23:53:57 2006 From: skip at pobox.com (skip at pobox.com) Date: Tue, 18 Apr 2006 16:53:57 -0500 Subject: [Python-checkins] r45505 - python/trunk/Modules/posixmodule.c In-Reply-To: <4444888A.60603@v.loewis.de> References: <20060418004950.454AC1E400A@bag.python.org> <4444888A.60603@v.loewis.de> Message-ID: <17477.24565.863267.639675@montanaro.dyndns.org> >> The return type is size_t and cannot return -1. Do we know if the >> previous code ever worked? Is there any system that does return -1? Martin> It's now broken in a different way. Previously, it was broken Martin> because it reset errno after the call. Now it is broken because Martin> it treats -1 as an error indication; the correct error Martin> indication is 0 (but 0 also indicates that there is no Martin> configuration-defined value, hence errno must be set to 0 before Martin> the call). I was working from the confstr man page on my Mac which reads, in part: SYNOPSIS #include size_t confstr(int name, char *buf, size_t len); DESCRIPTION This interface is specified by POSIX. A more flexible (but non-portable) interface is provided by sysctl(3). ... RETURN VALUES If the call to confstr() is not successful, -1 is returned and errno is set appropriately. Otherwise, if the variable does not have a configuration defined value, 0 is returned and errno is not modified. Otherwise, the buffer size needed to hold the entire configuration- defined value is returned. If this size is greater than the argument len, the string in buf was truncated. I just tried this little program: #include #include #include main(int argc, char **argv) { char buf[80]; errno = 0; printf("confstr(5) returned %u, errno == %d\n", confstr(5, buf, 79), errno); errno = 0; printf("confstr(-1) returned %u, errno == %d\n", confstr(-1, buf, 79), errno); } which, when run, outputs: confstr(5) returned 1, errno == 0 confstr(-1) returned 0, errno == 22 It appears the Mac's documentation is incorrect. Martin> Also, I suggest to use None as the return value for "no value Martin> available"; it might be that the configured value is an empty Martin> string (in which case confstr returns 1). Martin> Also, the PyString_FromStringAndSize is wrong: the confstr Martin> result includes the terminating \0, whereas Martin> PyString_FromStringAndSize doesn't. Martin> Also, for symmetry, it would be better if Martin> PyString_FromStringAndSize is also used in the case without Martin> resizing. This would also have the advantage of allowing for Martin> results that include \0 (although I doubt they are actually Martin> used). I'll work on all of this. Are you sure you want the API to change? Skip From python-checkins at python.org Wed Apr 19 01:04:00 2006 From: python-checkins at python.org (thomas.wouters) Date: Wed, 19 Apr 2006 01:04:00 +0200 (CEST) Subject: [Python-checkins] r45551 - in python/trunk/Lib/setuptools: command tests Message-ID: <20060418230400.D0E501E4002@bag.python.org> Author: thomas.wouters Date: Wed Apr 19 01:04:00 2006 New Revision: 45551 Modified: python/trunk/Lib/setuptools/ (props changed) python/trunk/Lib/setuptools/command/ (props changed) python/trunk/Lib/setuptools/tests/ (props changed) Log: Add proper svn magic to ignore .pyc/.pyo files in Lib/setuptools (and subdirs) like it already exists for the other Lib subdirs. From python-checkins at python.org Wed Apr 19 01:25:48 2006 From: python-checkins at python.org (david.goodger) Date: Wed, 19 Apr 2006 01:25:48 +0200 (CEST) Subject: [Python-checkins] r45552 - peps/trunk/pep-0359.txt peps/trunk/pep-3002.txt Message-ID: <20060418232548.284AD1E401E@bag.python.org> Author: david.goodger Date: Wed Apr 19 01:25:47 2006 New Revision: 45552 Modified: peps/trunk/pep-0359.txt peps/trunk/pep-3002.txt Log: updates from Steven Bethard Modified: peps/trunk/pep-0359.txt ============================================================================== --- peps/trunk/pep-0359.txt (original) +++ peps/trunk/pep-0359.txt Wed Apr 19 01:25:47 2006 @@ -8,7 +8,7 @@ Content-Type: text/x-rst Created: 05-Apr-2006 Python-Version: 2.6 -Post-History: 05-Apr-2006, 06-Apr-2006 +Post-History: 05-Apr-2006, 06-Apr-2006, 13-Apr-2006 Abstract @@ -26,6 +26,19 @@ = ("", , ) where ```` is the dict created by executing ````. +This is mostly syntactic sugar for:: + + class : + __metaclass__ = + + +and is intended to help more clearly express the intent of the +statement when something other than a class is being created. Of +course, other syntax for such a statement is possible, but it is hoped +that by keeping a strong parallel to the class statement, an +understanding of how classes and metaclasses work will translate into +an understanding of how the make-statement works as well. + The PEP is based on a suggestion [1]_ from Michele Simionato on the python-dev list. @@ -35,12 +48,11 @@ Class statements provide two nice facilities to Python: - (1) They are the standard Python means of creating a namespace. All - statements within a class body are executed, and the resulting - local name bindings are passed as a dict to the metaclass. +(1) They execute a block of statements and provide the resulting + bindings as a dict to the metaclass. - (2) They encourage DRY (don't repeat yourself) by allowing the class - being created to know the name it is being assigned. +(2) They encourage DRY (don't repeat yourself) by allowing the class + being created to know the name it is being assigned. Thus in a simple class statement like:: @@ -59,13 +71,153 @@ creation of the dict by allowing it to be expressed as a series of statements. -Historically, type instances (a.k.a. class objects) have been the only -objects blessed with this sort of syntactic support. But other sorts -of objects could benefit from such support. For example, property -objects take three function arguments, but because the property type -cannot be passed a namespace, these functions, though relevant only to -the property, must be declared before it and then passed as arguments -to the property call, e.g.:: +Historically, type instances (a.k.a. class objects) have been the +only objects blessed with this sort of syntactic support. The make +statement aims to extend this support to other sorts of objects where +such syntax would also be useful. + + +Example: simple namespaces +-------------------------- + +Let's say I have some attributes in a module that I access like:: + + mod.thematic_roletype + mod.opinion_roletype + + mod.text_format + mod.html_format + +and since "Namespaces are one honking great idea", I'd like to be able +to access these attributes instead as:: + + mod.roletypes.thematic + mod.roletypes.opinion + + mod.format.text + mod.format.html + +I currently have two main options: + +(1) Turn the module into a package, turn ``roletypes`` and ``format`` + into submodules, and move the attributes to the submodules. + +(2) Create ``roletypes`` and ``format`` classes, and move the + attributes to the classes. + +The former is a fair chunk of refactoring work, and produces two tiny +modules without much content. The latter keeps the attributes local +to the module, but creates classes when there is no intention of ever +creating instances of those classes. + +In situations like this, it would be nice to simply be able to declare +a "namespace" to hold the few attributes. With the new make +statement, I could introduce my new namespaces with something like:: + + make namespace roletypes: + thematic = ... + opinion = ... + + make namespace format: + text = ... + html = ... + +and keep my attributes local to the module without making classes that +are never intended to be instantiated. One definition of namespace +that would make this work is:: + + class namespace(object): + def __init__(self, name, args, kwargs): + self.__dict__.update(kwargs) + +Given this definition, at the end of the make-statements above, +``roletypes`` and ``format`` would be namespace instances. + + +Example: GUI objects +-------------------- + +In GUI toolkits, objects like frames and panels are often associated +with attributes and functions. With the make-statement, code that +looks something like:: + + root = Tkinter.Tk() + frame = Tkinter.Frame(root) + frame.pack() + def say_hi(): + print "hi there, everyone!" + hi_there = Tkinter.Button(frame, text="Hello", command=say_hi) + hi_there.pack(side=Tkinter.LEFT) + root.mainloop() + +could be rewritten to group the the Button's function with its +declaration:: + + root = Tkinter.Tk() + frame = Tkinter.Frame(root) + frame.pack() + make Tkinter.Button hi_there(frame): + text = "Hello" + def command(): + print "hi there, everyone!" + hi_there.pack(side=Tkinter.LEFT) + root.mainloop() + + +Example: custom descriptors +--------------------------- + +Since descriptors are used to customize access to an attribute, it's +often useful to know the name of that attribute. Current Python +doesn't give an easy way to find this name and so a lot of custom +descriptors, like Ian Bicking's setonce descriptor [2]_, have to hack +around this somehow. With the make-statement, you could create a +``setonce`` attribute like:: + + class A(object): + ... + make setonce x: + "A's x attribute" + ... + +where the ``setonce`` descriptor would be defined like:: + + class setonce(object): + + def __init__(self, name, args, kwargs): + self._name = '_setonce_attr_%s' % name + self.__doc__ = kwargs.pop('__doc__', None) + + def __get__(self, obj, type=None): + if obj is None: + return self + return getattr(obj, self._name) + + def __set__(self, obj, value): + try: + getattr(obj, self._name) + except AttributeError: + setattr(obj, self._name, value) + else: + raise AttributeError("Attribute already set") + + def set(self, obj, value): + setattr(obj, self._name, value) + + def __delete__(self, obj): + delattr(obj, self._name) + +Note that unlike the original implementation, the private attribute +name is stable since it uses the name of the descriptor, and therefore +instances of class A are pickleable. + + +Example: property namespaces +---------------------------- + +Python's property type takes three function arguments and a docstring +argument which, though relevant only to the property, must be declared +before it and then passed as arguments to the property call, e.g.:: class C(object): ... @@ -73,60 +225,73 @@ ... def set_x(self): ... - x = property(get_x, set_x, ...) + x = property(get_x, set_x, "the x of the frobulation") -There have been a few recipes [2]_ trying to work around this -behavior, but with the new make statement (and an appropriate -definition of property), the getter and setter functions can be -defined in the property's namespace like:: +This issue has been brought up before, and Guido [3]_ and others [4]_ +have briefly mused over alternate property syntaxes to make declaring +properties easier. With the make-statement, the following syntax +could be supported:: class C(object): ... - make property x: - def get(self): + make block_property x: + '''The x of the frobulation''' + def fget(self): ... - def set(self): + def fset(self): ... -The definition of such a property callable could be as simple as:: - - def property(name, args, namespace): - fget = namespace.get('get') - fset = namespace.get('set') - fdel = namespace.get('delete') - doc = namespace.get('__doc__') - return __builtin__.property(fget, fset, fdel, doc) - -Of course, properties are only one of the many possible uses of the -make statement. The make statement is useful in essentially any -situation where a name is associated with a namespace. So, for -example, namespaces could be created as simply as:: - - make namespace ns: - """This creates a namespace named ns with a badger attribute - and a spam function""" - - badger = 42 - - def spam(): - ... +with the following definition of ``block_property``:: -And if Python acquires interfaces, given an appropriately defined -``interface`` callable, the make statement can support interface -creation through the syntax:: - - make interface C(...): - ... + def block_property(name, args, block_dict): + fget = block_dict.pop('fget', None) + fset = block_dict.pop('fset', None) + fdel = block_dict.pop('fdel', None) + doc = block_dict.pop('__doc__', None) + assert not block_dict + return property(fget, fset, fdel, doc) + + +Example: interfaces +------------------- + +Guido [5]_ and others have occasionally suggested introducing +interfaces into python. Most suggestions have offered syntax along +the lines of:: + + interface IFoo: + """Foo blah blah""" + + def fumble(name, count): + """docstring""" + +but since there is currently no way in Python to declare an interface +in this manner, most implementations of Python interfaces use class +objects instead, e.g. Zope's:: + + class IFoo(Interface): + """Foo blah blah""" + + def fumble(name, count): + """docstring""" + +With the new make-statement, these interfaces could instead be +declared as:: + + make Interface IFoo: + """Foo blah blah""" + + def fumble(name, count): + """docstring""" -This would mean that interface systems like that of Zope would no -longer have to abuse the class syntax to create proper interface -instances. +which makes the intent (that this is an interface, not a class) much +clearer. Specification ============= -Python will translate a make statement:: +Python will translate a make-statement:: make : @@ -139,25 +304,39 @@ The ```` expression is optional; if not present, an empty tuple will be assumed. -A patch is available implementing these semantics [3]_. +A patch is available implementing these semantics [6]_. -The make statement introduces a new keyword, ``make``. Thus in Python -2.6, the make statement will have to be enabled using ``from +The make-statement introduces a new keyword, ``make``. Thus in Python +2.6, the make-statement will have to be enabled using ``from __future__ import make_statement``. Open Issues =========== +Keyword +------- + Does the ``make`` keyword break too much code? Originally, the make statement used the keyword ``create`` (a suggestion due to Nick -Coghlan). However, investigations into the standard library [4]_ and -Zope+Plone code [5]_ revealed that ``create`` would break a lot more +Coghlan). However, investigations into the standard library [7]_ and +Zope+Plone code [8]_ revealed that ``create`` would break a lot more code, so ``make`` was adopted as the keyword instead. However, there are still a few instances where ``make`` would break code. Is there a better keyword for the statement? -********** +Some possible keywords and their counts in the standard library (plus +some installed packages): + +* make - 2 (both in tests) +* create - 19 (including existing function in imaplib) +* build - 83 (including existing class in distutils.command.build) +* construct - 0 +* produce - 0 + + +The make-statement as an alternate constructor +---------------------------------------------- Currently, there are not many functions which have the signature ``(name, args, kwargs)``. That means that something like:: @@ -169,10 +348,10 @@ is currently impossible because the dict constructor has a different signature. Does this sort of thing need to be supported? One suggestion, by Carl Banks, would be to add a ``__make__`` magic method -that would be called before ``__call__``. For types, the ``__make__`` -method would be identical to ``__call__`` (and thus unnecessary), but -dicts could support the make statement by defining a ``__make__`` -method on the dict type that looks something like:: +that if found would be called instead of ``__call__``. For types, +the ``__make__`` method would be identical to ``__call__`` and thus +unnecessary, but dicts could support the make-statement by defining a +``__make__`` method on the dict type that looks something like:: def __make__(cls, name, args, kwargs): return cls(**kwargs) @@ -185,6 +364,112 @@ x = 1 y = 2 +So the question is, will many types want to use the make-statement as +an alternate constructor? And if so, does that alternate constructor +need to have the same name as the original constructor? + + +Customizing the dict in which the block is executed +--------------------------------------------------- + +Should users of the make-statement be able to determine in which dict +object the code is executed? This would allow the make-statement to +be used in situations where a normal dict object would not suffice, +e.g. if order and repeated names must be allowed. Allowing this sort +of customization could allow XML to be written without repeating +element names, and with nesting of make-statements corresponding to +nesting of XML elements:: + + make Element html: + make Element body: + text('before first h1') + make Element h1: + attrib(style='first') + text('first h1') + tail('after first h1') + make Element h1: + attrib(style='second') + text('second h1') + tail('after second h1') + +If the make-statement tried to get the dict in which to execute its +block by calling the callable's ``__make_dict__`` method, the +following code would allow the make-statement to be used as above:: + + class Element(object): + + class __make_dict__(dict): + + def __init__(self, *args, **kwargs): + self._super = super(Element.__make_dict__, self) + self._super.__init__(*args, **kwargs) + self.elements = [] + self.text = None + self.tail = None + self.attrib = {} + + def __getitem__(self, name): + try: + return self._super.__getitem__(name) + except KeyError: + if name in ['attrib', 'text', 'tail']: + return getattr(self, 'set_%s' % name) + else: + return globals()[name] + + def __setitem__(self, name, value): + self._super.__setitem__(name, value) + self.elements.append(value) + + def set_attrib(self, **kwargs): + self.attrib = kwargs + + def set_text(self, text): + self.text = text + + def set_tail(self, text): + self.tail = text + + def __new__(cls, name, args, edict): + get_element = etree.ElementTree.Element + result = get_element(name, attrib=edict.attrib) + result.text = edict.text + result.tail = edict.tail + for element in edict.elements: + result.append(element) + return result + +Note, however, that the code to support this is somewhat fragile -- +it has to magically populate the namespace with ``attrib``, ``text`` +and ``tail``, and it assumes that every name binding inside the make +statement body is creating an Element. As it stands, this code would +break with the introduction of a simple for-loop to any one of the +make-statement bodies, because the for-loop would bind a name to a +non-Element object. This could be worked around by adding some sort +of isinstance check or attribute examination, but this still results +in a somewhat fragile solution. + +It has also been pointed out that the with-statement can provide +equivalent nesting with a much more explicit syntax:: + + with Element('html') as html: + with Element('body') as body: + body.text = 'before first h1' + with Element('h1', style='first') as h1: + h1.text = 'first h1' + h1.tail = 'after first h1' + with Element('h1', style='second') as h1: + h1.text = 'second h1' + h1.tail = 'after second h1' + +And if the repetition of the element names here is too much of a DRY +violoation, it is also possible to eliminate all as-clauses except for +the first by adding a few methods to Element. [9]_ + +So are there real use-cases for executing the block in a dict of a +different type? And if so, should the make-statement be extended to +support them? + Optional Extensions =================== @@ -213,7 +498,7 @@ Removing __metaclass__ in Python 3000 ------------------------------------- -As a side-effect of its generality, the make statement mostly +As a side-effect of its generality, the make-statement mostly eliminates the need for the ``__metaclass__`` attribute in class objects. Thus in Python 3000, instead of:: @@ -222,7 +507,7 @@ metaclasses could be supported by using the metaclass as the callable -in a make statement:: +in a make-statement:: make : @@ -234,7 +519,7 @@ Removing class statements in Python 3000 ---------------------------------------- -In the most extreme application of make statements, the class +In the most extreme application of make-statements, the class statement itself could be deprecated in favor of ``make type`` statements. @@ -245,18 +530,30 @@ .. [1] Michele Simionato's original suggestion (http://mail.python.org/pipermail/python-dev/2005-October/057435.html) -.. [2] Namespace-based property recipe +.. [2] Ian Bicking's setonce descriptor + (http://blog.ianbicking.org/easy-readonly-attributes.html) + +.. [3] Guido ponders property syntax + (http://mail.python.org/pipermail/python-dev/2005-October/057404.html) + +.. [4] Namespace-based property recipe (http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/442418) -.. [3] Make Statement patch +.. [5] Python interfaces + (http://www.artima.com/weblogs/viewpost.jsp?thread=86641) + +.. [6] Make Statement patch (http://ucsu.colorado.edu/~bethard/py/make_statement.patch) -.. [4] Instances of create in the stdlib +.. [7] Instances of create in the stdlib (http://mail.python.org/pipermail/python-list/2006-April/335159.html) -.. [5] Instances of create in Zope+Plone +.. [8] Instances of create in Zope+Plone (http://mail.python.org/pipermail/python-list/2006-April/335284.html) +.. [9] Eliminate as-clauses in with-statement XML + (http://mail.python.org/pipermail/python-list/2006-April/336774.html) + Copyright ========= Modified: peps/trunk/pep-3002.txt ============================================================================== --- peps/trunk/pep-3002.txt (original) +++ peps/trunk/pep-3002.txt Wed Apr 19 01:25:47 2006 @@ -6,8 +6,8 @@ Status: Draft Type: Process Content-Type: text/x-rst -Created: 03-Mar-2006 -Post-History: 03-Mar-2006 +Created: 27-Mar-2006 +Post-History: 27-Mar-2006, 13-Apr-2006 Abstract From python-checkins at python.org Wed Apr 19 01:58:53 2006 From: python-checkins at python.org (brett.cannon) Date: Wed, 19 Apr 2006 01:58:53 +0200 (CEST) Subject: [Python-checkins] r45553 - python/trunk/Makefile.pre.in Message-ID: <20060418235853.294F71E4002@bag.python.org> Author: brett.cannon Date: Wed Apr 19 01:58:52 2006 New Revision: 45553 Modified: python/trunk/Makefile.pre.in Log: Added a pycremoval rule. Called by clean, it removes all .pyc and .pyo files. Handy to have as a separate rule from clean when mucking around with bytecode generation. Modified: python/trunk/Makefile.pre.in ============================================================================== --- python/trunk/Makefile.pre.in (original) +++ python/trunk/Makefile.pre.in Wed Apr 19 01:58:52 2006 @@ -988,11 +988,12 @@ # Sanitation targets -- clean leaves libraries, executables and tags # files, which clobber removes those as well +pycremoval: + find $(srcdir) -name '*.py[co]' -exec rm -f {} ';' -clean: +clean: pycremoval find . -name '*.o' -exec rm -f {} ';' find . -name '*.s[ol]' -exec rm -f {} ';' - find $(srcdir) -name '*.py[co]' -exec rm -f {} ';' find $(srcdir)/build -name 'fficonfig.h' -exec rm -f {} ';' || true find $(srcdir)/build -name 'fficonfig.py' -exec rm -f {} ';' || true From python-checkins at python.org Wed Apr 19 02:13:25 2006 From: python-checkins at python.org (phillip.eby) Date: Wed, 19 Apr 2006 02:13:25 +0200 (CEST) Subject: [Python-checkins] r45554 - in sandbox/trunk/setuptools: _pkgutil.py easy_install.py pkg_resources.py setuptools/__init__.py setuptools/archive_util.py setuptools/command/__init__.py setuptools/command/alias.py setuptools/command/bdist_egg.py setuptools/command/bdist_rpm.py setuptools/command/build_ext.py setuptools/command/build_py.py setuptools/command/develop.py setuptools/command/easy_install.py setuptools/command/egg_info.py setuptools/command/install.py setuptools/command/install_egg_info.py setuptools/command/install_lib.py setuptools/command/install_scripts.py setuptools/command/rotate.py setuptools/command/saveopts.py setuptools/command/sdist.py setuptools/command/setopt.py setuptools/command/test.py setuptools/depends.py setuptools/dist.py setuptools/extension.py setuptools/package_index.py setuptools/sandbox.py setuptools/site-patch.py setuptools/tests/__init__.py setuptools/tests/test_resources.py Message-ID: <20060419001325.212D21E4030@bag.python.org> Author: phillip.eby Date: Wed Apr 19 02:13:21 2006 New Revision: 45554 Modified: sandbox/trunk/setuptools/_pkgutil.py sandbox/trunk/setuptools/easy_install.py sandbox/trunk/setuptools/pkg_resources.py sandbox/trunk/setuptools/setuptools/__init__.py sandbox/trunk/setuptools/setuptools/archive_util.py sandbox/trunk/setuptools/setuptools/command/__init__.py sandbox/trunk/setuptools/setuptools/command/alias.py sandbox/trunk/setuptools/setuptools/command/bdist_egg.py sandbox/trunk/setuptools/setuptools/command/bdist_rpm.py sandbox/trunk/setuptools/setuptools/command/build_ext.py sandbox/trunk/setuptools/setuptools/command/build_py.py sandbox/trunk/setuptools/setuptools/command/develop.py sandbox/trunk/setuptools/setuptools/command/easy_install.py sandbox/trunk/setuptools/setuptools/command/egg_info.py sandbox/trunk/setuptools/setuptools/command/install.py sandbox/trunk/setuptools/setuptools/command/install_egg_info.py sandbox/trunk/setuptools/setuptools/command/install_lib.py sandbox/trunk/setuptools/setuptools/command/install_scripts.py sandbox/trunk/setuptools/setuptools/command/rotate.py sandbox/trunk/setuptools/setuptools/command/saveopts.py sandbox/trunk/setuptools/setuptools/command/sdist.py sandbox/trunk/setuptools/setuptools/command/setopt.py sandbox/trunk/setuptools/setuptools/command/test.py sandbox/trunk/setuptools/setuptools/depends.py sandbox/trunk/setuptools/setuptools/dist.py sandbox/trunk/setuptools/setuptools/extension.py sandbox/trunk/setuptools/setuptools/package_index.py sandbox/trunk/setuptools/setuptools/sandbox.py sandbox/trunk/setuptools/setuptools/site-patch.py sandbox/trunk/setuptools/setuptools/tests/__init__.py sandbox/trunk/setuptools/setuptools/tests/test_resources.py Log: Backport whitespace normalization from 2.5 trunk. Modified: sandbox/trunk/setuptools/_pkgutil.py ============================================================================== --- sandbox/trunk/setuptools/_pkgutil.py (original) +++ sandbox/trunk/setuptools/_pkgutil.py Wed Apr 19 02:13:21 2006 @@ -44,7 +44,7 @@ class cls(cls,object): pass mro = cls.__mro__[1:] except TypeError: - mro = object, # must be an ExtensionClass or some such :( + mro = object, # must be an ExtensionClass or some such :( for t in mro: if t in registry: return registry[t](*args,**kw) @@ -64,7 +64,7 @@ wrapper.__dict__ = func.__dict__ wrapper.__doc__ = func.__doc__ wrapper.register = register - return wrapper + return wrapper def walk_packages(path=None, prefix='', onerror=None): @@ -160,7 +160,7 @@ modname = inspect.getmodulename(fn) if modname=='__init__' or modname in yielded: continue - + path = os.path.join(self.path, fn) ispkg = False @@ -276,7 +276,7 @@ try: import zipimport from zipimport import zipimporter - + def iter_zipimport_modules(importer, prefix=''): dirlist = zipimport._zip_directory_cache[importer.archive].keys() dirlist.sort() Modified: sandbox/trunk/setuptools/easy_install.py ============================================================================== --- sandbox/trunk/setuptools/easy_install.py (original) +++ sandbox/trunk/setuptools/easy_install.py Wed Apr 19 02:13:21 2006 @@ -3,4 +3,3 @@ if __name__ == '__main__': from setuptools.command.easy_install import main main() - Modified: sandbox/trunk/setuptools/pkg_resources.py ============================================================================== --- sandbox/trunk/setuptools/pkg_resources.py (original) +++ sandbox/trunk/setuptools/pkg_resources.py Wed Apr 19 02:13:21 2006 @@ -58,7 +58,7 @@ # Exceptions 'ResolutionError','VersionConflict','DistributionNotFound','UnknownExtra', 'ExtractionError', - + # Parsing functions and string utilities 'parse_requirements', 'parse_version', 'safe_name', 'safe_version', 'get_platform', 'compatible_platforms', 'yield_lines', 'split_sections', @@ -823,7 +823,7 @@ old_exc = sys.exc_info()[1] cache_path = self.extraction_path or get_default_cache() - + err = ExtractionError("""Can't extract file(s) to egg cache The following error occurred while trying to extract file(s) to the Python egg @@ -878,7 +878,7 @@ ensure_directory(target_path) except: self.extraction_error() - + self.cached_files[target_path] = 1 return target_path @@ -1264,11 +1264,11 @@ try: rename(tmpnam, real_path) - - except os.error: + + except os.error: if os.path.isfile(real_path): stat = os.stat(real_path) - + if stat.st_size==size and stat.st_mtime==timestamp: # size and stamp match, somebody did it just ahead of # us, so we're done @@ -2375,4 +2375,3 @@ # calling ``require()``) will get activated as well. add_activation_listener(lambda dist: dist.activate()) working_set.entries=[]; map(working_set.add_entry,sys.path) # match order - Modified: sandbox/trunk/setuptools/setuptools/__init__.py ============================================================================== --- sandbox/trunk/setuptools/setuptools/__init__.py (original) +++ sandbox/trunk/setuptools/setuptools/__init__.py Wed Apr 19 02:13:21 2006 @@ -40,7 +40,7 @@ return out setup = distutils.core.setup - + _Command = _get_unpatched(_Command) class Command(_Command): @@ -53,7 +53,7 @@ _Command.__init__(self,dist) for k,v in kw.items(): setattr(self,k,v) - + def reinitialize_command(self, command, reinit_subcommands=0, **kw): cmd = _Command.reinitialize_command(self, command, reinit_subcommands) for k,v in kw.items(): @@ -62,21 +62,3 @@ import distutils.core distutils.core.Command = Command # we can't patch distutils.cmd, alas - - - - - - - - - - - - - - - - - - Modified: sandbox/trunk/setuptools/setuptools/archive_util.py ============================================================================== --- sandbox/trunk/setuptools/setuptools/archive_util.py (original) +++ sandbox/trunk/setuptools/setuptools/archive_util.py Wed Apr 19 02:13:21 2006 @@ -14,7 +14,7 @@ """Couldn't recognize the archive type""" def default_filter(src,dst): - """The default progress/filter callback; returns True for all files""" + """The default progress/filter callback; returns True for all files""" return dst @@ -184,7 +184,7 @@ name = member.name # don't extract absolute paths or ones with .. in them if not name.startswith('/') and '..' not in name: - dst = os.path.join(extract_dir, *name.split('/')) + dst = os.path.join(extract_dir, *name.split('/')) dst = progress_filter(name, dst) if dst: if dst.endswith(os.sep): @@ -198,8 +198,3 @@ extraction_drivers = unpack_directory, unpack_zipfile, unpack_tarfile - - - - - Modified: sandbox/trunk/setuptools/setuptools/command/__init__.py ============================================================================== --- sandbox/trunk/setuptools/setuptools/command/__init__.py (original) +++ sandbox/trunk/setuptools/setuptools/command/__init__.py Wed Apr 19 02:13:21 2006 @@ -8,7 +8,7 @@ if sys.version>='2.5': # In Python 2.5 and above, distutils includes its own upload command __all__.remove('upload') - + from distutils.command.bdist import bdist Modified: sandbox/trunk/setuptools/setuptools/command/alias.py ============================================================================== --- sandbox/trunk/setuptools/setuptools/command/alias.py (original) +++ sandbox/trunk/setuptools/setuptools/command/alias.py Wed Apr 19 02:13:21 2006 @@ -11,17 +11,17 @@ if c in arg: return repr(arg) if arg.split()<>[arg]: return repr(arg) - return arg + return arg class alias(option_base): """Define a shortcut that invokes one or more commands""" - + description = "define a shortcut to invoke one or more commands" command_consumes_arguments = True user_options = [ - ('remove', 'r', 'remove (unset) the alias'), + ('remove', 'r', 'remove (unset) the alias'), ] + option_base.user_options boolean_options = option_base.boolean_options + ['remove'] @@ -77,6 +77,3 @@ else: source = '--filename=%r' % source return source+name+' '+command - - - Modified: sandbox/trunk/setuptools/setuptools/command/bdist_egg.py ============================================================================== --- sandbox/trunk/setuptools/setuptools/command/bdist_egg.py (original) +++ sandbox/trunk/setuptools/setuptools/command/bdist_egg.py Wed Apr 19 02:13:21 2006 @@ -233,7 +233,7 @@ if self.exclude_source_files: self.zap_pyfiles() - + # Make the archive make_zipfile(self.egg_output, archive_root, verbose=self.verbose, dry_run=self.dry_run) @@ -262,7 +262,7 @@ def make_init_files(self): """Create missing package __init__ files""" - init_files = [] + init_files = [] for base,dirs,files in walk_egg(self.bdist_dir): if base==self.bdist_dir: # don't put an __init__ in the root @@ -276,7 +276,7 @@ filename = os.path.join(base,'__init__.py') if not self.dry_run: f = open(filename,'w'); f.write(NS_PKG_STUB) - f.close() + f.close() init_files.append(filename) break else: @@ -329,7 +329,7 @@ def walk_egg(egg_dir): """Walk an unpacked egg's contents, skipping the metadata directory""" walker = os.walk(egg_dir) - base,dirs,files = walker.next() + base,dirs,files = walker.next() if 'EGG-INFO' in dirs: dirs.remove('EGG-INFO') yield base,dirs,files @@ -447,5 +447,3 @@ os.path.walk(base_dir, visit, None) return zip_filename - - Modified: sandbox/trunk/setuptools/setuptools/command/bdist_rpm.py ============================================================================== --- sandbox/trunk/setuptools/setuptools/command/bdist_rpm.py (original) +++ sandbox/trunk/setuptools/setuptools/command/bdist_rpm.py Wed Apr 19 02:13:21 2006 @@ -35,34 +35,3 @@ ] spec.insert(spec.index(line24)+1, "%define unmangled_version "+version) return spec - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Modified: sandbox/trunk/setuptools/setuptools/command/build_ext.py ============================================================================== --- sandbox/trunk/setuptools/setuptools/command/build_ext.py (original) +++ sandbox/trunk/setuptools/setuptools/command/build_ext.py Wed Apr 19 02:13:21 2006 @@ -283,5 +283,3 @@ self.create_static_lib( objects, basename, output_dir, debug, target_lang ) - - Modified: sandbox/trunk/setuptools/setuptools/command/build_py.py ============================================================================== --- sandbox/trunk/setuptools/setuptools/command/build_py.py (original) +++ sandbox/trunk/setuptools/setuptools/command/build_py.py Wed Apr 19 02:13:21 2006 @@ -93,7 +93,7 @@ ei_cmd = self.get_finalized_command('egg_info') for path in ei_cmd.filelist.files: if path.endswith('.py'): - continue + continue d,f = os.path.split(assert_relative(path)) prev = None while d and d!=prev and d not in src_dirs: @@ -142,7 +142,7 @@ f = open(init_py,'rU') if 'declare_namespace' not in f.read(): - from distutils.errors import DistutilsError + from distutils.errors import DistutilsError raise DistutilsError( "Namespace package problem: %s is a namespace package, but its\n" "__init__.py does not call declare_namespace()! Please fix it.\n" @@ -167,7 +167,7 @@ globs = (self.exclude_package_data.get('', []) + self.exclude_package_data.get(package, [])) bad = [] - for pattern in globs: + for pattern in globs: bad.extend( fnmatch.filter( files, os.path.join(src_dir, convert_path(pattern)) @@ -190,16 +190,3 @@ setup.py directory, *never* absolute paths. """ % path ) - - - - - - - - - - - - - Modified: sandbox/trunk/setuptools/setuptools/command/develop.py ============================================================================== --- sandbox/trunk/setuptools/setuptools/command/develop.py (original) +++ sandbox/trunk/setuptools/setuptools/command/develop.py Wed Apr 19 02:13:21 2006 @@ -46,7 +46,7 @@ "Please rename %r to %r before using 'develop'" % (ei.egg_info, ei.broken_egg_info) ) - self.args = [ei.egg_name] + self.args = [ei.egg_name] easy_install.finalize_options(self) self.egg_link = os.path.join(self.install_dir, ei.egg_name+'.egg-link') self.egg_base = ei.egg_base @@ -104,7 +104,7 @@ # create wrapper scripts in the script dir, pointing to dist.scripts # new-style... - self.install_wrapper_scripts(dist) + self.install_wrapper_scripts(dist) # ...and old-style for script_name in self.distribution.scripts or []: @@ -114,10 +114,3 @@ script_text = f.read() f.close() self.install_script(dist, script_name, script_text, script_path) - - - - - - - Modified: sandbox/trunk/setuptools/setuptools/command/easy_install.py ============================================================================== --- sandbox/trunk/setuptools/setuptools/command/easy_install.py (original) +++ sandbox/trunk/setuptools/setuptools/command/easy_install.py Wed Apr 19 02:13:21 2006 @@ -1357,7 +1357,7 @@ """Write changed .pth file back to disk""" if not self.dirty: return - + data = '\n'.join(map(self.make_relative,self.paths)) if data: log.debug("Saving %s", self.filename) @@ -1434,7 +1434,7 @@ del zdc[p] return - + def get_script_args(dist, executable=sys_executable): """Yield write_script() argument tuples for a distribution's entrypoints""" spec = str(dist.as_requirement()) @@ -1553,8 +1553,3 @@ distclass=DistributionWithoutHelpCommands, **kw ) ) - - - - - Modified: sandbox/trunk/setuptools/setuptools/command/egg_info.py ============================================================================== --- sandbox/trunk/setuptools/setuptools/command/egg_info.py (original) +++ sandbox/trunk/setuptools/setuptools/command/egg_info.py Wed Apr 19 02:13:21 2006 @@ -363,7 +363,3 @@ if match: return int(match.group(1)) return 0 - - - - Modified: sandbox/trunk/setuptools/setuptools/command/install.py ============================================================================== --- sandbox/trunk/setuptools/setuptools/command/install.py (original) +++ sandbox/trunk/setuptools/setuptools/command/install.py Wed Apr 19 02:13:21 2006 @@ -60,7 +60,7 @@ caller = sys._getframe(2) caller_module = caller.f_globals.get('__name__','') caller_name = caller.f_code.co_name - + if caller_module != 'distutils.dist' or caller_name!='run_commands': # We weren't called from the command line or setup(), so we # should run in backward-compatibility mode to support bdist_* @@ -68,7 +68,7 @@ _install.run(self) else: self.do_egg_install() - + @@ -99,25 +99,3 @@ cmd.args = args cmd.run() setuptools.bootstrap_install_from = None - - - - - - - - - - - - - - - - - - - - - - Modified: sandbox/trunk/setuptools/setuptools/command/install_egg_info.py ============================================================================== --- sandbox/trunk/setuptools/setuptools/command/install_egg_info.py (original) +++ sandbox/trunk/setuptools/setuptools/command/install_egg_info.py Wed Apr 19 02:13:21 2006 @@ -22,7 +22,7 @@ None, None, ei_cmd.egg_name, ei_cmd.egg_version ).egg_name()+'.egg-info' self.source = ei_cmd.egg_info - self.target = os.path.join(self.install_dir, basename) + self.target = os.path.join(self.install_dir, basename) self.outputs = [self.target] def run(self): @@ -43,7 +43,7 @@ return self.outputs def copytree(self): - # Copy the .egg-info tree to site-packages + # Copy the .egg-info tree to site-packages def skimmer(src,dst): # filter out source-control directories; note that 'src' is always # a '/'-separated path, regardless of platform. 'dst' is a @@ -78,5 +78,4 @@ "(p not in mp) and mp.append(p)\n" % locals() ) - f.close() - + f.close() Modified: sandbox/trunk/setuptools/setuptools/command/install_lib.py ============================================================================== --- sandbox/trunk/setuptools/setuptools/command/install_lib.py (original) +++ sandbox/trunk/setuptools/setuptools/command/install_lib.py Wed Apr 19 02:13:21 2006 @@ -74,9 +74,3 @@ if exclude: return [f for f in outputs if f not in exclude] return outputs - - - - - - Modified: sandbox/trunk/setuptools/setuptools/command/install_scripts.py ============================================================================== --- sandbox/trunk/setuptools/setuptools/command/install_scripts.py (original) +++ sandbox/trunk/setuptools/setuptools/command/install_scripts.py Wed Apr 19 02:13:21 2006 @@ -11,7 +11,7 @@ def initialize_options(self): _install_scripts.initialize_options(self) self.no_ep = False - + def run(self): self.run_command("egg_info") if self.distribution.scripts: @@ -20,9 +20,9 @@ self.outfiles = [] if self.no_ep: # don't install entry point scripts into .egg file! - return + return - ei_cmd = self.get_finalized_command("egg_info") + ei_cmd = self.get_finalized_command("egg_info") dist = Distribution( ei_cmd.egg_base, PathMetadata(ei_cmd.egg_base, ei_cmd.egg_info), ei_cmd.egg_name, ei_cmd.egg_version, @@ -54,29 +54,3 @@ os.chmod(target,0755) except (AttributeError, os.error): pass - - - - - - - - - - - - - - - - - - - - - - - - - - Modified: sandbox/trunk/setuptools/setuptools/command/rotate.py ============================================================================== --- sandbox/trunk/setuptools/setuptools/command/rotate.py (original) +++ sandbox/trunk/setuptools/setuptools/command/rotate.py Wed Apr 19 02:13:21 2006 @@ -28,7 +28,7 @@ "(e.g. '.zip' or '.egg')" ) if self.keep is None: - raise DistutilsOptionError("Must specify number of files to keep") + raise DistutilsOptionError("Must specify number of files to keep") try: self.keep = int(self.keep) except ValueError: @@ -55,28 +55,3 @@ log.info("Deleting %s", f) if not self.dry_run: os.unlink(f) - - - - - - - - - - - - - - - - - - - - - - - - - Modified: sandbox/trunk/setuptools/setuptools/command/saveopts.py ============================================================================== --- sandbox/trunk/setuptools/setuptools/command/saveopts.py (original) +++ sandbox/trunk/setuptools/setuptools/command/saveopts.py Wed Apr 19 02:13:21 2006 @@ -22,4 +22,3 @@ settings.setdefault(cmd,{})[opt] = val edit_config(self.filename, settings, self.dry_run) - Modified: sandbox/trunk/setuptools/setuptools/command/sdist.py ============================================================================== --- sandbox/trunk/setuptools/setuptools/command/sdist.py (original) +++ sandbox/trunk/setuptools/setuptools/command/sdist.py Wed Apr 19 02:13:21 2006 @@ -144,7 +144,7 @@ self.filelist.append(os.path.join(ei_cmd.egg_info,'SOURCES.txt')) self.check_metadata() - self.make_distribution() + self.make_distribution() dist_files = getattr(self.distribution,'dist_files',[]) for file in self.archive_files: @@ -161,4 +161,3 @@ # dying and thus masking the real error sys.exc_info()[2].tb_next.tb_frame.f_locals['template'].close() raise - Modified: sandbox/trunk/setuptools/setuptools/command/setopt.py ============================================================================== --- sandbox/trunk/setuptools/setuptools/command/setopt.py (original) +++ sandbox/trunk/setuptools/setuptools/command/setopt.py Wed Apr 19 02:13:21 2006 @@ -82,7 +82,7 @@ class option_base(Command): """Abstract base class for commands that mess with config files""" - + user_options = [ ('global-config', 'g', "save options to the site-wide distutils.cfg file"), @@ -94,7 +94,7 @@ boolean_options = [ 'global-config', 'user-config', - ] + ] def initialize_options(self): self.global_config = None @@ -116,7 +116,7 @@ "Must specify only one configuration file option", filenames ) - self.filename, = filenames + self.filename, = filenames @@ -130,7 +130,7 @@ ('command=', 'c', 'command to set an option for'), ('option=', 'o', 'option to set'), ('set-value=', 's', 'value of the option'), - ('remove', 'r', 'remove (unset) the value'), + ('remove', 'r', 'remove (unset) the value'), ] + option_base.user_options boolean_options = option_base.boolean_options + ['remove'] @@ -156,9 +156,3 @@ }, self.dry_run ) - - - - - - Modified: sandbox/trunk/setuptools/setuptools/command/test.py ============================================================================== --- sandbox/trunk/setuptools/setuptools/command/test.py (original) +++ sandbox/trunk/setuptools/setuptools/command/test.py Wed Apr 19 02:13:21 2006 @@ -88,7 +88,7 @@ self.reinitialize_command('build_ext', inplace=1) self.run_command('build_ext') - if self.distribution.tests_require: + if self.distribution.tests_require: self.distribution.fetch_build_eggs(self.distribution.tests_require) if self.test_suite: @@ -117,7 +117,3 @@ None, None, [unittest.__file__]+self.test_args, testLoader = loader_class() ) - - - - Modified: sandbox/trunk/setuptools/setuptools/depends.py ============================================================================== --- sandbox/trunk/setuptools/setuptools/depends.py (original) +++ sandbox/trunk/setuptools/setuptools/depends.py Wed Apr 19 02:13:21 2006 @@ -237,10 +237,3 @@ return const else: const = default - - - - - - - Modified: sandbox/trunk/setuptools/setuptools/dist.py ============================================================================== --- sandbox/trunk/setuptools/setuptools/dist.py (original) +++ sandbox/trunk/setuptools/setuptools/dist.py Wed Apr 19 02:13:21 2006 @@ -796,25 +796,3 @@ " doesn't contain any packages or modules under %s" % (self.description, item, item) ) - - - - - - - - - - - - - - - - - - - - - - Modified: sandbox/trunk/setuptools/setuptools/extension.py ============================================================================== --- sandbox/trunk/setuptools/setuptools/extension.py (original) +++ sandbox/trunk/setuptools/setuptools/extension.py Wed Apr 19 02:13:21 2006 @@ -14,7 +14,7 @@ """Extension that uses '.c' files in place of '.pyx' files""" if not have_pyrex: - # convert .pyx extensions to .c + # convert .pyx extensions to .c def __init__(self,*args,**kw): _Extension.__init__(self,*args,**kw) sources = [] @@ -33,4 +33,3 @@ distutils.extension.Extension = Extension if 'distutils.command.build_ext' in sys.modules: sys.modules['distutils.command.build_ext'].Extension = Extension - Modified: sandbox/trunk/setuptools/setuptools/package_index.py ============================================================================== --- sandbox/trunk/setuptools/setuptools/package_index.py (original) +++ sandbox/trunk/setuptools/setuptools/package_index.py Wed Apr 19 02:13:21 2006 @@ -260,7 +260,7 @@ def find_packages(self, requirement): self.scan_url(self.index_url + requirement.unsafe_name+'/') - + if not self.package_pages.get(requirement.key): # Fall back to safe version of the name self.scan_url(self.index_url + requirement.project_name+'/') @@ -489,7 +489,7 @@ "Can't process plain .py files without an '#egg=name-version'" " suffix to enable automatic setup script generation." ) - + dl_blocksize = 8192 def _download_to(self, url, filename): self.url_ok(url,True) # raises error if not allowed @@ -672,26 +672,3 @@ # DNS-bl0ck1n9 f1r3w4llz sUx0rs! _sf_mirrors[:] = ['dl.sourceforge.net'] return random.choice(_sf_mirrors) - - - - - - - - - - - - - - - - - - - - - - - Modified: sandbox/trunk/setuptools/setuptools/sandbox.py ============================================================================== --- sandbox/trunk/setuptools/setuptools/sandbox.py (original) +++ sandbox/trunk/setuptools/setuptools/sandbox.py Wed Apr 19 02:13:21 2006 @@ -201,5 +201,3 @@ support alternate installation locations even if you run its setup script by hand. Please inform the package's author and the EasyInstall maintainers to find out if a fix or workaround is available.""" % self.args - - Modified: sandbox/trunk/setuptools/setuptools/site-patch.py ============================================================================== --- sandbox/trunk/setuptools/setuptools/site-patch.py (original) +++ sandbox/trunk/setuptools/setuptools/site-patch.py Wed Apr 19 02:13:21 2006 @@ -1,5 +1,5 @@ def __boot(): - import sys, imp, os, os.path + import sys, imp, os, os.path PYTHONPATH = os.environ.get('PYTHONPATH') if PYTHONPATH is None or (sys.platform=='win32' and not PYTHONPATH): PYTHONPATH = [] @@ -48,7 +48,7 @@ addsitedir(item) sys.__egginsert += oldpos # restore effective old position - + d,nd = makepath(stdpath[0]) insert_at = None new_path = [] @@ -66,17 +66,9 @@ # new path after the insert point, back-insert it new_path.insert(insert_at, item) insert_at += 1 - + sys.path[:] = new_path -if __name__=='site': +if __name__=='site': __boot() del __boot - - - - - - - - Modified: sandbox/trunk/setuptools/setuptools/tests/__init__.py ============================================================================== --- sandbox/trunk/setuptools/setuptools/tests/__init__.py (original) +++ sandbox/trunk/setuptools/setuptools/tests/__init__.py Wed Apr 19 02:13:21 2006 @@ -362,8 +362,3 @@ ts5 = makeSetup().get_command_obj('test') ts5.ensure_finalized() self.assertEqual(ts5.test_suite, None) - - - - - Modified: sandbox/trunk/setuptools/setuptools/tests/test_resources.py ============================================================================== --- sandbox/trunk/setuptools/setuptools/tests/test_resources.py (original) +++ sandbox/trunk/setuptools/setuptools/tests/test_resources.py Wed Apr 19 02:13:21 2006 @@ -143,7 +143,7 @@ self.assertRaises(VersionConflict, ws.resolve, parse_requirements("Foo==0.9"), ad) ws = WorkingSet([]) # reset - + # Request an extra that causes an unresolved dependency for "Baz" self.assertRaises( DistributionNotFound, ws.resolve,parse_requirements("Foo[bar]"), ad @@ -161,7 +161,7 @@ self.assertRaises( VersionConflict, ws.resolve, parse_requirements("Foo==1.2\nFoo!=1.2"), ad ) - + def testDistroDependsOptions(self): d = self.distRequires(""" Twisted>=1.5 @@ -481,12 +481,3 @@ for p,v1 in enumerate(torture): for v2 in torture[p+1:]: c(v2,v1) - - - - - - - - - From python-checkins at python.org Wed Apr 19 02:18:59 2006 From: python-checkins at python.org (phillip.eby) Date: Wed, 19 Apr 2006 02:18:59 +0200 (CEST) Subject: [Python-checkins] r45555 - sandbox/trunk/setuptools/setuptools/package_index.py Message-ID: <20060419001859.02D781E4009@bag.python.org> Author: phillip.eby Date: Wed Apr 19 02:18:58 2006 New Revision: 45555 Modified: sandbox/trunk/setuptools/setuptools/package_index.py Log: Support file:// links to directories in --find-links, so that easy_install can build packages from local source checkouts. Modified: sandbox/trunk/setuptools/setuptools/package_index.py ============================================================================== --- sandbox/trunk/setuptools/setuptools/package_index.py (original) +++ sandbox/trunk/setuptools/setuptools/package_index.py Wed Apr 19 02:18:58 2006 @@ -179,7 +179,7 @@ def process_filename(self, fn, nested=False): # process filenames or directories if not os.path.exists(fn): - self.warn("Not found: %s", url) + self.warn("Not found: %s", fn) return if os.path.isdir(fn) and not nested: @@ -450,7 +450,7 @@ def gen_setup(self, filename, fragment, tmpdir): - match = EGG_FRAGMENT.match(fragment); #import pdb; pdb.set_trace() + match = EGG_FRAGMENT.match(fragment) dists = match and [d for d in interpret_distro_name(filename, match.group(1), None) if d.version ] or [] @@ -582,7 +582,6 @@ def _download_url(self, scheme, url, tmpdir): - # Determine download filename # name = filter(None,urlparse.urlparse(url)[2].split('/')) @@ -602,6 +601,8 @@ # if scheme=='svn' or scheme.startswith('svn+'): return self._download_svn(url, filename) + elif scheme=='file': + return urllib2.url2pathname(urlparse.urlparse(url)[2]) else: headers = self.retry_sf_download(url, filename) if 'html' in headers['content-type'].lower(): @@ -612,7 +613,6 @@ def scan_url(self, url): self.process_url(url, True) - def _download_html(self, url, headers, filename, tmpdir): file = open(filename) for line in file: @@ -672,3 +672,26 @@ # DNS-bl0ck1n9 f1r3w4llz sUx0rs! _sf_mirrors[:] = ['dl.sourceforge.net'] return random.choice(_sf_mirrors) + + + + + + + + + + + + + + + + + + + + + + +# this line is a kludge to keep the trailing blank lines for pje's editor From python-checkins at python.org Wed Apr 19 02:21:59 2006 From: python-checkins at python.org (phillip.eby) Date: Wed, 19 Apr 2006 02:21:59 +0200 (CEST) Subject: [Python-checkins] r45556 - sandbox/branches/setuptools-0.6/setuptools/package_index.py Message-ID: <20060419002159.D7BB21E4002@bag.python.org> Author: phillip.eby Date: Wed Apr 19 02:21:57 2006 New Revision: 45556 Modified: sandbox/branches/setuptools-0.6/setuptools/package_index.py Log: Backport support for file:// directory URLs in --find-links to 0.6 branch. Modified: sandbox/branches/setuptools-0.6/setuptools/package_index.py ============================================================================== --- sandbox/branches/setuptools-0.6/setuptools/package_index.py (original) +++ sandbox/branches/setuptools-0.6/setuptools/package_index.py Wed Apr 19 02:21:57 2006 @@ -179,7 +179,7 @@ def process_filename(self, fn, nested=False): # process filenames or directories if not os.path.exists(fn): - self.warn("Not found: %s", url) + self.warn("Not found: %s", fn) return if os.path.isdir(fn) and not nested: @@ -260,7 +260,7 @@ def find_packages(self, requirement): self.scan_url(self.index_url + requirement.unsafe_name+'/') - + if not self.package_pages.get(requirement.key): # Fall back to safe version of the name self.scan_url(self.index_url + requirement.project_name+'/') @@ -450,7 +450,7 @@ def gen_setup(self, filename, fragment, tmpdir): - match = EGG_FRAGMENT.match(fragment); #import pdb; pdb.set_trace() + match = EGG_FRAGMENT.match(fragment) dists = match and [d for d in interpret_distro_name(filename, match.group(1), None) if d.version ] or [] @@ -489,7 +489,7 @@ "Can't process plain .py files without an '#egg=name-version'" " suffix to enable automatic setup script generation." ) - + dl_blocksize = 8192 def _download_to(self, url, filename): self.url_ok(url,True) # raises error if not allowed @@ -582,7 +582,6 @@ def _download_url(self, scheme, url, tmpdir): - # Determine download filename # name = filter(None,urlparse.urlparse(url)[2].split('/')) @@ -602,6 +601,8 @@ # if scheme=='svn' or scheme.startswith('svn+'): return self._download_svn(url, filename) + elif scheme=='file': + return urllib2.url2pathname(urlparse.urlparse(url)[2]) else: headers = self.retry_sf_download(url, filename) if 'html' in headers['content-type'].lower(): @@ -612,7 +613,6 @@ def scan_url(self, url): self.process_url(url, True) - def _download_html(self, url, headers, filename, tmpdir): file = open(filename) for line in file: @@ -694,4 +694,4 @@ - +# this line is a kludge to keep the trailing blank lines for pje's editor From python-checkins at python.org Wed Apr 19 06:12:57 2006 From: python-checkins at python.org (neal.norwitz) Date: Wed, 19 Apr 2006 06:12:57 +0200 (CEST) Subject: [Python-checkins] r45557 - peps/trunk/pep-0356.txt Message-ID: <20060419041257.D96AA1E400A@bag.python.org> Author: neal.norwitz Date: Wed Apr 19 06:12:56 2006 New Revision: 45557 Modified: peps/trunk/pep-0356.txt Log: These bugs (multiple ifs in comprehensions and PyRun_* API changes) have been fixed. Modified: peps/trunk/pep-0356.txt ============================================================================== --- peps/trunk/pep-0356.txt (original) +++ peps/trunk/pep-0356.txt Wed Apr 19 06:12:56 2006 @@ -153,9 +153,6 @@ Open issues - - 1465834: bdist_wininst preinstall scripts broken by lack of - PyRun_SimpleString (and others?) as symbols instead of macros. - - AST compiler problems (Owner: Jeremy Hylton) * eval(str(-sys.maxint - 1)) should produce an int, not long. @@ -172,11 +169,6 @@ (Owner: Jeremy Hylton) http://python.org/sf/1191458 - - list/generator comprehensions don't support multiple if stmts - (a grammar issue introduced by the PEP 308 patch) - (Owner: Thomas Wouters) - http://python.org/sf/1466641 - - xmlplus/xmlcore situation wrt ElementTree needs resolution (Owners: Fred Drake/Martin von Loewis/Fredrik Lundh) http://mail.python.org/pipermail/python-dev/2005-December/058752.html From python-checkins at python.org Wed Apr 19 07:30:16 2006 From: python-checkins at python.org (neal.norwitz) Date: Wed, 19 Apr 2006 07:30:16 +0200 (CEST) Subject: [Python-checkins] r45558 - peps/trunk/pep-0356.txt Message-ID: <20060419053016.125B51E400A@bag.python.org> Author: neal.norwitz Date: Wed Apr 19 07:30:15 2006 New Revision: 45558 Modified: peps/trunk/pep-0356.txt Log: Update planned date for a2. Re-org completed feature into sections (mostly new modules) Modified: peps/trunk/pep-0356.txt ============================================================================== --- peps/trunk/pep-0356.txt (original) +++ peps/trunk/pep-0356.txt Wed Apr 19 07:30:15 2006 @@ -39,7 +39,7 @@ unfolds. alpha 1: April 5, 2006 [completed] - alpha 2: April 29, 2006 [planned] + alpha 2: April 25, 2006 [planned] alpha 3: May 27, 2006 [planned] beta 1: June 24, 2006 [planned] beta 2: July 15, 2006 [planned] @@ -63,35 +63,40 @@ PEP 353: Using ssize_t as the index type PEP 357: Allowing Any Object to be Used for Slicing - - ASCII is the default coding + - ASCII became the default coding - AST-based compiler - Access to C AST from Python through new _ast module - - Add support for reading shadow passwords (http://python.org/sf/579435) - - any()/all() builtin truth functions - - new hashlib module add support for SHA-224, -256, -384, and -512 - (replaces old md5 and sha modules) + New standard library modules + + - cProfile -- suitable for profiling long running applications + with minimal overhead + + - ctypes -- optional component of the windows installer + + - ElementTree and cElementTree -- by Fredrik Lundh - - new cProfile module suitable for profiling long running applications - with minimal overhead + - hashlib -- adds support for SHA-224, -256, -384, and -512 + (replaces old md5 and sha modules) - - Fredrik Lundh's ElementTree and cElementTree + - msilib -- for creating MSI files and bdist_msi in distutils. + There are no docs yet. - - Add ctypes to the standard library, make it an optional component - in the windows installer. Thomas Heller will maintain it. + - pysqlite - - Add support for the Unicode 4.1 UCD + - setuptools - - Add msilib module for creating MSI files and bdist_msi in distutils. - There are no docs yet. + Other notable features - - pysqlite was added to the standard library + - Added support for reading shadow passwords (http://python.org/sf/579435) - - Add PEP 302 zipfile/__loader__ support to the following modules: - warnings, linecache, inspect, traceback, site, and doctest + - Added support for the Unicode 4.1 UCD + + - Added PEP 302 zipfile/__loader__ support to the following modules: + warnings, linecache, inspect, traceback, site, and doctest Possible features for 2.5 @@ -119,9 +124,6 @@ (Owner: ???) http://mail.python.org/pipermail/python-dev/2006-February/060926.html - - setuptools to the standard library - (Owner: Phillip Eby) - - wsgiref to the standard library (Owner: Phillip Eby) From python-checkins at python.org Wed Apr 19 07:51:06 2006 From: python-checkins at python.org (neal.norwitz) Date: Wed, 19 Apr 2006 07:51:06 +0200 (CEST) Subject: [Python-checkins] r45559 - peps/trunk/pep-0356.txt Message-ID: <20060419055106.7101F1E400A@bag.python.org> Author: neal.norwitz Date: Wed Apr 19 07:51:05 2006 New Revision: 45559 Modified: peps/trunk/pep-0356.txt Log: Add a list of missing docs Modified: peps/trunk/pep-0356.txt ============================================================================== --- peps/trunk/pep-0356.txt (original) +++ peps/trunk/pep-0356.txt Wed Apr 19 07:51:05 2006 @@ -155,6 +155,13 @@ Open issues + - Missing documentation + * ctypes + * ElementTree/cElementTree + * msilib + * pysqlite + * setuptools (written, needs conversion to proper format) + - AST compiler problems (Owner: Jeremy Hylton) * eval(str(-sys.maxint - 1)) should produce an int, not long. From neal at metaslash.com Wed Apr 19 11:06:08 2006 From: neal at metaslash.com (Neal Norwitz) Date: Wed, 19 Apr 2006 05:06:08 -0400 Subject: [Python-checkins] Python Regression Test Failures refleak (1) Message-ID: <20060419090608.GA4346@python.psfb.org> test_cmd_line leaked [0, 17, -17] references test_filecmp leaked [0, 13, 0] references test_threading_local leaked [-93, 0, 0] references test_urllib2 leaked [-121, 88, 99] references From python-checkins at python.org Wed Apr 19 13:50:28 2006 From: python-checkins at python.org (armin.rigo) Date: Wed, 19 Apr 2006 13:50:28 +0200 (CEST) Subject: [Python-checkins] r45560 - in python/trunk: Lib/test/test_socket.py Misc/NEWS Modules/socketmodule.c Message-ID: <20060419115028.4104B1E405D@bag.python.org> Author: armin.rigo Date: Wed Apr 19 13:50:27 2006 New Revision: 45560 Modified: python/trunk/Lib/test/test_socket.py python/trunk/Misc/NEWS python/trunk/Modules/socketmodule.c Log: SF Patch #1062014: AF_UNIX sockets under Linux have a special abstract namespace that is now fully supported. Modified: python/trunk/Lib/test/test_socket.py ============================================================================== --- python/trunk/Lib/test/test_socket.py (original) +++ python/trunk/Lib/test/test_socket.py Wed Apr 19 13:50:27 2006 @@ -825,6 +825,32 @@ self.assert_(issubclass(socket.gaierror, socket.error)) self.assert_(issubclass(socket.timeout, socket.error)) +class TestLinuxAbstractNamespace(unittest.TestCase): + + UNIX_PATH_MAX = 108 + + def testLinuxAbstractNamespace(self): + address = "\x00python-test-hello\x00\xff" + s1 = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) + s1.bind(address) + s1.listen(1) + s2 = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) + s2.connect(s1.getsockname()) + s1.accept() + self.assertEqual(s1.getsockname(), address) + self.assertEqual(s2.getpeername(), address) + + def testMaxName(self): + address = "\x00" + "h" * (self.UNIX_PATH_MAX - 1) + s = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) + s.bind(address) + self.assertEqual(s.getsockname(), address) + + def testNameOverflow(self): + address = "\x00" + "h" * self.UNIX_PATH_MAX + s = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) + self.assertRaises(socket.error, s.bind, address) + def test_main(): tests = [GeneralModuleTests, BasicTCPTest, TCPTimeoutTest, TestExceptions] @@ -840,6 +866,8 @@ ]) if hasattr(socket, "socketpair"): tests.append(BasicSocketPairTest) + if sys.platform == 'linux2': + tests.append(TestLinuxAbstractNamespace) test_support.run_unittest(*tests) if __name__ == "__main__": Modified: python/trunk/Misc/NEWS ============================================================================== --- python/trunk/Misc/NEWS (original) +++ python/trunk/Misc/NEWS Wed Apr 19 13:50:27 2006 @@ -80,6 +80,9 @@ - Bug #1332852: bsddb module minimum BerkeleyDB version raised to 3.3 as older versions cause excessive test failures. +- Patch #1062014: AF_UNIX sockets under Linux have a special + abstract namespace that is now fully supported. + Library ------- Modified: python/trunk/Modules/socketmodule.c ============================================================================== --- python/trunk/Modules/socketmodule.c (original) +++ python/trunk/Modules/socketmodule.c Wed Apr 19 13:50:27 2006 @@ -968,7 +968,18 @@ case AF_UNIX: { struct sockaddr_un *a = (struct sockaddr_un *) addr; - return PyString_FromString(a->sun_path); +#ifdef linux + if (a->sun_path[0] == 0) { /* Linux abstract namespace */ + addrlen -= (sizeof(*a) - sizeof(a->sun_path)); + return PyString_FromStringAndSize(a->sun_path, + addrlen); + } + else +#endif /* linux */ + { + /* regular NULL-terminated string */ + return PyString_FromString(a->sun_path); + } } #endif /* AF_UNIX */ @@ -1098,14 +1109,28 @@ addr = (struct sockaddr_un*)&(s->sock_addr).un; if (!PyArg_Parse(args, "t#", &path, &len)) return 0; - if (len >= sizeof addr->sun_path) { - PyErr_SetString(socket_error, - "AF_UNIX path too long"); - return 0; +#ifdef linux + if (len > 0 && path[0] == 0) { + /* Linux abstract namespace extension */ + if (len > sizeof addr->sun_path) { + PyErr_SetString(socket_error, + "AF_UNIX path too long"); + return 0; + } + } + else +#endif /* linux */ + { + /* regular NULL-terminated string */ + if (len >= sizeof addr->sun_path) { + PyErr_SetString(socket_error, + "AF_UNIX path too long"); + return 0; + } + addr->sun_path[len] = 0; } addr->sun_family = s->sock_family; memcpy(addr->sun_path, path, len); - addr->sun_path[len] = 0; *addr_ret = (struct sockaddr *) addr; #if defined(PYOS_OS2) *len_ret = sizeof(*addr); From python-checkins at python.org Wed Apr 19 14:37:16 2006 From: python-checkins at python.org (andrew.kuchling) Date: Wed, 19 Apr 2006 14:37:16 +0200 (CEST) Subject: [Python-checkins] r45561 - peps/trunk/pep-0356.txt Message-ID: <20060419123716.CFACB1E4004@bag.python.org> Author: andrew.kuchling Date: Wed Apr 19 14:37:16 2006 New Revision: 45561 Modified: peps/trunk/pep-0356.txt Log: Add an item I'll work on Modified: peps/trunk/pep-0356.txt ============================================================================== --- peps/trunk/pep-0356.txt (original) +++ peps/trunk/pep-0356.txt Wed Apr 19 14:37:16 2006 @@ -186,6 +186,11 @@ "timing" (listed as obsolete), "cl" (listed as possibly not up-to-date), and "sv" (listed as obsolete hardware specific). + - Add write support for mailboxes from the code in sandbox/mailbox. + AMK will take one last look at the code for backward-compatibility + problems and then apply the changes before 2.5a2. + (Owner: A.M. Kuchling. It would still be good if another person + would take a look at the new code.) Copyright From python-checkins at python.org Wed Apr 19 14:55:39 2006 From: python-checkins at python.org (andrew.kuchling) Date: Wed, 19 Apr 2006 14:55:39 +0200 (CEST) Subject: [Python-checkins] r45562 - python/trunk/Doc/whatsnew/whatsnew25.tex Message-ID: <20060419125539.BC1031E4004@bag.python.org> Author: andrew.kuchling Date: Wed Apr 19 14:55:39 2006 New Revision: 45562 Modified: python/trunk/Doc/whatsnew/whatsnew25.tex Log: Write datetime.strptime() item; show use of @contextmanager in defining __context__ methods; minor edits; add two names Modified: python/trunk/Doc/whatsnew/whatsnew25.tex ============================================================================== --- python/trunk/Doc/whatsnew/whatsnew25.tex (original) +++ python/trunk/Doc/whatsnew/whatsnew25.tex Wed Apr 19 14:55:39 2006 @@ -145,7 +145,7 @@ \begin{seealso} \seepep{308}{Conditional Expressions}{PEP written by -Guido van Rossum and Raymond D. Hettinger; implemented by Thomas +Guido van~Rossum and Raymond D. Hettinger; implemented by Thomas Wouters.} \end{seealso} @@ -549,7 +549,7 @@ therefore been removed. This seems like a minor bit of language trivia, but using generators and \code{try...finally} is actually necessary in order to implement the \keyword{with} statement -described by PEP 343. We'll look at this new statement in the following +described by PEP 343. I'll look at this new statement in the following section. Another even more esoteric effect of this change: previously, the @@ -560,7 +560,7 @@ \begin{seealso} \seepep{342}{Coroutines via Enhanced Generators}{PEP written by -Guido van Rossum and Phillip J. Eby; +Guido van~Rossum and Phillip J. Eby; implemented by Phillip J. Eby. Includes examples of some fancier uses of generators as coroutines.} @@ -581,10 +581,10 @@ uses \code{try...finally} blocks to ensure that clean-up code is executed. -First, I'll discuss the statement as it will commonly be used, and -then a subsection will examine the implementation details and how to -write objects (called ``context managers'') that can be used with this -statement. +In this section, I'll discuss the statement as it will commonly be +used. In the next section, I'll examine the implementation details +and show how to write objects called ``context managers'' and +``contexts'' for use with this statement. The \keyword{with} statement is a new control-flow structure whose basic structure is: @@ -830,10 +830,29 @@ ... \end{verbatim} -There's a \function{nested(\var{mgr1}, \var{mgr2}, ...)} manager that -combines a number of context managers so you don't need to write -nested \keyword{with} statements. This example -both uses a database transaction and also acquires a thread lock: +You can also use this decorator to write the \method{__context__()} method +for a class without creating a new class for the context: + +\begin{verbatim} +class DatabaseConnection: + + @contextmanager + def __context__ (self): + cursor = self.cursor() + try: + yield cursor + except: + self.rollback() + raise + else: + self.commit() +\end{verbatim} + + +There's a \function{nested(\var{mgr1}, \var{mgr2}, ...)} manager that +combines a number of context managers so you don't need to write +nested \keyword{with} statements. This example statement does two +things, starting a database transaction and acquiring a thread lock: \begin{verbatim} lock = threading.Lock() @@ -853,8 +872,8 @@ \begin{seealso} -\seepep{343}{The ``with'' statement}{PEP written by Guido van Rossum -and Nick Coghlan; implemented by Mike Bland, Guido van Rossum, and +\seepep{343}{The ``with'' statement}{PEP written by Guido van~Rossum +and Nick Coghlan; implemented by Mike Bland, Guido van~Rossum, and Neal Norwitz. The PEP shows the code generated for a \keyword{with} statement, which can be helpful in learning how context managers work.} @@ -926,7 +945,7 @@ \begin{seealso} \seepep{352}{Required Superclass for Exceptions}{PEP written by -Brett Cannon and Guido van Rossum; implemented by Brett Cannon.} +Brett Cannon and Guido van~Rossum; implemented by Brett Cannon.} \end{seealso} @@ -1174,9 +1193,6 @@ % the cPickle module no longer accepts the deprecated None option in the % args tuple returned by __reduce__(). -% XXX datetime.datetime() now has a strptime class method which can be used to -% create datetime object using a string and format. - % XXX fileinput: opening hook used to control how files are opened. % .input() now has a mode parameter % now has a fileno() function @@ -1250,6 +1266,19 @@ \member{line_num} is not the same as the number of records read. (Contributed by Skip Montanaro and Andrew McNamara.) +\item The \class{datetime} class in the \module{datetime} +module now has a \method{strptime(\var{string}, \var{format})} +method for parsing date strings, contributed by Josh Spoerri. +It uses the same format characters as \function{time.strptime()} and +\function{time.strftime()}: + +\begin{verbatim} +from datetime import datetime + +ts = datetime.strptime('10:13:15 2006-03-07', + '%H:%M:%S %Y-%m-%d') +\end{verbatim} + \item In the \module{gc} module, the new \function{get_count()} function returns a 3-tuple containing the current collection counts for the three GC generations. This is accounting information for the garbage @@ -1943,6 +1972,7 @@ The author would like to thank the following people for offering suggestions, corrections and assistance with various drafts of this -article: Martin von~L\"owis, Mike Rovner, Thomas Wouters. +article: Phillip J. Eby, Kent Johnson, Martin von~L\"owis, Mike +Rovner, Thomas Wouters. \end{document} From python-checkins at python.org Wed Apr 19 16:50:16 2006 From: python-checkins at python.org (thomas.wouters) Date: Wed, 19 Apr 2006 16:50:16 +0200 (CEST) Subject: [Python-checkins] r45563 - python/trunk/Objects/stringobject.c Message-ID: <20060419145016.CD68D1E4004@bag.python.org> Author: thomas.wouters Date: Wed Apr 19 16:50:15 2006 New Revision: 45563 Modified: python/trunk/Objects/stringobject.c Log: Use Py_ssize_t to hold the 'width' argument to the ljust, rjust, center and zfill stringmethods, so they can create strings larger than 2Gb on 64bit systems (even win64.) The unicode versions of these methods already did this right. Modified: python/trunk/Objects/stringobject.c ============================================================================== --- python/trunk/Objects/stringobject.c (original) +++ python/trunk/Objects/stringobject.c Wed Apr 19 16:50:15 2006 @@ -2860,10 +2860,10 @@ static PyObject * string_ljust(PyStringObject *self, PyObject *args) { - int width; + Py_ssize_t width; char fillchar = ' '; - if (!PyArg_ParseTuple(args, "i|c:ljust", &width, &fillchar)) + if (!PyArg_ParseTuple(args, "n|c:ljust", &width, &fillchar)) return NULL; if (PyString_GET_SIZE(self) >= width && PyString_CheckExact(self)) { @@ -2884,10 +2884,10 @@ static PyObject * string_rjust(PyStringObject *self, PyObject *args) { - int width; + Py_ssize_t width; char fillchar = ' '; - if (!PyArg_ParseTuple(args, "i|c:rjust", &width, &fillchar)) + if (!PyArg_ParseTuple(args, "n|c:rjust", &width, &fillchar)) return NULL; if (PyString_GET_SIZE(self) >= width && PyString_CheckExact(self)) { @@ -2909,10 +2909,10 @@ string_center(PyStringObject *self, PyObject *args) { Py_ssize_t marg, left; - long width; + Py_ssize_t width; char fillchar = ' '; - if (!PyArg_ParseTuple(args, "l|c:center", &width, &fillchar)) + if (!PyArg_ParseTuple(args, "n|c:center", &width, &fillchar)) return NULL; if (PyString_GET_SIZE(self) >= width && PyString_CheckExact(self)) { @@ -2938,9 +2938,9 @@ Py_ssize_t fill; PyObject *s; char *p; + Py_ssize_t width; - long width; - if (!PyArg_ParseTuple(args, "l:zfill", &width)) + if (!PyArg_ParseTuple(args, "n:zfill", &width)) return NULL; if (PyString_GET_SIZE(self) >= width) { From buildbot at python.org Wed Apr 19 16:58:07 2006 From: buildbot at python.org (buildbot at python.org) Date: Wed, 19 Apr 2006 14:58:07 +0000 Subject: [Python-checkins] buildbot failure in x86 cygwin trunk Message-ID: <20060419145807.3FA721E4004@bag.python.org> The Buildbot has detected a new failure of x86 cygwin trunk. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520cygwin%2520trunk/builds/224 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: thomas.wouters BUILD FAILED: failed svn sincerely, -The Buildbot From python-checkins at python.org Wed Apr 19 17:09:46 2006 From: python-checkins at python.org (thomas.wouters) Date: Wed, 19 Apr 2006 17:09:46 +0200 (CEST) Subject: [Python-checkins] r45564 - python/trunk/Python/ceval.c Message-ID: <20060419150946.4CBBC1E4004@bag.python.org> Author: thomas.wouters Date: Wed Apr 19 17:09:44 2006 New Revision: 45564 Modified: python/trunk/Python/ceval.c Log: Teach Python/ceval.c's inlining of 'str += str' about Py_ssize_t sizes; this was having funny effects when called on >2Gb strings ;P Modified: python/trunk/Python/ceval.c ============================================================================== --- python/trunk/Python/ceval.c (original) +++ python/trunk/Python/ceval.c Wed Apr 19 17:09:44 2006 @@ -4238,8 +4238,8 @@ /* Now we own the last reference to 'v', so we can resize it * in-place. */ - int v_len = PyString_GET_SIZE(v); - int w_len = PyString_GET_SIZE(w); + Py_ssize_t v_len = PyString_GET_SIZE(v); + Py_ssize_t w_len = PyString_GET_SIZE(w); if (_PyString_Resize(&v, v_len + w_len) != 0) { /* XXX if _PyString_Resize() fails, 'v' has been * deallocated so it cannot be put back into 'variable'. From python-checkins at python.org Wed Apr 19 17:22:01 2006 From: python-checkins at python.org (marc-andre.lemburg) Date: Wed, 19 Apr 2006 17:22:01 +0200 (CEST) Subject: [Python-checkins] r45565 - in external/pybench-1.3: Arithmetic.py Calls.py CommandLine.py Constructs.py Dict.py Exceptions.py Imports.py Instances.py LICENSE Lists.py Lookups.py Numbers.py README Setup.py Strings.py Tuples.py Unicode.py package package/__init__.py package/submodule.py pybench.py Message-ID: <20060419152201.86B5B1E4004@bag.python.org> Author: marc-andre.lemburg Date: Wed Apr 19 17:21:58 2006 New Revision: 45565 Added: external/pybench-1.3/ external/pybench-1.3/Arithmetic.py (contents, props changed) external/pybench-1.3/Calls.py (contents, props changed) external/pybench-1.3/CommandLine.py (contents, props changed) external/pybench-1.3/Constructs.py (contents, props changed) external/pybench-1.3/Dict.py (contents, props changed) external/pybench-1.3/Exceptions.py (contents, props changed) external/pybench-1.3/Imports.py (contents, props changed) external/pybench-1.3/Instances.py (contents, props changed) external/pybench-1.3/LICENSE (contents, props changed) external/pybench-1.3/Lists.py (contents, props changed) external/pybench-1.3/Lookups.py (contents, props changed) external/pybench-1.3/Numbers.py (contents, props changed) external/pybench-1.3/README (contents, props changed) external/pybench-1.3/Setup.py (contents, props changed) external/pybench-1.3/Strings.py (contents, props changed) external/pybench-1.3/Tuples.py (contents, props changed) external/pybench-1.3/Unicode.py (contents, props changed) external/pybench-1.3/package/ external/pybench-1.3/package/__init__.py (contents, props changed) external/pybench-1.3/package/submodule.py (contents, props changed) external/pybench-1.3/pybench.py (contents, props changed) Log: Import pybench 1.3. Added: external/pybench-1.3/Arithmetic.py ============================================================================== --- (empty file) +++ external/pybench-1.3/Arithmetic.py Wed Apr 19 17:21:58 2006 @@ -0,0 +1,778 @@ +from pybench import Test + +class SimpleIntegerArithmetic(Test): + + version = 0.3 + operations = 5 * (3 + 5 + 5 + 3 + 3 + 3) + rounds = 120000 + + def test(self): + + for i in xrange(self.rounds): + + a = 2 + b = 3 + c = 3 + + c = a + b + c = b + c + c = c + a + c = a + b + c = b + c + + c = c - a + c = a - b + c = b - c + c = c - a + c = b - c + + c = a / b + c = b / a + c = c / b + + c = a * b + c = b * a + c = c * b + + c = a / b + c = b / a + c = c / b + + a = 2 + b = 3 + c = 3 + + c = a + b + c = b + c + c = c + a + c = a + b + c = b + c + + c = c - a + c = a - b + c = b - c + c = c - a + c = b - c + + c = a / b + c = b / a + c = c / b + + c = a * b + c = b * a + c = c * b + + c = a / b + c = b / a + c = c / b + + a = 2 + b = 3 + c = 3 + + c = a + b + c = b + c + c = c + a + c = a + b + c = b + c + + c = c - a + c = a - b + c = b - c + c = c - a + c = b - c + + c = a / b + c = b / a + c = c / b + + c = a * b + c = b * a + c = c * b + + c = a / b + c = b / a + c = c / b + + a = 2 + b = 3 + c = 3 + + c = a + b + c = b + c + c = c + a + c = a + b + c = b + c + + c = c - a + c = a - b + c = b - c + c = c - a + c = b - c + + c = a / b + c = b / a + c = c / b + + c = a * b + c = b * a + c = c * b + + c = a / b + c = b / a + c = c / b + + a = 2 + b = 3 + c = 3 + + c = a + b + c = b + c + c = c + a + c = a + b + c = b + c + + c = c - a + c = a - b + c = b - c + c = c - a + c = b - c + + c = a / b + c = b / a + c = c / b + + c = a * b + c = b * a + c = c * b + + c = a / b + c = b / a + c = c / b + + def calibrate(self): + + for i in xrange(self.rounds): + pass + +class SimpleFloatArithmetic(Test): + + version = 0.3 + operations = 5 * (3 + 5 + 5 + 3 + 3 + 3) + rounds = 100000 + + def test(self): + + for i in xrange(self.rounds): + + a = 2.1 + b = 3.3332 + c = 3.14159 + + c = a + b + c = b + c + c = c + a + c = a + b + c = b + c + + c = c - a + c = a - b + c = b - c + c = c - a + c = b - c + + c = a / b + c = b / a + c = c / b + + c = a * b + c = b * a + c = c * b + + c = a / b + c = b / a + c = c / b + + a = 2.1 + b = 3.3332 + c = 3.14159 + + c = a + b + c = b + c + c = c + a + c = a + b + c = b + c + + c = c - a + c = a - b + c = b - c + c = c - a + c = b - c + + c = a / b + c = b / a + c = c / b + + c = a * b + c = b * a + c = c * b + + c = a / b + c = b / a + c = c / b + + a = 2.1 + b = 3.3332 + c = 3.14159 + + c = a + b + c = b + c + c = c + a + c = a + b + c = b + c + + c = c - a + c = a - b + c = b - c + c = c - a + c = b - c + + c = a / b + c = b / a + c = c / b + + c = a * b + c = b * a + c = c * b + + c = a / b + c = b / a + c = c / b + + a = 2.1 + b = 3.3332 + c = 3.14159 + + c = a + b + c = b + c + c = c + a + c = a + b + c = b + c + + c = c - a + c = a - b + c = b - c + c = c - a + c = b - c + + c = a / b + c = b / a + c = c / b + + c = a * b + c = b * a + c = c * b + + c = a / b + c = b / a + c = c / b + + a = 2.1 + b = 3.3332 + c = 3.14159 + + c = a + b + c = b + c + c = c + a + c = a + b + c = b + c + + c = c - a + c = a - b + c = b - c + c = c - a + c = b - c + + c = a / b + c = b / a + c = c / b + + c = a * b + c = b * a + c = c * b + + c = a / b + c = b / a + c = c / b + + def calibrate(self): + + for i in xrange(self.rounds): + pass + +class SimpleIntFloatArithmetic(Test): + + version = 0.3 + operations = 5 * (3 + 5 + 5 + 3 + 3 + 3) + rounds = 120000 + + def test(self): + + for i in xrange(self.rounds): + + a = 2 + b = 3 + c = 3.14159 + + c = a + b + c = b + c + c = c + a + c = a + b + c = b + c + + c = c - a + c = a - b + c = b - c + c = c - a + c = b - c + + c = a / b + c = b / a + c = c / b + + c = a * b + c = b * a + c = c * b + + c = a / b + c = b / a + c = c / b + + a = 2 + b = 3 + c = 3.14159 + + c = a + b + c = b + c + c = c + a + c = a + b + c = b + c + + c = c - a + c = a - b + c = b - c + c = c - a + c = b - c + + c = a / b + c = b / a + c = c / b + + c = a * b + c = b * a + c = c * b + + c = a / b + c = b / a + c = c / b + + a = 2 + b = 3 + c = 3.14159 + + c = a + b + c = b + c + c = c + a + c = a + b + c = b + c + + c = c - a + c = a - b + c = b - c + c = c - a + c = b - c + + c = a / b + c = b / a + c = c / b + + c = a * b + c = b * a + c = c * b + + c = a / b + c = b / a + c = c / b + + a = 2 + b = 3 + c = 3.14159 + + c = a + b + c = b + c + c = c + a + c = a + b + c = b + c + + c = c - a + c = a - b + c = b - c + c = c - a + c = b - c + + c = a / b + c = b / a + c = c / b + + c = a * b + c = b * a + c = c * b + + c = a / b + c = b / a + c = c / b + + a = 2 + b = 3 + c = 3.14159 + + c = a + b + c = b + c + c = c + a + c = a + b + c = b + c + + c = c - a + c = a - b + c = b - c + c = c - a + c = b - c + + c = a / b + c = b / a + c = c / b + + c = a * b + c = b * a + c = c * b + + c = a / b + c = b / a + c = c / b + + def calibrate(self): + + for i in xrange(self.rounds): + pass + + +class SimpleLongArithmetic(Test): + + version = 0.3 + operations = 5 * (3 + 5 + 5 + 3 + 3 + 3) + rounds = 30000 + + def test(self): + + for i in xrange(self.rounds): + + a = 2220001L + b = 100001L + c = 30005L + + c = a + b + c = b + c + c = c + a + c = a + b + c = b + c + + c = c - a + c = a - b + c = b - c + c = c - a + c = b - c + + c = a / b + c = b / a + c = c / b + + c = a * b + c = b * a + c = c * b + + c = a / b + c = b / a + c = c / b + + a = 2220001L + b = 100001L + c = 30005L + + c = a + b + c = b + c + c = c + a + c = a + b + c = b + c + + c = c - a + c = a - b + c = b - c + c = c - a + c = b - c + + c = a / b + c = b / a + c = c / b + + c = a * b + c = b * a + c = c * b + + c = a / b + c = b / a + c = c / b + + a = 2220001L + b = 100001L + c = 30005L + + c = a + b + c = b + c + c = c + a + c = a + b + c = b + c + + c = c - a + c = a - b + c = b - c + c = c - a + c = b - c + + c = a / b + c = b / a + c = c / b + + c = a * b + c = b * a + c = c * b + + c = a / b + c = b / a + c = c / b + + a = 2220001L + b = 100001L + c = 30005L + + c = a + b + c = b + c + c = c + a + c = a + b + c = b + c + + c = c - a + c = a - b + c = b - c + c = c - a + c = b - c + + c = a / b + c = b / a + c = c / b + + c = a * b + c = b * a + c = c * b + + c = a / b + c = b / a + c = c / b + + a = 2220001L + b = 100001L + c = 30005L + + c = a + b + c = b + c + c = c + a + c = a + b + c = b + c + + c = c - a + c = a - b + c = b - c + c = c - a + c = b - c + + c = a / b + c = b / a + c = c / b + + c = a * b + c = b * a + c = c * b + + c = a / b + c = b / a + c = c / b + + def calibrate(self): + + for i in xrange(self.rounds): + pass + +class SimpleComplexArithmetic(Test): + + version = 0.3 + operations = 5 * (3 + 5 + 5 + 3 + 3 + 3) + rounds = 40000 + + def test(self): + + for i in xrange(self.rounds): + + a = 2 + 3j + b = 2.5 + 4.5j + c = 1.2 + 6.2j + + c = a + b + c = b + c + c = c + a + c = a + b + c = b + c + + c = c - a + c = a - b + c = b - c + c = c - a + c = b - c + + c = a / b + c = b / a + c = c / b + + c = a * b + c = b * a + c = c * b + + c = a / b + c = b / a + c = c / b + + a = 2 + 3j + b = 2.5 + 4.5j + c = 1.2 + 6.2j + + c = a + b + c = b + c + c = c + a + c = a + b + c = b + c + + c = c - a + c = a - b + c = b - c + c = c - a + c = b - c + + c = a / b + c = b / a + c = c / b + + c = a * b + c = b * a + c = c * b + + c = a / b + c = b / a + c = c / b + + a = 2 + 3j + b = 2.5 + 4.5j + c = 1.2 + 6.2j + + c = a + b + c = b + c + c = c + a + c = a + b + c = b + c + + c = c - a + c = a - b + c = b - c + c = c - a + c = b - c + + c = a / b + c = b / a + c = c / b + + c = a * b + c = b * a + c = c * b + + c = a / b + c = b / a + c = c / b + + a = 2 + 3j + b = 2.5 + 4.5j + c = 1.2 + 6.2j + + c = a + b + c = b + c + c = c + a + c = a + b + c = b + c + + c = c - a + c = a - b + c = b - c + c = c - a + c = b - c + + c = a / b + c = b / a + c = c / b + + c = a * b + c = b * a + c = c * b + + c = a / b + c = b / a + c = c / b + + a = 2 + 3j + b = 2.5 + 4.5j + c = 1.2 + 6.2j + + c = a + b + c = b + c + c = c + a + c = a + b + c = b + c + + c = c - a + c = a - b + c = b - c + c = c - a + c = b - c + + c = a / b + c = b / a + c = c / b + + c = a * b + c = b * a + c = c * b + + c = a / b + c = b / a + c = c / b + + def calibrate(self): + + for i in xrange(self.rounds): + pass + Added: external/pybench-1.3/Calls.py ============================================================================== --- (empty file) +++ external/pybench-1.3/Calls.py Wed Apr 19 17:21:58 2006 @@ -0,0 +1,410 @@ +from pybench import Test + +class PythonFunctionCalls(Test): + + version = 0.3 + operations = 5*(1+4+4+2) + rounds = 60000 + + def test(self): + + global f,f1,g,h + + # define functions + def f(): + pass + + def f1(x): + pass + + def g(a,b,c): + return a,b,c + + def h(a,b,c,d=1,e=2,f=3): + return d,e,f + + # do calls + for i in xrange(self.rounds): + + f() + f1(i) + f1(i) + f1(i) + f1(i) + g(i,i,i) + g(i,i,i) + g(i,i,i) + g(i,i,i) + h(i,i,3,i,i) + h(i,i,i,2,i,3) + + f() + f1(i) + f1(i) + f1(i) + f1(i) + g(i,i,i) + g(i,i,i) + g(i,i,i) + g(i,i,i) + h(i,i,3,i,i) + h(i,i,i,2,i,3) + + f() + f1(i) + f1(i) + f1(i) + f1(i) + g(i,i,i) + g(i,i,i) + g(i,i,i) + g(i,i,i) + h(i,i,3,i,i) + h(i,i,i,2,i,3) + + f() + f1(i) + f1(i) + f1(i) + f1(i) + g(i,i,i) + g(i,i,i) + g(i,i,i) + g(i,i,i) + h(i,i,3,i,i) + h(i,i,i,2,i,3) + + f() + f1(i) + f1(i) + f1(i) + f1(i) + g(i,i,i) + g(i,i,i) + g(i,i,i) + g(i,i,i) + h(i,i,3,i,i) + h(i,i,i,2,i,3) + + def calibrate(self): + + global f,f1,g,h + + # define functions + def f(): + pass + + def f1(x): + pass + + def g(a,b,c): + return a,b,c + + def h(a,b,c,d=1,e=2,f=3): + return d,e,f + + # do calls + for i in xrange(self.rounds): + pass + +### + +class BuiltinFunctionCalls(Test): + + version = 0.4 + operations = 5*(2+5+5+5) + rounds = 30000 + + def test(self): + + # localize functions + f0 = globals + f1 = hash + f2 = cmp + f3 = range + + # do calls + for i in xrange(self.rounds): + + f0() + f0() + f1(i) + f1(i) + f1(i) + f1(i) + f1(i) + f2(1,2) + f2(1,2) + f2(1,2) + f2(1,2) + f2(1,2) + f3(1,3,2) + f3(1,3,2) + f3(1,3,2) + f3(1,3,2) + f3(1,3,2) + + f0() + f0() + f1(i) + f1(i) + f1(i) + f1(i) + f1(i) + f2(1,2) + f2(1,2) + f2(1,2) + f2(1,2) + f2(1,2) + f3(1,3,2) + f3(1,3,2) + f3(1,3,2) + f3(1,3,2) + f3(1,3,2) + + f0() + f0() + f1(i) + f1(i) + f1(i) + f1(i) + f1(i) + f2(1,2) + f2(1,2) + f2(1,2) + f2(1,2) + f2(1,2) + f3(1,3,2) + f3(1,3,2) + f3(1,3,2) + f3(1,3,2) + f3(1,3,2) + + f0() + f0() + f1(i) + f1(i) + f1(i) + f1(i) + f1(i) + f2(1,2) + f2(1,2) + f2(1,2) + f2(1,2) + f2(1,2) + f3(1,3,2) + f3(1,3,2) + f3(1,3,2) + f3(1,3,2) + f3(1,3,2) + + f0() + f0() + f1(i) + f1(i) + f1(i) + f1(i) + f1(i) + f2(1,2) + f2(1,2) + f2(1,2) + f2(1,2) + f2(1,2) + f3(1,3,2) + f3(1,3,2) + f3(1,3,2) + f3(1,3,2) + f3(1,3,2) + + def calibrate(self): + + # localize functions + f0 = dir + f1 = hash + f2 = range + f3 = range + + # do calls + for i in xrange(self.rounds): + pass + +### + +class PythonMethodCalls(Test): + + version = 0.3 + operations = 5*(6 + 5 + 4) + rounds = 20000 + + def test(self): + + class c: + + x = 2 + s = 'string' + + def f(self): + + return self.x + + def j(self,a,b): + + self.y = a + self.t = b + return self.y + + def k(self,a,b,c=3): + + self.y = a + self.s = b + self.t = c + + o = c() + + for i in xrange(self.rounds): + + o.f() + o.f() + o.f() + o.f() + o.f() + o.f() + o.j(i,i) + o.j(i,i) + o.j(i,2) + o.j(i,2) + o.j(2,2) + o.k(i,i) + o.k(i,2) + o.k(i,2,3) + o.k(i,i,c=4) + + o.f() + o.f() + o.f() + o.f() + o.f() + o.f() + o.j(i,i) + o.j(i,i) + o.j(i,2) + o.j(i,2) + o.j(2,2) + o.k(i,i) + o.k(i,2) + o.k(i,2,3) + o.k(i,i,c=4) + + o.f() + o.f() + o.f() + o.f() + o.f() + o.f() + o.j(i,i) + o.j(i,i) + o.j(i,2) + o.j(i,2) + o.j(2,2) + o.k(i,i) + o.k(i,2) + o.k(i,2,3) + o.k(i,i,c=4) + + o.f() + o.f() + o.f() + o.f() + o.f() + o.f() + o.j(i,i) + o.j(i,i) + o.j(i,2) + o.j(i,2) + o.j(2,2) + o.k(i,i) + o.k(i,2) + o.k(i,2,3) + o.k(i,i,c=4) + + o.f() + o.f() + o.f() + o.f() + o.f() + o.f() + o.j(i,i) + o.j(i,i) + o.j(i,2) + o.j(i,2) + o.j(2,2) + o.k(i,i) + o.k(i,2) + o.k(i,2,3) + o.k(i,i,c=4) + + def calibrate(self): + + class c: + + x = 2 + s = 'string' + + def f(self): + + return self.x + + def j(self,a,b): + + self.y = a + self.t = b + + def k(self,a,b,c=3): + + self.y = a + self.s = b + self.t = c + + o = c + + for i in xrange(self.rounds): + pass + +### + +class Recursion(Test): + + version = 0.3 + operations = 5 + rounds = 50000 + + def test(self): + + global f + + def f(x): + + if x > 1: + return f(x-1) + return 1 + + for i in xrange(self.rounds): + f(10) + f(10) + f(10) + f(10) + f(10) + + def calibrate(self): + + global f + + def f(x): + + if x > 0: + return f(x-1) + return 1 + + for i in xrange(self.rounds): + pass + Added: external/pybench-1.3/CommandLine.py ============================================================================== --- (empty file) +++ external/pybench-1.3/CommandLine.py Wed Apr 19 17:21:58 2006 @@ -0,0 +1,634 @@ +""" CommandLine - Get and parse command line options + + NOTE: This still is very much work in progress !!! + + Different version are likely to be incompatible. + + TODO: + + * Incorporate the changes made by (see Inbox) + * Add number range option using srange() + +""" + +__copyright__ = """\ +Copyright (c), 1997-2006, Marc-Andre Lemburg (mal at lemburg.com) +Copyright (c), 2000-2006, eGenix.com Software GmbH (info at egenix.com) +See the documentation for further information on copyrights, +or contact the author. All Rights Reserved. +""" + +__version__ = '1.2' + +import sys, getopt, string, glob, os, re, exceptions, traceback + +### Helpers + +def _getopt_flags(options): + + """ Convert the option list to a getopt flag string and long opt + list + + """ + s = [] + l = [] + for o in options: + if o.prefix == '-': + # short option + s.append(o.name) + if o.takes_argument: + s.append(':') + else: + # long option + if o.takes_argument: + l.append(o.name+'=') + else: + l.append(o.name) + return string.join(s,''),l + +def invisible_input(prompt='>>> '): + + """ Get raw input from a terminal without echoing the characters to + the terminal, e.g. for password queries. + + """ + import getpass + entry = getpass.getpass(prompt) + if entry is None: + raise KeyboardInterrupt + return entry + +def fileopen(name, mode='wb', encoding=None): + + """ Open a file using mode. + + Default mode is 'wb' meaning to open the file for writing in + binary mode. If encoding is given, I/O to and from the file is + transparently encoded using the given encoding. + + Files opened for writing are chmod()ed to 0600. + + """ + if name == 'stdout': + return sys.stdout + elif name == 'stderr': + return sys.stderr + elif name == 'stdin': + return sys.stdin + else: + if encoding is not None: + import codecs + f = codecs.open(name, mode, encoding) + else: + f = open(name, mode) + if 'w' in mode: + os.chmod(name, 0600) + return f + +def option_dict(options): + + """ Return a dictionary mapping option names to Option instances. + """ + d = {} + for option in options: + d[option.name] = option + return d + +# Alias +getpasswd = invisible_input + +_integerRE = re.compile('\s*(-?\d+)\s*$') +_integerRangeRE = re.compile('\s*(-?\d+)\s*-\s*(-?\d+)\s*$') + +def srange(s, + + split=string.split,integer=_integerRE, + integerRange=_integerRangeRE): + + """ Converts a textual representation of integer numbers and ranges + to a Python list. + + Supported formats: 2,3,4,2-10,-1 - -3, 5 - -2 + + Values are appended to the created list in the order specified + in the string. + + """ + l = [] + append = l.append + for entry in split(s,','): + m = integer.match(entry) + if m: + append(int(m.groups()[0])) + continue + m = integerRange.match(entry) + if m: + start,end = map(int,m.groups()) + l[len(l):] = range(start,end+1) + return l + +def abspath(path, + + expandvars=os.path.expandvars,expanduser=os.path.expanduser, + join=os.path.join,getcwd=os.getcwd): + + """ Return the corresponding absolute path for path. + + path is expanded in the usual shell ways before + joining it with the current working directory. + + """ + try: + path = expandvars(path) + except AttributeError: + pass + try: + path = expanduser(path) + except AttributeError: + pass + return join(getcwd(), path) + +### Option classes + +class Option: + + """ Option base class. Takes no argument. + + """ + default = None + helptext = '' + prefix = '-' + takes_argument = 0 + has_default = 0 + tab = 15 + + def __init__(self,name,help=None): + + if not name[:1] == '-': + raise TypeError,'option names must start with "-"' + if name[1:2] == '-': + self.prefix = '--' + self.name = name[2:] + else: + self.name = name[1:] + if help: + self.help = help + + def __str__(self): + + o = self + name = o.prefix + o.name + if o.takes_argument: + name = name + ' arg' + if len(name) > self.tab: + name = name + '\n' + ' ' * (self.tab + 1 + len(o.prefix)) + else: + name = '%-*s ' % (self.tab, name) + description = o.help + if o.has_default: + description = description + ' (%s)' % o.default + return '%s %s' % (name, description) + +class ArgumentOption(Option): + + """ Option that takes an argument. + + An optional default argument can be given. + + """ + def __init__(self,name,help=None,default=None): + + # Basemethod + Option.__init__(self,name,help) + + if default is not None: + self.default = default + self.has_default = 1 + self.takes_argument = 1 + +class SwitchOption(Option): + + """ Options that can be on or off. Has an optional default value. + + """ + def __init__(self,name,help=None,default=None): + + # Basemethod + Option.__init__(self,name,help) + + if default is not None: + self.default = default + self.has_default = 1 + +### Application baseclass + +class Application: + + """ Command line application interface with builtin argument + parsing. + + """ + # Options the program accepts (Option instances) + options = [] + + # Standard settings; these are appended to options in __init__ + preset_options = [SwitchOption('-v', + 'generate verbose output'), + SwitchOption('-h', + 'show this help text'), + SwitchOption('--help', + 'show this help text'), + SwitchOption('--debug', + 'enable debugging'), + SwitchOption('--copyright', + 'show copyright'), + SwitchOption('--examples', + 'show examples of usage')] + + # The help layout looks like this: + # [header] - defaults to '' + # + # [synopsis] - formatted as ' %s' % self.synopsis + # + # options: + # [options] - formatted from self.options + # + # [version] - formatted as 'Version:\n %s' % self.version, if given + # + # [about] - defaults to '' + # + # Note: all fields that do not behave as template are formatted + # using the instances dictionary as substitution namespace, + # e.g. %(name)s will be replaced by the applications name. + # + + # Header (default to program name) + header = '' + + # Name (defaults to program name) + name = '' + + # Synopsis (%(name)s is replaced by the program name) + synopsis = '%(name)s [option] files...' + + # Version (optional) + version = '' + + # General information printed after the possible options (optional) + about = '' + + # Examples of usage to show when the --examples option is given (optional) + examples = '' + + # Copyright to show + copyright = __copyright__ + + # Apply file globbing ? + globbing = 1 + + # Generate debug output ? + debug = 0 + + # Generate verbose output ? + verbose = 0 + + # Internal errors to catch + InternalError = exceptions.Exception + + # Instance variables: + values = None # Dictionary of passed options (or default values) + # indexed by the options name, e.g. '-h' + files = None # List of passed filenames + optionlist = None # List of passed options + + def __init__(self,argv=None): + + # Setup application specs + if argv is None: + argv = sys.argv + self.filename = os.path.split(argv[0])[1] + if not self.name: + self.name = os.path.split(self.filename)[1] + else: + self.name = self.name + if not self.header: + self.header = self.name + else: + self.header = self.header + + # Init .arguments list + self.arguments = argv[1:] + + # Setup Option mapping + self.option_map = option_dict(self.options) + + # Append preset options + for option in self.preset_options: + if not self.option_map.has_key(option.name): + self.add_option(option) + + # Init .files list + self.files = [] + + # Start Application + try: + # Process startup + rc = self.startup() + if rc is not None: + raise SystemExit,rc + + # Parse command line + rc = self.parse() + if rc is not None: + raise SystemExit,rc + + # Start application + rc = self.main() + if rc is None: + rc = 0 + + except SystemExit,rc: + pass + + except KeyboardInterrupt: + print + print '* User Break' + print + rc = 1 + + except self.InternalError: + print + print '* Internal Error' + if self.debug: + print + traceback.print_exc(20, sys.stdout) + elif self.verbose: + print ' %s: %s' % sys.exc_info()[:2] + print + rc = 1 + + raise SystemExit,rc + + def add_option(self, option): + + """ Add a new Option instance to the Application dynamically. + + Note that this has to be done *before* .parse() is being + executed. + + """ + self.options.append(option) + self.option_map[option.name] = option + + def startup(self): + + """ Set user defined instance variables. + + If this method returns anything other than None, the + process is terminated with the return value as exit code. + + """ + return None + + def exit(self, rc=0): + + """ Exit the program. + + rc is used as exit code and passed back to the calling + program. It defaults to 0 which usually means: OK. + + """ + raise SystemExit, rc + + def parse(self): + + """ Parse the command line and fill in self.values and self.files. + + After having parsed the options, the remaining command line + arguments are interpreted as files and passed to .handle_files() + for processing. + + As final step the option handlers are called in the order + of the options given on the command line. + + """ + # Parse arguments + self.values = values = {} + for o in self.options: + if o.has_default: + values[o.prefix+o.name] = o.default + else: + values[o.prefix+o.name] = 0 + flags,lflags = _getopt_flags(self.options) + try: + optlist,files = getopt.getopt(self.arguments,flags,lflags) + if self.globbing: + l = [] + for f in files: + gf = glob.glob(f) + if not gf: + l.append(f) + else: + l[len(l):] = gf + files = l + self.optionlist = optlist + self.files = files + self.files + except getopt.error,why: + self.help(why) + sys.exit(1) + + # Call file handler + rc = self.handle_files(self.files) + if rc is not None: + sys.exit(rc) + + # Call option handlers + for optionname, value in optlist: + + # Try to convert value to integer + try: + value = string.atoi(value) + except ValueError: + pass + + # Find handler and call it (or count the number of option + # instances on the command line) + handlername = 'handle' + string.replace(optionname, '-', '_') + try: + handler = getattr(self, handlername) + except AttributeError: + if value == '': + # count the number of occurances + if values.has_key(optionname): + values[optionname] = values[optionname] + 1 + else: + values[optionname] = 1 + else: + values[optionname] = value + else: + rc = handler(value) + if rc is not None: + raise SystemExit, rc + + # Apply final file check (for backward compatibility) + rc = self.check_files(self.files) + if rc is not None: + sys.exit(rc) + + def check_files(self,filelist): + + """ Apply some user defined checks on the files given in filelist. + + This may modify filelist in place. A typical application + is checking that at least n files are given. + + If this method returns anything other than None, the + process is terminated with the return value as exit code. + + """ + return None + + def help(self,note=''): + + self.print_header() + if self.synopsis: + print 'Synopsis:' + # To remain backward compatible: + try: + synopsis = self.synopsis % self.name + except (NameError, KeyError, TypeError): + synopsis = self.synopsis % self.__dict__ + print ' ' + synopsis + print + self.print_options() + if self.version: + print 'Version:' + print ' %s' % self.version + print + if self.about: + print string.strip(self.about % self.__dict__) + print + if note: + print '-'*72 + print 'Note:',note + print + + def notice(self,note): + + print '-'*72 + print 'Note:',note + print '-'*72 + print + + def print_header(self): + + print '-'*72 + print self.header % self.__dict__ + print '-'*72 + print + + def print_options(self): + + options = self.options + print 'Options and default settings:' + if not options: + print ' None' + return + long = filter(lambda x: x.prefix == '--', options) + short = filter(lambda x: x.prefix == '-', options) + items = short + long + for o in options: + print ' ',o + print + + # + # Example handlers: + # + # If a handler returns anything other than None, processing stops + # and the return value is passed to sys.exit() as argument. + # + + # File handler + def handle_files(self,files): + + """ This may process the files list in place. + """ + return None + + # Short option handler + def handle_h(self,arg): + + self.help() + return 0 + + def handle_v(self, value): + + """ Turn on verbose output. + """ + self.verbose = 1 + + # Handlers for long options have two underscores in their name + def handle__help(self,arg): + + self.help() + return 0 + + def handle__debug(self,arg): + + self.debug = 1 + # We don't want to catch internal errors: + self.InternalError = None + + def handle__copyright(self,arg): + + self.print_header() + print string.strip(self.copyright % self.__dict__) + print + return 0 + + def handle__examples(self,arg): + + self.print_header() + if self.examples: + print 'Examples:' + print + print string.strip(self.examples % self.__dict__) + print + else: + print 'No examples available.' + print + return 0 + + def main(self): + + """ Override this method as program entry point. + + The return value is passed to sys.exit() as argument. If + it is None, 0 is assumed (meaning OK). Unhandled + exceptions are reported with exit status code 1 (see + __init__ for further details). + + """ + return None + +# Alias +CommandLine = Application + +def _test(): + + class MyApplication(Application): + header = 'Test Application' + version = __version__ + options = [Option('-v','verbose')] + + def handle_v(self,arg): + print 'VERBOSE, Yeah !' + + cmd = MyApplication() + if not cmd.values['-h']: + cmd.help() + print 'files:',cmd.files + print 'Bye...' + +if __name__ == '__main__': + _test() Added: external/pybench-1.3/Constructs.py ============================================================================== --- (empty file) +++ external/pybench-1.3/Constructs.py Wed Apr 19 17:21:58 2006 @@ -0,0 +1,565 @@ +from pybench import Test + +class IfThenElse(Test): + + version = 0.31 + operations = 30*3 # hard to say... + rounds = 150000 + + def test(self): + + a,b,c = 1,2,3 + for i in xrange(self.rounds): + + if a == 1: + if b == 2: + if c != 3: + c = 3 + b = 3 + else: + c = 2 + elif b == 3: + b = 2 + a = 2 + elif a == 2: + a = 3 + else: + a = 1 + + if a == 1: + if b == 2: + if c != 3: + c = 3 + b = 3 + else: + c = 2 + elif b == 3: + b = 2 + a = 2 + elif a == 2: + a = 3 + else: + a = 1 + + if a == 1: + if b == 2: + if c != 3: + c = 3 + b = 3 + else: + c = 2 + elif b == 3: + b = 2 + a = 2 + elif a == 2: + a = 3 + else: + a = 1 + + if a == 1: + if b == 2: + if c != 3: + c = 3 + b = 3 + else: + c = 2 + elif b == 3: + b = 2 + a = 2 + elif a == 2: + a = 3 + else: + a = 1 + + if a == 1: + if b == 2: + if c != 3: + c = 3 + b = 3 + else: + c = 2 + elif b == 3: + b = 2 + a = 2 + elif a == 2: + a = 3 + else: + a = 1 + + if a == 1: + if b == 2: + if c != 3: + c = 3 + b = 3 + else: + c = 2 + elif b == 3: + b = 2 + a = 2 + elif a == 2: + a = 3 + else: + a = 1 + + if a == 1: + if b == 2: + if c != 3: + c = 3 + b = 3 + else: + c = 2 + elif b == 3: + b = 2 + a = 2 + elif a == 2: + a = 3 + else: + a = 1 + + if a == 1: + if b == 2: + if c != 3: + c = 3 + b = 3 + else: + c = 2 + elif b == 3: + b = 2 + a = 2 + elif a == 2: + a = 3 + else: + a = 1 + + if a == 1: + if b == 2: + if c != 3: + c = 3 + b = 3 + else: + c = 2 + elif b == 3: + b = 2 + a = 2 + elif a == 2: + a = 3 + else: + a = 1 + + if a == 1: + if b == 2: + if c != 3: + c = 3 + b = 3 + else: + c = 2 + elif b == 3: + b = 2 + a = 2 + elif a == 2: + a = 3 + else: + a = 1 + + if a == 1: + if b == 2: + if c != 3: + c = 3 + b = 3 + else: + c = 2 + elif b == 3: + b = 2 + a = 2 + elif a == 2: + a = 3 + else: + a = 1 + + if a == 1: + if b == 2: + if c != 3: + c = 3 + b = 3 + else: + c = 2 + elif b == 3: + b = 2 + a = 2 + elif a == 2: + a = 3 + else: + a = 1 + + if a == 1: + if b == 2: + if c != 3: + c = 3 + b = 3 + else: + c = 2 + elif b == 3: + b = 2 + a = 2 + elif a == 2: + a = 3 + else: + a = 1 + + if a == 1: + if b == 2: + if c != 3: + c = 3 + b = 3 + else: + c = 2 + elif b == 3: + b = 2 + a = 2 + elif a == 2: + a = 3 + else: + a = 1 + + if a == 1: + if b == 2: + if c != 3: + c = 3 + b = 3 + else: + c = 2 + elif b == 3: + b = 2 + a = 2 + elif a == 2: + a = 3 + else: + a = 1 + + if a == 1: + if b == 2: + if c != 3: + c = 3 + b = 3 + else: + c = 2 + elif b == 3: + b = 2 + a = 2 + elif a == 2: + a = 3 + else: + a = 1 + + if a == 1: + if b == 2: + if c != 3: + c = 3 + b = 3 + else: + c = 2 + elif b == 3: + b = 2 + a = 2 + elif a == 2: + a = 3 + else: + a = 1 + + if a == 1: + if b == 2: + if c != 3: + c = 3 + b = 3 + else: + c = 2 + elif b == 3: + b = 2 + a = 2 + elif a == 2: + a = 3 + else: + a = 1 + + if a == 1: + if b == 2: + if c != 3: + c = 3 + b = 3 + else: + c = 2 + elif b == 3: + b = 2 + a = 2 + elif a == 2: + a = 3 + else: + a = 1 + + if a == 1: + if b == 2: + if c != 3: + c = 3 + b = 3 + else: + c = 2 + elif b == 3: + b = 2 + a = 2 + elif a == 2: + a = 3 + else: + a = 1 + + if a == 1: + if b == 2: + if c != 3: + c = 3 + b = 3 + else: + c = 2 + elif b == 3: + b = 2 + a = 2 + elif a == 2: + a = 3 + else: + a = 1 + + if a == 1: + if b == 2: + if c != 3: + c = 3 + b = 3 + else: + c = 2 + elif b == 3: + b = 2 + a = 2 + elif a == 2: + a = 3 + else: + a = 1 + + if a == 1: + if b == 2: + if c != 3: + c = 3 + b = 3 + else: + c = 2 + elif b == 3: + b = 2 + a = 2 + elif a == 2: + a = 3 + else: + a = 1 + + if a == 1: + if b == 2: + if c != 3: + c = 3 + b = 3 + else: + c = 2 + elif b == 3: + b = 2 + a = 2 + elif a == 2: + a = 3 + else: + a = 1 + + if a == 1: + if b == 2: + if c != 3: + c = 3 + b = 3 + else: + c = 2 + elif b == 3: + b = 2 + a = 2 + elif a == 2: + a = 3 + else: + a = 1 + + if a == 1: + if b == 2: + if c != 3: + c = 3 + b = 3 + else: + c = 2 + elif b == 3: + b = 2 + a = 2 + elif a == 2: + a = 3 + else: + a = 1 + + if a == 1: + if b == 2: + if c != 3: + c = 3 + b = 3 + else: + c = 2 + elif b == 3: + b = 2 + a = 2 + elif a == 2: + a = 3 + else: + a = 1 + + if a == 1: + if b == 2: + if c != 3: + c = 3 + b = 3 + else: + c = 2 + elif b == 3: + b = 2 + a = 2 + elif a == 2: + a = 3 + else: + a = 1 + + if a == 1: + if b == 2: + if c != 3: + c = 3 + b = 3 + else: + c = 2 + elif b == 3: + b = 2 + a = 2 + elif a == 2: + a = 3 + else: + a = 1 + + if a == 1: + if b == 2: + if c != 3: + c = 3 + b = 3 + else: + c = 2 + elif b == 3: + b = 2 + a = 2 + elif a == 2: + a = 3 + else: + a = 1 + + def calibrate(self): + + a,b,c = 1,2,3 + for i in xrange(self.rounds): + pass + +class NestedForLoops(Test): + + version = 0.3 + operations = 1000*10*5 + rounds = 150 + + def test(self): + + l1 = range(1000) + l2 = range(10) + l3 = range(5) + for i in xrange(self.rounds): + for i in l1: + for j in l2: + for k in l3: + pass + + def calibrate(self): + + l1 = range(1000) + l2 = range(10) + l3 = range(5) + for i in xrange(self.rounds): + pass + +class ForLoops(Test): + + version = 0.1 + operations = 5 * 5 + rounds = 8000 + + def test(self): + + l1 = range(100) + for i in xrange(self.rounds): + for i in l1: + pass + for i in l1: + pass + for i in l1: + pass + for i in l1: + pass + for i in l1: + pass + + for i in l1: + pass + for i in l1: + pass + for i in l1: + pass + for i in l1: + pass + for i in l1: + pass + + for i in l1: + pass + for i in l1: + pass + for i in l1: + pass + for i in l1: + pass + for i in l1: + pass + + for i in l1: + pass + for i in l1: + pass + for i in l1: + pass + for i in l1: + pass + for i in l1: + pass + + for i in l1: + pass + for i in l1: + pass + for i in l1: + pass + for i in l1: + pass + for i in l1: + pass + + def calibrate(self): + + l1 = range(1000) + for i in xrange(self.rounds): + pass + Added: external/pybench-1.3/Dict.py ============================================================================== --- (empty file) +++ external/pybench-1.3/Dict.py Wed Apr 19 17:21:58 2006 @@ -0,0 +1,503 @@ +from pybench import Test + +class DictCreation(Test): + + version = 0.3 + operations = 5*(5 + 5) + rounds = 60000 + + def test(self): + + for i in xrange(self.rounds): + + d1 = {} + d2 = {} + d3 = {} + d4 = {} + d5 = {} + + d1 = {1:2,3:4,5:6} + d2 = {2:3,4:5,6:7} + d3 = {3:4,5:6,7:8} + d4 = {4:5,6:7,8:9} + d5 = {6:7,8:9,10:11} + + d1 = {} + d2 = {} + d3 = {} + d4 = {} + d5 = {} + + d1 = {1:2,3:4,5:6} + d2 = {2:3,4:5,6:7} + d3 = {3:4,5:6,7:8} + d4 = {4:5,6:7,8:9} + d5 = {6:7,8:9,10:11} + + d1 = {} + d2 = {} + d3 = {} + d4 = {} + d5 = {} + + d1 = {1:2,3:4,5:6} + d2 = {2:3,4:5,6:7} + d3 = {3:4,5:6,7:8} + d4 = {4:5,6:7,8:9} + d5 = {6:7,8:9,10:11} + + d1 = {} + d2 = {} + d3 = {} + d4 = {} + d5 = {} + + d1 = {1:2,3:4,5:6} + d2 = {2:3,4:5,6:7} + d3 = {3:4,5:6,7:8} + d4 = {4:5,6:7,8:9} + d5 = {6:7,8:9,10:11} + + d1 = {} + d2 = {} + d3 = {} + d4 = {} + d5 = {} + + d1 = {1:2,3:4,5:6} + d2 = {2:3,4:5,6:7} + d3 = {3:4,5:6,7:8} + d4 = {4:5,6:7,8:9} + d5 = {6:7,8:9,10:11} + + def calibrate(self): + + for i in xrange(self.rounds): + pass + +class DictWithStringKeys(Test): + + version = 0.1 + operations = 5*(6 + 6) + rounds = 200000 + + def test(self): + + d = {} + + for i in xrange(self.rounds): + + d['abc'] = 1 + d['def'] = 2 + d['ghi'] = 3 + d['jkl'] = 4 + d['mno'] = 5 + d['pqr'] = 6 + + d['abc'] + d['def'] + d['ghi'] + d['jkl'] + d['mno'] + d['pqr'] + + d['abc'] = 1 + d['def'] = 2 + d['ghi'] = 3 + d['jkl'] = 4 + d['mno'] = 5 + d['pqr'] = 6 + + d['abc'] + d['def'] + d['ghi'] + d['jkl'] + d['mno'] + d['pqr'] + + d['abc'] = 1 + d['def'] = 2 + d['ghi'] = 3 + d['jkl'] = 4 + d['mno'] = 5 + d['pqr'] = 6 + + d['abc'] + d['def'] + d['ghi'] + d['jkl'] + d['mno'] + d['pqr'] + + d['abc'] = 1 + d['def'] = 2 + d['ghi'] = 3 + d['jkl'] = 4 + d['mno'] = 5 + d['pqr'] = 6 + + d['abc'] + d['def'] + d['ghi'] + d['jkl'] + d['mno'] + d['pqr'] + + d['abc'] = 1 + d['def'] = 2 + d['ghi'] = 3 + d['jkl'] = 4 + d['mno'] = 5 + d['pqr'] = 6 + + d['abc'] + d['def'] + d['ghi'] + d['jkl'] + d['mno'] + d['pqr'] + + def calibrate(self): + + d = {} + + for i in xrange(self.rounds): + pass + +class DictWithFloatKeys(Test): + + version = 0.1 + operations = 5*(6 + 6) + rounds = 200000 + + def test(self): + + d = {} + + for i in xrange(self.rounds): + + d[1.234] = 1 + d[2.345] = 2 + d[3.456] = 3 + d[4.567] = 4 + d[5.678] = 5 + d[6.789] = 6 + + d[1.234] + d[2.345] + d[3.456] + d[4.567] + d[5.678] + d[6.789] + + d[1.234] = 1 + d[2.345] = 2 + d[3.456] = 3 + d[4.567] = 4 + d[5.678] = 5 + d[6.789] = 6 + + d[1.234] + d[2.345] + d[3.456] + d[4.567] + d[5.678] + d[6.789] + + d[1.234] = 1 + d[2.345] = 2 + d[3.456] = 3 + d[4.567] = 4 + d[5.678] = 5 + d[6.789] = 6 + + d[1.234] + d[2.345] + d[3.456] + d[4.567] + d[5.678] + d[6.789] + + d[1.234] = 1 + d[2.345] = 2 + d[3.456] = 3 + d[4.567] = 4 + d[5.678] = 5 + d[6.789] = 6 + + d[1.234] + d[2.345] + d[3.456] + d[4.567] + d[5.678] + d[6.789] + + d[1.234] = 1 + d[2.345] = 2 + d[3.456] = 3 + d[4.567] = 4 + d[5.678] = 5 + d[6.789] = 6 + + d[1.234] + d[2.345] + d[3.456] + d[4.567] + d[5.678] + d[6.789] + + def calibrate(self): + + d = {} + + for i in xrange(self.rounds): + pass + +class DictWithIntegerKeys(Test): + + version = 0.1 + operations = 5*(6 + 6) + rounds = 200000 + + def test(self): + + d = {} + + for i in xrange(self.rounds): + + d[1] = 1 + d[2] = 2 + d[3] = 3 + d[4] = 4 + d[5] = 5 + d[6] = 6 + + d[1] + d[2] + d[3] + d[4] + d[5] + d[6] + + d[1] = 1 + d[2] = 2 + d[3] = 3 + d[4] = 4 + d[5] = 5 + d[6] = 6 + + d[1] + d[2] + d[3] + d[4] + d[5] + d[6] + + d[1] = 1 + d[2] = 2 + d[3] = 3 + d[4] = 4 + d[5] = 5 + d[6] = 6 + + d[1] + d[2] + d[3] + d[4] + d[5] + d[6] + + d[1] = 1 + d[2] = 2 + d[3] = 3 + d[4] = 4 + d[5] = 5 + d[6] = 6 + + d[1] + d[2] + d[3] + d[4] + d[5] + d[6] + + d[1] = 1 + d[2] = 2 + d[3] = 3 + d[4] = 4 + d[5] = 5 + d[6] = 6 + + d[1] + d[2] + d[3] + d[4] + d[5] + d[6] + + def calibrate(self): + + d = {} + + for i in xrange(self.rounds): + pass + +class SimpleDictManipulation(Test): + + version = 0.3 + operations = 5*(6 + 6 + 6 + 6) + rounds = 50000 + + def test(self): + + d = {} + + for i in xrange(self.rounds): + + d[0] = 3 + d[1] = 4 + d[2] = 5 + d[3] = 3 + d[4] = 4 + d[5] = 5 + + x = d[0] + x = d[1] + x = d[2] + x = d[3] + x = d[4] + x = d[5] + + d.has_key(0) + d.has_key(2) + d.has_key(4) + d.has_key(6) + d.has_key(8) + d.has_key(10) + + del d[0] + del d[1] + del d[2] + del d[3] + del d[4] + del d[5] + + d[0] = 3 + d[1] = 4 + d[2] = 5 + d[3] = 3 + d[4] = 4 + d[5] = 5 + + x = d[0] + x = d[1] + x = d[2] + x = d[3] + x = d[4] + x = d[5] + + d.has_key(0) + d.has_key(2) + d.has_key(4) + d.has_key(6) + d.has_key(8) + d.has_key(10) + + del d[0] + del d[1] + del d[2] + del d[3] + del d[4] + del d[5] + + d[0] = 3 + d[1] = 4 + d[2] = 5 + d[3] = 3 + d[4] = 4 + d[5] = 5 + + x = d[0] + x = d[1] + x = d[2] + x = d[3] + x = d[4] + x = d[5] + + d.has_key(0) + d.has_key(2) + d.has_key(4) + d.has_key(6) + d.has_key(8) + d.has_key(10) + + del d[0] + del d[1] + del d[2] + del d[3] + del d[4] + del d[5] + + d[0] = 3 + d[1] = 4 + d[2] = 5 + d[3] = 3 + d[4] = 4 + d[5] = 5 + + x = d[0] + x = d[1] + x = d[2] + x = d[3] + x = d[4] + x = d[5] + + d.has_key(0) + d.has_key(2) + d.has_key(4) + d.has_key(6) + d.has_key(8) + d.has_key(10) + + del d[0] + del d[1] + del d[2] + del d[3] + del d[4] + del d[5] + + d[0] = 3 + d[1] = 4 + d[2] = 5 + d[3] = 3 + d[4] = 4 + d[5] = 5 + + x = d[0] + x = d[1] + x = d[2] + x = d[3] + x = d[4] + x = d[5] + + d.has_key(0) + d.has_key(2) + d.has_key(4) + d.has_key(6) + d.has_key(8) + d.has_key(10) + + del d[0] + del d[1] + del d[2] + del d[3] + del d[4] + del d[5] + + def calibrate(self): + + d = {} + + for i in xrange(self.rounds): + pass + Added: external/pybench-1.3/Exceptions.py ============================================================================== --- (empty file) +++ external/pybench-1.3/Exceptions.py Wed Apr 19 17:21:58 2006 @@ -0,0 +1,681 @@ +from pybench import Test + +class TryRaiseExcept(Test): + + version = 0.1 + operations = 2 + 3 + rounds = 60000 + + def test(self): + + error = ValueError + + for i in xrange(self.rounds): + try: + raise error + except: + pass + try: + raise error + except: + pass + try: + raise error,"something" + except: + pass + try: + raise error,"something" + except: + pass + try: + raise error,"something" + except: + pass + + def calibrate(self): + + error = ValueError + + for i in xrange(self.rounds): + pass + + +class TryExcept(Test): + + version = 0.1 + operations = 15 * 10 + rounds = 200000 + + def test(self): + + for i in xrange(self.rounds): + try: + pass + except: + pass + try: + pass + except: + pass + try: + pass + except: + pass + try: + pass + except: + pass + try: + pass + except: + pass + try: + pass + except: + pass + try: + pass + except: + pass + try: + pass + except: + pass + try: + pass + except: + pass + try: + pass + except: + pass + + try: + pass + except: + pass + try: + pass + except: + pass + try: + pass + except: + pass + try: + pass + except: + pass + try: + pass + except: + pass + try: + pass + except: + pass + try: + pass + except: + pass + try: + pass + except: + pass + try: + pass + except: + pass + try: + pass + except: + pass + + + try: + pass + except: + pass + try: + pass + except: + pass + try: + pass + except: + pass + try: + pass + except: + pass + try: + pass + except: + pass + try: + pass + except: + pass + try: + pass + except: + pass + try: + pass + except: + pass + try: + pass + except: + pass + try: + pass + except: + pass + + + try: + pass + except: + pass + try: + pass + except: + pass + try: + pass + except: + pass + try: + pass + except: + pass + try: + pass + except: + pass + try: + pass + except: + pass + try: + pass + except: + pass + try: + pass + except: + pass + try: + pass + except: + pass + try: + pass + except: + pass + + + try: + pass + except: + pass + try: + pass + except: + pass + try: + pass + except: + pass + try: + pass + except: + pass + try: + pass + except: + pass + try: + pass + except: + pass + try: + pass + except: + pass + try: + pass + except: + pass + try: + pass + except: + pass + try: + pass + except: + pass + + try: + pass + except: + pass + try: + pass + except: + pass + try: + pass + except: + pass + try: + pass + except: + pass + try: + pass + except: + pass + try: + pass + except: + pass + try: + pass + except: + pass + try: + pass + except: + pass + try: + pass + except: + pass + try: + pass + except: + pass + + try: + pass + except: + pass + try: + pass + except: + pass + try: + pass + except: + pass + try: + pass + except: + pass + try: + pass + except: + pass + try: + pass + except: + pass + try: + pass + except: + pass + try: + pass + except: + pass + try: + pass + except: + pass + try: + pass + except: + pass + + + try: + pass + except: + pass + try: + pass + except: + pass + try: + pass + except: + pass + try: + pass + except: + pass + try: + pass + except: + pass + try: + pass + except: + pass + try: + pass + except: + pass + try: + pass + except: + pass + try: + pass + except: + pass + try: + pass + except: + pass + + + try: + pass + except: + pass + try: + pass + except: + pass + try: + pass + except: + pass + try: + pass + except: + pass + try: + pass + except: + pass + try: + pass + except: + pass + try: + pass + except: + pass + try: + pass + except: + pass + try: + pass + except: + pass + try: + pass + except: + pass + + + try: + pass + except: + pass + try: + pass + except: + pass + try: + pass + except: + pass + try: + pass + except: + pass + try: + pass + except: + pass + try: + pass + except: + pass + try: + pass + except: + pass + try: + pass + except: + pass + try: + pass + except: + pass + try: + pass + except: + pass + + try: + pass + except: + pass + try: + pass + except: + pass + try: + pass + except: + pass + try: + pass + except: + pass + try: + pass + except: + pass + try: + pass + except: + pass + try: + pass + except: + pass + try: + pass + except: + pass + try: + pass + except: + pass + try: + pass + except: + pass + + try: + pass + except: + pass + try: + pass + except: + pass + try: + pass + except: + pass + try: + pass + except: + pass + try: + pass + except: + pass + try: + pass + except: + pass + try: + pass + except: + pass + try: + pass + except: + pass + try: + pass + except: + pass + try: + pass + except: + pass + + + try: + pass + except: + pass + try: + pass + except: + pass + try: + pass + except: + pass + try: + pass + except: + pass + try: + pass + except: + pass + try: + pass + except: + pass + try: + pass + except: + pass + try: + pass + except: + pass + try: + pass + except: + pass + try: + pass + except: + pass + + + try: + pass + except: + pass + try: + pass + except: + pass + try: + pass + except: + pass + try: + pass + except: + pass + try: + pass + except: + pass + try: + pass + except: + pass + try: + pass + except: + pass + try: + pass + except: + pass + try: + pass + except: + pass + try: + pass + except: + pass + + + try: + pass + except: + pass + try: + pass + except: + pass + try: + pass + except: + pass + try: + pass + except: + pass + try: + pass + except: + pass + try: + pass + except: + pass + try: + pass + except: + pass + try: + pass + except: + pass + try: + pass + except: + pass + try: + pass + except: + pass + + def calibrate(self): + + for i in xrange(self.rounds): + pass + + Added: external/pybench-1.3/Imports.py ============================================================================== --- (empty file) +++ external/pybench-1.3/Imports.py Wed Apr 19 17:21:58 2006 @@ -0,0 +1,139 @@ +from pybench import Test + +# First imports: +import os +import package.submodule + +class SecondImport(Test): + + version = 0.1 + operations = 5 * 5 + rounds = 20000 + + def test(self): + + for i in xrange(self.rounds): + import os + import os + import os + import os + import os + + import os + import os + import os + import os + import os + + import os + import os + import os + import os + import os + + import os + import os + import os + import os + import os + + import os + import os + import os + import os + import os + + def calibrate(self): + + for i in xrange(self.rounds): + pass + + +class SecondPackageImport(Test): + + version = 0.1 + operations = 5 * 5 + rounds = 20000 + + def test(self): + + for i in xrange(self.rounds): + import package + import package + import package + import package + import package + + import package + import package + import package + import package + import package + + import package + import package + import package + import package + import package + + import package + import package + import package + import package + import package + + import package + import package + import package + import package + import package + + def calibrate(self): + + for i in xrange(self.rounds): + pass + +class SecondSubmoduleImport(Test): + + version = 0.1 + operations = 5 * 5 + rounds = 20000 + + def test(self): + + for i in xrange(self.rounds): + import package.submodule + import package.submodule + import package.submodule + import package.submodule + import package.submodule + + import package.submodule + import package.submodule + import package.submodule + import package.submodule + import package.submodule + + import package.submodule + import package.submodule + import package.submodule + import package.submodule + import package.submodule + + import package.submodule + import package.submodule + import package.submodule + import package.submodule + import package.submodule + + import package.submodule + import package.submodule + import package.submodule + import package.submodule + import package.submodule + + def calibrate(self): + + for i in xrange(self.rounds): + pass + Added: external/pybench-1.3/Instances.py ============================================================================== --- (empty file) +++ external/pybench-1.3/Instances.py Wed Apr 19 17:21:58 2006 @@ -0,0 +1,68 @@ +from pybench import Test + +class CreateInstances(Test): + + version = 0.2 + operations = 3 + 7 + 4 + rounds = 60000 + + def test(self): + + class c: + pass + + class d: + def __init__(self,a,b,c): + self.a = a + self.b = b + self.c = c + + class e: + def __init__(self,a,b,c=4): + self.a = a + self.b = b + self.c = c + self.d = a + self.e = b + self.f = c + + for i in xrange(self.rounds): + o = c() + o1 = c() + o2 = c() + p = d(i,i,3) + p1 = d(i,i,3) + p2 = d(i,3,3) + p3 = d(3,i,3) + p4 = d(i,i,i) + p5 = d(3,i,3) + p6 = d(i,i,i) + q = e(i,i,3) + q1 = e(i,i,3) + q2 = e(i,i,3) + q3 = e(i,i) + + def calibrate(self): + + class c: + pass + + class d: + def __init__(self,a,b,c): + self.a = a + self.b = b + self.c = c + + class e: + def __init__(self,a,b,c=4): + self.a = a + self.b = b + self.c = c + self.d = a + self.e = b + self.f = c + + for i in xrange(self.rounds): + pass + + Added: external/pybench-1.3/LICENSE ============================================================================== --- (empty file) +++ external/pybench-1.3/LICENSE Wed Apr 19 17:21:58 2006 @@ -0,0 +1,25 @@ +pybench License +--------------- + +This copyright notice and license applies to all files in the pybench +directory of the pybench distribution. + +Copyright (c), 1997-2006, Marc-Andre Lemburg (mal at lemburg.com) +Copyright (c), 2000-2006, eGenix.com Software GmbH (info at egenix.com) + + All Rights Reserved. + +Permission to use, copy, modify, and distribute this software and its +documentation for any purpose and without fee or royalty is hereby +granted, provided that the above copyright notice appear in all copies +and that both that copyright notice and this permission notice appear +in supporting documentation or portions thereof, including +modifications, that you make. + +THE AUTHOR MARC-ANDRE LEMBURG DISCLAIMS ALL WARRANTIES WITH REGARD TO +THIS SOFTWARE, INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY AND +FITNESS, IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY SPECIAL, +INDIRECT OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING +FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, +NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION +WITH THE USE OR PERFORMANCE OF THIS SOFTWARE ! Added: external/pybench-1.3/Lists.py ============================================================================== --- (empty file) +++ external/pybench-1.3/Lists.py Wed Apr 19 17:21:58 2006 @@ -0,0 +1,292 @@ +from pybench import Test + +class SimpleListManipulation(Test): + + version = 0.3 + operations = 5* (6 + 6 + 6) + rounds = 60000 + + def test(self): + + l = [] + + for i in xrange(self.rounds): + + l.append(2) + l.append(3) + l.append(4) + l.append(2) + l.append(3) + l.append(4) + + l[0] = 3 + l[1] = 4 + l[2] = 5 + l[3] = 3 + l[4] = 4 + l[5] = 5 + + x = l[0] + x = l[1] + x = l[2] + x = l[3] + x = l[4] + x = l[5] + + l.append(2) + l.append(3) + l.append(4) + l.append(2) + l.append(3) + l.append(4) + + l[0] = 3 + l[1] = 4 + l[2] = 5 + l[3] = 3 + l[4] = 4 + l[5] = 5 + + x = l[0] + x = l[1] + x = l[2] + x = l[3] + x = l[4] + x = l[5] + + l.append(2) + l.append(3) + l.append(4) + l.append(2) + l.append(3) + l.append(4) + + l[0] = 3 + l[1] = 4 + l[2] = 5 + l[3] = 3 + l[4] = 4 + l[5] = 5 + + x = l[0] + x = l[1] + x = l[2] + x = l[3] + x = l[4] + x = l[5] + + l.append(2) + l.append(3) + l.append(4) + l.append(2) + l.append(3) + l.append(4) + + l[0] = 3 + l[1] = 4 + l[2] = 5 + l[3] = 3 + l[4] = 4 + l[5] = 5 + + x = l[0] + x = l[1] + x = l[2] + x = l[3] + x = l[4] + x = l[5] + + l.append(2) + l.append(3) + l.append(4) + l.append(2) + l.append(3) + l.append(4) + + l[0] = 3 + l[1] = 4 + l[2] = 5 + l[3] = 3 + l[4] = 4 + l[5] = 5 + + x = l[0] + x = l[1] + x = l[2] + x = l[3] + x = l[4] + x = l[5] + + if len(l) > 10000: + # cut down the size + del l[:] + + def calibrate(self): + + l = [] + + for i in xrange(self.rounds): + pass + +class ListSlicing(Test): + + version = 0.4 + operations = 25*(3+1+2+1) + rounds = 400 + + def test(self): + + n = range(100) + r = range(25) + + for i in xrange(self.rounds): + + l = range(100) + + for j in r: + + m = l[50:] + m = l[:25] + m = l[50:55] + l[:3] = n + m = l[:-1] + m = l[1:] + l[-1:] = n + + def calibrate(self): + + n = range(100) + r = range(25) + + for i in xrange(self.rounds): + + l = range(100) + + for j in r: + pass + +class SmallLists(Test): + + version = 0.3 + operations = 5*(1+ 6 + 6 + 3 + 1) + rounds = 60000 + + def test(self): + + for i in xrange(self.rounds): + + l = [] + + l.append(2) + l.append(3) + l.append(4) + l.append(2) + l.append(3) + l.append(4) + + l[0] = 3 + l[1] = 4 + l[2] = 5 + l[3] = 3 + l[4] = 4 + l[5] = 5 + + l[:3] = [1,2,3] + m = l[:-1] + m = l[1:] + + l[-1:] = [4,5,6] + + l = [] + + l.append(2) + l.append(3) + l.append(4) + l.append(2) + l.append(3) + l.append(4) + + l[0] = 3 + l[1] = 4 + l[2] = 5 + l[3] = 3 + l[4] = 4 + l[5] = 5 + + l[:3] = [1,2,3] + m = l[:-1] + m = l[1:] + + l[-1:] = [4,5,6] + + l = [] + + l.append(2) + l.append(3) + l.append(4) + l.append(2) + l.append(3) + l.append(4) + + l[0] = 3 + l[1] = 4 + l[2] = 5 + l[3] = 3 + l[4] = 4 + l[5] = 5 + + l[:3] = [1,2,3] + m = l[:-1] + m = l[1:] + + l[-1:] = [4,5,6] + + l = [] + + l.append(2) + l.append(3) + l.append(4) + l.append(2) + l.append(3) + l.append(4) + + l[0] = 3 + l[1] = 4 + l[2] = 5 + l[3] = 3 + l[4] = 4 + l[5] = 5 + + l[:3] = [1,2,3] + m = l[:-1] + m = l[1:] + + l[-1:] = [4,5,6] + + l = [] + + l.append(2) + l.append(3) + l.append(4) + l.append(2) + l.append(3) + l.append(4) + + l[0] = 3 + l[1] = 4 + l[2] = 5 + l[3] = 3 + l[4] = 4 + l[5] = 5 + + l[:3] = [1,2,3] + m = l[:-1] + m = l[1:] + + l[-1:] = [4,5,6] + + def calibrate(self): + + for i in xrange(self.rounds): + l = [] + Added: external/pybench-1.3/Lookups.py ============================================================================== --- (empty file) +++ external/pybench-1.3/Lookups.py Wed Apr 19 17:21:58 2006 @@ -0,0 +1,946 @@ +from pybench import Test + +class SpecialClassAttribute(Test): + + version = 0.3 + operations = 5*(12 + 12) + rounds = 100000 + + def test(self): + + class c: + pass + + for i in xrange(self.rounds): + + c.__a = 2 + c.__b = 3 + c.__c = 4 + + c.__a = 2 + c.__b = 3 + c.__c = 4 + + c.__a = 2 + c.__b = 3 + c.__c = 4 + + c.__a = 2 + c.__b = 3 + c.__c = 4 + + x = c.__a + x = c.__b + x = c.__c + + x = c.__a + x = c.__b + x = c.__c + + x = c.__a + x = c.__b + x = c.__c + + x = c.__a + x = c.__b + x = c.__c + + c.__a = 2 + c.__b = 3 + c.__c = 4 + + c.__a = 2 + c.__b = 3 + c.__c = 4 + + c.__a = 2 + c.__b = 3 + c.__c = 4 + + c.__a = 2 + c.__b = 3 + c.__c = 4 + + x = c.__a + x = c.__b + x = c.__c + + x = c.__a + x = c.__b + x = c.__c + + x = c.__a + x = c.__b + x = c.__c + + x = c.__a + x = c.__b + x = c.__c + + c.__a = 2 + c.__b = 3 + c.__c = 4 + + c.__a = 2 + c.__b = 3 + c.__c = 4 + + c.__a = 2 + c.__b = 3 + c.__c = 4 + + c.__a = 2 + c.__b = 3 + c.__c = 4 + + x = c.__a + x = c.__b + x = c.__c + + x = c.__a + x = c.__b + x = c.__c + + x = c.__a + x = c.__b + x = c.__c + + x = c.__a + x = c.__b + x = c.__c + + c.__a = 2 + c.__b = 3 + c.__c = 4 + + c.__a = 2 + c.__b = 3 + c.__c = 4 + + c.__a = 2 + c.__b = 3 + c.__c = 4 + + c.__a = 2 + c.__b = 3 + c.__c = 4 + + x = c.__a + x = c.__b + x = c.__c + + x = c.__a + x = c.__b + x = c.__c + + x = c.__a + x = c.__b + x = c.__c + + x = c.__a + x = c.__b + x = c.__c + + c.__a = 2 + c.__b = 3 + c.__c = 4 + + c.__a = 2 + c.__b = 3 + c.__c = 4 + + c.__a = 2 + c.__b = 3 + c.__c = 4 + + c.__a = 2 + c.__b = 3 + c.__c = 4 + + x = c.__a + x = c.__b + x = c.__c + + x = c.__a + x = c.__b + x = c.__c + + x = c.__a + x = c.__b + x = c.__c + + x = c.__a + x = c.__b + x = c.__c + + def calibrate(self): + + class c: + pass + + for i in xrange(self.rounds): + pass + +class NormalClassAttribute(Test): + + version = 0.3 + operations = 5*(12 + 12) + rounds = 100000 + + def test(self): + + class c: + pass + + for i in xrange(self.rounds): + + c.a = 2 + c.b = 3 + c.c = 4 + + c.a = 2 + c.b = 3 + c.c = 4 + + c.a = 2 + c.b = 3 + c.c = 4 + + c.a = 2 + c.b = 3 + c.c = 4 + + + x = c.a + x = c.b + x = c.c + + x = c.a + x = c.b + x = c.c + + x = c.a + x = c.b + x = c.c + + x = c.a + x = c.b + x = c.c + + c.a = 2 + c.b = 3 + c.c = 4 + + c.a = 2 + c.b = 3 + c.c = 4 + + c.a = 2 + c.b = 3 + c.c = 4 + + c.a = 2 + c.b = 3 + c.c = 4 + + + x = c.a + x = c.b + x = c.c + + x = c.a + x = c.b + x = c.c + + x = c.a + x = c.b + x = c.c + + x = c.a + x = c.b + x = c.c + + c.a = 2 + c.b = 3 + c.c = 4 + + c.a = 2 + c.b = 3 + c.c = 4 + + c.a = 2 + c.b = 3 + c.c = 4 + + c.a = 2 + c.b = 3 + c.c = 4 + + + x = c.a + x = c.b + x = c.c + + x = c.a + x = c.b + x = c.c + + x = c.a + x = c.b + x = c.c + + x = c.a + x = c.b + x = c.c + + c.a = 2 + c.b = 3 + c.c = 4 + + c.a = 2 + c.b = 3 + c.c = 4 + + c.a = 2 + c.b = 3 + c.c = 4 + + c.a = 2 + c.b = 3 + c.c = 4 + + + x = c.a + x = c.b + x = c.c + + x = c.a + x = c.b + x = c.c + + x = c.a + x = c.b + x = c.c + + x = c.a + x = c.b + x = c.c + + c.a = 2 + c.b = 3 + c.c = 4 + + c.a = 2 + c.b = 3 + c.c = 4 + + c.a = 2 + c.b = 3 + c.c = 4 + + c.a = 2 + c.b = 3 + c.c = 4 + + + x = c.a + x = c.b + x = c.c + + x = c.a + x = c.b + x = c.c + + x = c.a + x = c.b + x = c.c + + x = c.a + x = c.b + x = c.c + + def calibrate(self): + + class c: + pass + + for i in xrange(self.rounds): + pass + +class SpecialInstanceAttribute(Test): + + version = 0.3 + operations = 5*(12 + 12) + rounds = 100000 + + def test(self): + + class c: + pass + o = c() + + for i in xrange(self.rounds): + + o.__a__ = 2 + o.__b__ = 3 + o.__c__ = 4 + + o.__a__ = 2 + o.__b__ = 3 + o.__c__ = 4 + + o.__a__ = 2 + o.__b__ = 3 + o.__c__ = 4 + + o.__a__ = 2 + o.__b__ = 3 + o.__c__ = 4 + + + x = o.__a__ + x = o.__b__ + x = o.__c__ + + x = o.__a__ + x = o.__b__ + x = o.__c__ + + x = o.__a__ + x = o.__b__ + x = o.__c__ + + x = o.__a__ + x = o.__b__ + x = o.__c__ + + o.__a__ = 2 + o.__b__ = 3 + o.__c__ = 4 + + o.__a__ = 2 + o.__b__ = 3 + o.__c__ = 4 + + o.__a__ = 2 + o.__b__ = 3 + o.__c__ = 4 + + o.__a__ = 2 + o.__b__ = 3 + o.__c__ = 4 + + + x = o.__a__ + x = o.__b__ + x = o.__c__ + + x = o.__a__ + x = o.__b__ + x = o.__c__ + + x = o.__a__ + x = o.__b__ + x = o.__c__ + + x = o.__a__ + x = o.__b__ + x = o.__c__ + + o.__a__ = 2 + o.__b__ = 3 + o.__c__ = 4 + + o.__a__ = 2 + o.__b__ = 3 + o.__c__ = 4 + + o.__a__ = 2 + o.__b__ = 3 + o.__c__ = 4 + + o.__a__ = 2 + o.__b__ = 3 + o.__c__ = 4 + + + x = o.__a__ + x = o.__b__ + x = o.__c__ + + x = o.__a__ + x = o.__b__ + x = o.__c__ + + x = o.__a__ + x = o.__b__ + x = o.__c__ + + x = o.__a__ + x = o.__b__ + x = o.__c__ + + o.__a__ = 2 + o.__b__ = 3 + o.__c__ = 4 + + o.__a__ = 2 + o.__b__ = 3 + o.__c__ = 4 + + o.__a__ = 2 + o.__b__ = 3 + o.__c__ = 4 + + o.__a__ = 2 + o.__b__ = 3 + o.__c__ = 4 + + + x = o.__a__ + x = o.__b__ + x = o.__c__ + + x = o.__a__ + x = o.__b__ + x = o.__c__ + + x = o.__a__ + x = o.__b__ + x = o.__c__ + + x = o.__a__ + x = o.__b__ + x = o.__c__ + + o.__a__ = 2 + o.__b__ = 3 + o.__c__ = 4 + + o.__a__ = 2 + o.__b__ = 3 + o.__c__ = 4 + + o.__a__ = 2 + o.__b__ = 3 + o.__c__ = 4 + + o.__a__ = 2 + o.__b__ = 3 + o.__c__ = 4 + + + x = o.__a__ + x = o.__b__ + x = o.__c__ + + x = o.__a__ + x = o.__b__ + x = o.__c__ + + x = o.__a__ + x = o.__b__ + x = o.__c__ + + x = o.__a__ + x = o.__b__ + x = o.__c__ + + def calibrate(self): + + class c: + pass + o = c() + + for i in xrange(self.rounds): + pass + +class NormalInstanceAttribute(Test): + + version = 0.3 + operations = 5*(12 + 12) + rounds = 100000 + + def test(self): + + class c: + pass + o = c() + + for i in xrange(self.rounds): + + o.a = 2 + o.b = 3 + o.c = 4 + + o.a = 2 + o.b = 3 + o.c = 4 + + o.a = 2 + o.b = 3 + o.c = 4 + + o.a = 2 + o.b = 3 + o.c = 4 + + + x = o.a + x = o.b + x = o.c + + x = o.a + x = o.b + x = o.c + + x = o.a + x = o.b + x = o.c + + x = o.a + x = o.b + x = o.c + + o.a = 2 + o.b = 3 + o.c = 4 + + o.a = 2 + o.b = 3 + o.c = 4 + + o.a = 2 + o.b = 3 + o.c = 4 + + o.a = 2 + o.b = 3 + o.c = 4 + + + x = o.a + x = o.b + x = o.c + + x = o.a + x = o.b + x = o.c + + x = o.a + x = o.b + x = o.c + + x = o.a + x = o.b + x = o.c + + o.a = 2 + o.b = 3 + o.c = 4 + + o.a = 2 + o.b = 3 + o.c = 4 + + o.a = 2 + o.b = 3 + o.c = 4 + + o.a = 2 + o.b = 3 + o.c = 4 + + + x = o.a + x = o.b + x = o.c + + x = o.a + x = o.b + x = o.c + + x = o.a + x = o.b + x = o.c + + x = o.a + x = o.b + x = o.c + + o.a = 2 + o.b = 3 + o.c = 4 + + o.a = 2 + o.b = 3 + o.c = 4 + + o.a = 2 + o.b = 3 + o.c = 4 + + o.a = 2 + o.b = 3 + o.c = 4 + + + x = o.a + x = o.b + x = o.c + + x = o.a + x = o.b + x = o.c + + x = o.a + x = o.b + x = o.c + + x = o.a + x = o.b + x = o.c + + o.a = 2 + o.b = 3 + o.c = 4 + + o.a = 2 + o.b = 3 + o.c = 4 + + o.a = 2 + o.b = 3 + o.c = 4 + + o.a = 2 + o.b = 3 + o.c = 4 + + + x = o.a + x = o.b + x = o.c + + x = o.a + x = o.b + x = o.c + + x = o.a + x = o.b + x = o.c + + x = o.a + x = o.b + x = o.c + + def calibrate(self): + + class c: + pass + o = c() + + for i in xrange(self.rounds): + pass + +class BuiltinMethodLookup(Test): + + version = 0.3 + operations = 5*(3*5 + 3*5) + rounds = 70000 + + def test(self): + + l = [] + d = {} + + for i in xrange(self.rounds): + + l.append + l.append + l.append + l.append + l.append + + l.insert + l.insert + l.insert + l.insert + l.insert + + l.sort + l.sort + l.sort + l.sort + l.sort + + d.has_key + d.has_key + d.has_key + d.has_key + d.has_key + + d.items + d.items + d.items + d.items + d.items + + d.get + d.get + d.get + d.get + d.get + + l.append + l.append + l.append + l.append + l.append + + l.insert + l.insert + l.insert + l.insert + l.insert + + l.sort + l.sort + l.sort + l.sort + l.sort + + d.has_key + d.has_key + d.has_key + d.has_key + d.has_key + + d.items + d.items + d.items + d.items + d.items + + d.get + d.get + d.get + d.get + d.get + + l.append + l.append + l.append + l.append + l.append + + l.insert + l.insert + l.insert + l.insert + l.insert + + l.sort + l.sort + l.sort + l.sort + l.sort + + d.has_key + d.has_key + d.has_key + d.has_key + d.has_key + + d.items + d.items + d.items + d.items + d.items + + d.get + d.get + d.get + d.get + d.get + + l.append + l.append + l.append + l.append + l.append + + l.insert + l.insert + l.insert + l.insert + l.insert + + l.sort + l.sort + l.sort + l.sort + l.sort + + d.has_key + d.has_key + d.has_key + d.has_key + d.has_key + + d.items + d.items + d.items + d.items + d.items + + d.get + d.get + d.get + d.get + d.get + + l.append + l.append + l.append + l.append + l.append + + l.insert + l.insert + l.insert + l.insert + l.insert + + l.sort + l.sort + l.sort + l.sort + l.sort + + d.has_key + d.has_key + d.has_key + d.has_key + d.has_key + + d.items + d.items + d.items + d.items + d.items + + d.get + d.get + d.get + d.get + d.get + + def calibrate(self): + + l = [] + d = {} + + for i in xrange(self.rounds): + pass + Added: external/pybench-1.3/Numbers.py ============================================================================== --- (empty file) +++ external/pybench-1.3/Numbers.py Wed Apr 19 17:21:58 2006 @@ -0,0 +1,784 @@ +from pybench import Test + +class CompareIntegers(Test): + + version = 0.1 + operations = 30 * 5 + rounds = 120000 + + def test(self): + + for i in xrange(self.rounds): + + 2 < 3 + 2 > 3 + 2 == 3 + 2 > 3 + 2 < 3 + + 2 < 3 + 2 > 3 + 2 == 3 + 2 > 3 + 2 < 3 + + 2 < 3 + 2 > 3 + 2 == 3 + 2 > 3 + 2 < 3 + + 2 < 3 + 2 > 3 + 2 == 3 + 2 > 3 + 2 < 3 + + 2 < 3 + 2 > 3 + 2 == 3 + 2 > 3 + 2 < 3 + + 2 < 3 + 2 > 3 + 2 == 3 + 2 > 3 + 2 < 3 + + 2 < 3 + 2 > 3 + 2 == 3 + 2 > 3 + 2 < 3 + + 2 < 3 + 2 > 3 + 2 == 3 + 2 > 3 + 2 < 3 + + 2 < 3 + 2 > 3 + 2 == 3 + 2 > 3 + 2 < 3 + + 2 < 3 + 2 > 3 + 2 == 3 + 2 > 3 + 2 < 3 + + 2 < 3 + 2 > 3 + 2 == 3 + 2 > 3 + 2 < 3 + + 2 < 3 + 2 > 3 + 2 == 3 + 2 > 3 + 2 < 3 + + 2 < 3 + 2 > 3 + 2 == 3 + 2 > 3 + 2 < 3 + + 2 < 3 + 2 > 3 + 2 == 3 + 2 > 3 + 2 < 3 + + 2 < 3 + 2 > 3 + 2 == 3 + 2 > 3 + 2 < 3 + + 2 < 3 + 2 > 3 + 2 == 3 + 2 > 3 + 2 < 3 + + 2 < 3 + 2 > 3 + 2 == 3 + 2 > 3 + 2 < 3 + + 2 < 3 + 2 > 3 + 2 == 3 + 2 > 3 + 2 < 3 + + 2 < 3 + 2 > 3 + 2 == 3 + 2 > 3 + 2 < 3 + + 2 < 3 + 2 > 3 + 2 == 3 + 2 > 3 + 2 < 3 + + 2 < 3 + 2 > 3 + 2 == 3 + 2 > 3 + 2 < 3 + + 2 < 3 + 2 > 3 + 2 == 3 + 2 > 3 + 2 < 3 + + 2 < 3 + 2 > 3 + 2 == 3 + 2 > 3 + 2 < 3 + + 2 < 3 + 2 > 3 + 2 == 3 + 2 > 3 + 2 < 3 + + 2 < 3 + 2 > 3 + 2 == 3 + 2 > 3 + 2 < 3 + + 2 < 3 + 2 > 3 + 2 == 3 + 2 > 3 + 2 < 3 + + 2 < 3 + 2 > 3 + 2 == 3 + 2 > 3 + 2 < 3 + + 2 < 3 + 2 > 3 + 2 == 3 + 2 > 3 + 2 < 3 + + 2 < 3 + 2 > 3 + 2 == 3 + 2 > 3 + 2 < 3 + + 2 < 3 + 2 > 3 + 2 == 3 + 2 > 3 + 2 < 3 + + def calibrate(self): + + for i in xrange(self.rounds): + pass + + +class CompareFloats(Test): + + version = 0.1 + operations = 30 * 5 + rounds = 60000 + + def test(self): + + for i in xrange(self.rounds): + + 2.1 < 3.31 + 2.1 > 3.31 + 2.1 == 3.31 + 2.1 > 3.31 + 2.1 < 3.31 + + 2.1 < 3.31 + 2.1 > 3.31 + 2.1 == 3.31 + 2.1 > 3.31 + 2.1 < 3.31 + + 2.1 < 3.31 + 2.1 > 3.31 + 2.1 == 3.31 + 2.1 > 3.31 + 2.1 < 3.31 + + 2.1 < 3.31 + 2.1 > 3.31 + 2.1 == 3.31 + 2.1 > 3.31 + 2.1 < 3.31 + + 2.1 < 3.31 + 2.1 > 3.31 + 2.1 == 3.31 + 2.1 > 3.31 + 2.1 < 3.31 + + 2.1 < 3.31 + 2.1 > 3.31 + 2.1 == 3.31 + 2.1 > 3.31 + 2.1 < 3.31 + + 2.1 < 3.31 + 2.1 > 3.31 + 2.1 == 3.31 + 2.1 > 3.31 + 2.1 < 3.31 + + 2.1 < 3.31 + 2.1 > 3.31 + 2.1 == 3.31 + 2.1 > 3.31 + 2.1 < 3.31 + + 2.1 < 3.31 + 2.1 > 3.31 + 2.1 == 3.31 + 2.1 > 3.31 + 2.1 < 3.31 + + 2.1 < 3.31 + 2.1 > 3.31 + 2.1 == 3.31 + 2.1 > 3.31 + 2.1 < 3.31 + + 2.1 < 3.31 + 2.1 > 3.31 + 2.1 == 3.31 + 2.1 > 3.31 + 2.1 < 3.31 + + 2.1 < 3.31 + 2.1 > 3.31 + 2.1 == 3.31 + 2.1 > 3.31 + 2.1 < 3.31 + + 2.1 < 3.31 + 2.1 > 3.31 + 2.1 == 3.31 + 2.1 > 3.31 + 2.1 < 3.31 + + 2.1 < 3.31 + 2.1 > 3.31 + 2.1 == 3.31 + 2.1 > 3.31 + 2.1 < 3.31 + + 2.1 < 3.31 + 2.1 > 3.31 + 2.1 == 3.31 + 2.1 > 3.31 + 2.1 < 3.31 + + 2.1 < 3.31 + 2.1 > 3.31 + 2.1 == 3.31 + 2.1 > 3.31 + 2.1 < 3.31 + + 2.1 < 3.31 + 2.1 > 3.31 + 2.1 == 3.31 + 2.1 > 3.31 + 2.1 < 3.31 + + 2.1 < 3.31 + 2.1 > 3.31 + 2.1 == 3.31 + 2.1 > 3.31 + 2.1 < 3.31 + + 2.1 < 3.31 + 2.1 > 3.31 + 2.1 == 3.31 + 2.1 > 3.31 + 2.1 < 3.31 + + 2.1 < 3.31 + 2.1 > 3.31 + 2.1 == 3.31 + 2.1 > 3.31 + 2.1 < 3.31 + + 2.1 < 3.31 + 2.1 > 3.31 + 2.1 == 3.31 + 2.1 > 3.31 + 2.1 < 3.31 + + 2.1 < 3.31 + 2.1 > 3.31 + 2.1 == 3.31 + 2.1 > 3.31 + 2.1 < 3.31 + + 2.1 < 3.31 + 2.1 > 3.31 + 2.1 == 3.31 + 2.1 > 3.31 + 2.1 < 3.31 + + 2.1 < 3.31 + 2.1 > 3.31 + 2.1 == 3.31 + 2.1 > 3.31 + 2.1 < 3.31 + + 2.1 < 3.31 + 2.1 > 3.31 + 2.1 == 3.31 + 2.1 > 3.31 + 2.1 < 3.31 + + 2.1 < 3.31 + 2.1 > 3.31 + 2.1 == 3.31 + 2.1 > 3.31 + 2.1 < 3.31 + + 2.1 < 3.31 + 2.1 > 3.31 + 2.1 == 3.31 + 2.1 > 3.31 + 2.1 < 3.31 + + 2.1 < 3.31 + 2.1 > 3.31 + 2.1 == 3.31 + 2.1 > 3.31 + 2.1 < 3.31 + + 2.1 < 3.31 + 2.1 > 3.31 + 2.1 == 3.31 + 2.1 > 3.31 + 2.1 < 3.31 + + 2.1 < 3.31 + 2.1 > 3.31 + 2.1 == 3.31 + 2.1 > 3.31 + 2.1 < 3.31 + + def calibrate(self): + + for i in xrange(self.rounds): + pass + + +class CompareFloatsIntegers(Test): + + version = 0.1 + operations = 30 * 5 + rounds = 60000 + + def test(self): + + for i in xrange(self.rounds): + + 2.1 < 4 + 2.1 > 4 + 2.1 == 4 + 2.1 > 4 + 2.1 < 4 + + 2.1 < 4 + 2.1 > 4 + 2.1 == 4 + 2.1 > 4 + 2.1 < 4 + + 2.1 < 4 + 2.1 > 4 + 2.1 == 4 + 2.1 > 4 + 2.1 < 4 + + 2.1 < 4 + 2.1 > 4 + 2.1 == 4 + 2.1 > 4 + 2.1 < 4 + + 2.1 < 4 + 2.1 > 4 + 2.1 == 4 + 2.1 > 4 + 2.1 < 4 + + 2.1 < 4 + 2.1 > 4 + 2.1 == 4 + 2.1 > 4 + 2.1 < 4 + + 2.1 < 4 + 2.1 > 4 + 2.1 == 4 + 2.1 > 4 + 2.1 < 4 + + 2.1 < 4 + 2.1 > 4 + 2.1 == 4 + 2.1 > 4 + 2.1 < 4 + + 2.1 < 4 + 2.1 > 4 + 2.1 == 4 + 2.1 > 4 + 2.1 < 4 + + 2.1 < 4 + 2.1 > 4 + 2.1 == 4 + 2.1 > 4 + 2.1 < 4 + + 2.1 < 4 + 2.1 > 4 + 2.1 == 4 + 2.1 > 4 + 2.1 < 4 + + 2.1 < 4 + 2.1 > 4 + 2.1 == 4 + 2.1 > 4 + 2.1 < 4 + + 2.1 < 4 + 2.1 > 4 + 2.1 == 4 + 2.1 > 4 + 2.1 < 4 + + 2.1 < 4 + 2.1 > 4 + 2.1 == 4 + 2.1 > 4 + 2.1 < 4 + + 2.1 < 4 + 2.1 > 4 + 2.1 == 4 + 2.1 > 4 + 2.1 < 4 + + 2.1 < 4 + 2.1 > 4 + 2.1 == 4 + 2.1 > 4 + 2.1 < 4 + + 2.1 < 4 + 2.1 > 4 + 2.1 == 4 + 2.1 > 4 + 2.1 < 4 + + 2.1 < 4 + 2.1 > 4 + 2.1 == 4 + 2.1 > 4 + 2.1 < 4 + + 2.1 < 4 + 2.1 > 4 + 2.1 == 4 + 2.1 > 4 + 2.1 < 4 + + 2.1 < 4 + 2.1 > 4 + 2.1 == 4 + 2.1 > 4 + 2.1 < 4 + + 2.1 < 4 + 2.1 > 4 + 2.1 == 4 + 2.1 > 4 + 2.1 < 4 + + 2.1 < 4 + 2.1 > 4 + 2.1 == 4 + 2.1 > 4 + 2.1 < 4 + + 2.1 < 4 + 2.1 > 4 + 2.1 == 4 + 2.1 > 4 + 2.1 < 4 + + 2.1 < 4 + 2.1 > 4 + 2.1 == 4 + 2.1 > 4 + 2.1 < 4 + + 2.1 < 4 + 2.1 > 4 + 2.1 == 4 + 2.1 > 4 + 2.1 < 4 + + 2.1 < 4 + 2.1 > 4 + 2.1 == 4 + 2.1 > 4 + 2.1 < 4 + + 2.1 < 4 + 2.1 > 4 + 2.1 == 4 + 2.1 > 4 + 2.1 < 4 + + 2.1 < 4 + 2.1 > 4 + 2.1 == 4 + 2.1 > 4 + 2.1 < 4 + + 2.1 < 4 + 2.1 > 4 + 2.1 == 4 + 2.1 > 4 + 2.1 < 4 + + 2.1 < 4 + 2.1 > 4 + 2.1 == 4 + 2.1 > 4 + 2.1 < 4 + + def calibrate(self): + + for i in xrange(self.rounds): + pass + + +class CompareLongs(Test): + + version = 0.1 + operations = 30 * 5 + rounds = 60000 + + def test(self): + + for i in xrange(self.rounds): + + 1234567890L < 3456789012345L + 1234567890L > 3456789012345L + 1234567890L == 3456789012345L + 1234567890L > 3456789012345L + 1234567890L < 3456789012345L + + 1234567890L < 3456789012345L + 1234567890L > 3456789012345L + 1234567890L == 3456789012345L + 1234567890L > 3456789012345L + 1234567890L < 3456789012345L + + 1234567890L < 3456789012345L + 1234567890L > 3456789012345L + 1234567890L == 3456789012345L + 1234567890L > 3456789012345L + 1234567890L < 3456789012345L + + 1234567890L < 3456789012345L + 1234567890L > 3456789012345L + 1234567890L == 3456789012345L + 1234567890L > 3456789012345L + 1234567890L < 3456789012345L + + 1234567890L < 3456789012345L + 1234567890L > 3456789012345L + 1234567890L == 3456789012345L + 1234567890L > 3456789012345L + 1234567890L < 3456789012345L + + 1234567890L < 3456789012345L + 1234567890L > 3456789012345L + 1234567890L == 3456789012345L + 1234567890L > 3456789012345L + 1234567890L < 3456789012345L + + 1234567890L < 3456789012345L + 1234567890L > 3456789012345L + 1234567890L == 3456789012345L + 1234567890L > 3456789012345L + 1234567890L < 3456789012345L + + 1234567890L < 3456789012345L + 1234567890L > 3456789012345L + 1234567890L == 3456789012345L + 1234567890L > 3456789012345L + 1234567890L < 3456789012345L + + 1234567890L < 3456789012345L + 1234567890L > 3456789012345L + 1234567890L == 3456789012345L + 1234567890L > 3456789012345L + 1234567890L < 3456789012345L + + 1234567890L < 3456789012345L + 1234567890L > 3456789012345L + 1234567890L == 3456789012345L + 1234567890L > 3456789012345L + 1234567890L < 3456789012345L + + 1234567890L < 3456789012345L + 1234567890L > 3456789012345L + 1234567890L == 3456789012345L + 1234567890L > 3456789012345L + 1234567890L < 3456789012345L + + 1234567890L < 3456789012345L + 1234567890L > 3456789012345L + 1234567890L == 3456789012345L + 1234567890L > 3456789012345L + 1234567890L < 3456789012345L + + 1234567890L < 3456789012345L + 1234567890L > 3456789012345L + 1234567890L == 3456789012345L + 1234567890L > 3456789012345L + 1234567890L < 3456789012345L + + 1234567890L < 3456789012345L + 1234567890L > 3456789012345L + 1234567890L == 3456789012345L + 1234567890L > 3456789012345L + 1234567890L < 3456789012345L + + 1234567890L < 3456789012345L + 1234567890L > 3456789012345L + 1234567890L == 3456789012345L + 1234567890L > 3456789012345L + 1234567890L < 3456789012345L + + 1234567890L < 3456789012345L + 1234567890L > 3456789012345L + 1234567890L == 3456789012345L + 1234567890L > 3456789012345L + 1234567890L < 3456789012345L + + 1234567890L < 3456789012345L + 1234567890L > 3456789012345L + 1234567890L == 3456789012345L + 1234567890L > 3456789012345L + 1234567890L < 3456789012345L + + 1234567890L < 3456789012345L + 1234567890L > 3456789012345L + 1234567890L == 3456789012345L + 1234567890L > 3456789012345L + 1234567890L < 3456789012345L + + 1234567890L < 3456789012345L + 1234567890L > 3456789012345L + 1234567890L == 3456789012345L + 1234567890L > 3456789012345L + 1234567890L < 3456789012345L + + 1234567890L < 3456789012345L + 1234567890L > 3456789012345L + 1234567890L == 3456789012345L + 1234567890L > 3456789012345L + 1234567890L < 3456789012345L + + 1234567890L < 3456789012345L + 1234567890L > 3456789012345L + 1234567890L == 3456789012345L + 1234567890L > 3456789012345L + 1234567890L < 3456789012345L + + 1234567890L < 3456789012345L + 1234567890L > 3456789012345L + 1234567890L == 3456789012345L + 1234567890L > 3456789012345L + 1234567890L < 3456789012345L + + 1234567890L < 3456789012345L + 1234567890L > 3456789012345L + 1234567890L == 3456789012345L + 1234567890L > 3456789012345L + 1234567890L < 3456789012345L + + 1234567890L < 3456789012345L + 1234567890L > 3456789012345L + 1234567890L == 3456789012345L + 1234567890L > 3456789012345L + 1234567890L < 3456789012345L + + 1234567890L < 3456789012345L + 1234567890L > 3456789012345L + 1234567890L == 3456789012345L + 1234567890L > 3456789012345L + 1234567890L < 3456789012345L + + 1234567890L < 3456789012345L + 1234567890L > 3456789012345L + 1234567890L == 3456789012345L + 1234567890L > 3456789012345L + 1234567890L < 3456789012345L + + 1234567890L < 3456789012345L + 1234567890L > 3456789012345L + 1234567890L == 3456789012345L + 1234567890L > 3456789012345L + 1234567890L < 3456789012345L + + 1234567890L < 3456789012345L + 1234567890L > 3456789012345L + 1234567890L == 3456789012345L + 1234567890L > 3456789012345L + 1234567890L < 3456789012345L + + 1234567890L < 3456789012345L + 1234567890L > 3456789012345L + 1234567890L == 3456789012345L + 1234567890L > 3456789012345L + 1234567890L < 3456789012345L + + 1234567890L < 3456789012345L + 1234567890L > 3456789012345L + 1234567890L == 3456789012345L + 1234567890L > 3456789012345L + 1234567890L < 3456789012345L + + def calibrate(self): + + for i in xrange(self.rounds): + pass Added: external/pybench-1.3/README ============================================================================== --- (empty file) +++ external/pybench-1.3/README Wed Apr 19 17:21:58 2006 @@ -0,0 +1,372 @@ +________________________________________________________________________ + +PYBENCH - A Python Benchmark Suite +________________________________________________________________________ + + Extendable suite of of low-level benchmarks for measuring + the performance of the Python implementation + (interpreter, compiler or VM). + +pybench is a collection of tests that provides a standardized way to +measure the performance of Python implementations. It takes a very +close look at different aspects of Python programs and let's you +decide which factors are more important to you than others, rather +than wrapping everything up in one number, like the other performance +tests do (e.g. pystone which is included in the Python Standard +Library). + +pybench has been used in the past by several Python developers to +track down performance bottlenecks or to demonstrate the impact of +optimizations and new features in Python. + +The command line interface for pybench is the file pybench.py. Run +this script with option '--help' to get a listing of the possible +options. Without options, pybench will simply execute the benchmark +and then print out a report to stdout. + + +Micro-Manual +------------ + +Run 'pybench.py -h' to see the help screen. +Run 'pybench.py' to just let the benchmark suite do it's thing and +'pybench.py -f ' to have it store the results in a file too. + +This is the current output of pybench.py --help: + +Synopsis: + pybench.py [option] files... + +Options and default settings: + -n arg number of rounds (10) + -f arg save benchmark to file arg () + -c arg compare benchmark with the one in file arg () + -s arg show benchmark in file arg, then exit () + -S show statistics of benchmarks (0) + -w arg set warp factor to arg (20) + -d hide noise in compares (0) + --no-gc disable garbage collection (0) + -v generate verbose output + -h show this help text + --help show this help text + --debug enable debugging + --copyright show copyright + --examples show examples of usage + +Version: + 1.3 + +The normal operation is to run the suite and display the +results. Use -f to save them for later reuse or comparisms. + +Examples: + +python1.5 pybench.py -w 100 -f p15 +python1.4 pybench.py -w 100 -f p14 +python pybench.py -s p15 -c p14 + + +License +------- + +See LICENSE file. + + +Sample output +------------- + +PYBENCH 1.3 + +Machine Details: + Platform ID: Linux-2.6.8-24.19-default-x86_64-with-SuSE-9.2-x86-64 + Executable: /home/lemburg/projects/Python/Installation/bin/python + Python: 2.5a1.0 + Compiler: GCC 3.3.4 (pre 3.3.5 20040809) + Build: Apr 9 2006 01:50:57 (#trunk) + +Searching for tests... + BuiltinFunctionCalls + BuiltinMethodLookup + CompareFloats + CompareFloatsIntegers + CompareIntegers + CompareInternedStrings + CompareLongs + CompareStrings + CompareUnicode + ConcatStrings + ConcatUnicode + CreateInstances + CreateStringsWithConcat + CreateUnicodeWithConcat + DictCreation + DictWithFloatKeys + DictWithIntegerKeys + DictWithStringKeys + ForLoops + IfThenElse + ListSlicing + NestedForLoops + NormalClassAttribute + NormalInstanceAttribute + PythonFunctionCalls + PythonMethodCalls + Recursion + SecondImport + SecondPackageImport + SecondSubmoduleImport + SimpleComplexArithmetic + SimpleDictManipulation + SimpleFloatArithmetic + SimpleIntFloatArithmetic + SimpleIntegerArithmetic + SimpleListManipulation + SimpleLongArithmetic + SmallLists + SmallTuples + SpecialClassAttribute + SpecialInstanceAttribute + StringMappings + StringPredicates + StringSlicing + TryExcept + TryRaiseExcept + TupleSlicing + UnicodeMappings + UnicodePredicates + UnicodeProperties + UnicodeSlicing + +Running 10 round(s) of the suite: + +... + + Round 10 real abs overhead + BuiltinFunctionCalls: 0.030r 0.030a 0.000o + BuiltinMethodLookup: 0.059r 0.060a 0.001o + CompareFloats: 0.050r 0.050a 0.000o + CompareFloatsIntegers: 0.050r 0.050a 0.000o + CompareIntegers: 0.070r 0.070a 0.000o + CompareInternedStrings: 0.039r 0.040a 0.001o + CompareLongs: 0.050r 0.050a 0.000o + CompareStrings: 0.060r 0.060a 0.000o + CompareUnicode: 0.060r 0.060a 0.000o + ConcatStrings: 0.040r 0.040a 0.000o + ConcatUnicode: 0.050r 0.050a 0.000o + CreateInstances: 0.050r 0.050a 0.000o + CreateStringsWithConcat: 0.029r 0.030a 0.001o + CreateUnicodeWithConcat: 0.060r 0.060a 0.000o + DictCreation: 0.040r 0.040a 0.000o + DictWithFloatKeys: 0.089r 0.090a 0.000o + DictWithIntegerKeys: 0.059r 0.060a 0.001o + DictWithStringKeys: 0.070r 0.070a 0.001o + ForLoops: 0.050r 0.050a 0.000o + IfThenElse: 0.070r 0.070a 0.000o + ListSlicing: 0.030r 0.030a 0.000o + NestedForLoops: 0.030r 0.030a 0.000o + NormalClassAttribute: 0.060r 0.060a 0.000o + NormalInstanceAttribute: 0.060r 0.060a 0.000o + PythonFunctionCalls: 0.060r 0.060a 0.000o + PythonMethodCalls: 0.050r 0.050a 0.000o + Recursion: 0.050r 0.050a 0.000o + SecondImport: 0.030r 0.030a 0.000o + SecondPackageImport: 0.030r 0.030a 0.000o + SecondSubmoduleImport: 0.040r 0.040a 0.000o + SimpleComplexArithmetic: 0.030r 0.030a 0.000o + SimpleDictManipulation: 0.040r 0.040a 0.000o + SimpleFloatArithmetic: 0.050r 0.050a 0.001o + SimpleIntFloatArithmetic: 0.060r 0.060a 0.000o + SimpleIntegerArithmetic: 0.060r 0.060a 0.000o + SimpleListManipulation: 0.030r 0.030a 0.000o + SimpleLongArithmetic: 0.030r 0.030a 0.000o + SmallLists: 0.050r 0.050a 0.000o + SmallTuples: 0.050r 0.050a 0.000o + SpecialClassAttribute: 0.060r 0.060a 0.000o + SpecialInstanceAttribute: 0.079r 0.080a 0.001o + StringMappings: 0.060r 0.060a 0.000o + StringPredicates: 0.049r 0.050a 0.001o + StringSlicing: 0.039r 0.040a 0.000o + TryExcept: 0.079r 0.080a 0.001o + TryRaiseExcept: 0.059r 0.060a 0.001o + TupleSlicing: 0.050r 0.050a 0.000o + UnicodeMappings: 0.070r 0.070a 0.001o + UnicodePredicates: 0.059r 0.060a 0.001o + UnicodeProperties: 0.059r 0.060a 0.001o + UnicodeSlicing: 0.050r 0.050a 0.000o + ---------------------- + Average round time: 2.937 seconds + + +Tests: per run per oper. overhead +------------------------------------------------------------------------ + BuiltinFunctionCalls: 29.85 ms 0.23 us 0.00 ms + BuiltinMethodLookup: 66.85 ms 0.13 us 0.50 ms + CompareFloats: 43.00 ms 0.10 us 0.00 ms + CompareFloatsIntegers: 51.80 ms 0.12 us 0.00 ms + CompareIntegers: 70.70 ms 0.08 us 0.50 ms + CompareInternedStrings: 41.40 ms 0.08 us 0.50 ms + CompareLongs: 47.90 ms 0.11 us 0.00 ms + CompareStrings: 58.50 ms 0.12 us 0.50 ms + CompareUnicode: 56.55 ms 0.15 us 0.50 ms + ConcatStrings: 44.75 ms 0.30 us 0.00 ms + ConcatUnicode: 54.55 ms 0.36 us 0.50 ms + CreateInstances: 50.95 ms 1.21 us 0.00 ms + CreateStringsWithConcat: 28.85 ms 0.14 us 0.50 ms + CreateUnicodeWithConcat: 53.75 ms 0.27 us 0.00 ms + DictCreation: 41.90 ms 0.28 us 0.00 ms + DictWithFloatKeys: 88.50 ms 0.15 us 0.50 ms + DictWithIntegerKeys: 62.55 ms 0.10 us 0.50 ms + DictWithStringKeys: 60.50 ms 0.10 us 0.50 ms + ForLoops: 46.90 ms 4.69 us 0.00 ms + IfThenElse: 60.55 ms 0.09 us 0.00 ms + ListSlicing: 29.90 ms 8.54 us 0.00 ms + NestedForLoops: 33.95 ms 0.10 us 0.00 ms + NormalClassAttribute: 62.75 ms 0.10 us 0.50 ms + NormalInstanceAttribute: 61.80 ms 0.10 us 0.50 ms + PythonFunctionCalls: 60.00 ms 0.36 us 0.00 ms + PythonMethodCalls: 50.00 ms 0.67 us 0.00 ms + Recursion: 46.85 ms 3.75 us 0.00 ms + SecondImport: 35.00 ms 1.40 us 0.00 ms + SecondPackageImport: 32.00 ms 1.28 us 0.00 ms + SecondSubmoduleImport: 38.00 ms 1.52 us 0.00 ms + SimpleComplexArithmetic: 26.85 ms 0.12 us 0.00 ms + SimpleDictManipulation: 40.85 ms 0.14 us 0.00 ms + SimpleFloatArithmetic: 48.70 ms 0.09 us 0.50 ms + SimpleIntFloatArithmetic: 57.70 ms 0.09 us 0.00 ms + SimpleIntegerArithmetic: 58.75 ms 0.09 us 0.50 ms + SimpleListManipulation: 34.80 ms 0.13 us 0.00 ms + SimpleLongArithmetic: 30.95 ms 0.19 us 0.50 ms + SmallLists: 47.60 ms 0.19 us 0.00 ms + SmallTuples: 48.80 ms 0.20 us 0.50 ms + SpecialClassAttribute: 61.70 ms 0.10 us 0.00 ms + SpecialInstanceAttribute: 76.70 ms 0.13 us 0.50 ms + StringMappings: 58.70 ms 0.47 us 0.00 ms + StringPredicates: 50.00 ms 0.18 us 1.00 ms + StringSlicing: 39.65 ms 0.23 us 0.50 ms + TryExcept: 84.45 ms 0.06 us 0.50 ms + TryRaiseExcept: 61.75 ms 4.12 us 0.50 ms + TupleSlicing: 48.95 ms 0.47 us 0.00 ms + UnicodeMappings: 71.50 ms 3.97 us 0.50 ms + UnicodePredicates: 52.75 ms 0.23 us 1.00 ms + UnicodeProperties: 61.90 ms 0.31 us 1.00 ms + UnicodeSlicing: 53.75 ms 0.31 us 0.50 ms +------------------------------------------------------------------------ + Average round time: 2937.00 ms + +________________________________________________________________________ + +Writing New Tests +________________________________________________________________________ + +pybench tests are simple modules defining one or more pybench.Test +subclasses. + +Writing a test essentially boils down to providing two methods: +.test() which runs .rounds number of .operations test operations each +and .calibrate() which does the same except that it doesn't actually +execute the operations. + + +Here's an example: +------------------ + +from pybench import Test + +class IntegerCounting(Test): + + # Version number of the test as float (x.yy); this is important + # for comparisons of benchmark runs - tests with unequal version + # number will not get compared. + version = 1.0 + + # The number of abstract operations done in each round of the + # test. An operation is the basic unit of what you want to + # measure. The benchmark will output the amount of run-time per + # operation. Note that in order to raise the measured timings + # significantly above noise level, it is often required to repeat + # sets of operations more than once per test round. The measured + # overhead per test round should be less than 1 second. + operations = 20 + + # Number of rounds to execute per test run. This should be + # adjusted to a figure that results in a test run-time of between + # 20-50 seconds. + rounds = 100000 + + def test(self): + + """ Run the test. + + The test needs to run self.rounds executing + self.operations number of operations each. + + """ + # Init the test + a = 1 + + # Run test rounds + # + # NOTE: Use xrange() for all test loops unless you want to face + # a 20MB process ! + # + for i in xrange(self.rounds): + + # Repeat the operations per round to raise the run-time + # per operation significantly above the noise level of the + # for-loop overhead. + + # Execute 20 operations (a += 1): + a += 1 + a += 1 + a += 1 + a += 1 + a += 1 + a += 1 + a += 1 + a += 1 + a += 1 + a += 1 + a += 1 + a += 1 + a += 1 + a += 1 + a += 1 + a += 1 + a += 1 + a += 1 + a += 1 + a += 1 + + def calibrate(self): + + """ Calibrate the test. + + This method should execute everything that is needed to + setup and run the test - except for the actual operations + that you intend to measure. pybench uses this method to + measure the test implementation overhead. + + """ + # Init the test + a = 1 + + # Run test rounds (without actually doing any operation) + for i in xrange(self.rounds): + + # Skip the actual execution of the operations, since we + # only want to measure the test's administration overhead. + pass + +Registering a new test module +----------------------------- + +To register a test module with pybench, the classes need to be +imported into the pybench.Setup module. pybench will then scan all the +symbols defined in that module for subclasses of pybench.Test and +automatically add them to the benchmark suite. + + +Have fun, +-- +Marc-Andre Lemburg +mal at lemburg.com Added: external/pybench-1.3/Setup.py ============================================================================== --- (empty file) +++ external/pybench-1.3/Setup.py Wed Apr 19 17:21:58 2006 @@ -0,0 +1,35 @@ +#!python + +# Setup file for pybench +# +# This file has to import all tests to be run; it is executed as +# Python source file, so you can do all kinds of manipulations here +# rather than having to edit the tests themselves. +# +# Note: Please keep this module compatible to Python 1.5.2. +# +# Tests may include features in later Python versions, but these +# should then be embedded in try-except clauses in this configuration +# module. + +# Defaults +Number_of_rounds = 10 +Warp_factor = 20 + +# Import tests +from Arithmetic import * +from Calls import * +from Constructs import * +from Lookups import * +from Instances import * +from Lists import * +from Tuples import * +from Dict import * +from Exceptions import * +from Imports import * +from Strings import * +from Numbers import * +try: + from Unicode import * +except (ImportError, SyntaxError): + pass Added: external/pybench-1.3/Strings.py ============================================================================== --- (empty file) +++ external/pybench-1.3/Strings.py Wed Apr 19 17:21:58 2006 @@ -0,0 +1,564 @@ +from pybench import Test +from string import join + +class ConcatStrings(Test): + + version = 0.1 + operations = 10 * 5 + rounds = 60000 + + def test(self): + + # Make sure the strings are *not* interned + s = join(map(str,range(100))) + t = join(map(str,range(1,101))) + + for i in xrange(self.rounds): + t + s + t + s + t + s + t + s + t + s + + t + s + t + s + t + s + t + s + t + s + + t + s + t + s + t + s + t + s + t + s + + t + s + t + s + t + s + t + s + t + s + + t + s + t + s + t + s + t + s + t + s + + t + s + t + s + t + s + t + s + t + s + + t + s + t + s + t + s + t + s + t + s + + t + s + t + s + t + s + t + s + t + s + + t + s + t + s + t + s + t + s + t + s + + t + s + t + s + t + s + t + s + t + s + + def calibrate(self): + + s = join(map(str,range(100))) + t = join(map(str,range(1,101))) + + for i in xrange(self.rounds): + pass + + +class CompareStrings(Test): + + version = 0.2 + operations = 10 * 5 + rounds = 200000 + + def test(self): + + # Make sure the strings are *not* interned + s = join(map(str,range(10))) + t = join(map(str,range(10))) + "abc" + + for i in xrange(self.rounds): + t < s + t > s + t == s + t > s + t < s + + t < s + t > s + t == s + t > s + t < s + + t < s + t > s + t == s + t > s + t < s + + t < s + t > s + t == s + t > s + t < s + + t < s + t > s + t == s + t > s + t < s + + t < s + t > s + t == s + t > s + t < s + + t < s + t > s + t == s + t > s + t < s + + t < s + t > s + t == s + t > s + t < s + + t < s + t > s + t == s + t > s + t < s + + t < s + t > s + t == s + t > s + t < s + + def calibrate(self): + + s = join(map(str,range(10))) + t = join(map(str,range(10))) + "abc" + + for i in xrange(self.rounds): + pass + + +class CompareInternedStrings(Test): + + version = 0.1 + operations = 10 * 5 + rounds = 200000 + + def test(self): + + # Make sure the strings *are* interned + s = intern(join(map(str,range(10)))) + t = s + + for i in xrange(self.rounds): + t == s + t == s + t >= s + t > s + t < s + + t == s + t == s + t >= s + t > s + t < s + + t == s + t == s + t >= s + t > s + t < s + + t == s + t == s + t >= s + t > s + t < s + + t == s + t == s + t >= s + t > s + t < s + + t == s + t == s + t >= s + t > s + t < s + + t == s + t == s + t >= s + t > s + t < s + + t == s + t == s + t >= s + t > s + t < s + + t == s + t == s + t >= s + t > s + t < s + + t == s + t == s + t >= s + t > s + t < s + + def calibrate(self): + + s = intern(join(map(str,range(10)))) + t = s + + for i in xrange(self.rounds): + pass + + +class CreateStringsWithConcat(Test): + + version = 0.1 + operations = 10 * 5 + rounds = 80000 + + def test(self): + + for i in xrange(self.rounds): + s = 'om' + s = s + 'xbx' + s = s + 'xcx' + s = s + 'xdx' + s = s + 'xex' + + s = s + 'xax' + s = s + 'xbx' + s = s + 'xcx' + s = s + 'xdx' + s = s + 'xex' + + s = s + 'xax' + s = s + 'xbx' + s = s + 'xcx' + s = s + 'xdx' + s = s + 'xex' + + s = s + 'xax' + s = s + 'xbx' + s = s + 'xcx' + s = s + 'xdx' + s = s + 'xex' + + s = s + 'xax' + s = s + 'xbx' + s = s + 'xcx' + s = s + 'xdx' + s = s + 'xex' + + s = s + 'xax' + s = s + 'xbx' + s = s + 'xcx' + s = s + 'xdx' + s = s + 'xex' + + s = s + 'xax' + s = s + 'xbx' + s = s + 'xcx' + s = s + 'xdx' + s = s + 'xex' + + s = s + 'xax' + s = s + 'xbx' + s = s + 'xcx' + s = s + 'xdx' + s = s + 'xex' + + s = s + 'xax' + s = s + 'xbx' + s = s + 'xcx' + s = s + 'xdx' + s = s + 'xex' + + s = s + 'xax' + s = s + 'xbx' + s = s + 'xcx' + s = s + 'xdx' + s = s + 'xex' + + def calibrate(self): + + for i in xrange(self.rounds): + pass + + +class StringSlicing(Test): + + version = 0.1 + operations = 5 * 7 + rounds = 100000 + + def test(self): + + s = join(map(str,range(100))) + + for i in xrange(self.rounds): + + s[50:] + s[:25] + s[50:55] + s[-1:] + s[:1] + s[2:] + s[11:-11] + + s[50:] + s[:25] + s[50:55] + s[-1:] + s[:1] + s[2:] + s[11:-11] + + s[50:] + s[:25] + s[50:55] + s[-1:] + s[:1] + s[2:] + s[11:-11] + + s[50:] + s[:25] + s[50:55] + s[-1:] + s[:1] + s[2:] + s[11:-11] + + s[50:] + s[:25] + s[50:55] + s[-1:] + s[:1] + s[2:] + s[11:-11] + + def calibrate(self): + + s = join(map(str,range(100))) + + for i in xrange(self.rounds): + pass + +### String methods + +if hasattr('', 'lower'): + + class StringMappings(Test): + + version = 0.1 + operations = 3 * (5 + 4 + 2 + 1) + rounds = 70000 + + def test(self): + + s = join(map(chr,range(20)),'') + t = join(map(chr,range(50)),'') + u = join(map(chr,range(100)),'') + v = join(map(chr,range(256)),'') + + for i in xrange(self.rounds): + + s.lower() + s.lower() + s.lower() + s.lower() + s.lower() + + s.upper() + s.upper() + s.upper() + s.upper() + s.upper() + + s.title() + s.title() + s.title() + s.title() + s.title() + + t.lower() + t.lower() + t.lower() + t.lower() + + t.upper() + t.upper() + t.upper() + t.upper() + + t.title() + t.title() + t.title() + t.title() + + u.lower() + u.lower() + + u.upper() + u.upper() + + u.title() + u.title() + + v.lower() + + v.upper() + + v.title() + + def calibrate(self): + + s = join(map(chr,range(20)),'') + t = join(map(chr,range(50)),'') + u = join(map(chr,range(100)),'') + v = join(map(chr,range(256)),'') + + for i in xrange(self.rounds): + pass + + class StringPredicates(Test): + + version = 0.1 + operations = 10 * 7 + rounds = 80000 + + def test(self): + + data = ('abc', '123', ' ', '\xe4\xf6\xfc', '\xdf'*10) + len_data = len(data) + + for i in xrange(self.rounds): + s = data[i % len_data] + + s.isalnum() + s.isalpha() + s.isdigit() + s.islower() + s.isspace() + s.istitle() + s.isupper() + + s.isalnum() + s.isalpha() + s.isdigit() + s.islower() + s.isspace() + s.istitle() + s.isupper() + + s.isalnum() + s.isalpha() + s.isdigit() + s.islower() + s.isspace() + s.istitle() + s.isupper() + + s.isalnum() + s.isalpha() + s.isdigit() + s.islower() + s.isspace() + s.istitle() + s.isupper() + + s.isalnum() + s.isalpha() + s.isdigit() + s.islower() + s.isspace() + s.istitle() + s.isupper() + + s.isalnum() + s.isalpha() + s.isdigit() + s.islower() + s.isspace() + s.istitle() + s.isupper() + + s.isalnum() + s.isalpha() + s.isdigit() + s.islower() + s.isspace() + s.istitle() + s.isupper() + + s.isalnum() + s.isalpha() + s.isdigit() + s.islower() + s.isspace() + s.istitle() + s.isupper() + + s.isalnum() + s.isalpha() + s.isdigit() + s.islower() + s.isspace() + s.istitle() + s.isupper() + + s.isalnum() + s.isalpha() + s.isdigit() + s.islower() + s.isspace() + s.istitle() + s.isupper() + + def calibrate(self): + + data = ('abc', '123', ' ', '\u1234\u2345\u3456', '\uFFFF'*10) + data = ('abc', '123', ' ', '\xe4\xf6\xfc', '\xdf'*10) + len_data = len(data) + + for i in xrange(self.rounds): + s = data[i % len_data] + + Added: external/pybench-1.3/Tuples.py ============================================================================== --- (empty file) +++ external/pybench-1.3/Tuples.py Wed Apr 19 17:21:58 2006 @@ -0,0 +1,365 @@ +from pybench import Test + +class TupleSlicing(Test): + + version = 0.31 + operations = 3 * 25 * 10 * 7 + rounds = 400 + + def test(self): + + r = range(25) + + for i in xrange(self.rounds): + + t = tuple(range(100)) + + for j in r: + + m = t[50:] + m = t[:25] + m = t[50:55] + m = t[:-1] + m = t[1:] + m = t[-10:] + m = t[:10] + + m = t[50:] + m = t[:25] + m = t[50:55] + m = t[:-1] + m = t[1:] + m = t[-10:] + m = t[:10] + + m = t[50:] + m = t[:25] + m = t[50:55] + m = t[:-1] + m = t[1:] + m = t[-10:] + m = t[:10] + + m = t[50:] + m = t[:25] + m = t[50:55] + m = t[:-1] + m = t[1:] + m = t[-10:] + m = t[:10] + + m = t[50:] + m = t[:25] + m = t[50:55] + m = t[:-1] + m = t[1:] + m = t[-10:] + m = t[:10] + + m = t[50:] + m = t[:25] + m = t[50:55] + m = t[:-1] + m = t[1:] + m = t[-10:] + m = t[:10] + + m = t[50:] + m = t[:25] + m = t[50:55] + m = t[:-1] + m = t[1:] + m = t[-10:] + m = t[:10] + + m = t[50:] + m = t[:25] + m = t[50:55] + m = t[:-1] + m = t[1:] + m = t[-10:] + m = t[:10] + + m = t[50:] + m = t[:25] + m = t[50:55] + m = t[:-1] + m = t[1:] + m = t[-10:] + m = t[:10] + + m = t[50:] + m = t[:25] + m = t[50:55] + m = t[:-1] + m = t[1:] + m = t[-10:] + m = t[:10] + + m = t[50:] + m = t[:25] + m = t[50:55] + m = t[:-1] + m = t[1:] + m = t[-10:] + m = t[:10] + + m = t[50:] + m = t[:25] + m = t[50:55] + m = t[:-1] + m = t[1:] + m = t[-10:] + m = t[:10] + + m = t[50:] + m = t[:25] + m = t[50:55] + m = t[:-1] + m = t[1:] + m = t[-10:] + m = t[:10] + + m = t[50:] + m = t[:25] + m = t[50:55] + m = t[:-1] + m = t[1:] + m = t[-10:] + m = t[:10] + + m = t[50:] + m = t[:25] + m = t[50:55] + m = t[:-1] + m = t[1:] + m = t[-10:] + m = t[:10] + + m = t[50:] + m = t[:25] + m = t[50:55] + m = t[:-1] + m = t[1:] + m = t[-10:] + m = t[:10] + + m = t[50:] + m = t[:25] + m = t[50:55] + m = t[:-1] + m = t[1:] + m = t[-10:] + m = t[:10] + + m = t[50:] + m = t[:25] + m = t[50:55] + m = t[:-1] + m = t[1:] + m = t[-10:] + m = t[:10] + + m = t[50:] + m = t[:25] + m = t[50:55] + m = t[:-1] + m = t[1:] + m = t[-10:] + m = t[:10] + + m = t[50:] + m = t[:25] + m = t[50:55] + m = t[:-1] + m = t[1:] + m = t[-10:] + m = t[:10] + + m = t[50:] + m = t[:25] + m = t[50:55] + m = t[:-1] + m = t[1:] + m = t[-10:] + m = t[:10] + + m = t[50:] + m = t[:25] + m = t[50:55] + m = t[:-1] + m = t[1:] + m = t[-10:] + m = t[:10] + + m = t[50:] + m = t[:25] + m = t[50:55] + m = t[:-1] + m = t[1:] + m = t[-10:] + m = t[:10] + + m = t[50:] + m = t[:25] + m = t[50:55] + m = t[:-1] + m = t[1:] + m = t[-10:] + m = t[:10] + + m = t[50:] + m = t[:25] + m = t[50:55] + m = t[:-1] + m = t[1:] + m = t[-10:] + m = t[:10] + + m = t[50:] + m = t[:25] + m = t[50:55] + m = t[:-1] + m = t[1:] + m = t[-10:] + m = t[:10] + + m = t[50:] + m = t[:25] + m = t[50:55] + m = t[:-1] + m = t[1:] + m = t[-10:] + m = t[:10] + + m = t[50:] + m = t[:25] + m = t[50:55] + m = t[:-1] + m = t[1:] + m = t[-10:] + m = t[:10] + + m = t[50:] + m = t[:25] + m = t[50:55] + m = t[:-1] + m = t[1:] + m = t[-10:] + m = t[:10] + + m = t[50:] + m = t[:25] + m = t[50:55] + m = t[:-1] + m = t[1:] + m = t[-10:] + m = t[:10] + + def calibrate(self): + + r = range(25) + + for i in xrange(self.rounds): + + t = tuple(range(100)) + + for j in r: + + pass + +class SmallTuples(Test): + + version = 0.3 + operations = 5*(1 + 3 + 6 + 2) + rounds = 80000 + + def test(self): + + for i in xrange(self.rounds): + + t = (1,2,3,4,5,6) + + a,b,c,d,e,f = t + a,b,c,d,e,f = t + a,b,c,d,e,f = t + + a,b,c = t[:3] + a,b,c = t[:3] + a,b,c = t[:3] + a,b,c = t[:3] + a,b,c = t[:3] + a,b,c = t[:3] + + l = list(t) + t = tuple(l) + + t = (1,2,3,4,5,6) + + a,b,c,d,e,f = t + a,b,c,d,e,f = t + a,b,c,d,e,f = t + + a,b,c = t[:3] + a,b,c = t[:3] + a,b,c = t[:3] + a,b,c = t[:3] + a,b,c = t[:3] + a,b,c = t[:3] + + l = list(t) + t = tuple(l) + + t = (1,2,3,4,5,6) + + a,b,c,d,e,f = t + a,b,c,d,e,f = t + a,b,c,d,e,f = t + + a,b,c = t[:3] + a,b,c = t[:3] + a,b,c = t[:3] + a,b,c = t[:3] + a,b,c = t[:3] + a,b,c = t[:3] + + l = list(t) + t = tuple(l) + + t = (1,2,3,4,5,6) + + a,b,c,d,e,f = t + a,b,c,d,e,f = t + a,b,c,d,e,f = t + + a,b,c = t[:3] + a,b,c = t[:3] + a,b,c = t[:3] + a,b,c = t[:3] + a,b,c = t[:3] + a,b,c = t[:3] + + l = list(t) + t = tuple(l) + + t = (1,2,3,4,5,6) + + a,b,c,d,e,f = t + a,b,c,d,e,f = t + a,b,c,d,e,f = t + + a,b,c = t[:3] + a,b,c = t[:3] + a,b,c = t[:3] + a,b,c = t[:3] + a,b,c = t[:3] + a,b,c = t[:3] + + l = list(t) + t = tuple(l) + + def calibrate(self): + + for i in xrange(self.rounds): + pass + Added: external/pybench-1.3/Unicode.py ============================================================================== --- (empty file) +++ external/pybench-1.3/Unicode.py Wed Apr 19 17:21:58 2006 @@ -0,0 +1,542 @@ +try: + unicode +except NameError: + raise ImportError + +from pybench import Test +from string import join + +class ConcatUnicode(Test): + + version = 0.1 + operations = 10 * 5 + rounds = 60000 + + def test(self): + + # Make sure the strings are *not* interned + s = unicode(join(map(str,range(100)))) + t = unicode(join(map(str,range(1,101)))) + + for i in xrange(self.rounds): + t + s + t + s + t + s + t + s + t + s + + t + s + t + s + t + s + t + s + t + s + + t + s + t + s + t + s + t + s + t + s + + t + s + t + s + t + s + t + s + t + s + + t + s + t + s + t + s + t + s + t + s + + t + s + t + s + t + s + t + s + t + s + + t + s + t + s + t + s + t + s + t + s + + t + s + t + s + t + s + t + s + t + s + + t + s + t + s + t + s + t + s + t + s + + t + s + t + s + t + s + t + s + t + s + + def calibrate(self): + + s = unicode(join(map(str,range(100)))) + t = unicode(join(map(str,range(1,101)))) + + for i in xrange(self.rounds): + pass + + +class CompareUnicode(Test): + + version = 0.1 + operations = 10 * 5 + rounds = 150000 + + def test(self): + + # Make sure the strings are *not* interned + s = unicode(join(map(str,range(10)))) + t = unicode(join(map(str,range(10))) + "abc") + + for i in xrange(self.rounds): + t < s + t > s + t == s + t > s + t < s + + t < s + t > s + t == s + t > s + t < s + + t < s + t > s + t == s + t > s + t < s + + t < s + t > s + t == s + t > s + t < s + + t < s + t > s + t == s + t > s + t < s + + t < s + t > s + t == s + t > s + t < s + + t < s + t > s + t == s + t > s + t < s + + t < s + t > s + t == s + t > s + t < s + + t < s + t > s + t == s + t > s + t < s + + t < s + t > s + t == s + t > s + t < s + + def calibrate(self): + + s = unicode(join(map(str,range(10)))) + t = unicode(join(map(str,range(10))) + "abc") + + for i in xrange(self.rounds): + pass + + +class CreateUnicodeWithConcat(Test): + + version = 0.1 + operations = 10 * 5 + rounds = 80000 + + def test(self): + + for i in xrange(self.rounds): + s = u'om' + s = s + u'xbx' + s = s + u'xcx' + s = s + u'xdx' + s = s + u'xex' + + s = s + u'xax' + s = s + u'xbx' + s = s + u'xcx' + s = s + u'xdx' + s = s + u'xex' + + s = s + u'xax' + s = s + u'xbx' + s = s + u'xcx' + s = s + u'xdx' + s = s + u'xex' + + s = s + u'xax' + s = s + u'xbx' + s = s + u'xcx' + s = s + u'xdx' + s = s + u'xex' + + s = s + u'xax' + s = s + u'xbx' + s = s + u'xcx' + s = s + u'xdx' + s = s + u'xex' + + s = s + u'xax' + s = s + u'xbx' + s = s + u'xcx' + s = s + u'xdx' + s = s + u'xex' + + s = s + u'xax' + s = s + u'xbx' + s = s + u'xcx' + s = s + u'xdx' + s = s + u'xex' + + s = s + u'xax' + s = s + u'xbx' + s = s + u'xcx' + s = s + u'xdx' + s = s + u'xex' + + s = s + u'xax' + s = s + u'xbx' + s = s + u'xcx' + s = s + u'xdx' + s = s + u'xex' + + s = s + u'xax' + s = s + u'xbx' + s = s + u'xcx' + s = s + u'xdx' + s = s + u'xex' + + def calibrate(self): + + for i in xrange(self.rounds): + pass + + +class UnicodeSlicing(Test): + + version = 0.1 + operations = 5 * 7 + rounds = 100000 + + def test(self): + + s = unicode(join(map(str,range(100)))) + + for i in xrange(self.rounds): + + s[50:] + s[:25] + s[50:55] + s[-1:] + s[:1] + s[2:] + s[11:-11] + + s[50:] + s[:25] + s[50:55] + s[-1:] + s[:1] + s[2:] + s[11:-11] + + s[50:] + s[:25] + s[50:55] + s[-1:] + s[:1] + s[2:] + s[11:-11] + + s[50:] + s[:25] + s[50:55] + s[-1:] + s[:1] + s[2:] + s[11:-11] + + s[50:] + s[:25] + s[50:55] + s[-1:] + s[:1] + s[2:] + s[11:-11] + + def calibrate(self): + + s = unicode(join(map(str,range(100)))) + + for i in xrange(self.rounds): + pass + +### String methods + +class UnicodeMappings(Test): + + version = 0.1 + operations = 3 * (5 + 4 + 2 + 1) + rounds = 10000 + + def test(self): + + s = join(map(unichr,range(20)),'') + t = join(map(unichr,range(100)),'') + u = join(map(unichr,range(500)),'') + v = join(map(unichr,range(1000)),'') + + for i in xrange(self.rounds): + + s.lower() + s.lower() + s.lower() + s.lower() + s.lower() + + s.upper() + s.upper() + s.upper() + s.upper() + s.upper() + + s.title() + s.title() + s.title() + s.title() + s.title() + + t.lower() + t.lower() + t.lower() + t.lower() + + t.upper() + t.upper() + t.upper() + t.upper() + + t.title() + t.title() + t.title() + t.title() + + u.lower() + u.lower() + + u.upper() + u.upper() + + u.title() + u.title() + + v.lower() + + v.upper() + + v.title() + + def calibrate(self): + + s = join(map(unichr,range(20)),'') + t = join(map(unichr,range(100)),'') + u = join(map(unichr,range(500)),'') + v = join(map(unichr,range(1000)),'') + + for i in xrange(self.rounds): + pass + +class UnicodePredicates(Test): + + version = 0.1 + operations = 5 * 9 + rounds = 100000 + + def test(self): + + data = (u'abc', u'123', u' ', u'\u1234\u2345\u3456', u'\uFFFF'*10) + len_data = len(data) + + for i in xrange(self.rounds): + s = data[i % len_data] + + s.isalnum() + s.isalpha() + s.isdecimal() + s.isdigit() + s.islower() + s.isnumeric() + s.isspace() + s.istitle() + s.isupper() + + s.isalnum() + s.isalpha() + s.isdecimal() + s.isdigit() + s.islower() + s.isnumeric() + s.isspace() + s.istitle() + s.isupper() + + s.isalnum() + s.isalpha() + s.isdecimal() + s.isdigit() + s.islower() + s.isnumeric() + s.isspace() + s.istitle() + s.isupper() + + s.isalnum() + s.isalpha() + s.isdecimal() + s.isdigit() + s.islower() + s.isnumeric() + s.isspace() + s.istitle() + s.isupper() + + s.isalnum() + s.isalpha() + s.isdecimal() + s.isdigit() + s.islower() + s.isnumeric() + s.isspace() + s.istitle() + s.isupper() + + def calibrate(self): + + data = (u'abc', u'123', u' ', u'\u1234\u2345\u3456', u'\uFFFF'*10) + len_data = len(data) + + for i in xrange(self.rounds): + s = data[i % len_data] + +try: + import unicodedata +except ImportError: + pass +else: + class UnicodeProperties(Test): + + version = 0.1 + operations = 5 * 8 + rounds = 100000 + + def test(self): + + data = (u'a', u'1', u' ', u'\u1234', u'\uFFFF') + len_data = len(data) + digit = unicodedata.digit + numeric = unicodedata.numeric + decimal = unicodedata.decimal + category = unicodedata.category + bidirectional = unicodedata.bidirectional + decomposition = unicodedata.decomposition + mirrored = unicodedata.mirrored + combining = unicodedata.combining + + for i in xrange(self.rounds): + + c = data[i % len_data] + + digit(c, None) + numeric(c, None) + decimal(c, None) + category(c) + bidirectional(c) + decomposition(c) + mirrored(c) + combining(c) + + digit(c, None) + numeric(c, None) + decimal(c, None) + category(c) + bidirectional(c) + decomposition(c) + mirrored(c) + combining(c) + + digit(c, None) + numeric(c, None) + decimal(c, None) + category(c) + bidirectional(c) + decomposition(c) + mirrored(c) + combining(c) + + digit(c, None) + numeric(c, None) + decimal(c, None) + category(c) + bidirectional(c) + decomposition(c) + mirrored(c) + combining(c) + + digit(c, None) + numeric(c, None) + decimal(c, None) + category(c) + bidirectional(c) + decomposition(c) + mirrored(c) + combining(c) + + def calibrate(self): + + data = (u'a', u'1', u' ', u'\u1234', u'\uFFFF') + len_data = len(data) + digit = unicodedata.digit + numeric = unicodedata.numeric + decimal = unicodedata.decimal + category = unicodedata.category + bidirectional = unicodedata.bidirectional + decomposition = unicodedata.decomposition + mirrored = unicodedata.mirrored + combining = unicodedata.combining + + for i in xrange(self.rounds): + + c = data[i % len_data] Added: external/pybench-1.3/package/__init__.py ============================================================================== Added: external/pybench-1.3/package/submodule.py ============================================================================== Added: external/pybench-1.3/pybench.py ============================================================================== --- (empty file) +++ external/pybench-1.3/pybench.py Wed Apr 19 17:21:58 2006 @@ -0,0 +1,461 @@ +#!/usr/local/bin/python -O + +""" A Python Benchmark Suite + +""" +# +# Note: Please keep this module compatible to Python 1.5.2. +# +# Tests may include features in later Python versions, but these +# should then be embedded in try-except clauses in the configuration +# module Setup.py. +# + +# pybench Copyright +__copyright__ = """\ +Copyright (c), 1997-2006, Marc-Andre Lemburg (mal at lemburg.com) +Copyright (c), 2000-2006, eGenix.com Software GmbH (info at egenix.com) + + All Rights Reserved. + +Permission to use, copy, modify, and distribute this software and its +documentation for any purpose and without fee or royalty is hereby +granted, provided that the above copyright notice appear in all copies +and that both that copyright notice and this permission notice appear +in supporting documentation or portions thereof, including +modifications, that you make. + +THE AUTHOR MARC-ANDRE LEMBURG DISCLAIMS ALL WARRANTIES WITH REGARD TO +THIS SOFTWARE, INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY AND +FITNESS, IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY SPECIAL, +INDIRECT OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING +FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, +NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION +WITH THE USE OR PERFORMANCE OF THIS SOFTWARE ! +""" + +# Version number +__version__ = '1.3' + +# +# NOTE: Use xrange for all test loops unless you want to face +# a 20MB process ! +# +# All tests should have rounds set to values so that a run() +# takes between 20-50 seconds. This is to get fairly good +# clock() values. You can use option -w to speedup the tests +# by a fixed integer factor (the "warp factor"). +# + +import sys,time,operator +from CommandLine import * + +try: + import cPickle + pickle = cPickle +except ImportError: + import pickle + +### Test baseclass + +class Test: + + """ All test must have this class as baseclass. It provides + the necessary interface to the benchmark machinery. + + The tests must set .rounds to a value high enough to let the + test run between 20-50 seconds. This is needed because + clock()-timing only gives rather inaccurate values (on Linux, + for example, it is accurate to a few hundreths of a + second). If you don't want to wait that long, use a warp + factor larger than 1. + + It is also important to set the .operations variable to a + value representing the number of "virtual operations" done per + call of .run(). + + If you change a test in some way, don't forget to increase + it's version number. + + """ + + ### Instance variables that each test should override + + # Version number of the test as float (x.yy); this is important + # for comparisons of benchmark runs - tests with unequal version + # number will not get compared. + version = 1.0 + + # The number of abstract operations done in each round of the + # test. An operation is the basic unit of what you want to + # measure. The benchmark will output the amount of run-time per + # operation. Note that in order to raise the measured timings + # significantly above noise level, it is often required to repeat + # sets of operations more than once per test round. The measured + # overhead per test round should be less than 1 second. + operations = 1 + + # Number of rounds to execute per test run. This should be + # adjusted to a figure that results in a test run-time of between + # 20-50 seconds. + rounds = 100000 + + ### Internal variables + + # Mark this class as implementing a test + is_a_test = 1 + + # Misc. internal variables + last_timing = (0,0,0) # last timing (real,run,calibration) + warp = 1 # warp factor this test uses + cruns = 20 # number of calibration runs + overhead = None # list of calibration timings + + def __init__(self,warp=1): + + if warp > 1: + self.rounds = self.rounds / warp + self.warp = warp + self.times = [] + self.overhead = [] + # We want these to be in the instance dict, so that pickle + # saves them + self.version = self.version + self.operations = self.operations + self.rounds = self.rounds + + def run(self): + + """ Run the test in two phases: first calibrate, then + do the actual test. Be careful to keep the calibration + timing low w/r to the test timing. + + """ + test = self.test + calibrate = self.calibrate + clock = time.clock + cruns = self.cruns + # first calibrate + offset = 0.0 + for i in range(cruns): + t = clock() + calibrate() + t = clock() - t + offset = offset + t + offset = offset / cruns + # now the real thing + t = clock() + test() + t = clock() - t + self.last_timing = (t-offset,t,offset) + self.times.append(t-offset) + + def calibrate(self): + + """ Calibrate the test. + + This method should execute everything that is needed to + setup and run the test - except for the actual operations + that you intend to measure. pybench uses this method to + measure the test implementation overhead. + + """ + return + + def test(self): + + """ Run the test. + + The test needs to run self.rounds executing + self.operations number of operations each. + + """ + # do some tests + return + + def stat(self): + + """ Returns two value: average time per run and average per + operation. + + """ + runs = len(self.times) + if runs == 0: + return 0,0 + totaltime = reduce(operator.add,self.times,0.0) + avg = totaltime / float(runs) + op_avg = totaltime / float(runs * self.rounds * self.operations) + if self.overhead: + totaloverhead = reduce(operator.add,self.overhead,0.0) + ov_avg = totaloverhead / float(runs) + else: + # use self.last_timing - not too accurate + ov_avg = self.last_timing[2] + return avg,op_avg,ov_avg + +### Load Setup + +# This has to be done after the definition of the Test class, since +# the Setup module will import subclasses using this class. + +import Setup + +### Benchmark base class + +class Benchmark: + + name = '?' # Name of the benchmark + rounds = 1 # Number of rounds to run + warp = 1 # Warp factor + roundtime = 0 # Average round time + version = None # Benchmark version number (see __init__) + # as float x.yy + starttime = None # Benchmark start time + + def __init__(self): + + self.tests = {} + self.version = 0.31 + + def load_tests(self,setupmod,warp=1): + + self.warp = warp + tests = self.tests + print 'Searching for tests...' + setupmod.__dict__.values() + for c in setupmod.__dict__.values(): + if hasattr(c,'is_a_test') and c.__name__ != 'Test': + tests[c.__name__] = c(warp) + l = tests.keys() + l.sort() + for t in l: + print ' ',t + print + + def run(self): + + tests = self.tests.items() + tests.sort() + clock = time.clock + print 'Running %i round(s) of the suite: ' % self.rounds + print + self.starttime = time.time() + roundtime = clock() + for i in range(self.rounds): + print ' Round %-25i real abs overhead' % (i+1) + for j in range(len(tests)): + name,t = tests[j] + print '%30s:' % name, + t.run() + print ' %.3fr %.3fa %.3fo' % t.last_timing + print ' ----------------------' + print ' Average round time: %.3f seconds' % \ + ((clock() - roundtime)/(i+1)) + print + self.roundtime = (clock() - roundtime) / self.rounds + print + + def print_stat(self, compare_to=None, hidenoise=0): + + if not compare_to: + print '%-30s per run per oper. overhead' % 'Tests:' + print '-'*72 + tests = self.tests.items() + tests.sort() + for name,t in tests: + avg,op_avg,ov_avg = t.stat() + print '%30s: %10.2f ms %7.2f us %7.2f ms' % \ + (name,avg*1000.0,op_avg*1000000.0,ov_avg*1000.0) + print '-'*72 + print '%30s: %10.2f ms' % \ + ('Average round time',self.roundtime * 1000.0) + + else: + print '%-30s per run per oper. diff *)' % \ + 'Tests:' + print '-'*72 + tests = self.tests.items() + tests.sort() + compatible = 1 + for name,t in tests: + avg,op_avg,ov_avg = t.stat() + try: + other = compare_to.tests[name] + except KeyError: + other = None + if other and other.version == t.version and \ + other.operations == t.operations: + avg1,op_avg1,ov_avg1 = other.stat() + qop_avg = (op_avg/op_avg1-1.0)*100.0 + if hidenoise and abs(qop_avg) < 10: + qop_avg = '' + else: + qop_avg = '%+7.2f%%' % qop_avg + else: + qavg,qop_avg = 'n/a', 'n/a' + compatible = 0 + print '%30s: %10.2f ms %7.2f us %8s' % \ + (name,avg*1000.0,op_avg*1000000.0,qop_avg) + print '-'*72 + if compatible and compare_to.roundtime > 0 and \ + compare_to.version == self.version: + print '%30s: %10.2f ms %+7.2f%%' % \ + ('Average round time',self.roundtime * 1000.0, + ((self.roundtime*self.warp)/ + (compare_to.roundtime*compare_to.warp)-1.0)*100.0) + else: + print '%30s: %10.2f ms n/a' % \ + ('Average round time',self.roundtime * 1000.0) + print + print '*) measured against: %s (rounds=%i, warp=%i)' % \ + (compare_to.name,compare_to.rounds,compare_to.warp) + print + +def print_machine(): + + import platform + print 'Machine Details:' + print ' Platform ID: %s' % platform.platform() + print ' Executable: %s' % sys.executable + # There's a bug in Python 2.2b1+... + if sys.version[:6] == '2.2b1+': + return + print ' Python: %s' % platform.python_version() + print ' Compiler: %s' % platform.python_compiler() + buildno, builddate = platform.python_build() + print ' Build: %s (#%s)' % (builddate, buildno) + +class PyBenchCmdline(Application): + + header = ("PYBENCH - a benchmark test suite for Python " + "interpreters/compilers.") + + version = __version__ + + options = [ArgumentOption('-n','number of rounds',Setup.Number_of_rounds), + ArgumentOption('-f','save benchmark to file arg',''), + ArgumentOption('-c','compare benchmark with the one in file arg',''), + ArgumentOption('-s','show benchmark in file arg, then exit',''), + SwitchOption('-S','show statistics of benchmarks',0), + ArgumentOption('-w','set warp factor to arg',Setup.Warp_factor), + SwitchOption('-d','hide noise in compares', 0), + SwitchOption('--no-gc','disable garbage collection', 0), + ] + + about = """\ +The normal operation is to run the suite and display the +results. Use -f to save them for later reuse or comparisms. + +Examples: + +python1.5 pybench.py -w 100 -f p15 +python1.4 pybench.py -w 100 -f p14 +python pybench.py -s p15 -c p14 +""" + copyright = __copyright__ + + def handle_S(self, value): + + """ Display one line stats for each benchmark file given on the + command line. + + """ + for benchmark in self.files: + try: + f = open(benchmark, 'rb') + bench = pickle.load(f) + f.close() + except IOError: + print '* Error opening/reading file %s' % repr(benchmark) + else: + print '%s,%-.2f,ms' % (benchmark, bench.roundtime*1000.0) + return 0 + + def main(self): + + rounds = self.values['-n'] + reportfile = self.values['-f'] + show_bench = self.values['-s'] + compare_to = self.values['-c'] + hidenoise = self.values['-d'] + warp = self.values['-w'] + nogc = self.values['--no-gc'] + + # Switch off GC + if nogc: + try: + import gc + except ImportError: + nogc = 0 + else: + if self.values['--no-gc']: + gc.disable() + + print 'PYBENCH',__version__ + print + + if not compare_to: + print_machine() + print + + if compare_to: + try: + f = open(compare_to,'rb') + bench = pickle.load(f) + bench.name = compare_to + f.close() + compare_to = bench + except IOError: + print '* Error opening/reading file',compare_to + compare_to = None + + if show_bench: + try: + f = open(show_bench,'rb') + bench = pickle.load(f) + bench.name = show_bench + f.close() + print 'Benchmark: %s (rounds=%i, warp=%i)' % \ + (bench.name,bench.rounds,bench.warp) + print + bench.print_stat(compare_to, hidenoise) + except IOError: + print '* Error opening/reading file',show_bench + print + return + + if reportfile: + if nogc: + print 'Benchmark: %s (rounds=%i, warp=%i, no GC)' % \ + (reportfile,rounds,warp) + else: + print 'Benchmark: %s (rounds=%i, warp=%i)' % \ + (reportfile,rounds,warp) + print + + # Create benchmark object + bench = Benchmark() + bench.rounds = rounds + bench.load_tests(Setup,warp) + try: + bench.run() + except KeyboardInterrupt: + print + print '*** KeyboardInterrupt -- Aborting' + print + return + bench.print_stat(compare_to) + # ring bell + sys.stderr.write('\007') + + if reportfile: + try: + f = open(reportfile,'wb') + bench.name = reportfile + pickle.dump(bench,f) + f.close() + except IOError: + print '* Error opening/writing reportfile' + +if __name__ == '__main__': + PyBenchCmdline() From python-checkins at python.org Wed Apr 19 17:27:34 2006 From: python-checkins at python.org (marc-andre.lemburg) Date: Wed, 19 Apr 2006 17:27:34 +0200 (CEST) Subject: [Python-checkins] r45566 - python/trunk/Tools/pybench Message-ID: <20060419152734.DD8D31E4047@bag.python.org> Author: marc-andre.lemburg Date: Wed Apr 19 17:27:33 2006 New Revision: 45566 Added: python/trunk/Tools/pybench/ - copied from r45565, external/pybench-1.3/ Log: Adding pybench 1.3 to the Tools/ directory. From buildbot at python.org Wed Apr 19 17:29:21 2006 From: buildbot at python.org (buildbot at python.org) Date: Wed, 19 Apr 2006 15:29:21 +0000 Subject: [Python-checkins] buildbot warnings in sparc solaris10 gcc trunk Message-ID: <20060419152921.5072B1E4004@bag.python.org> The Buildbot has detected a new failure of sparc solaris10 gcc trunk. Full details are available at: http://www.python.org/dev/buildbot/all/sparc%2520solaris10%2520gcc%2520trunk/builds/491 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: thomas.wouters Build Had Warnings: warnings test sincerely, -The Buildbot From python-checkins at python.org Wed Apr 19 17:38:03 2006 From: python-checkins at python.org (thomas.wouters) Date: Wed, 19 Apr 2006 17:38:03 +0200 (CEST) Subject: [Python-checkins] r45567 - python/trunk/Objects/stringobject.c Message-ID: <20060419153803.49ECC1E4004@bag.python.org> Author: thomas.wouters Date: Wed Apr 19 17:38:01 2006 New Revision: 45567 Modified: python/trunk/Objects/stringobject.c Log: Make s.replace() work with explicit counts exceeding 2Gb. Modified: python/trunk/Objects/stringobject.c ============================================================================== --- python/trunk/Objects/stringobject.c (original) +++ python/trunk/Objects/stringobject.c Wed Apr 19 17:38:01 2006 @@ -2524,11 +2524,11 @@ char *new_s; const Py_ssize_t len = PyString_GET_SIZE(self); Py_ssize_t sub_len, repl_len, out_len; - int count = -1; + Py_ssize_t count = -1; PyObject *newobj; PyObject *subobj, *replobj; - if (!PyArg_ParseTuple(args, "OO|i:replace", + if (!PyArg_ParseTuple(args, "OO|n:replace", &subobj, &replobj, &count)) return NULL; From python-checkins at python.org Wed Apr 19 17:49:01 2006 From: python-checkins at python.org (marc-andre.lemburg) Date: Wed, 19 Apr 2006 17:49:01 +0200 (CEST) Subject: [Python-checkins] r45568 - python/trunk/Misc/NEWS Message-ID: <20060419154901.39ABA1E4004@bag.python.org> Author: marc-andre.lemburg Date: Wed Apr 19 17:48:59 2006 New Revision: 45568 Modified: python/trunk/Misc/NEWS Log: Add news item for pybench addition. Modified: python/trunk/Misc/NEWS ============================================================================== --- python/trunk/Misc/NEWS (original) +++ python/trunk/Misc/NEWS Wed Apr 19 17:48:59 2006 @@ -147,6 +147,11 @@ it separately and by hand. It also wasn't cleaning up its changes to the current Decimal context. +Tools +----- + +- Added the Python benchmark suite pybench to the Tools/ directory; + contributed by Marc-Andre Lemburg. Documentation ------------- From buildbot at python.org Wed Apr 19 18:00:22 2006 From: buildbot at python.org (buildbot at python.org) Date: Wed, 19 Apr 2006 16:00:22 +0000 Subject: [Python-checkins] buildbot warnings in x86 Ubuntu dapper (icc) trunk Message-ID: <20060419160022.D1B101E401A@bag.python.org> The Buildbot has detected a new failure of x86 Ubuntu dapper (icc) trunk. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520Ubuntu%2520dapper%2520%2528icc%2529%2520trunk/builds/197 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: thomas.wouters Build Had Warnings: warnings test sincerely, -The Buildbot From buildbot at python.org Wed Apr 19 18:50:09 2006 From: buildbot at python.org (buildbot at python.org) Date: Wed, 19 Apr 2006 16:50:09 +0000 Subject: [Python-checkins] buildbot warnings in x86 cygwin trunk Message-ID: <20060419165009.BCB591E400E@bag.python.org> The Buildbot has detected a new failure of x86 cygwin trunk. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520cygwin%2520trunk/builds/225 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: thomas.wouters Build Had Warnings: warnings test sincerely, -The Buildbot From python-checkins at python.org Wed Apr 19 20:34:17 2006 From: python-checkins at python.org (andrew.kuchling) Date: Wed, 19 Apr 2006 20:34:17 +0200 (CEST) Subject: [Python-checkins] r45569 - peps/trunk/pep-0042.txt Message-ID: <20060419183417.DADCA1E4004@bag.python.org> Author: andrew.kuchling Date: Wed Apr 19 20:34:16 2006 New Revision: 45569 Modified: peps/trunk/pep-0042.txt Log: Update one item; should it just be removed? Modified: peps/trunk/pep-0042.txt ============================================================================== --- peps/trunk/pep-0042.txt (original) +++ peps/trunk/pep-0042.txt Wed Apr 19 20:34:16 2006 @@ -280,7 +280,7 @@ http://www.python.org/sf/558745 - Make 'as' a keyword. It has been a pseudo-keyword long enough. - Too much effort to bother. + (It's deprecated in 2.5, and will become a keyword in 2.6.) C API wishes From python-checkins at python.org Wed Apr 19 20:35:24 2006 From: python-checkins at python.org (andrew.kuchling) Date: Wed, 19 Apr 2006 20:35:24 +0200 (CEST) Subject: [Python-checkins] r45570 - peps/trunk/pep-0042.txt Message-ID: <20060419183524.D1E311E4004@bag.python.org> Author: andrew.kuchling Date: Wed Apr 19 20:35:24 2006 New Revision: 45570 Modified: peps/trunk/pep-0042.txt Log: Typo fix Modified: peps/trunk/pep-0042.txt ============================================================================== --- peps/trunk/pep-0042.txt (original) +++ peps/trunk/pep-0042.txt Wed Apr 19 20:35:24 2006 @@ -91,7 +91,7 @@ Standard Library - The urllib module should support proxies which require - authenication. See SourceForge bug #210619 for information: + authentication. See SourceForge bug #210619 for information: http://www.python.org/sf/210619 From neal at metaslash.com Wed Apr 19 23:06:42 2006 From: neal at metaslash.com (Neal Norwitz) Date: Wed, 19 Apr 2006 17:06:42 -0400 Subject: [Python-checkins] Python Regression Test Failures refleak (2) Message-ID: <20060419210642.GA20745@python.psfb.org> test_cmd_line leaked [0, -17, 17] references test_threaded_import leaked [0, 0, 11] references test_threadedtempfile leaked [0, 0, -11] references test_threading_local leaked [-93, 0, 0] references test_urllib2 leaked [-121, 88, 99] references From python-checkins at python.org Wed Apr 19 23:56:53 2006 From: python-checkins at python.org (guido.van.rossum) Date: Wed, 19 Apr 2006 23:56:53 +0200 (CEST) Subject: [Python-checkins] r45571 - peps/trunk/pep-3100.txt Message-ID: <20060419215653.9BD131E402E@bag.python.org> Author: guido.van.rossum Date: Wed Apr 19 23:56:53 2006 New Revision: 45571 Modified: peps/trunk/pep-3100.txt Log: Add making ... a general expression. Modified: peps/trunk/pep-3100.txt ============================================================================== --- peps/trunk/pep-3100.txt (original) +++ peps/trunk/pep-3100.txt Wed Apr 19 23:56:53 2006 @@ -59,10 +59,11 @@ * Support only new-style classes; classic classes will be gone [1]_ * Replace ``print`` by a function [16]_ * Do something so you can catch multiple exceptions using ``except E1, - E2, E3:``. Maybe use ``except E1, E2, E3 as err:`` if you want the + E2:``. Probably use ``except E1, E2, E3 as err:`` if you want the error variable? [3]_ * ``None``, ``True`` and ``False`` become keywords [4]_ (Or perhaps just ``None``?) +* ``...`` to become a general expression element [24]_ * ``as`` becomes a keyword [5]_ (probably in 2.6 already) * Have list comprehensions be syntactic sugar for passing an equivalent generator expression to ``list()``; as a consequence the @@ -284,6 +285,9 @@ .. [23] PEP 308 ("Conditional Expressions") http://www.python.org/dev/peps/pep-0308 +.. [24] python-3000 email + http://mail.python.org/pipermail/python-3000/2006-April/000996.html + .. [#pep238] PEP 238 (Changing the Division Operator) http://www.python.org/dev/peps/pep-0238 From python-checkins at python.org Thu Apr 20 00:38:34 2006 From: python-checkins at python.org (brett.cannon) Date: Thu, 20 Apr 2006 00:38:34 +0200 (CEST) Subject: [Python-checkins] r45572 - peps/trunk/pep-3100.txt Message-ID: <20060419223834.7443E1E4022@bag.python.org> Author: brett.cannon Date: Thu Apr 20 00:38:33 2006 New Revision: 45572 Modified: peps/trunk/pep-3100.txt Log: Make ellipsis a legal expression in places other than in slices. Modified: peps/trunk/pep-3100.txt ============================================================================== --- peps/trunk/pep-3100.txt (original) +++ peps/trunk/pep-3100.txt Thu Apr 20 00:38:33 2006 @@ -92,6 +92,7 @@ * __builtins__ should get a different name *or* completely unified with __builtin__. Keeping both with confusingly similar spellings and semantics is evil. +* Make ellipsis (``...``) a legal expression outside of slices [25]_ To be removed: @@ -288,6 +289,9 @@ .. [24] python-3000 email http://mail.python.org/pipermail/python-3000/2006-April/000996.html +.. [25] python-3000 email ("Type Expressions") + http://mail.python.org/pipermail/python-3000/2006-April/000996.html + .. [#pep238] PEP 238 (Changing the Division Operator) http://www.python.org/dev/peps/pep-0238 From python-checkins at python.org Thu Apr 20 03:29:50 2006 From: python-checkins at python.org (skip.montanaro) Date: Thu, 20 Apr 2006 03:29:50 +0200 (CEST) Subject: [Python-checkins] r45573 - in python/trunk: Doc/lib/libos.tex Lib/test/test_posix.py Modules/posixmodule.c Message-ID: <20060420012950.B820D1E4004@bag.python.org> Author: skip.montanaro Date: Thu Apr 20 03:29:48 2006 New Revision: 45573 Modified: python/trunk/Doc/lib/libos.tex python/trunk/Lib/test/test_posix.py python/trunk/Modules/posixmodule.c Log: Correct implementation and documentation of os.confstr. Add a simple test case. I've yet to figure out how to provoke a None return I can test. Modified: python/trunk/Doc/lib/libos.tex ============================================================================== --- python/trunk/Doc/lib/libos.tex (original) +++ python/trunk/Doc/lib/libos.tex Thu Apr 20 03:29:48 2006 @@ -1844,14 +1844,14 @@ string which is the name of a defined system value; these names are specified in a number of standards (\POSIX, \UNIX{} 95, \UNIX{} 98, and others). Some platforms define additional names as well. The names -known to the host operating system are given in the +known to the host operating system are given as the keys of the \code{confstr_names} dictionary. For configuration variables not included in that mapping, passing an integer for \var{name} is also accepted. Availability: Macintosh, \UNIX. -If the configuration value specified by \var{name} isn't defined, the -empty string is returned. +If the configuration value specified by \var{name} isn't defined, +\code{None} is returned. If \var{name} is a string and is not known, \exception{ValueError} is raised. If a specific value for \var{name} is not supported by the Modified: python/trunk/Lib/test/test_posix.py ============================================================================== --- python/trunk/Lib/test/test_posix.py (original) +++ python/trunk/Lib/test/test_posix.py Thu Apr 20 03:29:48 2006 @@ -73,6 +73,11 @@ finally: fp.close() + def test_confstr(self): + if hasattr(posix, 'confstr'): + self.assertRaises(ValueError, posix.confstr, "CS_garbage") + self.assertEqual(len(posix.confstr("CS_PATH")) > 0, True) + def test_dup2(self): if hasattr(posix, 'dup2'): fp1 = open(test_support.TESTFN) Modified: python/trunk/Modules/posixmodule.c ============================================================================== --- python/trunk/Modules/posixmodule.c (original) +++ python/trunk/Modules/posixmodule.c Thu Apr 20 03:29:48 2006 @@ -6817,15 +6817,18 @@ errno = 0; len = confstr(name, buffer, sizeof(buffer)); - if (len == -1) { - posix_error(); - } - else if (len == 0) { - result = PyString_FromString(""); + if (len == 0) { + if (errno) { + posix_error(); + } + else { + result = Py_None; + Py_INCREF(Py_None); + } } else { if ((unsigned int)len >= sizeof(buffer)) { - result = PyString_FromStringAndSize(NULL, len); + result = PyString_FromStringAndSize(NULL, len+1); if (result != NULL) confstr(name, PyString_AS_STRING(result), len+1); } From buildbot at python.org Thu Apr 20 04:16:44 2006 From: buildbot at python.org (buildbot at python.org) Date: Thu, 20 Apr 2006 02:16:44 +0000 Subject: [Python-checkins] buildbot warnings in ppc Debian unstable trunk Message-ID: <20060420021644.4B3A71E4004@bag.python.org> The Buildbot has detected a new failure of ppc Debian unstable trunk. Full details are available at: http://www.python.org/dev/buildbot/all/ppc%2520Debian%2520unstable%2520trunk/builds/221 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: skip.montanaro Build Had Warnings: warnings test sincerely, -The Buildbot From python-checkins at python.org Thu Apr 20 06:54:24 2006 From: python-checkins at python.org (martin.v.loewis) Date: Thu, 20 Apr 2006 06:54:24 +0200 (CEST) Subject: [Python-checkins] r45574 - python/trunk/Misc/NEWS Message-ID: <20060420045424.8BDD21E4004@bag.python.org> Author: martin.v.loewis Date: Thu Apr 20 06:54:23 2006 New Revision: 45574 Modified: python/trunk/Misc/NEWS Log: Document r43622. Modified: python/trunk/Misc/NEWS ============================================================================== --- python/trunk/Misc/NEWS (original) +++ python/trunk/Misc/NEWS Thu Apr 20 06:54:23 2006 @@ -166,6 +166,10 @@ Core and builtins ----------------- +- On Windows, .DLL is not an accepted file name extension for + extension modules anymore; extensions are only found if they + end in .PYD. + - Bug #1421664: sys.stderr.encoding is now set to the same value as sys.stdout.encoding. From martin at v.loewis.de Thu Apr 20 08:11:37 2006 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 20 Apr 2006 08:11:37 +0200 Subject: [Python-checkins] r45573 - in python/trunk: Doc/lib/libos.tex Lib/test/test_posix.py Modules/posixmodule.c In-Reply-To: <20060420012950.B820D1E4004@bag.python.org> References: <20060420012950.B820D1E4004@bag.python.org> Message-ID: <44472619.7080303@v.loewis.de> skip.montanaro wrote: > Correct implementation and documentation of os.confstr. Add a simple test > case. I've yet to figure out how to provoke a None return I can test. I couldn't find an "unconfigured" value on Linux, either. I could find some values which return an empty string (e.g. 1006, _CS_LFS64_LIBS), so I still think it is good to tell these cases apart. > if ((unsigned int)len >= sizeof(buffer)) { > - result = PyString_FromStringAndSize(NULL, len); > + result = PyString_FromStringAndSize(NULL, len+1); > if (result != NULL) > confstr(name, PyString_AS_STRING(result), len+1); > } Somebody please correct me, but this still doesn't look right. As "len" includes the terminating null byte as returned from confstr, and as PyString_FromStringAndSize requires the number of characters *in* the string (i.e. excluding the terminating null byte), I think this should be if ((unsigned int)len >= sizeof(buffer)) { result = PyString_FromStringAndSize(NULL, len-1); if (result != NULL) confstr(name, PyString_AS_STRING(result), len); } Also, for symmetry, I would like the second case to read result = PyString_FromStringAndSize(buffer, len-1); What do you think? Regards, Martin From python-checkins at python.org Thu Apr 20 08:44:42 2006 From: python-checkins at python.org (neal.norwitz) Date: Thu, 20 Apr 2006 08:44:42 +0200 (CEST) Subject: [Python-checkins] r45575 - python/trunk/Modules/posixmodule.c Message-ID: <20060420064442.E44191E4009@bag.python.org> Author: neal.norwitz Date: Thu Apr 20 08:44:42 2006 New Revision: 45575 Modified: python/trunk/Modules/posixmodule.c Log: Whitespace, fix indentation Modified: python/trunk/Modules/posixmodule.c ============================================================================== --- python/trunk/Modules/posixmodule.c (original) +++ python/trunk/Modules/posixmodule.c Thu Apr 20 08:44:42 2006 @@ -6816,7 +6816,6 @@ errno = 0; len = confstr(name, buffer, sizeof(buffer)); - if (len == 0) { if (errno) { posix_error(); @@ -6827,7 +6826,7 @@ } } else { - if ((unsigned int)len >= sizeof(buffer)) { + if ((unsigned int)len >= sizeof(buffer)) { result = PyString_FromStringAndSize(NULL, len+1); if (result != NULL) confstr(name, PyString_AS_STRING(result), len+1); From python-checkins at python.org Thu Apr 20 08:56:06 2006 From: python-checkins at python.org (neal.norwitz) Date: Thu, 20 Apr 2006 08:56:06 +0200 (CEST) Subject: [Python-checkins] r45576 - python/trunk/Modules/posixmodule.c Message-ID: <20060420065606.333631E4009@bag.python.org> Author: neal.norwitz Date: Thu Apr 20 08:56:05 2006 New Revision: 45576 Modified: python/trunk/Modules/posixmodule.c Log: Address issues brought up by MvL on python-checkins. I tested this with valgrind on amd64. The man pages I found for diff architectures are inconsistent on this. I'm not entirely sure this change is correct for all architectures either. Perhaps we should just over-allocate and not worry about it? Modified: python/trunk/Modules/posixmodule.c ============================================================================== --- python/trunk/Modules/posixmodule.c (original) +++ python/trunk/Modules/posixmodule.c Thu Apr 20 08:56:05 2006 @@ -6809,7 +6809,7 @@ { PyObject *result = NULL; int name; - char buffer[64]; + char buffer[256]; if (PyArg_ParseTuple(args, "O&:confstr", conv_confstr_confname, &name)) { int len; @@ -6827,12 +6827,12 @@ } else { if ((unsigned int)len >= sizeof(buffer)) { - result = PyString_FromStringAndSize(NULL, len+1); + result = PyString_FromStringAndSize(NULL, len-1); if (result != NULL) - confstr(name, PyString_AS_STRING(result), len+1); + confstr(name, PyString_AS_STRING(result), len); } else - result = PyString_FromString(buffer); + result = PyString_FromStringAndSize(buffer, len-1); } } return result; From nnorwitz at gmail.com Thu Apr 20 09:00:07 2006 From: nnorwitz at gmail.com (Neal Norwitz) Date: Thu, 20 Apr 2006 00:00:07 -0700 Subject: [Python-checkins] r45573 - in python/trunk: Doc/lib/libos.tex Lib/test/test_posix.py Modules/posixmodule.c In-Reply-To: <44472619.7080303@v.loewis.de> References: <20060420012950.B820D1E4004@bag.python.org> <44472619.7080303@v.loewis.de> Message-ID: On 4/19/06, "Martin v. L?wis" wrote: > > Somebody please correct me, but this still doesn't look right. As "len" > includes the terminating null byte as returned from confstr, and as > PyString_FromStringAndSize requires the number of characters *in* > the string (i.e. excluding the terminating null byte), I think this > should be > > if ((unsigned int)len >= sizeof(buffer)) { > result = PyString_FromStringAndSize(NULL, len-1); > if (result != NULL) > confstr(name, PyString_AS_STRING(result), len); > } > > Also, for symmetry, I would like the second case to read > > result = PyString_FromStringAndSize(buffer, len-1); I couldn't determine you were wrong. I tested your version with valgrind and I set the buffer to 4 bytes to ensure the code was exercised. There were no complaints. I'm not sure. I found man pages on a bunch of diff architectures, they seemed really inconsistent and not explicit about what len should be. It's probably safer to over allocate. So I'm not sure what is the best thing to do. I didn't find one usage when searching just man pages. I wonder if anyone really uses this API? n From buildbot at python.org Thu Apr 20 09:12:04 2006 From: buildbot at python.org (buildbot at python.org) Date: Thu, 20 Apr 2006 07:12:04 +0000 Subject: [Python-checkins] buildbot warnings in x86 XP trunk Message-ID: <20060420071204.678731E403C@bag.python.org> The Buildbot has detected a new failure of x86 XP trunk. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520XP%2520trunk/builds/512 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: neal.norwitz Build Had Warnings: warnings failed slave lost sincerely, -The Buildbot From martin at v.loewis.de Thu Apr 20 10:06:50 2006 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 20 Apr 2006 10:06:50 +0200 Subject: [Python-checkins] r45573 - in python/trunk: Doc/lib/libos.tex Lib/test/test_posix.py Modules/posixmodule.c In-Reply-To: References: <20060420012950.B820D1E4004@bag.python.org> <44472619.7080303@v.loewis.de> Message-ID: <4447411A.8070106@v.loewis.de> Neal Norwitz wrote: > It's probably safer to over allocate. So I'm not sure what is the > best thing to do. I didn't find one usage when searching just man > pages. I wonder if anyone really uses this API? Clearly not, or else we would have heard about it. However, over time, systems will find more interesting uses for it, and users will some day stop hard-coding things based on uname output, and instead start using confstr where available. Regards, Martin From neal at metaslash.com Thu Apr 20 11:06:14 2006 From: neal at metaslash.com (Neal Norwitz) Date: Thu, 20 Apr 2006 05:06:14 -0400 Subject: [Python-checkins] Python Regression Test Failures refleak (1) Message-ID: <20060420090614.GA30421@python.psfb.org> test_cmd_line leaked [0, 0, 17] references test_filecmp leaked [0, 0, 13] references test_threading_local leaked [-93, 0, 0] references test_urllib2 leaked [-121, 88, 99] references From python-checkins at python.org Thu Apr 20 11:43:01 2006 From: python-checkins at python.org (thomas.wouters) Date: Thu, 20 Apr 2006 11:43:01 +0200 (CEST) Subject: [Python-checkins] r45577 - peps/trunk/pep-3100.txt Message-ID: <20060420094301.D3FBC1E400B@bag.python.org> Author: thomas.wouters Date: Thu Apr 20 11:43:01 2006 New Revision: 45577 Modified: peps/trunk/pep-3100.txt Log: Fix dupe entry. Modified: peps/trunk/pep-3100.txt ============================================================================== --- peps/trunk/pep-3100.txt (original) +++ peps/trunk/pep-3100.txt Thu Apr 20 11:43:01 2006 @@ -92,7 +92,6 @@ * __builtins__ should get a different name *or* completely unified with __builtin__. Keeping both with confusingly similar spellings and semantics is evil. -* Make ellipsis (``...``) a legal expression outside of slices [25]_ To be removed: @@ -289,9 +288,6 @@ .. [24] python-3000 email http://mail.python.org/pipermail/python-3000/2006-April/000996.html -.. [25] python-3000 email ("Type Expressions") - http://mail.python.org/pipermail/python-3000/2006-April/000996.html - .. [#pep238] PEP 238 (Changing the Division Operator) http://www.python.org/dev/peps/pep-0238 From python-checkins at python.org Thu Apr 20 13:36:27 2006 From: python-checkins at python.org (anthony.baxter) Date: Thu, 20 Apr 2006 13:36:27 +0200 (CEST) Subject: [Python-checkins] r45578 - peps/trunk/pep-0356.txt Message-ID: <20060420113627.96EDD1E4009@bag.python.org> Author: anthony.baxter Date: Thu Apr 20 13:36:26 2006 New Revision: 45578 Modified: peps/trunk/pep-0356.txt Log: few new possible items Modified: peps/trunk/pep-0356.txt ============================================================================== --- peps/trunk/pep-0356.txt (original) +++ peps/trunk/pep-0356.txt Thu Apr 20 13:36:26 2006 @@ -146,6 +146,15 @@ - All modules in Modules/ should be updated to be ssize_t clean. (Owner: Neal) + - Python core should compile cleanly with g++ + (Owner: Anthony) + + - optparse should be updated to the latest Optik codebase + (Owner: Anthony) + + - upgrade stdlib to use newer hashlib module instead of older + deprecated md5/sha modules. + (Owner: Anthony) Deferred until 2.6: From python-checkins at python.org Thu Apr 20 15:36:06 2006 From: python-checkins at python.org (andrew.kuchling) Date: Thu, 20 Apr 2006 15:36:06 +0200 (CEST) Subject: [Python-checkins] r45579 - python/trunk/Doc/whatsnew/whatsnew25.tex Message-ID: <20060420133606.F22781E4009@bag.python.org> Author: andrew.kuchling Date: Thu Apr 20 15:36:06 2006 New Revision: 45579 Modified: python/trunk/Doc/whatsnew/whatsnew25.tex Log: Add some items; add "New module" consistently; make contextlib.closing example more interesting and more correct (thanks Gustavo!); add a name Modified: python/trunk/Doc/whatsnew/whatsnew25.tex ============================================================================== --- python/trunk/Doc/whatsnew/whatsnew25.tex (original) +++ python/trunk/Doc/whatsnew/whatsnew25.tex Thu Apr 20 15:36:06 2006 @@ -865,9 +865,12 @@ and calls \code{\var{object}.close()} at the end of the block. \begin{verbatim} -with closing(open('/tmp/file', 'r')) as f: +import urllib, sys +from contextlib import closing + +with closing(urllib.urlopen('http://www.yahoo.com')) as f: for line in f: - ... + sys.stdout.write(line) \end{verbatim} \begin{seealso} @@ -1193,11 +1196,6 @@ % the cPickle module no longer accepts the deprecated None option in the % args tuple returned by __reduce__(). -% XXX fileinput: opening hook used to control how files are opened. -% .input() now has a mode parameter -% now has a fileno() function -% accepts Unicode filenames - \item The \module{audioop} module now supports the a-LAW encoding, and the code for u-LAW encoding has been improved. (Contributed by Lars Immisch.) @@ -1242,11 +1240,12 @@ method that removes the first occurrence of \var{value} in the queue, raising \exception{ValueError} if the value isn't found. -\item The \module{contextlib} module contains helper functions for use -with the new \keyword{with} statement. See section~\ref{module-contextlib} -for more about this module. (Contributed by Phillip J. Eby.) +\item New module: The \module{contextlib} module contains helper functions for use +with the new \keyword{with} statement. See +section~\ref{module-contextlib} for more about this module. +(Contributed by Phillip J. Eby.) -\item The \module{cProfile} module is a C implementation of +\item New module: The \module{cProfile} module is a C implementation of the existing \module{profile} module that has much lower overhead. The module's interface is the same as \module{profile}: you run \code{cProfile.run('main()')} to profile a function, can save profile @@ -1279,6 +1278,17 @@ '%H:%M:%S %Y-%m-%d') \end{verbatim} +\item The \module{fileinput} module was made more flexible. +Unicode filenames are now supported, and a \var{mode} parameter that +defaults to \code{"r"} was added to the +\function{input()} function to allow opening files in binary or +universal-newline mode. Another new parameter, \var{openhook}, +lets you use a function other than \function{open()} +to open the input files. Once you're iterating over +the set of files, the \class{FileInput} object's new +\method{fileno()} returns the file descriptor for the currently opened file. +(Contributed by Georg Brandl.) + \item In the \module{gc} module, the new \function{get_count()} function returns a 3-tuple containing the current collection counts for the three GC generations. This is accounting information for the garbage @@ -1385,9 +1395,9 @@ \method{gettype()}, and \method{getproto()} methods to retrieve the family, type, and protocol values for the socket. -\item New module: \module{spwd} provides functions for accessing the -shadow password database on systems that support it. -% XXX give example +\item New module: the \module{spwd} module provides functions for +accessing the shadow password database on systems that support +shadow passwords. \item The Python developers switched from CVS to Subversion during the 2.5 development process. Information about the exact build version is @@ -1418,7 +1428,20 @@ by some specifications, so it's still available as \member{unicodedata.db_3_2_0}. -% patch #754022: Greatly enhanced webbrowser.py (by Oleg Broytmann). +\item The \module{webbrowser} module received a number of +enhancements. +It's now usable as a script with \code{python -m webbrowser}, taking a +URL as the argument; there are a number of switches +to control the behaviour (\programopt{-n} for a new browser window, +\programopt{-t} for a new tab). New module-level functions, +\function{open_new()} and \function{open_new_tab()}, were added +to support this. The module's \function{open()} function supports an +additional feature, an \var{autoraise} parameter that signals whether +to raise the open window when possible. A number of additional +browsers were added to the supported list such as Firefox, Opera, +Konqueror, and elinks. (Contributed by Oleg Broytmann and George +Brandl.) +% Patch #754022 \item The \module{xmlrpclib} module now supports returning @@ -1434,9 +1457,6 @@ %====================================================================== -% whole new modules get described in subsections here - -%====================================================================== \subsection{The ctypes package} The \module{ctypes} package, written by Thomas Heller, has been added @@ -1878,6 +1898,10 @@ now uses the \cfunction{dlopen()} function instead of MacOS-specific functions. +\item Windows: \file{.dll} is no longer supported as a filename extension for +extension modules. \file{.pyd} is now the only filename extension that will +be searched for. + \end{itemize} @@ -1972,7 +1996,7 @@ The author would like to thank the following people for offering suggestions, corrections and assistance with various drafts of this -article: Phillip J. Eby, Kent Johnson, Martin von~L\"owis, Mike -Rovner, Thomas Wouters. +article: Phillip J. Eby, Kent Johnson, Martin von~L\"owis, Gustavo +Niemeyer, Mike Rovner, Thomas Wouters. \end{document} From python-checkins at python.org Thu Apr 20 15:38:36 2006 From: python-checkins at python.org (andrew.kuchling) Date: Thu, 20 Apr 2006 15:38:36 +0200 (CEST) Subject: [Python-checkins] r45580 - python/trunk/Doc/whatsnew/whatsnew25.tex Message-ID: <20060420133836.CB1AA1E4009@bag.python.org> Author: andrew.kuchling Date: Thu Apr 20 15:38:36 2006 New Revision: 45580 Modified: python/trunk/Doc/whatsnew/whatsnew25.tex Log: Markup fix Modified: python/trunk/Doc/whatsnew/whatsnew25.tex ============================================================================== --- python/trunk/Doc/whatsnew/whatsnew25.tex (original) +++ python/trunk/Doc/whatsnew/whatsnew25.tex Thu Apr 20 15:38:36 2006 @@ -1978,7 +1978,7 @@ on 64-bit machines. Extension code may need to make the same change to avoid warnings and to support 64-bit machines. See the earlier -section~ref{section-353} for a discussion of this change. +section~\ref{section-353} for a discussion of this change. \item C API: The obmalloc changes mean that From python-checkins at python.org Thu Apr 20 15:39:41 2006 From: python-checkins at python.org (andrew.kuchling) Date: Thu, 20 Apr 2006 15:39:41 +0200 (CEST) Subject: [Python-checkins] r45581 - python/trunk/Doc/whatsnew/whatsnew25.tex Message-ID: <20060420133941.2DBE81E4009@bag.python.org> Author: andrew.kuchling Date: Thu Apr 20 15:39:40 2006 New Revision: 45581 Modified: python/trunk/Doc/whatsnew/whatsnew25.tex Log: Argh, make another markup fix Modified: python/trunk/Doc/whatsnew/whatsnew25.tex ============================================================================== --- python/trunk/Doc/whatsnew/whatsnew25.tex (original) +++ python/trunk/Doc/whatsnew/whatsnew25.tex Thu Apr 20 15:39:40 2006 @@ -1821,7 +1821,7 @@ \item The largest change to the C API came from \pep{353}, which modifies the interpreter to use a \ctype{Py_ssize_t} type definition instead of \ctype{int}. See the earlier -section~ref{section-353} for a discussion of this change. +section~\ref{section-353} for a discussion of this change. \item The design of the bytecode compiler has changed a great deal, to no longer generate bytecode by traversing the parse tree. Instead From python-checkins at python.org Thu Apr 20 15:43:22 2006 From: python-checkins at python.org (andrew.kuchling) Date: Thu, 20 Apr 2006 15:43:22 +0200 (CEST) Subject: [Python-checkins] r45582 - python/trunk/Doc/whatsnew/whatsnew25.tex Message-ID: <20060420134322.718B21E4009@bag.python.org> Author: andrew.kuchling Date: Thu Apr 20 15:43:21 2006 New Revision: 45582 Modified: python/trunk/Doc/whatsnew/whatsnew25.tex Log: Change a footnote to a parenthetical (in two senses) paragraph Modified: python/trunk/Doc/whatsnew/whatsnew25.tex ============================================================================== --- python/trunk/Doc/whatsnew/whatsnew25.tex (original) +++ python/trunk/Doc/whatsnew/whatsnew25.tex Thu Apr 20 15:43:21 2006 @@ -452,11 +452,14 @@ expression when you're doing something with the returned value, as in the above example. The parentheses aren't always necessary, but it's easier to always add them instead of having to remember when they're -needed.\footnote{The exact rules are that a \keyword{yield}-expression must -always be parenthesized except when it occurs at the top-level -expression on the right-hand side of an assignment, meaning you can -write \code{val = yield i} but have to use parentheses when there's an -operation, as in \code{val = (yield i) + 12}.} +needed. + +(\pep{342} explains the exact rules, which are that a +\keyword{yield}-expression must always be parenthesized except when it +occurs at the top-level expression on the right-hand side of an +assignment. This means you can write \code{val = yield i} but have to +use parentheses when there's an operation, as in \code{val = (yield i) ++ 12}.) Values are sent into a generator by calling its \method{send(\var{value})} method. The generator's code is then From python-checkins at python.org Thu Apr 20 19:14:36 2006 From: python-checkins at python.org (thomas.heller) Date: Thu, 20 Apr 2006 19:14:36 +0200 (CEST) Subject: [Python-checkins] r45584 - external/ctypes Message-ID: <20060420171436.9A5761E4010@bag.python.org> Author: thomas.heller Date: Thu Apr 20 19:14:35 2006 New Revision: 45584 Added: external/ctypes/ - copied from r45583, external/ctypes-0.9.9.4/ Log: tagging 0.9.9.4 as current ctypes From python-checkins at python.org Thu Apr 20 21:27:12 2006 From: python-checkins at python.org (thomas.heller) Date: Thu, 20 Apr 2006 21:27:12 +0200 (CEST) Subject: [Python-checkins] r45586 - external/ctypes-0.9.9.6 Message-ID: <20060420192712.CEF2F1E400B@bag.python.org> Author: thomas.heller Date: Thu Apr 20 21:27:12 2006 New Revision: 45586 Added: external/ctypes-0.9.9.6/ - copied from r45585, external/ctypes/ Log: tagging ctypes-0.9.9.6 From neal at metaslash.com Thu Apr 20 23:06:05 2006 From: neal at metaslash.com (Neal Norwitz) Date: Thu, 20 Apr 2006 17:06:05 -0400 Subject: [Python-checkins] Python Regression Test Failures refleak (1) Message-ID: <20060420210605.GA31650@python.psfb.org> test_cmd_line leaked [0, 17, 0] references test_threading_local leaked [-93, 0, 0] references test_urllib2 leaked [-121, 88, 99] references From python-checkins at python.org Thu Apr 20 23:38:18 2006 From: python-checkins at python.org (jack.jansen) Date: Thu, 20 Apr 2006 23:38:18 +0200 (CEST) Subject: [Python-checkins] r45587 - python/trunk/Tools/bgen/bgen/bgenObjectDefinition.py Message-ID: <20060420213818.378B31E401D@bag.python.org> Author: jack.jansen Date: Thu Apr 20 23:38:17 2006 New Revision: 45587 Modified: python/trunk/Tools/bgen/bgen/bgenObjectDefinition.py Log: - tp_init shouldn't call base class tp_init by default - tp_new (which was apparently always overridden:-) called base class tp_init in stead of tp_new. Modified: python/trunk/Tools/bgen/bgen/bgenObjectDefinition.py ============================================================================== --- python/trunk/Tools/bgen/bgen/bgenObjectDefinition.py (original) +++ python/trunk/Tools/bgen/bgen/bgenObjectDefinition.py Thu Apr 20 23:38:17 2006 @@ -383,6 +383,8 @@ Output("%s_tp_free, /* tp_free */", self.prefix) def output_tp_initBody_basecall(self): + """If a type shares its init call with its base type set output_tp_initBody + to output_tp_initBody_basecall""" if self.basetype: Output("if (%s.tp_init)", self.basetype) OutLbrace() @@ -395,7 +397,6 @@ if self.output_tp_initBody: Output("static int %s_tp_init(PyObject *_self, PyObject *_args, PyObject *_kwds)", self.prefix) OutLbrace() - self.output_tp_initBody_basecall() self.output_tp_initBody() OutRbrace() else: @@ -425,7 +426,7 @@ if self.basetype: Output("if (%s.tp_new)", self.basetype) OutLbrace() - Output("if ( (*%s.tp_init)(_self, _args, _kwds) == NULL) return NULL;", self.basetype) + Output("if ( (*%s.tp_new)(type, _args, _kwds) == NULL) return NULL;", self.basetype) Dedent() Output("} else {") Indent() From buildbot at python.org Fri Apr 21 00:05:37 2006 From: buildbot at python.org (buildbot at python.org) Date: Thu, 20 Apr 2006 22:05:37 +0000 Subject: [Python-checkins] buildbot warnings in x86 W2k trunk Message-ID: <20060420220538.10A551E4013@bag.python.org> The Buildbot has detected a new failure of x86 W2k trunk. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520W2k%2520trunk/builds/556 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: jack.jansen Build Had Warnings: warnings test sincerely, -The Buildbot From python-checkins at python.org Fri Apr 21 03:33:41 2006 From: python-checkins at python.org (skip.montanaro) Date: Fri, 21 Apr 2006 03:33:41 +0200 (CEST) Subject: [Python-checkins] r45590 - python/trunk/Modules/gcmodule.c Message-ID: <20060421013341.29E681E400B@bag.python.org> Author: skip.montanaro Date: Fri Apr 21 03:33:40 2006 New Revision: 45590 Modified: python/trunk/Modules/gcmodule.c Log: This is a long-ago patch I submitted to SF (1100924) to time the gc passes. Barry approved it awhile ago. Been sitting in my sandbox for awhile as well. Modified: python/trunk/Modules/gcmodule.c ============================================================================== --- python/trunk/Modules/gcmodule.c (original) +++ python/trunk/Modules/gcmodule.c Fri Apr 21 03:33:40 2006 @@ -734,6 +734,8 @@ PyGC_Head unreachable; /* non-problematic unreachable trash */ PyGC_Head finalizers; /* objects with, & reachable from, __del__ */ PyGC_Head *gc; + static PyObject *tmod = NULL; + double t1 = 0.0; if (delstr == NULL) { delstr = PyString_InternFromString("__del__"); @@ -741,7 +743,23 @@ Py_FatalError("gc couldn't allocate \"__del__\""); } + if (tmod == NULL) { + tmod = PyImport_ImportModule("time"); + if (tmod == NULL) + PyErr_Clear(); + } + if (debug & DEBUG_STATS) { + if (tmod != NULL) { + PyObject *f = PyObject_CallMethod(tmod, "time", NULL); + if (f == NULL) { + PyErr_Clear(); + } + else { + t1 = PyFloat_AsDouble(f); + Py_DECREF(f); + } + } PySys_WriteStderr("gc: collecting generation %d...\n", generation); PySys_WriteStderr("gc: objects in each generation:"); @@ -814,6 +832,17 @@ if (debug & DEBUG_COLLECTABLE) { debug_cycle("collectable", FROM_GC(gc)); } + if (tmod != NULL && (debug & DEBUG_STATS)) { + PyObject *f = PyObject_CallMethod(tmod, "time", NULL); + if (f == NULL) { + PyErr_Clear(); + } + else { + t1 = PyFloat_AsDouble(f)-t1; + Py_DECREF(f); + PySys_WriteStderr("gc: %.4fs elapsed.\n", t1); + } + } } /* Clear weakrefs and invoke callbacks as necessary. */ From buildbot at python.org Fri Apr 21 03:41:38 2006 From: buildbot at python.org (buildbot at python.org) Date: Fri, 21 Apr 2006 01:41:38 +0000 Subject: [Python-checkins] buildbot warnings in x86 OpenBSD trunk Message-ID: <20060421014138.F2FEB1E400B@bag.python.org> The Buildbot has detected a new failure of x86 OpenBSD trunk. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520OpenBSD%2520trunk/builds/392 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: skip.montanaro Build Had Warnings: warnings test sincerely, -The Buildbot From python-checkins at python.org Fri Apr 21 04:31:09 2006 From: python-checkins at python.org (skip.montanaro) Date: Fri, 21 Apr 2006 04:31:09 +0200 (CEST) Subject: [Python-checkins] r45591 - in python/trunk: Doc/lib/libprofile.tex Lib/pstats.py Misc/NEWS Message-ID: <20060421023109.4E6471E400B@bag.python.org> Author: skip.montanaro Date: Fri Apr 21 04:31:07 2006 New Revision: 45591 Modified: python/trunk/Doc/lib/libprofile.tex python/trunk/Lib/pstats.py python/trunk/Misc/NEWS Log: Allow pstats.Stats creator to specify an alternate to stdout. Modified: python/trunk/Doc/lib/libprofile.tex ============================================================================== --- python/trunk/Doc/lib/libprofile.tex (original) +++ python/trunk/Doc/lib/libprofile.tex Fri Apr 21 04:31:07 2006 @@ -391,17 +391,17 @@ % (This \stmodindex use may be hard to change ;-( ) \stmodindex{pstats} -\begin{classdesc}{Stats}{filename\optional{, \moreargs}} +\begin{classdesc}{Stats}{filename\optional{, \moreargs\optional{, stream=sys.stdout}}} This class constructor creates an instance of a ``statistics object'' from a \var{filename} (or set of filenames). \class{Stats} objects are -manipulated by methods, in order to print useful reports. +manipulated by methods, in order to print useful reports. You may specify +an alternate output stream by giving the keyword argument, \code{stream}. -The file selected by the above constructor must have been created by -the corresponding version of \module{profile} or \module{cProfile}. -To be specific, there is -\emph{no} file compatibility guaranteed with future versions of this -profiler, and there is no compatibility with files produced by other -profilers. +The file selected by the above constructor must have been created by the +corresponding version of \module{profile} or \module{cProfile}. To be +specific, there is \emph{no} file compatibility guaranteed with future +versions of this profiler, and there is no compatibility with files produced +by other profilers. %(such as the old system profiler). If several files are provided, all the statistics for identical Modified: python/trunk/Lib/pstats.py ============================================================================== --- python/trunk/Lib/pstats.py (original) +++ python/trunk/Lib/pstats.py Fri Apr 21 04:31:07 2006 @@ -32,6 +32,7 @@ # CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. +import sys import os import time import marshal @@ -58,18 +59,31 @@ printed. The sort_stats() method now processes some additional options (i.e., in - addition to the old -1, 0, 1, or 2). It takes an arbitrary number of quoted - strings to select the sort order. For example sort_stats('time', 'name') - sorts on the major key of "internal function time", and on the minor - key of 'the name of the function'. Look at the two tables in sort_stats() - and get_sort_arg_defs(self) for more examples. + addition to the old -1, 0, 1, or 2). It takes an arbitrary number of + quoted strings to select the sort order. For example sort_stats('time', + 'name') sorts on the major key of 'internal function time', and on the + minor key of 'the name of the function'. Look at the two tables in + sort_stats() and get_sort_arg_defs(self) for more examples. - All methods now return "self", so you can string together commands like: + All methods return self, so you can string together commands like: Stats('foo', 'goo').strip_dirs().sort_stats('calls').\ print_stats(5).print_callers(5) """ - def __init__(self, *args): + def __init__(self, *args, **kwds): + # I can't figure out how to explictly specify a stream keyword arg + # with *args: + # def __init__(self, *args, stream=sys.stdout): ... + # so I use **kwds and sqauwk if something unexpected is passed in. + self.stream = sys.stdout + if "stream" in kwds: + self.stream = kwds["stream"] + del kwds["stream"] + if kwds: + keys = kwds.keys() + keys.sort() + extras = ", ".join(["%s=%s" % (k, kwds[k]) for k in keys]) + raise ValueError, "unrecognized keyword args: %s" % extras if not len(args): arg = None else: @@ -96,9 +110,9 @@ trouble = 0 finally: if trouble: - print "Invalid timing data", - if self.files: print self.files[-1], - print + print >> self.stream, "Invalid timing data", + if self.files: print >> self.stream, self.files[-1], + print >> self.stream def load_stats(self, arg): if not arg: self.stats = {} @@ -320,7 +334,7 @@ if not list: return 0, list - print msg + print >> self.stream, msg if count < len(self.stats): width = 0 for func in list: @@ -330,24 +344,24 @@ def print_stats(self, *amount): for filename in self.files: - print filename - if self.files: print + print >> self.stream, filename + if self.files: print >> self.stream indent = ' ' * 8 for func in self.top_level: - print indent, func_get_function_name(func) + print >> self.stream, indent, func_get_function_name(func) - print indent, self.total_calls, "function calls", + print >> self.stream, indent, self.total_calls, "function calls", if self.total_calls != self.prim_calls: - print "(%d primitive calls)" % self.prim_calls, - print "in %.3f CPU seconds" % self.total_tt - print + print >> self.stream, "(%d primitive calls)" % self.prim_calls, + print >> self.stream, "in %.3f CPU seconds" % self.total_tt + print >> self.stream width, list = self.get_print_list(amount) if list: self.print_title() for func in list: self.print_line(func) - print - print + print >> self.stream + print >> self.stream return self def print_callees(self, *amount): @@ -361,8 +375,8 @@ self.print_call_line(width, func, self.all_callees[func]) else: self.print_call_line(width, func, {}) - print - print + print >> self.stream + print >> self.stream return self def print_callers(self, *amount): @@ -372,12 +386,12 @@ for func in list: cc, nc, tt, ct, callers = self.stats[func] self.print_call_line(width, func, callers, "<-") - print - print + print >> self.stream + print >> self.stream return self def print_call_heading(self, name_size, column_title): - print "Function ".ljust(name_size) + column_title + print >> self.stream, "Function ".ljust(name_size) + column_title # print sub-header only if we have new-style callers subheader = False for cc, nc, tt, ct, callers in self.stats.itervalues(): @@ -386,12 +400,12 @@ subheader = isinstance(value, tuple) break if subheader: - print " "*name_size + " ncalls tottime cumtime" + print >> self.stream, " "*name_size + " ncalls tottime cumtime" def print_call_line(self, name_size, source, call_dict, arrow="->"): - print func_std_string(source).ljust(name_size) + arrow, + print >> self.stream, func_std_string(source).ljust(name_size) + arrow, if not call_dict: - print + print >> self.stream return clist = call_dict.keys() clist.sort() @@ -411,30 +425,30 @@ else: substats = '%s(%r) %s' % (name, value, f8(self.stats[func][3])) left_width = name_size + 3 - print indent*left_width + substats + print >> self.stream, indent*left_width + substats indent = " " def print_title(self): - print ' ncalls tottime percall cumtime percall', \ - 'filename:lineno(function)' + print >> self.stream, ' ncalls tottime percall cumtime percall', + print >> self.stream, 'filename:lineno(function)' def print_line(self, func): # hack : should print percentages cc, nc, tt, ct, callers = self.stats[func] c = str(nc) if nc != cc: c = c + '/' + str(cc) - print c.rjust(9), - print f8(tt), + print >> self.stream, c.rjust(9), + print >> self.stream, f8(tt), if nc == 0: - print ' '*8, + print >> self.stream, ' '*8, else: - print f8(tt/nc), - print f8(ct), + print >> self.stream, f8(tt/nc), + print >> self.stream, f8(ct), if cc == 0: - print ' '*8, + print >> self.stream, ' '*8, else: - print f8(ct/cc), - print func_std_string(func) + print >> self.stream, f8(ct/cc), + print >> self.stream, func_std_string(func) class TupleComp: """This class provides a generic function for comparing any two tuples. @@ -549,7 +563,7 @@ try: frac = float(term) if frac > 1 or frac < 0: - print "Fraction argument mus be in [0, 1]" + print >> self.stream, "Fraction argument must be in [0, 1]" continue processed.append(frac) continue @@ -559,93 +573,93 @@ if self.stats: getattr(self.stats, fn)(*processed) else: - print "No statistics object is loaded." + print >> self.stream, "No statistics object is loaded." return 0 def generic_help(self): - print "Arguments may be:" - print "* An integer maximum number of entries to print." - print "* A decimal fractional number between 0 and 1, controlling" - print " what fraction of selected entries to print." - print "* A regular expression; only entries with function names" - print " that match it are printed." + print >> self.stream, "Arguments may be:" + print >> self.stream, "* An integer maximum number of entries to print." + print >> self.stream, "* A decimal fractional number between 0 and 1, controlling" + print >> self.stream, " what fraction of selected entries to print." + print >> self.stream, "* A regular expression; only entries with function names" + print >> self.stream, " that match it are printed." def do_add(self, line): self.stats.add(line) return 0 def help_add(self): - print "Add profile info from given file to current statistics object." + print >> self.stream, "Add profile info from given file to current statistics object." def do_callees(self, line): return self.generic('print_callees', line) def help_callees(self): - print "Print callees statistics from the current stat object." + print >> self.stream, "Print callees statistics from the current stat object." self.generic_help() def do_callers(self, line): return self.generic('print_callers', line) def help_callers(self): - print "Print callers statistics from the current stat object." + print >> self.stream, "Print callers statistics from the current stat object." self.generic_help() def do_EOF(self, line): - print "" + print >> self.stream, "" return 1 def help_EOF(self): - print "Leave the profile brower." + print >> self.stream, "Leave the profile brower." def do_quit(self, line): return 1 def help_quit(self): - print "Leave the profile brower." + print >> self.stream, "Leave the profile brower." def do_read(self, line): if line: try: self.stats = Stats(line) except IOError, args: - print args[1] + print >> self.stream, args[1] return self.prompt = line + "% " elif len(self.prompt) > 2: line = self.prompt[-2:] else: - print "No statistics object is current -- cannot reload." + print >> self.stream, "No statistics object is current -- cannot reload." return 0 def help_read(self): - print "Read in profile data from a specified file." + print >> self.stream, "Read in profile data from a specified file." def do_reverse(self, line): self.stats.reverse_order() return 0 def help_reverse(self): - print "Reverse the sort order of the profiling report." + print >> self.stream, "Reverse the sort order of the profiling report." def do_sort(self, line): abbrevs = self.stats.get_sort_arg_defs() if line and not filter(lambda x,a=abbrevs: x not in a,line.split()): self.stats.sort_stats(*line.split()) else: - print "Valid sort keys (unique prefixes are accepted):" + print >> self.stream, "Valid sort keys (unique prefixes are accepted):" for (key, value) in Stats.sort_arg_dict_default.iteritems(): - print "%s -- %s" % (key, value[1]) + print >> self.stream, "%s -- %s" % (key, value[1]) return 0 def help_sort(self): - print "Sort profile data according to specified keys." - print "(Typing `sort' without arguments lists valid keys.)" + print >> self.stream, "Sort profile data according to specified keys." + print >> self.stream, "(Typing `sort' without arguments lists valid keys.)" def complete_sort(self, text, *args): return [a for a in Stats.sort_arg_dict_default if a.startswith(text)] def do_stats(self, line): return self.generic('print_stats', line) def help_stats(self): - print "Print statistics from the current stat object." + print >> self.stream, "Print statistics from the current stat object." self.generic_help() def do_strip(self, line): self.stats.strip_dirs() return 0 def help_strip(self): - print "Strip leading path information from filenames in the report." + print >> self.stream, "Strip leading path information from filenames in the report." def postcmd(self, stop, line): if stop: @@ -653,14 +667,14 @@ return None import sys - print "Welcome to the profile statistics browser." + print >> self.stream, "Welcome to the profile statistics browser." if len(sys.argv) > 1: initprofile = sys.argv[1] else: initprofile = None try: ProfileBrowser(initprofile).cmdloop() - print "Goodbye." + print >> self.stream, "Goodbye." except KeyboardInterrupt: pass Modified: python/trunk/Misc/NEWS ============================================================================== --- python/trunk/Misc/NEWS (original) +++ python/trunk/Misc/NEWS Fri Apr 21 04:31:07 2006 @@ -120,6 +120,9 @@ - Fix exception when doing glob.glob('anything*/') +- The pstats.Stats class accepts an optional stream keyword argument to + direct output to an alternate file-like object. + Build ----- From buildbot at python.org Fri Apr 21 04:38:59 2006 From: buildbot at python.org (buildbot at python.org) Date: Fri, 21 Apr 2006 02:38:59 +0000 Subject: [Python-checkins] buildbot warnings in x86 OpenBSD trunk Message-ID: <20060421023900.0C2851E400B@bag.python.org> The Buildbot has detected a new failure of x86 OpenBSD trunk. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520OpenBSD%2520trunk/builds/394 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: skip.montanaro Build Had Warnings: warnings test sincerely, -The Buildbot From nnorwitz at gmail.com Fri Apr 21 07:40:16 2006 From: nnorwitz at gmail.com (Neal Norwitz) Date: Thu, 20 Apr 2006 22:40:16 -0700 Subject: [Python-checkins] r45585 - in external/ctypes: .cvsignore ACKS ANNOUNCE ChangeLog LICENSE.txt MANIFEST.in README.CVS README.txt ctypes-dev.el ctypes/.CTYPES_DEVEL ctypes/.cvsignore ctypes/__init__.py ctypes/_endian.py ctypes/_loader.py ctypes/mac Message-ID: On 4/20/06, thomas.heller wrote: > Author: thomas.heller > Date: Thu Apr 20 21:23:42 2006 > New Revision: 45585 I tried to diff this to see what changed, but I don't get anything returned. Does anyone know how to view the full diff of this checkin? n From python-checkins at python.org Fri Apr 21 11:43:32 2006 From: python-checkins at python.org (thomas.wouters) Date: Fri, 21 Apr 2006 11:43:32 +0200 (CEST) Subject: [Python-checkins] r45593 - in python/branches/p3yk: Doc/lib/libcodecs.tex Doc/whatsnew/whatsnew25.tex Include/codecs.h Lib/StringIO.py Lib/codecs.py Lib/encodings/__init__.py Lib/encodings/ascii.py Lib/encodings/base64_codec.py Lib/encodings/bz2_codec.py Lib/encodings/charmap.py Lib/encodings/cp037.py Lib/encodings/cp1006.py Lib/encodings/cp1026.py Lib/encodings/cp1140.py Lib/encodings/cp1250.py Lib/encodings/cp1251.py Lib/encodings/cp1252.py Lib/encodings/cp1253.py Lib/encodings/cp1254.py Lib/encodings/cp1255.py Lib/encodings/cp1256.py Lib/encodings/cp1257.py Lib/encodings/cp1258.py Lib/encodings/cp424.py Lib/encodings/cp437.py Lib/encodings/cp500.py Lib/encodings/cp737.py Lib/encodings/cp775.py Lib/encodings/cp850.py Lib/encodings/cp852.py Lib/encodings/cp855.py Lib/encodings/cp856.py Lib/encodings/cp857.py Lib/encodings/cp860.py Lib/encodings/cp861.py Lib/encodings/cp862.py Lib/encodings/cp863.py Lib/encodings/cp864.py Lib/encodings/cp865.py Lib/encodings/cp866.py Lib/encodings/cp869.py Lib/encodings/cp874.py Lib/encodings/cp875.py Lib/encodings/hex_codec.py Lib/encodings/hp_roman8.py Lib/encodings/idna.py Lib/encodings/iso8859_1.py Lib/encodings/iso8859_10.py Lib/encodings/iso8859_11.py Lib/encodings/iso8859_13.py Lib/encodings/iso8859_14.py Lib/encodings/iso8859_15.py Lib/encodings/iso8859_16.py Lib/encodings/iso8859_2.py Lib/encodings/iso8859_3.py Lib/encodings/iso8859_4.py Lib/encodings/iso8859_5.py Lib/encodings/iso8859_6.py Lib/encodings/iso8859_7.py Lib/encodings/iso8859_8.py Lib/encodings/iso8859_9.py Lib/encodings/koi8_r.py Lib/encodings/koi8_u.py Lib/encodings/latin_1.py Lib/encodings/mac_arabic.py Lib/encodings/mac_centeuro.py Lib/encodings/mac_croatian.py Lib/encodings/mac_cyrillic.py Lib/encodings/mac_farsi.py Lib/encodings/mac_greek.py Lib/encodings/mac_iceland.py Lib/encodings/mac_latin2.py Lib/encodings/mac_roman.py Lib/encodings/mac_romanian.py Lib/encodings/mac_turkish.py Lib/encodings/mbcs.py Lib/encodings/palmos.py Lib/encodings/ptcp154.py Lib/encodings/punycode.py Lib/encodings/quopri_codec.py Lib/encodings/raw_unicode_escape.py Lib/encodings/rot_13.py Lib/encodings/string_escape.py Lib/encodings/tis_620.py Lib/encodings/undefined.py Lib/encodings/unicode_escape.py Lib/encodings/unicode_internal.py Lib/encodings/utf_16.py Lib/encodings/utf_16_be.py Lib/encodings/utf_16_le.py Lib/encodings/utf_7.py Lib/encodings/utf_8.py Lib/encodings/utf_8_sig.py Lib/encodings/uu_codec.py Lib/encodings/zlib_codec.py Lib/logging/__init__.py Lib/runpy.py Lib/test/regrtest.py Lib/test/test_StringIO.py Lib/test/test___all__.py Lib/test/test_codecs.py Lib/test/test_hashlib_speed.py Lib/test/test_runpy.py Lib/test/time_hashlib.py Makefile.pre.in Modules/_ctypes/_ctypes.c Modules/_ctypes/cfield.c Modules/_ctypes/ctypes.h Modules/_testcapimodule.c Modules/cStringIO.c Modules/main.c Modules/xxmodule.c Objects/obmalloc.c Python/codecs.c Tools/buildbot/clean.bat Tools/unicode/Makefile Tools/unicode/gencodec.py Message-ID: <20060421094332.2C4571E400B@bag.python.org> Author: thomas.wouters Date: Fri Apr 21 11:43:23 2006 New Revision: 45593 Added: python/branches/p3yk/Lib/runpy.py - copied unchanged from r43067, python/trunk/Lib/runpy.py python/branches/p3yk/Lib/test/test_runpy.py - copied unchanged from r43067, python/trunk/Lib/test/test_runpy.py python/branches/p3yk/Lib/test/time_hashlib.py - copied unchanged from r43067, python/trunk/Lib/test/time_hashlib.py Removed: python/branches/p3yk/Lib/test/test_hashlib_speed.py Modified: python/branches/p3yk/ (props changed) python/branches/p3yk/Doc/lib/libcodecs.tex python/branches/p3yk/Doc/whatsnew/whatsnew25.tex python/branches/p3yk/Include/codecs.h python/branches/p3yk/Lib/StringIO.py python/branches/p3yk/Lib/codecs.py python/branches/p3yk/Lib/encodings/__init__.py python/branches/p3yk/Lib/encodings/ascii.py python/branches/p3yk/Lib/encodings/base64_codec.py python/branches/p3yk/Lib/encodings/bz2_codec.py python/branches/p3yk/Lib/encodings/charmap.py python/branches/p3yk/Lib/encodings/cp037.py python/branches/p3yk/Lib/encodings/cp1006.py python/branches/p3yk/Lib/encodings/cp1026.py python/branches/p3yk/Lib/encodings/cp1140.py python/branches/p3yk/Lib/encodings/cp1250.py python/branches/p3yk/Lib/encodings/cp1251.py python/branches/p3yk/Lib/encodings/cp1252.py python/branches/p3yk/Lib/encodings/cp1253.py python/branches/p3yk/Lib/encodings/cp1254.py python/branches/p3yk/Lib/encodings/cp1255.py python/branches/p3yk/Lib/encodings/cp1256.py python/branches/p3yk/Lib/encodings/cp1257.py python/branches/p3yk/Lib/encodings/cp1258.py python/branches/p3yk/Lib/encodings/cp424.py python/branches/p3yk/Lib/encodings/cp437.py python/branches/p3yk/Lib/encodings/cp500.py python/branches/p3yk/Lib/encodings/cp737.py python/branches/p3yk/Lib/encodings/cp775.py python/branches/p3yk/Lib/encodings/cp850.py python/branches/p3yk/Lib/encodings/cp852.py python/branches/p3yk/Lib/encodings/cp855.py python/branches/p3yk/Lib/encodings/cp856.py python/branches/p3yk/Lib/encodings/cp857.py python/branches/p3yk/Lib/encodings/cp860.py python/branches/p3yk/Lib/encodings/cp861.py python/branches/p3yk/Lib/encodings/cp862.py python/branches/p3yk/Lib/encodings/cp863.py python/branches/p3yk/Lib/encodings/cp864.py python/branches/p3yk/Lib/encodings/cp865.py python/branches/p3yk/Lib/encodings/cp866.py python/branches/p3yk/Lib/encodings/cp869.py python/branches/p3yk/Lib/encodings/cp874.py python/branches/p3yk/Lib/encodings/cp875.py python/branches/p3yk/Lib/encodings/hex_codec.py python/branches/p3yk/Lib/encodings/hp_roman8.py python/branches/p3yk/Lib/encodings/idna.py python/branches/p3yk/Lib/encodings/iso8859_1.py python/branches/p3yk/Lib/encodings/iso8859_10.py python/branches/p3yk/Lib/encodings/iso8859_11.py python/branches/p3yk/Lib/encodings/iso8859_13.py python/branches/p3yk/Lib/encodings/iso8859_14.py python/branches/p3yk/Lib/encodings/iso8859_15.py python/branches/p3yk/Lib/encodings/iso8859_16.py python/branches/p3yk/Lib/encodings/iso8859_2.py python/branches/p3yk/Lib/encodings/iso8859_3.py python/branches/p3yk/Lib/encodings/iso8859_4.py python/branches/p3yk/Lib/encodings/iso8859_5.py python/branches/p3yk/Lib/encodings/iso8859_6.py python/branches/p3yk/Lib/encodings/iso8859_7.py python/branches/p3yk/Lib/encodings/iso8859_8.py python/branches/p3yk/Lib/encodings/iso8859_9.py python/branches/p3yk/Lib/encodings/koi8_r.py python/branches/p3yk/Lib/encodings/koi8_u.py python/branches/p3yk/Lib/encodings/latin_1.py python/branches/p3yk/Lib/encodings/mac_arabic.py python/branches/p3yk/Lib/encodings/mac_centeuro.py python/branches/p3yk/Lib/encodings/mac_croatian.py python/branches/p3yk/Lib/encodings/mac_cyrillic.py python/branches/p3yk/Lib/encodings/mac_farsi.py python/branches/p3yk/Lib/encodings/mac_greek.py python/branches/p3yk/Lib/encodings/mac_iceland.py python/branches/p3yk/Lib/encodings/mac_latin2.py python/branches/p3yk/Lib/encodings/mac_roman.py python/branches/p3yk/Lib/encodings/mac_romanian.py python/branches/p3yk/Lib/encodings/mac_turkish.py python/branches/p3yk/Lib/encodings/mbcs.py python/branches/p3yk/Lib/encodings/palmos.py python/branches/p3yk/Lib/encodings/ptcp154.py python/branches/p3yk/Lib/encodings/punycode.py python/branches/p3yk/Lib/encodings/quopri_codec.py python/branches/p3yk/Lib/encodings/raw_unicode_escape.py python/branches/p3yk/Lib/encodings/rot_13.py python/branches/p3yk/Lib/encodings/string_escape.py python/branches/p3yk/Lib/encodings/tis_620.py python/branches/p3yk/Lib/encodings/undefined.py python/branches/p3yk/Lib/encodings/unicode_escape.py python/branches/p3yk/Lib/encodings/unicode_internal.py python/branches/p3yk/Lib/encodings/utf_16.py python/branches/p3yk/Lib/encodings/utf_16_be.py python/branches/p3yk/Lib/encodings/utf_16_le.py python/branches/p3yk/Lib/encodings/utf_7.py python/branches/p3yk/Lib/encodings/utf_8.py python/branches/p3yk/Lib/encodings/utf_8_sig.py python/branches/p3yk/Lib/encodings/uu_codec.py python/branches/p3yk/Lib/encodings/zlib_codec.py python/branches/p3yk/Lib/logging/__init__.py python/branches/p3yk/Lib/test/regrtest.py python/branches/p3yk/Lib/test/test_StringIO.py python/branches/p3yk/Lib/test/test___all__.py python/branches/p3yk/Lib/test/test_codecs.py python/branches/p3yk/Makefile.pre.in python/branches/p3yk/Modules/_ctypes/_ctypes.c python/branches/p3yk/Modules/_ctypes/cfield.c python/branches/p3yk/Modules/_ctypes/ctypes.h python/branches/p3yk/Modules/_testcapimodule.c python/branches/p3yk/Modules/cStringIO.c python/branches/p3yk/Modules/main.c python/branches/p3yk/Modules/xxmodule.c python/branches/p3yk/Objects/obmalloc.c python/branches/p3yk/Python/codecs.c python/branches/p3yk/Tools/buildbot/clean.bat python/branches/p3yk/Tools/unicode/Makefile python/branches/p3yk/Tools/unicode/gencodec.py Log: Merge part of the trunk changes into the p3yk branch. This merges from 43030 (branch-creation time) up to 43067. 43068 and 43069 contain a little swapping action between re.py and sre.py, and this mightily confuses svn merge, so later changes are going in separately. This merge should break no additional tests. The last-merged revision is going in a 'last_merge' property on '.' (the branch directory.) Arbitrarily chosen, really; if there's a BCP for this, I couldn't find it, but we can easily change it afterwards ;) Modified: python/branches/p3yk/Doc/lib/libcodecs.tex ============================================================================== --- python/branches/p3yk/Doc/lib/libcodecs.tex (original) +++ python/branches/p3yk/Doc/lib/libcodecs.tex Fri Apr 21 11:43:23 2006 @@ -24,8 +24,19 @@ \begin{funcdesc}{register}{search_function} Register a codec search function. Search functions are expected to take one argument, the encoding name in all lower case letters, and -return a tuple of functions \code{(\var{encoder}, \var{decoder}, \var{stream_reader}, -\var{stream_writer})} taking the following arguments: +return a \class{CodecInfo} object having the following attributes: + +\begin{itemize} + \item \code{name} The name of the encoding; + \item \code{encoder} The stateless encoding function; + \item \code{decoder} The stateless decoding function; + \item \code{incrementalencoder} An incremental encoder class or factory function; + \item \code{incrementaldecoder} An incremental decoder class or factory function; + \item \code{streamwriter} A stream writer class or factory function; + \item \code{streamreader} A stream reader class or factory function. +\end{itemize} + +The various functions or classes take the following arguments: \var{encoder} and \var{decoder}: These must be functions or methods which have the same interface as the @@ -33,7 +44,17 @@ Codec Interface). The functions/methods are expected to work in a stateless mode. - \var{stream_reader} and \var{stream_writer}: These have to be + \var{incrementalencoder} and \var{incrementalencoder}: These have to be + factory functions providing the following interface: + + \code{factory(\var{errors}='strict')} + + The factory functions must return objects providing the interfaces + defined by the base classes \class{IncrementalEncoder} and + \class{IncrementalEncoder}, respectively. Incremental codecs can maintain + state. + + \var{streamreader} and \var{streamwriter}: These have to be factory functions providing the following interface: \code{factory(\var{stream}, \var{errors}='strict')} @@ -58,13 +79,13 @@ \end{funcdesc} \begin{funcdesc}{lookup}{encoding} -Looks up a codec tuple in the Python codec registry and returns the -function tuple as defined above. +Looks up the codec info in the Python codec registry and returns a +\class{CodecInfo} object as defined above. Encodings are first looked up in the registry's cache. If not found, -the list of registered search functions is scanned. If no codecs tuple -is found, a \exception{LookupError} is raised. Otherwise, the codecs -tuple is stored in the cache and returned to the caller. +the list of registered search functions is scanned. If no \class{CodecInfo} +object is found, a \exception{LookupError} is raised. Otherwise, the +\class{CodecInfo} object is stored in the cache and returned to the caller. \end{funcdesc} To simplify access to the various codecs, the module provides these @@ -85,6 +106,22 @@ Raises a \exception{LookupError} in case the encoding cannot be found. \end{funcdesc} +\begin{funcdesc}{getincrementalencoder}{encoding} +Lookup up the codec for the given encoding and return its incremental encoder +class or factory function. + +Raises a \exception{LookupError} in case the encoding cannot be found or the +codec doesn't support an incremental encoder. +\end{funcdesc} + +\begin{funcdesc}{getincrementaldecoder}{encoding} +Lookup up the codec for the given encoding and return its incremental decoder +class or factory function. + +Raises a \exception{LookupError} in case the encoding cannot be found or the +codec doesn't support an incremental decoder. +\end{funcdesc} + \begin{funcdesc}{getreader}{encoding} Lookup up the codec for the given encoding and return its StreamReader class or factory function. @@ -188,6 +225,18 @@ an encoding error occurs. \end{funcdesc} +\begin{funcdesc}{iterencode}{iterable, encoding\optional{, errors}} +Uses an incremental encoder to iteratively encode the input provided by +\var{iterable}. This function is a generator. \var{errors} (as well as +any other keyword argument) is passed through to the incremental encoder. +\end{funcdesc} + +\begin{funcdesc}{iterdecode}{iterable, encoding\optional{, errors}} +Uses an incremental decoder to iteratively decode the input provided by +\var{iterable}. This function is a generator. \var{errors} (as well as +any other keyword argument) is passed through to the incremental encoder. +\end{funcdesc} + The module also provides the following constants which are useful for reading and writing to platform dependent files: @@ -292,6 +341,109 @@ empty object of the output object type in this situation. \end{methoddesc} +The \class{IncrementalEncoder} and \class{IncrementalDecoder} classes provide +the basic interface for incremental encoding and decoding. Encoding/decoding the +input isn't done with one call to the stateless encoder/decoder function, +but with multiple calls to the \method{encode}/\method{decode} method of the +incremental encoder/decoder. The incremental encoder/decoder keeps track of +the encoding/decoding process during method calls. + +The joined output of calls to the \method{encode}/\method{decode} method is the +same as if the all single inputs where joined into one, and this input was +encoded/decoded with the stateless encoder/decoder. + + +\subsubsection{IncrementalEncoder Objects \label{incremental-encoder-objects}} + +The \class{IncrementalEncoder} class is used for encoding an input in multiple +steps. It defines the following methods which every incremental encoder must +define in order to be compatible to the Python codec registry. + +\begin{classdesc}{IncrementalEncoder}{\optional{errors}} + Constructor for a \class{IncrementalEncoder} instance. + + All incremental encoders must provide this constructor interface. They are + free to add additional keyword arguments, but only the ones defined + here are used by the Python codec registry. + + The \class{IncrementalEncoder} may implement different error handling + schemes by providing the \var{errors} keyword argument. These + parameters are predefined: + + \begin{itemize} + \item \code{'strict'} Raise \exception{ValueError} (or a subclass); + this is the default. + \item \code{'ignore'} Ignore the character and continue with the next. + \item \code{'replace'} Replace with a suitable replacement character + \item \code{'xmlcharrefreplace'} Replace with the appropriate XML + character reference + \item \code{'backslashreplace'} Replace with backslashed escape sequences. + \end{itemize} + + The \var{errors} argument will be assigned to an attribute of the + same name. Assigning to this attribute makes it possible to switch + between different error handling strategies during the lifetime + of the \class{IncrementalEncoder} object. + + The set of allowed values for the \var{errors} argument can + be extended with \function{register_error()}. +\end{classdesc} + +\begin{methoddesc}{encode}{object\optional{, final}} + Encodes \var{object} (taking the current state of the encoder into account) + and returns the resulting encoded object. If this is the last call to + \method{encode} \var{final} must be true (the default is false). +\end{methoddesc} + +\begin{methoddesc}{reset}{} + Reset the encoder to the initial state. +\end{methoddesc} + + +\subsubsection{IncrementalDecoder Objects \label{incremental-decoder-objects}} + +The \class{IncrementalDecoder} class is used for decoding an input in multiple +steps. It defines the following methods which every incremental decoder must +define in order to be compatible to the Python codec registry. + +\begin{classdesc}{IncrementalDecoder}{\optional{errors}} + Constructor for a \class{IncrementalDecoder} instance. + + All incremental decoders must provide this constructor interface. They are + free to add additional keyword arguments, but only the ones defined + here are used by the Python codec registry. + + The \class{IncrementalDecoder} may implement different error handling + schemes by providing the \var{errors} keyword argument. These + parameters are predefined: + + \begin{itemize} + \item \code{'strict'} Raise \exception{ValueError} (or a subclass); + this is the default. + \item \code{'ignore'} Ignore the character and continue with the next. + \item \code{'replace'} Replace with a suitable replacement character. + \end{itemize} + + The \var{errors} argument will be assigned to an attribute of the + same name. Assigning to this attribute makes it possible to switch + between different error handling strategies during the lifetime + of the \class{IncrementalEncoder} object. + + The set of allowed values for the \var{errors} argument can + be extended with \function{register_error()}. +\end{classdesc} + +\begin{methoddesc}{decode}{object\optional{, final}} + Decodes \var{object} (taking the current state of the decoder into account) + and returns the resulting decoded object. If this is the last call to + \method{decode} \var{final} must be true (the default is false). +\end{methoddesc} + +\begin{methoddesc}{reset}{} + Reset the decoder to the initial state. +\end{methoddesc} + + The \class{StreamWriter} and \class{StreamReader} classes provide generic working interfaces which can be used to implement new encodings submodules very easily. See \module{encodings.utf_8} for an Modified: python/branches/p3yk/Doc/whatsnew/whatsnew25.tex ============================================================================== --- python/branches/p3yk/Doc/whatsnew/whatsnew25.tex (original) +++ python/branches/p3yk/Doc/whatsnew/whatsnew25.tex Fri Apr 21 11:43:23 2006 @@ -210,6 +210,12 @@ %====================================================================== +\section{PEP 338: Executing Modules as Scripts} + +% XXX write this + + +%====================================================================== \section{PEP 341: Unified try/except/finally} % XXX write this Modified: python/branches/p3yk/Include/codecs.h ============================================================================== --- python/branches/p3yk/Include/codecs.h (original) +++ python/branches/p3yk/Include/codecs.h Fri Apr 21 11:43:23 2006 @@ -29,15 +29,15 @@ /* Codec register lookup API. - Looks up the given encoding and returns a tuple (encoder, decoder, - stream reader, stream writer) of functions which implement the - different aspects of processing the encoding. + Looks up the given encoding and returns a CodecInfo object with + function attributes which implement the different aspects of + processing the encoding. The encoding string is looked up converted to all lower-case characters. This makes encodings looked up through this mechanism effectively case-insensitive. - If no codec is found, a KeyError is set and NULL returned. + If no codec is found, a KeyError is set and NULL returned. As side effect, this tries to load the encodings package, if not yet done. This is part of the lazy load strategy for the encodings @@ -101,6 +101,20 @@ const char *encoding ); +/* Get a IncrementalEncoder object for the given encoding. */ + +PyAPI_FUNC(PyObject *) PyCodec_IncrementalEncoder( + const char *encoding, + const char *errors + ); + +/* Get a IncrementalDecoder object function for the given encoding. */ + +PyAPI_FUNC(PyObject *) PyCodec_IncrementalDecoder( + const char *encoding, + const char *errors + ); + /* Get a StreamReader factory function for the given encoding. */ PyAPI_FUNC(PyObject *) PyCodec_StreamReader( Modified: python/branches/p3yk/Lib/StringIO.py ============================================================================== --- python/branches/p3yk/Lib/StringIO.py (original) +++ python/branches/p3yk/Lib/StringIO.py Fri Apr 21 11:43:23 2006 @@ -72,8 +72,7 @@ method is called repeatedly. This method returns the next input line, or raises StopIteration when EOF is hit. """ - if self.closed: - raise StopIteration + _complain_ifclosed(self.closed) r = self.readline() if not r: raise StopIteration Modified: python/branches/p3yk/Lib/codecs.py ============================================================================== --- python/branches/p3yk/Lib/codecs.py (original) +++ python/branches/p3yk/Lib/codecs.py Fri Apr 21 11:43:23 2006 @@ -73,6 +73,23 @@ ### Codec base classes (defining the API) +class CodecInfo(tuple): + + def __new__(cls, encode, decode, streamreader=None, streamwriter=None, + incrementalencoder=None, incrementaldecoder=None, name=None): + self = tuple.__new__(cls, (encode, decode, streamreader, streamwriter)) + self.name = name + self.encode = encode + self.decode = decode + self.incrementalencoder = incrementalencoder + self.incrementaldecoder = incrementaldecoder + self.streamwriter = streamwriter + self.streamreader = streamreader + return self + + def __repr__(self): + return "<%s.%s object for encoding %s at 0x%x>" % (self.__class__.__module__, self.__class__.__name__, self.name, id(self)) + class Codec: """ Defines the interface for stateless encoders/decoders. @@ -137,6 +154,88 @@ """ raise NotImplementedError +class IncrementalEncoder(object): + """ + A IncrementalEncoder encodes an input in multiple steps. The input can be + passed piece by piece to the encode() method. The IncrementalEncoder remembers + the state of the Encoding process between calls to encode(). + """ + def __init__(self, errors='strict'): + """ + Creates a IncrementalEncoder instance. + + The IncrementalEncoder may use different error handling schemes by + providing the errors keyword argument. See the module docstring + for a list of possible values. + """ + self.errors = errors + self.buffer = "" + + def encode(self, input, final=False): + """ + Encodes input and returns the resulting object. + """ + raise NotImplementedError + + def reset(self): + """ + Resets the encoder to the initial state. + """ + +class IncrementalDecoder(object): + """ + An IncrementalDecoder decodes an input in multiple steps. The input can be + passed piece by piece to the decode() method. The IncrementalDecoder + remembers the state of the decoding process between calls to decode(). + """ + def __init__(self, errors='strict'): + """ + Creates a IncrementalDecoder instance. + + The IncrementalDecoder may use different error handling schemes by + providing the errors keyword argument. See the module docstring + for a list of possible values. + """ + self.errors = errors + + def decode(self, input, final=False): + """ + Decodes input and returns the resulting object. + """ + raise NotImplementedError + + def reset(self): + """ + Resets the decoder to the initial state. + """ + +class BufferedIncrementalDecoder(IncrementalDecoder): + """ + This subclass of IncrementalDecoder can be used as the baseclass for an + incremental decoder if the decoder must be able to handle incomplete byte + sequences. + """ + def __init__(self, errors='strict'): + IncrementalDecoder.__init__(self, errors) + self.buffer = "" # undecoded input that is kept between calls to decode() + + def _buffer_decode(self, input, errors, final): + # Overwrite this method in subclasses: It must decode input + # and return an (output, length consumed) tuple + raise NotImplementedError + + def decode(self, input, final=False): + # decode input (taking the buffer into account) + data = self.buffer + input + (result, consumed) = self._buffer_decode(data, self.errors, final) + # keep undecoded input until the next call + self.buffer = data[consumed:] + return result + + def reset(self): + IncrementalDecoder.reset(self) + self.bytebuffer = "" + # # The StreamWriter and StreamReader class provide generic working # interfaces which can be used to implement new encoding submodules @@ -666,8 +765,8 @@ file = __builtin__.open(filename, mode, buffering) if encoding is None: return file - (e, d, sr, sw) = lookup(encoding) - srw = StreamReaderWriter(file, sr, sw, errors) + info = lookup(encoding) + srw = StreamReaderWriter(file, info.streamreader, info.streamwriter, errors) # Add attributes to simplify introspection srw.encoding = encoding return srw @@ -699,11 +798,9 @@ """ if file_encoding is None: file_encoding = data_encoding - encode, decode = lookup(data_encoding)[:2] - Reader, Writer = lookup(file_encoding)[2:] - sr = StreamRecoder(file, - encode, decode, Reader, Writer, - errors) + info = lookup(data_encoding) + sr = StreamRecoder(file, info.encode, info.decode, + info.streamreader, info.streamwriter, errors) # Add attributes to simplify introspection sr.data_encoding = data_encoding sr.file_encoding = file_encoding @@ -719,7 +816,7 @@ Raises a LookupError in case the encoding cannot be found. """ - return lookup(encoding)[0] + return lookup(encoding).encode def getdecoder(encoding): @@ -729,7 +826,35 @@ Raises a LookupError in case the encoding cannot be found. """ - return lookup(encoding)[1] + return lookup(encoding).decode + +def getincrementalencoder(encoding): + + """ Lookup up the codec for the given encoding and return + its IncrementalEncoder class or factory function. + + Raises a LookupError in case the encoding cannot be found + or the codecs doesn't provide an incremental encoder. + + """ + encoder = lookup(encoding).incrementalencoder + if encoder is None: + raise LookupError(encoding) + return encoder + +def getincrementaldecoder(encoding): + + """ Lookup up the codec for the given encoding and return + its IncrementalDecoder class or factory function. + + Raises a LookupError in case the encoding cannot be found + or the codecs doesn't provide an incremental decoder. + + """ + decoder = lookup(encoding).incrementaldecoder + if decoder is None: + raise LookupError(encoding) + return decoder def getreader(encoding): @@ -739,7 +864,7 @@ Raises a LookupError in case the encoding cannot be found. """ - return lookup(encoding)[2] + return lookup(encoding).streamreader def getwriter(encoding): @@ -749,7 +874,43 @@ Raises a LookupError in case the encoding cannot be found. """ - return lookup(encoding)[3] + return lookup(encoding).streamwriter + +def iterencode(iterator, encoding, errors='strict', **kwargs): + """ + Encoding iterator. + + Encodes the input strings from the iterator using a IncrementalEncoder. + + errors and kwargs are passed through to the IncrementalEncoder + constructor. + """ + encoder = getincrementalencoder(encoding)(errors, **kwargs) + for input in iterator: + output = encoder.encode(input) + if output: + yield output + output = encoder.encode("", True) + if output: + yield output + +def iterdecode(iterator, encoding, errors='strict', **kwargs): + """ + Decoding iterator. + + Decodes the input strings from the iterator using a IncrementalDecoder. + + errors and kwargs are passed through to the IncrementalDecoder + constructor. + """ + decoder = getincrementaldecoder(encoding)(errors, **kwargs) + for input in iterator: + output = decoder.decode(input) + if output: + yield output + output = decoder.decode("", True) + if output: + yield output ### Helpers for charmap-based codecs Modified: python/branches/p3yk/Lib/encodings/__init__.py ============================================================================== --- python/branches/p3yk/Lib/encodings/__init__.py (original) +++ python/branches/p3yk/Lib/encodings/__init__.py Fri Apr 21 11:43:23 2006 @@ -9,9 +9,10 @@ Each codec module must export the following interface: - * getregentry() -> (encoder, decoder, stream_reader, stream_writer) - The getregentry() API must return callable objects which adhere to - the Python Codec Interface Standard. + * getregentry() -> codecs.CodecInfo object + The getregentry() API must a CodecInfo object with encoder, decoder, + incrementalencoder, incrementaldecoder, streamwriter and streamreader + atttributes which adhere to the Python Codec Interface Standard. In addition, a module may optionally also define the following APIs which are then used by the package's codec search function: @@ -113,16 +114,24 @@ return None # Now ask the module for the registry entry - entry = tuple(getregentry()) - if len(entry) != 4: - raise CodecRegistryError,\ - 'module "%s" (%s) failed to register' % \ - (mod.__name__, mod.__file__) - for obj in entry: - if not callable(obj): + entry = getregentry() + if not isinstance(entry, codecs.CodecInfo): + if not 4 <= len(entry) <= 7: raise CodecRegistryError,\ - 'incompatible codecs in module "%s" (%s)' % \ + 'module "%s" (%s) failed to register' % \ (mod.__name__, mod.__file__) + if not callable(entry[0]) or \ + not callable(entry[1]) or \ + (entry[2] is not None and not callable(entry[2])) or \ + (entry[3] is not None and not callable(entry[3])) or \ + (len(entry) > 4 and entry[4] is not None and not callable(entry[4])) or \ + (len(entry) > 5 and entry[5] is not None and not callable(entry[5])): + raise CodecRegistryError,\ + 'incompatible codecs in module "%s" (%s)' % \ + (mod.__name__, mod.__file__) + if len(entry)<7 or entry[6] is None: + entry += (None,)*(6-len(entry)) + (mod.__name__.split(".", 1)[1],) + entry = codecs.CodecInfo(*entry) # Cache the codec registry entry _cache[encoding] = entry Modified: python/branches/p3yk/Lib/encodings/ascii.py ============================================================================== --- python/branches/p3yk/Lib/encodings/ascii.py (original) +++ python/branches/p3yk/Lib/encodings/ascii.py Fri Apr 21 11:43:23 2006 @@ -17,6 +17,14 @@ encode = codecs.ascii_encode decode = codecs.ascii_decode +class IncrementalEncoder(codecs.IncrementalEncoder): + def encode(self, input, final=False): + return codecs.ascii_encode(input, self.errors)[0] + +class IncrementalDecoder(codecs.IncrementalDecoder): + def decode(self, input, final=False): + return codecs.ascii_decode(input, self.errors)[0] + class StreamWriter(Codec,codecs.StreamWriter): pass @@ -31,5 +39,12 @@ ### encodings module API def getregentry(): - - return (Codec.encode,Codec.decode,StreamReader,StreamWriter) + return codecs.CodecInfo( + name='ascii', + encode=Codec.encode, + decode=Codec.decode, + incrementalencoder=IncrementalEncoder, + incrementaldecoder=IncrementalDecoder, + streamwriter=StreamWriter, + streamreader=StreamReader, + ) Modified: python/branches/p3yk/Lib/encodings/base64_codec.py ============================================================================== --- python/branches/p3yk/Lib/encodings/base64_codec.py (original) +++ python/branches/p3yk/Lib/encodings/base64_codec.py Fri Apr 21 11:43:23 2006 @@ -49,6 +49,16 @@ def decode(self, input,errors='strict'): return base64_decode(input,errors) +class IncrementalEncoder(codecs.IncrementalEncoder): + def encode(self, input, final=False): + assert self.errors == 'strict' + return base64.encodestring(input) + +class IncrementalDecoder(codecs.IncrementalDecoder): + def decode(self, input, final=False): + assert self.errors == 'strict' + return base64.decodestring(input) + class StreamWriter(Codec,codecs.StreamWriter): pass @@ -58,5 +68,12 @@ ### encodings module API def getregentry(): - - return (base64_encode,base64_decode,StreamReader,StreamWriter) + return codecs.CodecInfo( + name='base64', + encode=base64_encode, + decode=base64_decode, + incrementalencoder=IncrementalEncoder, + incrementaldecoder=IncrementalDecoder, + streamwriter=StreamWriter, + streamreader=StreamReader, + ) Modified: python/branches/p3yk/Lib/encodings/bz2_codec.py ============================================================================== --- python/branches/p3yk/Lib/encodings/bz2_codec.py (original) +++ python/branches/p3yk/Lib/encodings/bz2_codec.py Fri Apr 21 11:43:23 2006 @@ -51,6 +51,16 @@ def decode(self, input, errors='strict'): return bz2_decode(input, errors) +class IncrementalEncoder(codecs.IncrementalEncoder): + def encode(self, input, final=False): + assert self.errors == 'strict' + return bz2.compress(input) + +class IncrementalDecoder(codecs.IncrementalDecoder): + def decode(self, input, final=False): + assert self.errors == 'strict' + return bz2.decompress(input) + class StreamWriter(Codec,codecs.StreamWriter): pass @@ -60,5 +70,12 @@ ### encodings module API def getregentry(): - - return (bz2_encode,bz2_decode,StreamReader,StreamWriter) + return codecs.CodecInfo( + name="bz2", + encode=bz2_encode, + decode=bz2_decode, + incrementalencoder=IncrementalEncoder, + incrementaldecoder=IncrementalDecoder, + streamwriter=StreamWriter, + streamreader=StreamReader, + ) Modified: python/branches/p3yk/Lib/encodings/charmap.py ============================================================================== --- python/branches/p3yk/Lib/encodings/charmap.py (original) +++ python/branches/p3yk/Lib/encodings/charmap.py Fri Apr 21 11:43:23 2006 @@ -21,30 +21,49 @@ encode = codecs.charmap_encode decode = codecs.charmap_decode +class IncrementalEncoder(codecs.IncrementalEncoder): + def __init__(self, errors='strict', mapping=None): + codecs.IncrementalEncoder.__init__(self, errors) + self.mapping = mapping + + def encode(self, input, final=False): + return codecs.charmap_encode(input, self.errors, self.mapping)[0] + +class IncrementalDecoder(codecs.IncrementalDecoder): + def __init__(self, errors='strict', mapping=None): + codecs.IncrementalDecoder.__init__(self, errors) + self.mapping = mapping + + def decode(self, input, final=False): + return codecs.charmap_decode(input, self.errors, self.mapping)[0] + class StreamWriter(Codec,codecs.StreamWriter): def __init__(self,stream,errors='strict',mapping=None): - codecs.StreamWriter.__init__(self,stream,errors) self.mapping = mapping def encode(self,input,errors='strict'): - return Codec.encode(input,errors,self.mapping) class StreamReader(Codec,codecs.StreamReader): def __init__(self,stream,errors='strict',mapping=None): - codecs.StreamReader.__init__(self,stream,errors) self.mapping = mapping def decode(self,input,errors='strict'): - return Codec.decode(input,errors,self.mapping) ### encodings module API def getregentry(): - - return (Codec.encode,Codec.decode,StreamReader,StreamWriter) + return codecs.CodecInfo( + name='charmap', + encode=Codec.encode, + decode=Codec.decode, + incrementalencoder=IncrementalEncoder, + incrementaldecoder=IncrementalDecoder, + streamwriter=StreamWriter, + streamreader=StreamReader, + ) Modified: python/branches/p3yk/Lib/encodings/cp037.py ============================================================================== --- python/branches/p3yk/Lib/encodings/cp037.py (original) +++ python/branches/p3yk/Lib/encodings/cp037.py Fri Apr 21 11:43:23 2006 @@ -1,4 +1,4 @@ -""" Python Character Mapping Codec generated from 'MAPPINGS/VENDORS/MICSFT/EBCDIC/CP037.TXT' with gencodec.py. +""" Python Character Mapping Codec cp037 generated from 'MAPPINGS/VENDORS/MICSFT/EBCDIC/CP037.TXT' with gencodec.py. """#" @@ -9,13 +9,19 @@ class Codec(codecs.Codec): def encode(self,input,errors='strict'): - return codecs.charmap_encode(input,errors,encoding_map) def decode(self,input,errors='strict'): - return codecs.charmap_decode(input,errors,decoding_table) +class IncrementalEncoder(codecs.IncrementalEncoder): + def encode(self, input, final=False): + return codecs.charmap_encode(input,self.errors,encoding_map)[0] + +class IncrementalDecoder(codecs.IncrementalDecoder): + def decode(self, input, final=False): + return codecs.charmap_decode(input,self.errors,decoding_table)[0] + class StreamWriter(Codec,codecs.StreamWriter): pass @@ -25,8 +31,15 @@ ### encodings module API def getregentry(): - - return (Codec().encode,Codec().decode,StreamReader,StreamWriter) + return codecs.CodecInfo( + name='cp037', + encode=Codec().encode, + decode=Codec().decode, + incrementalencoder=IncrementalEncoder, + incrementaldecoder=IncrementalDecoder, + streamreader=StreamReader, + streamwriter=StreamWriter, + ) ### Decoding Table Modified: python/branches/p3yk/Lib/encodings/cp1006.py ============================================================================== --- python/branches/p3yk/Lib/encodings/cp1006.py (original) +++ python/branches/p3yk/Lib/encodings/cp1006.py Fri Apr 21 11:43:23 2006 @@ -1,4 +1,4 @@ -""" Python Character Mapping Codec generated from 'MAPPINGS/VENDORS/MISC/CP1006.TXT' with gencodec.py. +""" Python Character Mapping Codec cp1006 generated from 'MAPPINGS/VENDORS/MISC/CP1006.TXT' with gencodec.py. """#" @@ -9,13 +9,19 @@ class Codec(codecs.Codec): def encode(self,input,errors='strict'): - return codecs.charmap_encode(input,errors,encoding_map) def decode(self,input,errors='strict'): - return codecs.charmap_decode(input,errors,decoding_table) +class IncrementalEncoder(codecs.IncrementalEncoder): + def encode(self, input, final=False): + return codecs.charmap_encode(input,self.errors,encoding_map)[0] + +class IncrementalDecoder(codecs.IncrementalDecoder): + def decode(self, input, final=False): + return codecs.charmap_decode(input,self.errors,decoding_table)[0] + class StreamWriter(Codec,codecs.StreamWriter): pass @@ -25,8 +31,15 @@ ### encodings module API def getregentry(): - - return (Codec().encode,Codec().decode,StreamReader,StreamWriter) + return codecs.CodecInfo( + name='cp1006', + encode=Codec().encode, + decode=Codec().decode, + incrementalencoder=IncrementalEncoder, + incrementaldecoder=IncrementalDecoder, + streamreader=StreamReader, + streamwriter=StreamWriter, + ) ### Decoding Table Modified: python/branches/p3yk/Lib/encodings/cp1026.py ============================================================================== --- python/branches/p3yk/Lib/encodings/cp1026.py (original) +++ python/branches/p3yk/Lib/encodings/cp1026.py Fri Apr 21 11:43:23 2006 @@ -1,4 +1,4 @@ -""" Python Character Mapping Codec generated from 'MAPPINGS/VENDORS/MICSFT/EBCDIC/CP1026.TXT' with gencodec.py. +""" Python Character Mapping Codec cp1026 generated from 'MAPPINGS/VENDORS/MICSFT/EBCDIC/CP1026.TXT' with gencodec.py. """#" @@ -9,13 +9,19 @@ class Codec(codecs.Codec): def encode(self,input,errors='strict'): - return codecs.charmap_encode(input,errors,encoding_map) def decode(self,input,errors='strict'): - return codecs.charmap_decode(input,errors,decoding_table) +class IncrementalEncoder(codecs.IncrementalEncoder): + def encode(self, input, final=False): + return codecs.charmap_encode(input,self.errors,encoding_map)[0] + +class IncrementalDecoder(codecs.IncrementalDecoder): + def decode(self, input, final=False): + return codecs.charmap_decode(input,self.errors,decoding_table)[0] + class StreamWriter(Codec,codecs.StreamWriter): pass @@ -25,8 +31,15 @@ ### encodings module API def getregentry(): - - return (Codec().encode,Codec().decode,StreamReader,StreamWriter) + return codecs.CodecInfo( + name='cp1026', + encode=Codec().encode, + decode=Codec().decode, + incrementalencoder=IncrementalEncoder, + incrementaldecoder=IncrementalDecoder, + streamreader=StreamReader, + streamwriter=StreamWriter, + ) ### Decoding Table Modified: python/branches/p3yk/Lib/encodings/cp1140.py ============================================================================== --- python/branches/p3yk/Lib/encodings/cp1140.py (original) +++ python/branches/p3yk/Lib/encodings/cp1140.py Fri Apr 21 11:43:23 2006 @@ -1,4 +1,4 @@ -""" Python Character Mapping Codec generated from 'python-mappings/CP1140.TXT' with gencodec.py. +""" Python Character Mapping Codec cp1140 generated from 'python-mappings/CP1140.TXT' with gencodec.py. """#" @@ -9,13 +9,19 @@ class Codec(codecs.Codec): def encode(self,input,errors='strict'): - return codecs.charmap_encode(input,errors,encoding_map) def decode(self,input,errors='strict'): - return codecs.charmap_decode(input,errors,decoding_table) +class IncrementalEncoder(codecs.IncrementalEncoder): + def encode(self, input, final=False): + return codecs.charmap_encode(input,self.errors,encoding_map)[0] + +class IncrementalDecoder(codecs.IncrementalDecoder): + def decode(self, input, final=False): + return codecs.charmap_decode(input,self.errors,decoding_table)[0] + class StreamWriter(Codec,codecs.StreamWriter): pass @@ -25,8 +31,15 @@ ### encodings module API def getregentry(): - - return (Codec().encode,Codec().decode,StreamReader,StreamWriter) + return codecs.CodecInfo( + name='cp1140', + encode=Codec().encode, + decode=Codec().decode, + incrementalencoder=IncrementalEncoder, + incrementaldecoder=IncrementalDecoder, + streamreader=StreamReader, + streamwriter=StreamWriter, + ) ### Decoding Table Modified: python/branches/p3yk/Lib/encodings/cp1250.py ============================================================================== --- python/branches/p3yk/Lib/encodings/cp1250.py (original) +++ python/branches/p3yk/Lib/encodings/cp1250.py Fri Apr 21 11:43:23 2006 @@ -1,4 +1,4 @@ -""" Python Character Mapping Codec generated from 'MAPPINGS/VENDORS/MICSFT/WINDOWS/CP1250.TXT' with gencodec.py. +""" Python Character Mapping Codec cp1250 generated from 'MAPPINGS/VENDORS/MICSFT/WINDOWS/CP1250.TXT' with gencodec.py. """#" @@ -9,13 +9,19 @@ class Codec(codecs.Codec): def encode(self,input,errors='strict'): - return codecs.charmap_encode(input,errors,encoding_map) def decode(self,input,errors='strict'): - return codecs.charmap_decode(input,errors,decoding_table) +class IncrementalEncoder(codecs.IncrementalEncoder): + def encode(self, input, final=False): + return codecs.charmap_encode(input,self.errors,encoding_map)[0] + +class IncrementalDecoder(codecs.IncrementalDecoder): + def decode(self, input, final=False): + return codecs.charmap_decode(input,self.errors,decoding_table)[0] + class StreamWriter(Codec,codecs.StreamWriter): pass @@ -25,8 +31,15 @@ ### encodings module API def getregentry(): - - return (Codec().encode,Codec().decode,StreamReader,StreamWriter) + return codecs.CodecInfo( + name='cp1250', + encode=Codec().encode, + decode=Codec().decode, + incrementalencoder=IncrementalEncoder, + incrementaldecoder=IncrementalDecoder, + streamreader=StreamReader, + streamwriter=StreamWriter, + ) ### Decoding Table Modified: python/branches/p3yk/Lib/encodings/cp1251.py ============================================================================== --- python/branches/p3yk/Lib/encodings/cp1251.py (original) +++ python/branches/p3yk/Lib/encodings/cp1251.py Fri Apr 21 11:43:23 2006 @@ -1,4 +1,4 @@ -""" Python Character Mapping Codec generated from 'MAPPINGS/VENDORS/MICSFT/WINDOWS/CP1251.TXT' with gencodec.py. +""" Python Character Mapping Codec cp1251 generated from 'MAPPINGS/VENDORS/MICSFT/WINDOWS/CP1251.TXT' with gencodec.py. """#" @@ -9,13 +9,19 @@ class Codec(codecs.Codec): def encode(self,input,errors='strict'): - return codecs.charmap_encode(input,errors,encoding_map) def decode(self,input,errors='strict'): - return codecs.charmap_decode(input,errors,decoding_table) +class IncrementalEncoder(codecs.IncrementalEncoder): + def encode(self, input, final=False): + return codecs.charmap_encode(input,self.errors,encoding_map)[0] + +class IncrementalDecoder(codecs.IncrementalDecoder): + def decode(self, input, final=False): + return codecs.charmap_decode(input,self.errors,decoding_table)[0] + class StreamWriter(Codec,codecs.StreamWriter): pass @@ -25,8 +31,15 @@ ### encodings module API def getregentry(): - - return (Codec().encode,Codec().decode,StreamReader,StreamWriter) + return codecs.CodecInfo( + name='cp1251', + encode=Codec().encode, + decode=Codec().decode, + incrementalencoder=IncrementalEncoder, + incrementaldecoder=IncrementalDecoder, + streamreader=StreamReader, + streamwriter=StreamWriter, + ) ### Decoding Table Modified: python/branches/p3yk/Lib/encodings/cp1252.py ============================================================================== --- python/branches/p3yk/Lib/encodings/cp1252.py (original) +++ python/branches/p3yk/Lib/encodings/cp1252.py Fri Apr 21 11:43:23 2006 @@ -1,4 +1,4 @@ -""" Python Character Mapping Codec generated from 'MAPPINGS/VENDORS/MICSFT/WINDOWS/CP1252.TXT' with gencodec.py. +""" Python Character Mapping Codec cp1252 generated from 'MAPPINGS/VENDORS/MICSFT/WINDOWS/CP1252.TXT' with gencodec.py. """#" @@ -9,13 +9,19 @@ class Codec(codecs.Codec): def encode(self,input,errors='strict'): - return codecs.charmap_encode(input,errors,encoding_map) def decode(self,input,errors='strict'): - return codecs.charmap_decode(input,errors,decoding_table) +class IncrementalEncoder(codecs.IncrementalEncoder): + def encode(self, input, final=False): + return codecs.charmap_encode(input,self.errors,encoding_map)[0] + +class IncrementalDecoder(codecs.IncrementalDecoder): + def decode(self, input, final=False): + return codecs.charmap_decode(input,self.errors,decoding_table)[0] + class StreamWriter(Codec,codecs.StreamWriter): pass @@ -25,8 +31,15 @@ ### encodings module API def getregentry(): - - return (Codec().encode,Codec().decode,StreamReader,StreamWriter) + return codecs.CodecInfo( + name='cp1252', + encode=Codec().encode, + decode=Codec().decode, + incrementalencoder=IncrementalEncoder, + incrementaldecoder=IncrementalDecoder, + streamreader=StreamReader, + streamwriter=StreamWriter, + ) ### Decoding Table Modified: python/branches/p3yk/Lib/encodings/cp1253.py ============================================================================== --- python/branches/p3yk/Lib/encodings/cp1253.py (original) +++ python/branches/p3yk/Lib/encodings/cp1253.py Fri Apr 21 11:43:23 2006 @@ -1,4 +1,4 @@ -""" Python Character Mapping Codec generated from 'MAPPINGS/VENDORS/MICSFT/WINDOWS/CP1253.TXT' with gencodec.py. +""" Python Character Mapping Codec cp1253 generated from 'MAPPINGS/VENDORS/MICSFT/WINDOWS/CP1253.TXT' with gencodec.py. """#" @@ -9,13 +9,19 @@ class Codec(codecs.Codec): def encode(self,input,errors='strict'): - return codecs.charmap_encode(input,errors,encoding_map) def decode(self,input,errors='strict'): - return codecs.charmap_decode(input,errors,decoding_table) +class IncrementalEncoder(codecs.IncrementalEncoder): + def encode(self, input, final=False): + return codecs.charmap_encode(input,self.errors,encoding_map)[0] + +class IncrementalDecoder(codecs.IncrementalDecoder): + def decode(self, input, final=False): + return codecs.charmap_decode(input,self.errors,decoding_table)[0] + class StreamWriter(Codec,codecs.StreamWriter): pass @@ -25,8 +31,15 @@ ### encodings module API def getregentry(): - - return (Codec().encode,Codec().decode,StreamReader,StreamWriter) + return codecs.CodecInfo( + name='cp1253', + encode=Codec().encode, + decode=Codec().decode, + incrementalencoder=IncrementalEncoder, + incrementaldecoder=IncrementalDecoder, + streamreader=StreamReader, + streamwriter=StreamWriter, + ) ### Decoding Table Modified: python/branches/p3yk/Lib/encodings/cp1254.py ============================================================================== --- python/branches/p3yk/Lib/encodings/cp1254.py (original) +++ python/branches/p3yk/Lib/encodings/cp1254.py Fri Apr 21 11:43:23 2006 @@ -1,4 +1,4 @@ -""" Python Character Mapping Codec generated from 'MAPPINGS/VENDORS/MICSFT/WINDOWS/CP1254.TXT' with gencodec.py. +""" Python Character Mapping Codec cp1254 generated from 'MAPPINGS/VENDORS/MICSFT/WINDOWS/CP1254.TXT' with gencodec.py. """#" @@ -9,13 +9,19 @@ class Codec(codecs.Codec): def encode(self,input,errors='strict'): - return codecs.charmap_encode(input,errors,encoding_map) def decode(self,input,errors='strict'): - return codecs.charmap_decode(input,errors,decoding_table) +class IncrementalEncoder(codecs.IncrementalEncoder): + def encode(self, input, final=False): + return codecs.charmap_encode(input,self.errors,encoding_map)[0] + +class IncrementalDecoder(codecs.IncrementalDecoder): + def decode(self, input, final=False): + return codecs.charmap_decode(input,self.errors,decoding_table)[0] + class StreamWriter(Codec,codecs.StreamWriter): pass @@ -25,8 +31,15 @@ ### encodings module API def getregentry(): - - return (Codec().encode,Codec().decode,StreamReader,StreamWriter) + return codecs.CodecInfo( + name='cp1254', + encode=Codec().encode, + decode=Codec().decode, + incrementalencoder=IncrementalEncoder, + incrementaldecoder=IncrementalDecoder, + streamreader=StreamReader, + streamwriter=StreamWriter, + ) ### Decoding Table Modified: python/branches/p3yk/Lib/encodings/cp1255.py ============================================================================== --- python/branches/p3yk/Lib/encodings/cp1255.py (original) +++ python/branches/p3yk/Lib/encodings/cp1255.py Fri Apr 21 11:43:23 2006 @@ -1,4 +1,4 @@ -""" Python Character Mapping Codec generated from 'MAPPINGS/VENDORS/MICSFT/WINDOWS/CP1255.TXT' with gencodec.py. +""" Python Character Mapping Codec cp1255 generated from 'MAPPINGS/VENDORS/MICSFT/WINDOWS/CP1255.TXT' with gencodec.py. """#" @@ -9,13 +9,19 @@ class Codec(codecs.Codec): def encode(self,input,errors='strict'): - return codecs.charmap_encode(input,errors,encoding_map) def decode(self,input,errors='strict'): - return codecs.charmap_decode(input,errors,decoding_table) +class IncrementalEncoder(codecs.IncrementalEncoder): + def encode(self, input, final=False): + return codecs.charmap_encode(input,self.errors,encoding_map)[0] + +class IncrementalDecoder(codecs.IncrementalDecoder): + def decode(self, input, final=False): + return codecs.charmap_decode(input,self.errors,decoding_table)[0] + class StreamWriter(Codec,codecs.StreamWriter): pass @@ -25,8 +31,15 @@ ### encodings module API def getregentry(): - - return (Codec().encode,Codec().decode,StreamReader,StreamWriter) + return codecs.CodecInfo( + name='cp1255', + encode=Codec().encode, + decode=Codec().decode, + incrementalencoder=IncrementalEncoder, + incrementaldecoder=IncrementalDecoder, + streamreader=StreamReader, + streamwriter=StreamWriter, + ) ### Decoding Table Modified: python/branches/p3yk/Lib/encodings/cp1256.py ============================================================================== --- python/branches/p3yk/Lib/encodings/cp1256.py (original) +++ python/branches/p3yk/Lib/encodings/cp1256.py Fri Apr 21 11:43:23 2006 @@ -1,4 +1,4 @@ -""" Python Character Mapping Codec generated from 'MAPPINGS/VENDORS/MICSFT/WINDOWS/CP1256.TXT' with gencodec.py. +""" Python Character Mapping Codec cp1256 generated from 'MAPPINGS/VENDORS/MICSFT/WINDOWS/CP1256.TXT' with gencodec.py. """#" @@ -9,13 +9,19 @@ class Codec(codecs.Codec): def encode(self,input,errors='strict'): - return codecs.charmap_encode(input,errors,encoding_map) def decode(self,input,errors='strict'): - return codecs.charmap_decode(input,errors,decoding_table) +class IncrementalEncoder(codecs.IncrementalEncoder): + def encode(self, input, final=False): + return codecs.charmap_encode(input,self.errors,encoding_map)[0] + +class IncrementalDecoder(codecs.IncrementalDecoder): + def decode(self, input, final=False): + return codecs.charmap_decode(input,self.errors,decoding_table)[0] + class StreamWriter(Codec,codecs.StreamWriter): pass @@ -25,8 +31,15 @@ ### encodings module API def getregentry(): - - return (Codec().encode,Codec().decode,StreamReader,StreamWriter) + return codecs.CodecInfo( + name='cp1256', + encode=Codec().encode, + decode=Codec().decode, + incrementalencoder=IncrementalEncoder, + incrementaldecoder=IncrementalDecoder, + streamreader=StreamReader, + streamwriter=StreamWriter, + ) ### Decoding Table Modified: python/branches/p3yk/Lib/encodings/cp1257.py ============================================================================== --- python/branches/p3yk/Lib/encodings/cp1257.py (original) +++ python/branches/p3yk/Lib/encodings/cp1257.py Fri Apr 21 11:43:23 2006 @@ -1,4 +1,4 @@ -""" Python Character Mapping Codec generated from 'MAPPINGS/VENDORS/MICSFT/WINDOWS/CP1257.TXT' with gencodec.py. +""" Python Character Mapping Codec cp1257 generated from 'MAPPINGS/VENDORS/MICSFT/WINDOWS/CP1257.TXT' with gencodec.py. """#" @@ -9,13 +9,19 @@ class Codec(codecs.Codec): def encode(self,input,errors='strict'): - return codecs.charmap_encode(input,errors,encoding_map) def decode(self,input,errors='strict'): - return codecs.charmap_decode(input,errors,decoding_table) +class IncrementalEncoder(codecs.IncrementalEncoder): + def encode(self, input, final=False): + return codecs.charmap_encode(input,self.errors,encoding_map)[0] + +class IncrementalDecoder(codecs.IncrementalDecoder): + def decode(self, input, final=False): + return codecs.charmap_decode(input,self.errors,decoding_table)[0] + class StreamWriter(Codec,codecs.StreamWriter): pass @@ -25,8 +31,15 @@ ### encodings module API def getregentry(): - - return (Codec().encode,Codec().decode,StreamReader,StreamWriter) + return codecs.CodecInfo( + name='cp1257', + encode=Codec().encode, + decode=Codec().decode, + incrementalencoder=IncrementalEncoder, + incrementaldecoder=IncrementalDecoder, + streamreader=StreamReader, + streamwriter=StreamWriter, + ) ### Decoding Table Modified: python/branches/p3yk/Lib/encodings/cp1258.py ============================================================================== --- python/branches/p3yk/Lib/encodings/cp1258.py (original) +++ python/branches/p3yk/Lib/encodings/cp1258.py Fri Apr 21 11:43:23 2006 @@ -1,4 +1,4 @@ -""" Python Character Mapping Codec generated from 'MAPPINGS/VENDORS/MICSFT/WINDOWS/CP1258.TXT' with gencodec.py. +""" Python Character Mapping Codec cp1258 generated from 'MAPPINGS/VENDORS/MICSFT/WINDOWS/CP1258.TXT' with gencodec.py. """#" @@ -9,13 +9,19 @@ class Codec(codecs.Codec): def encode(self,input,errors='strict'): - return codecs.charmap_encode(input,errors,encoding_map) def decode(self,input,errors='strict'): - return codecs.charmap_decode(input,errors,decoding_table) +class IncrementalEncoder(codecs.IncrementalEncoder): + def encode(self, input, final=False): + return codecs.charmap_encode(input,self.errors,encoding_map)[0] + +class IncrementalDecoder(codecs.IncrementalDecoder): + def decode(self, input, final=False): + return codecs.charmap_decode(input,self.errors,decoding_table)[0] + class StreamWriter(Codec,codecs.StreamWriter): pass @@ -25,8 +31,15 @@ ### encodings module API def getregentry(): - - return (Codec().encode,Codec().decode,StreamReader,StreamWriter) + return codecs.CodecInfo( + name='cp1258', + encode=Codec().encode, + decode=Codec().decode, + incrementalencoder=IncrementalEncoder, + incrementaldecoder=IncrementalDecoder, + streamreader=StreamReader, + streamwriter=StreamWriter, + ) ### Decoding Table Modified: python/branches/p3yk/Lib/encodings/cp424.py ============================================================================== --- python/branches/p3yk/Lib/encodings/cp424.py (original) +++ python/branches/p3yk/Lib/encodings/cp424.py Fri Apr 21 11:43:23 2006 @@ -1,4 +1,4 @@ -""" Python Character Mapping Codec generated from 'MAPPINGS/VENDORS/MISC/CP424.TXT' with gencodec.py. +""" Python Character Mapping Codec cp424 generated from 'MAPPINGS/VENDORS/MISC/CP424.TXT' with gencodec.py. """#" @@ -9,13 +9,19 @@ class Codec(codecs.Codec): def encode(self,input,errors='strict'): - return codecs.charmap_encode(input,errors,encoding_map) def decode(self,input,errors='strict'): - return codecs.charmap_decode(input,errors,decoding_table) +class IncrementalEncoder(codecs.IncrementalEncoder): + def encode(self, input, final=False): + return codecs.charmap_encode(input,self.errors,encoding_map)[0] + +class IncrementalDecoder(codecs.IncrementalDecoder): + def decode(self, input, final=False): + return codecs.charmap_decode(input,self.errors,decoding_table)[0] + class StreamWriter(Codec,codecs.StreamWriter): pass @@ -25,8 +31,15 @@ ### encodings module API def getregentry(): - - return (Codec().encode,Codec().decode,StreamReader,StreamWriter) + return codecs.CodecInfo( + name='cp424', + encode=Codec().encode, + decode=Codec().decode, + incrementalencoder=IncrementalEncoder, + incrementaldecoder=IncrementalDecoder, + streamreader=StreamReader, + streamwriter=StreamWriter, + ) ### Decoding Table Modified: python/branches/p3yk/Lib/encodings/cp437.py ============================================================================== --- python/branches/p3yk/Lib/encodings/cp437.py (original) +++ python/branches/p3yk/Lib/encodings/cp437.py Fri Apr 21 11:43:23 2006 @@ -1,4 +1,4 @@ -""" Python Character Mapping Codec generated from 'VENDORS/MICSFT/PC/CP437.TXT' with gencodec.py. +""" Python Character Mapping Codec cp437 generated from 'VENDORS/MICSFT/PC/CP437.TXT' with gencodec.py. """#" @@ -9,13 +9,19 @@ class Codec(codecs.Codec): def encode(self,input,errors='strict'): - return codecs.charmap_encode(input,errors,encoding_map) def decode(self,input,errors='strict'): - return codecs.charmap_decode(input,errors,decoding_table) +class IncrementalEncoder(codecs.IncrementalEncoder): + def encode(self, input, final=False): + return codecs.charmap_encode(input,self.errors,encoding_map)[0] + +class IncrementalDecoder(codecs.IncrementalDecoder): + def decode(self, input, final=False): + return codecs.charmap_decode(input,self.errors,decoding_table)[0] + class StreamWriter(Codec,codecs.StreamWriter): pass @@ -25,8 +31,15 @@ ### encodings module API def getregentry(): - - return (Codec().encode,Codec().decode,StreamReader,StreamWriter) + return codecs.CodecInfo( + name='cp437', + encode=Codec().encode, + decode=Codec().decode, + incrementalencoder=IncrementalEncoder, + incrementaldecoder=IncrementalDecoder, + streamreader=StreamReader, + streamwriter=StreamWriter, + ) ### Decoding Map Modified: python/branches/p3yk/Lib/encodings/cp500.py ============================================================================== --- python/branches/p3yk/Lib/encodings/cp500.py (original) +++ python/branches/p3yk/Lib/encodings/cp500.py Fri Apr 21 11:43:23 2006 @@ -1,4 +1,4 @@ -""" Python Character Mapping Codec generated from 'MAPPINGS/VENDORS/MICSFT/EBCDIC/CP500.TXT' with gencodec.py. +""" Python Character Mapping Codec cp500 generated from 'MAPPINGS/VENDORS/MICSFT/EBCDIC/CP500.TXT' with gencodec.py. """#" @@ -9,13 +9,19 @@ class Codec(codecs.Codec): def encode(self,input,errors='strict'): - return codecs.charmap_encode(input,errors,encoding_map) def decode(self,input,errors='strict'): - return codecs.charmap_decode(input,errors,decoding_table) +class IncrementalEncoder(codecs.IncrementalEncoder): + def encode(self, input, final=False): + return codecs.charmap_encode(input,self.errors,encoding_map)[0] + +class IncrementalDecoder(codecs.IncrementalDecoder): + def decode(self, input, final=False): + return codecs.charmap_decode(input,self.errors,decoding_table)[0] + class StreamWriter(Codec,codecs.StreamWriter): pass @@ -25,8 +31,15 @@ ### encodings module API def getregentry(): - - return (Codec().encode,Codec().decode,StreamReader,StreamWriter) + return codecs.CodecInfo( + name='cp500', + encode=Codec().encode, + decode=Codec().decode, + incrementalencoder=IncrementalEncoder, + incrementaldecoder=IncrementalDecoder, + streamreader=StreamReader, + streamwriter=StreamWriter, + ) ### Decoding Table Modified: python/branches/p3yk/Lib/encodings/cp737.py ============================================================================== --- python/branches/p3yk/Lib/encodings/cp737.py (original) +++ python/branches/p3yk/Lib/encodings/cp737.py Fri Apr 21 11:43:23 2006 @@ -1,4 +1,4 @@ -""" Python Character Mapping Codec generated from 'VENDORS/MICSFT/PC/CP737.TXT' with gencodec.py. +""" Python Character Mapping Codec cp737 generated from 'VENDORS/MICSFT/PC/CP737.TXT' with gencodec.py. """#" @@ -9,13 +9,19 @@ class Codec(codecs.Codec): def encode(self,input,errors='strict'): - return codecs.charmap_encode(input,errors,encoding_map) def decode(self,input,errors='strict'): - return codecs.charmap_decode(input,errors,decoding_table) +class IncrementalEncoder(codecs.IncrementalEncoder): + def encode(self, input, final=False): + return codecs.charmap_encode(input,self.errors,encoding_map)[0] + +class IncrementalDecoder(codecs.IncrementalDecoder): + def decode(self, input, final=False): + return codecs.charmap_decode(input,self.errors,decoding_table)[0] + class StreamWriter(Codec,codecs.StreamWriter): pass @@ -25,8 +31,15 @@ ### encodings module API def getregentry(): - - return (Codec().encode,Codec().decode,StreamReader,StreamWriter) + return codecs.CodecInfo( + name='cp737', + encode=Codec().encode, + decode=Codec().decode, + incrementalencoder=IncrementalEncoder, + incrementaldecoder=IncrementalDecoder, + streamreader=StreamReader, + streamwriter=StreamWriter, + ) ### Decoding Map Modified: python/branches/p3yk/Lib/encodings/cp775.py ============================================================================== --- python/branches/p3yk/Lib/encodings/cp775.py (original) +++ python/branches/p3yk/Lib/encodings/cp775.py Fri Apr 21 11:43:23 2006 @@ -1,4 +1,4 @@ -""" Python Character Mapping Codec generated from 'VENDORS/MICSFT/PC/CP775.TXT' with gencodec.py. +""" Python Character Mapping Codec cp775 generated from 'VENDORS/MICSFT/PC/CP775.TXT' with gencodec.py. """#" @@ -9,13 +9,19 @@ class Codec(codecs.Codec): def encode(self,input,errors='strict'): - return codecs.charmap_encode(input,errors,encoding_map) def decode(self,input,errors='strict'): - return codecs.charmap_decode(input,errors,decoding_table) +class IncrementalEncoder(codecs.IncrementalEncoder): + def encode(self, input, final=False): + return codecs.charmap_encode(input,self.errors,encoding_map)[0] + +class IncrementalDecoder(codecs.IncrementalDecoder): + def decode(self, input, final=False): + return codecs.charmap_decode(input,self.errors,decoding_table)[0] + class StreamWriter(Codec,codecs.StreamWriter): pass @@ -25,9 +31,15 @@ ### encodings module API def getregentry(): - - return (Codec().encode,Codec().decode,StreamReader,StreamWriter) - + return codecs.CodecInfo( + name='cp775', + encode=Codec().encode, + decode=Codec().decode, + incrementalencoder=IncrementalEncoder, + incrementaldecoder=IncrementalDecoder, + streamreader=StreamReader, + streamwriter=StreamWriter, + ) ### Decoding Map decoding_map = codecs.make_identity_dict(range(256)) Modified: python/branches/p3yk/Lib/encodings/cp850.py ============================================================================== --- python/branches/p3yk/Lib/encodings/cp850.py (original) +++ python/branches/p3yk/Lib/encodings/cp850.py Fri Apr 21 11:43:23 2006 @@ -9,13 +9,19 @@ class Codec(codecs.Codec): def encode(self,input,errors='strict'): - return codecs.charmap_encode(input,errors,encoding_map) def decode(self,input,errors='strict'): - return codecs.charmap_decode(input,errors,decoding_table) +class IncrementalEncoder(codecs.IncrementalEncoder): + def encode(self, input, final=False): + return codecs.charmap_encode(input,self.errors,encoding_map)[0] + +class IncrementalDecoder(codecs.IncrementalDecoder): + def decode(self, input, final=False): + return codecs.charmap_decode(input,self.errors,decoding_table)[0] + class StreamWriter(Codec,codecs.StreamWriter): pass @@ -25,8 +31,15 @@ ### encodings module API def getregentry(): - - return (Codec().encode,Codec().decode,StreamReader,StreamWriter) + return codecs.CodecInfo( + name='cp850', + encode=Codec().encode, + decode=Codec().decode, + incrementalencoder=IncrementalEncoder, + incrementaldecoder=IncrementalDecoder, + streamreader=StreamReader, + streamwriter=StreamWriter, + ) ### Decoding Map Modified: python/branches/p3yk/Lib/encodings/cp852.py ============================================================================== --- python/branches/p3yk/Lib/encodings/cp852.py (original) +++ python/branches/p3yk/Lib/encodings/cp852.py Fri Apr 21 11:43:23 2006 @@ -9,13 +9,19 @@ class Codec(codecs.Codec): def encode(self,input,errors='strict'): - return codecs.charmap_encode(input,errors,encoding_map) def decode(self,input,errors='strict'): - return codecs.charmap_decode(input,errors,decoding_table) +class IncrementalEncoder(codecs.IncrementalEncoder): + def encode(self, input, final=False): + return codecs.charmap_encode(input,self.errors,encoding_map)[0] + +class IncrementalDecoder(codecs.IncrementalDecoder): + def decode(self, input, final=False): + return codecs.charmap_decode(input,self.errors,decoding_table)[0] + class StreamWriter(Codec,codecs.StreamWriter): pass @@ -25,8 +31,15 @@ ### encodings module API def getregentry(): - - return (Codec().encode,Codec().decode,StreamReader,StreamWriter) + return codecs.CodecInfo( + name='cp852', + encode=Codec().encode, + decode=Codec().decode, + incrementalencoder=IncrementalEncoder, + incrementaldecoder=IncrementalDecoder, + streamreader=StreamReader, + streamwriter=StreamWriter, + ) ### Decoding Map Modified: python/branches/p3yk/Lib/encodings/cp855.py ============================================================================== --- python/branches/p3yk/Lib/encodings/cp855.py (original) +++ python/branches/p3yk/Lib/encodings/cp855.py Fri Apr 21 11:43:23 2006 @@ -9,13 +9,19 @@ class Codec(codecs.Codec): def encode(self,input,errors='strict'): - return codecs.charmap_encode(input,errors,encoding_map) def decode(self,input,errors='strict'): - return codecs.charmap_decode(input,errors,decoding_table) +class IncrementalEncoder(codecs.IncrementalEncoder): + def encode(self, input, final=False): + return codecs.charmap_encode(input,self.errors,encoding_map)[0] + +class IncrementalDecoder(codecs.IncrementalDecoder): + def decode(self, input, final=False): + return codecs.charmap_decode(input,self.errors,decoding_table)[0] + class StreamWriter(Codec,codecs.StreamWriter): pass @@ -25,8 +31,15 @@ ### encodings module API def getregentry(): - - return (Codec().encode,Codec().decode,StreamReader,StreamWriter) + return codecs.CodecInfo( + name='cp855', + encode=Codec().encode, + decode=Codec().decode, + incrementalencoder=IncrementalEncoder, + incrementaldecoder=IncrementalDecoder, + streamreader=StreamReader, + streamwriter=StreamWriter, + ) ### Decoding Map Modified: python/branches/p3yk/Lib/encodings/cp856.py ============================================================================== --- python/branches/p3yk/Lib/encodings/cp856.py (original) +++ python/branches/p3yk/Lib/encodings/cp856.py Fri Apr 21 11:43:23 2006 @@ -1,4 +1,4 @@ -""" Python Character Mapping Codec generated from 'MAPPINGS/VENDORS/MISC/CP856.TXT' with gencodec.py. +""" Python Character Mapping Codec cp856 generated from 'MAPPINGS/VENDORS/MISC/CP856.TXT' with gencodec.py. """#" @@ -9,13 +9,19 @@ class Codec(codecs.Codec): def encode(self,input,errors='strict'): - return codecs.charmap_encode(input,errors,encoding_map) def decode(self,input,errors='strict'): - return codecs.charmap_decode(input,errors,decoding_table) +class IncrementalEncoder(codecs.IncrementalEncoder): + def encode(self, input, final=False): + return codecs.charmap_encode(input,self.errors,encoding_map)[0] + +class IncrementalDecoder(codecs.IncrementalDecoder): + def decode(self, input, final=False): + return codecs.charmap_decode(input,self.errors,decoding_table)[0] + class StreamWriter(Codec,codecs.StreamWriter): pass @@ -25,8 +31,15 @@ ### encodings module API def getregentry(): - - return (Codec().encode,Codec().decode,StreamReader,StreamWriter) + return codecs.CodecInfo( + name='cp856', + encode=Codec().encode, + decode=Codec().decode, + incrementalencoder=IncrementalEncoder, + incrementaldecoder=IncrementalDecoder, + streamreader=StreamReader, + streamwriter=StreamWriter, + ) ### Decoding Table Modified: python/branches/p3yk/Lib/encodings/cp857.py ============================================================================== --- python/branches/p3yk/Lib/encodings/cp857.py (original) +++ python/branches/p3yk/Lib/encodings/cp857.py Fri Apr 21 11:43:23 2006 @@ -9,13 +9,19 @@ class Codec(codecs.Codec): def encode(self,input,errors='strict'): - return codecs.charmap_encode(input,errors,encoding_map) def decode(self,input,errors='strict'): - return codecs.charmap_decode(input,errors,decoding_table) +class IncrementalEncoder(codecs.IncrementalEncoder): + def encode(self, input, final=False): + return codecs.charmap_encode(input,self.errors,encoding_map)[0] + +class IncrementalDecoder(codecs.IncrementalDecoder): + def decode(self, input, final=False): + return codecs.charmap_decode(input,self.errors,decoding_table)[0] + class StreamWriter(Codec,codecs.StreamWriter): pass @@ -25,8 +31,15 @@ ### encodings module API def getregentry(): - - return (Codec().encode,Codec().decode,StreamReader,StreamWriter) + return codecs.CodecInfo( + name='cp857', + encode=Codec().encode, + decode=Codec().decode, + incrementalencoder=IncrementalEncoder, + incrementaldecoder=IncrementalDecoder, + streamreader=StreamReader, + streamwriter=StreamWriter, + ) ### Decoding Map Modified: python/branches/p3yk/Lib/encodings/cp860.py ============================================================================== --- python/branches/p3yk/Lib/encodings/cp860.py (original) +++ python/branches/p3yk/Lib/encodings/cp860.py Fri Apr 21 11:43:23 2006 @@ -9,13 +9,19 @@ class Codec(codecs.Codec): def encode(self,input,errors='strict'): - return codecs.charmap_encode(input,errors,encoding_map) def decode(self,input,errors='strict'): - return codecs.charmap_decode(input,errors,decoding_table) +class IncrementalEncoder(codecs.IncrementalEncoder): + def encode(self, input, final=False): + return codecs.charmap_encode(input,self.errors,encoding_map)[0] + +class IncrementalDecoder(codecs.IncrementalDecoder): + def decode(self, input, final=False): + return codecs.charmap_decode(input,self.errors,decoding_table)[0] + class StreamWriter(Codec,codecs.StreamWriter): pass @@ -25,8 +31,15 @@ ### encodings module API def getregentry(): - - return (Codec().encode,Codec().decode,StreamReader,StreamWriter) + return codecs.CodecInfo( + name='cp860', + encode=Codec().encode, + decode=Codec().decode, + incrementalencoder=IncrementalEncoder, + incrementaldecoder=IncrementalDecoder, + streamreader=StreamReader, + streamwriter=StreamWriter, + ) ### Decoding Map Modified: python/branches/p3yk/Lib/encodings/cp861.py ============================================================================== --- python/branches/p3yk/Lib/encodings/cp861.py (original) +++ python/branches/p3yk/Lib/encodings/cp861.py Fri Apr 21 11:43:23 2006 @@ -9,13 +9,19 @@ class Codec(codecs.Codec): def encode(self,input,errors='strict'): - return codecs.charmap_encode(input,errors,encoding_map) def decode(self,input,errors='strict'): - return codecs.charmap_decode(input,errors,decoding_table) +class IncrementalEncoder(codecs.IncrementalEncoder): + def encode(self, input, final=False): + return codecs.charmap_encode(input,self.errors,encoding_map)[0] + +class IncrementalDecoder(codecs.IncrementalDecoder): + def decode(self, input, final=False): + return codecs.charmap_decode(input,self.errors,decoding_table)[0] + class StreamWriter(Codec,codecs.StreamWriter): pass @@ -25,8 +31,15 @@ ### encodings module API def getregentry(): - - return (Codec().encode,Codec().decode,StreamReader,StreamWriter) + return codecs.CodecInfo( + name='cp861', + encode=Codec().encode, + decode=Codec().decode, + incrementalencoder=IncrementalEncoder, + incrementaldecoder=IncrementalDecoder, + streamreader=StreamReader, + streamwriter=StreamWriter, + ) ### Decoding Map Modified: python/branches/p3yk/Lib/encodings/cp862.py ============================================================================== --- python/branches/p3yk/Lib/encodings/cp862.py (original) +++ python/branches/p3yk/Lib/encodings/cp862.py Fri Apr 21 11:43:23 2006 @@ -9,13 +9,19 @@ class Codec(codecs.Codec): def encode(self,input,errors='strict'): - return codecs.charmap_encode(input,errors,encoding_map) def decode(self,input,errors='strict'): - return codecs.charmap_decode(input,errors,decoding_table) +class IncrementalEncoder(codecs.IncrementalEncoder): + def encode(self, input, final=False): + return codecs.charmap_encode(input,self.errors,encoding_map)[0] + +class IncrementalDecoder(codecs.IncrementalDecoder): + def decode(self, input, final=False): + return codecs.charmap_decode(input,self.errors,decoding_table)[0] + class StreamWriter(Codec,codecs.StreamWriter): pass @@ -25,8 +31,15 @@ ### encodings module API def getregentry(): - - return (Codec().encode,Codec().decode,StreamReader,StreamWriter) + return codecs.CodecInfo( + name='cp862', + encode=Codec().encode, + decode=Codec().decode, + incrementalencoder=IncrementalEncoder, + incrementaldecoder=IncrementalDecoder, + streamreader=StreamReader, + streamwriter=StreamWriter, + ) ### Decoding Map Modified: python/branches/p3yk/Lib/encodings/cp863.py ============================================================================== --- python/branches/p3yk/Lib/encodings/cp863.py (original) +++ python/branches/p3yk/Lib/encodings/cp863.py Fri Apr 21 11:43:23 2006 @@ -9,13 +9,19 @@ class Codec(codecs.Codec): def encode(self,input,errors='strict'): - return codecs.charmap_encode(input,errors,encoding_map) def decode(self,input,errors='strict'): - return codecs.charmap_decode(input,errors,decoding_table) +class IncrementalEncoder(codecs.IncrementalEncoder): + def encode(self, input, final=False): + return codecs.charmap_encode(input,self.errors,encoding_map)[0] + +class IncrementalDecoder(codecs.IncrementalDecoder): + def decode(self, input, final=False): + return codecs.charmap_decode(input,self.errors,decoding_table)[0] + class StreamWriter(Codec,codecs.StreamWriter): pass @@ -25,8 +31,15 @@ ### encodings module API def getregentry(): - - return (Codec().encode,Codec().decode,StreamReader,StreamWriter) + return codecs.CodecInfo( + name='cp863', + encode=Codec().encode, + decode=Codec().decode, + incrementalencoder=IncrementalEncoder, + incrementaldecoder=IncrementalDecoder, + streamreader=StreamReader, + streamwriter=StreamWriter, + ) ### Decoding Map Modified: python/branches/p3yk/Lib/encodings/cp864.py ============================================================================== --- python/branches/p3yk/Lib/encodings/cp864.py (original) +++ python/branches/p3yk/Lib/encodings/cp864.py Fri Apr 21 11:43:23 2006 @@ -9,13 +9,19 @@ class Codec(codecs.Codec): def encode(self,input,errors='strict'): - return codecs.charmap_encode(input,errors,encoding_map) def decode(self,input,errors='strict'): - return codecs.charmap_decode(input,errors,decoding_table) +class IncrementalEncoder(codecs.IncrementalEncoder): + def encode(self, input, final=False): + return codecs.charmap_encode(input,self.errors,encoding_map)[0] + +class IncrementalDecoder(codecs.IncrementalDecoder): + def decode(self, input, final=False): + return codecs.charmap_decode(input,self.errors,decoding_table)[0] + class StreamWriter(Codec,codecs.StreamWriter): pass @@ -25,8 +31,15 @@ ### encodings module API def getregentry(): - - return (Codec().encode,Codec().decode,StreamReader,StreamWriter) + return codecs.CodecInfo( + name='cp864', + encode=Codec().encode, + decode=Codec().decode, + incrementalencoder=IncrementalEncoder, + incrementaldecoder=IncrementalDecoder, + streamreader=StreamReader, + streamwriter=StreamWriter, + ) ### Decoding Map Modified: python/branches/p3yk/Lib/encodings/cp865.py ============================================================================== --- python/branches/p3yk/Lib/encodings/cp865.py (original) +++ python/branches/p3yk/Lib/encodings/cp865.py Fri Apr 21 11:43:23 2006 @@ -9,13 +9,19 @@ class Codec(codecs.Codec): def encode(self,input,errors='strict'): - return codecs.charmap_encode(input,errors,encoding_map) def decode(self,input,errors='strict'): - return codecs.charmap_decode(input,errors,decoding_table) +class IncrementalEncoder(codecs.IncrementalEncoder): + def encode(self, input, final=False): + return codecs.charmap_encode(input,self.errors,encoding_map)[0] + +class IncrementalDecoder(codecs.IncrementalDecoder): + def decode(self, input, final=False): + return codecs.charmap_decode(input,self.errors,decoding_table)[0] + class StreamWriter(Codec,codecs.StreamWriter): pass @@ -25,8 +31,15 @@ ### encodings module API def getregentry(): - - return (Codec().encode,Codec().decode,StreamReader,StreamWriter) + return codecs.CodecInfo( + name='cp865', + encode=Codec().encode, + decode=Codec().decode, + incrementalencoder=IncrementalEncoder, + incrementaldecoder=IncrementalDecoder, + streamreader=StreamReader, + streamwriter=StreamWriter, + ) ### Decoding Map Modified: python/branches/p3yk/Lib/encodings/cp866.py ============================================================================== --- python/branches/p3yk/Lib/encodings/cp866.py (original) +++ python/branches/p3yk/Lib/encodings/cp866.py Fri Apr 21 11:43:23 2006 @@ -9,13 +9,19 @@ class Codec(codecs.Codec): def encode(self,input,errors='strict'): - return codecs.charmap_encode(input,errors,encoding_map) def decode(self,input,errors='strict'): - return codecs.charmap_decode(input,errors,decoding_table) +class IncrementalEncoder(codecs.IncrementalEncoder): + def encode(self, input, final=False): + return codecs.charmap_encode(input,self.errors,encoding_map)[0] + +class IncrementalDecoder(codecs.IncrementalDecoder): + def decode(self, input, final=False): + return codecs.charmap_decode(input,self.errors,decoding_table)[0] + class StreamWriter(Codec,codecs.StreamWriter): pass @@ -25,8 +31,15 @@ ### encodings module API def getregentry(): - - return (Codec().encode,Codec().decode,StreamReader,StreamWriter) + return codecs.CodecInfo( + name='cp866', + encode=Codec().encode, + decode=Codec().decode, + incrementalencoder=IncrementalEncoder, + incrementaldecoder=IncrementalDecoder, + streamreader=StreamReader, + streamwriter=StreamWriter, + ) ### Decoding Map Modified: python/branches/p3yk/Lib/encodings/cp869.py ============================================================================== --- python/branches/p3yk/Lib/encodings/cp869.py (original) +++ python/branches/p3yk/Lib/encodings/cp869.py Fri Apr 21 11:43:23 2006 @@ -9,13 +9,19 @@ class Codec(codecs.Codec): def encode(self,input,errors='strict'): - return codecs.charmap_encode(input,errors,encoding_map) def decode(self,input,errors='strict'): - return codecs.charmap_decode(input,errors,decoding_table) +class IncrementalEncoder(codecs.IncrementalEncoder): + def encode(self, input, final=False): + return codecs.charmap_encode(input,self.errors,encoding_map)[0] + +class IncrementalDecoder(codecs.IncrementalDecoder): + def decode(self, input, final=False): + return codecs.charmap_decode(input,self.errors,decoding_table)[0] + class StreamWriter(Codec,codecs.StreamWriter): pass @@ -25,8 +31,15 @@ ### encodings module API def getregentry(): - - return (Codec().encode,Codec().decode,StreamReader,StreamWriter) + return codecs.CodecInfo( + name='cp869', + encode=Codec().encode, + decode=Codec().decode, + incrementalencoder=IncrementalEncoder, + incrementaldecoder=IncrementalDecoder, + streamreader=StreamReader, + streamwriter=StreamWriter, + ) ### Decoding Map Modified: python/branches/p3yk/Lib/encodings/cp874.py ============================================================================== --- python/branches/p3yk/Lib/encodings/cp874.py (original) +++ python/branches/p3yk/Lib/encodings/cp874.py Fri Apr 21 11:43:23 2006 @@ -1,4 +1,4 @@ -""" Python Character Mapping Codec generated from 'MAPPINGS/VENDORS/MICSFT/WINDOWS/CP874.TXT' with gencodec.py. +""" Python Character Mapping Codec cp874 generated from 'MAPPINGS/VENDORS/MICSFT/WINDOWS/CP874.TXT' with gencodec.py. """#" @@ -9,13 +9,19 @@ class Codec(codecs.Codec): def encode(self,input,errors='strict'): - return codecs.charmap_encode(input,errors,encoding_map) def decode(self,input,errors='strict'): - return codecs.charmap_decode(input,errors,decoding_table) +class IncrementalEncoder(codecs.IncrementalEncoder): + def encode(self, input, final=False): + return codecs.charmap_encode(input,self.errors,encoding_map)[0] + +class IncrementalDecoder(codecs.IncrementalDecoder): + def decode(self, input, final=False): + return codecs.charmap_decode(input,self.errors,decoding_table)[0] + class StreamWriter(Codec,codecs.StreamWriter): pass @@ -25,8 +31,15 @@ ### encodings module API def getregentry(): - - return (Codec().encode,Codec().decode,StreamReader,StreamWriter) + return codecs.CodecInfo( + name='cp874', + encode=Codec().encode, + decode=Codec().decode, + incrementalencoder=IncrementalEncoder, + incrementaldecoder=IncrementalDecoder, + streamreader=StreamReader, + streamwriter=StreamWriter, + ) ### Decoding Table Modified: python/branches/p3yk/Lib/encodings/cp875.py ============================================================================== --- python/branches/p3yk/Lib/encodings/cp875.py (original) +++ python/branches/p3yk/Lib/encodings/cp875.py Fri Apr 21 11:43:23 2006 @@ -1,4 +1,4 @@ -""" Python Character Mapping Codec generated from 'MAPPINGS/VENDORS/MICSFT/EBCDIC/CP875.TXT' with gencodec.py. +""" Python Character Mapping Codec cp875 generated from 'MAPPINGS/VENDORS/MICSFT/EBCDIC/CP875.TXT' with gencodec.py. """#" @@ -9,13 +9,19 @@ class Codec(codecs.Codec): def encode(self,input,errors='strict'): - return codecs.charmap_encode(input,errors,encoding_map) def decode(self,input,errors='strict'): - return codecs.charmap_decode(input,errors,decoding_table) +class IncrementalEncoder(codecs.IncrementalEncoder): + def encode(self, input, final=False): + return codecs.charmap_encode(input,self.errors,encoding_map)[0] + +class IncrementalDecoder(codecs.IncrementalDecoder): + def decode(self, input, final=False): + return codecs.charmap_decode(input,self.errors,decoding_table)[0] + class StreamWriter(Codec,codecs.StreamWriter): pass @@ -25,8 +31,15 @@ ### encodings module API def getregentry(): - - return (Codec().encode,Codec().decode,StreamReader,StreamWriter) + return codecs.CodecInfo( + name='cp875', + encode=Codec().encode, + decode=Codec().decode, + incrementalencoder=IncrementalEncoder, + incrementaldecoder=IncrementalDecoder, + streamreader=StreamReader, + streamwriter=StreamWriter, + ) ### Decoding Table Modified: python/branches/p3yk/Lib/encodings/hex_codec.py ============================================================================== --- python/branches/p3yk/Lib/encodings/hex_codec.py (original) +++ python/branches/p3yk/Lib/encodings/hex_codec.py Fri Apr 21 11:43:23 2006 @@ -49,6 +49,16 @@ def decode(self, input,errors='strict'): return hex_decode(input,errors) +class IncrementalEncoder(codecs.IncrementalEncoder): + def encode(self, input, final=False): + assert self.errors == 'strict' + return binascii.b2a_hex(input) + +class IncrementalDecoder(codecs.IncrementalDecoder): + def decode(self, input, final=False): + assert self.errors == 'strict' + return binascii.a2b_hex(input) + class StreamWriter(Codec,codecs.StreamWriter): pass @@ -58,5 +68,12 @@ ### encodings module API def getregentry(): - - return (hex_encode,hex_decode,StreamReader,StreamWriter) + return codecs.CodecInfo( + name='hex', + encode=hex_encode, + decode=hex_decode, + incrementalencoder=IncrementalEncoder, + incrementaldecoder=IncrementalDecoder, + streamwriter=StreamWriter, + streamreader=StreamReader, + ) Modified: python/branches/p3yk/Lib/encodings/hp_roman8.py ============================================================================== --- python/branches/p3yk/Lib/encodings/hp_roman8.py (original) +++ python/branches/p3yk/Lib/encodings/hp_roman8.py Fri Apr 21 11:43:23 2006 @@ -14,13 +14,19 @@ class Codec(codecs.Codec): def encode(self,input,errors='strict'): - return codecs.charmap_encode(input,errors,encoding_map) def decode(self,input,errors='strict'): - return codecs.charmap_decode(input,errors,decoding_map) +class IncrementalEncoder(codecs.IncrementalEncoder): + def encode(self, input, final=False): + return codecs.charmap_encode(input,self.errors,encoding_map)[0] + +class IncrementalDecoder(codecs.IncrementalDecoder): + def decode(self, input, final=False): + return codecs.charmap_decode(input,self.errors,decoding_map)[0] + class StreamWriter(Codec,codecs.StreamWriter): pass @@ -30,8 +36,15 @@ ### encodings module API def getregentry(): - - return (Codec().encode,Codec().decode,StreamReader,StreamWriter) + return codecs.CodecInfo( + name='hp-roman8', + encode=Codec().encode, + decode=Codec().decode, + incrementalencoder=IncrementalEncoder, + incrementaldecoder=IncrementalDecoder, + streamwriter=StreamWriter, + streamreader=StreamReader, + ) ### Decoding Map Modified: python/branches/p3yk/Lib/encodings/idna.py ============================================================================== --- python/branches/p3yk/Lib/encodings/idna.py (original) +++ python/branches/p3yk/Lib/encodings/idna.py Fri Apr 21 11:43:23 2006 @@ -194,6 +194,14 @@ return u".".join(result)+trailing_dot, len(input) +class IncrementalEncoder(codecs.IncrementalEncoder): + def encode(self, input, final=False): + return Codec().encode(input, self.errors)[0] + +class IncrementalDecoder(codecs.IncrementalDecoder): + def decode(self, input, final=False): + return Codec().decode(input, self.errors)[0] + class StreamWriter(Codec,codecs.StreamWriter): pass @@ -203,5 +211,12 @@ ### encodings module API def getregentry(): - - return (Codec().encode,Codec().decode,StreamReader,StreamWriter) + return codecs.CodecInfo( + name='idna', + encode=Codec().encode, + decode=Codec().decode, + incrementalencoder=IncrementalEncoder, + incrementaldecoder=IncrementalDecoder, + streamwriter=StreamWriter, + streamreader=StreamReader, + ) Modified: python/branches/p3yk/Lib/encodings/iso8859_1.py ============================================================================== --- python/branches/p3yk/Lib/encodings/iso8859_1.py (original) +++ python/branches/p3yk/Lib/encodings/iso8859_1.py Fri Apr 21 11:43:23 2006 @@ -1,4 +1,4 @@ -""" Python Character Mapping Codec generated from 'MAPPINGS/ISO8859/8859-1.TXT' with gencodec.py. +""" Python Character Mapping Codec iso8859_1 generated from 'MAPPINGS/ISO8859/8859-1.TXT' with gencodec.py. """#" @@ -9,13 +9,19 @@ class Codec(codecs.Codec): def encode(self,input,errors='strict'): - return codecs.charmap_encode(input,errors,encoding_map) def decode(self,input,errors='strict'): - return codecs.charmap_decode(input,errors,decoding_table) +class IncrementalEncoder(codecs.IncrementalEncoder): + def encode(self, input, final=False): + return codecs.charmap_encode(input,self.errors,encoding_map)[0] + +class IncrementalDecoder(codecs.IncrementalDecoder): + def decode(self, input, final=False): + return codecs.charmap_decode(input,self.errors,decoding_table)[0] + class StreamWriter(Codec,codecs.StreamWriter): pass @@ -25,8 +31,15 @@ ### encodings module API def getregentry(): - - return (Codec().encode,Codec().decode,StreamReader,StreamWriter) + return codecs.CodecInfo( + name='iso8859-1', + encode=Codec().encode, + decode=Codec().decode, + incrementalencoder=IncrementalEncoder, + incrementaldecoder=IncrementalDecoder, + streamreader=StreamReader, + streamwriter=StreamWriter, + ) ### Decoding Table Modified: python/branches/p3yk/Lib/encodings/iso8859_10.py ============================================================================== --- python/branches/p3yk/Lib/encodings/iso8859_10.py (original) +++ python/branches/p3yk/Lib/encodings/iso8859_10.py Fri Apr 21 11:43:23 2006 @@ -1,4 +1,4 @@ -""" Python Character Mapping Codec generated from 'MAPPINGS/ISO8859/8859-10.TXT' with gencodec.py. +""" Python Character Mapping Codec iso8859_10 generated from 'MAPPINGS/ISO8859/8859-10.TXT' with gencodec.py. """#" @@ -9,13 +9,19 @@ class Codec(codecs.Codec): def encode(self,input,errors='strict'): - return codecs.charmap_encode(input,errors,encoding_map) def decode(self,input,errors='strict'): - return codecs.charmap_decode(input,errors,decoding_table) +class IncrementalEncoder(codecs.IncrementalEncoder): + def encode(self, input, final=False): + return codecs.charmap_encode(input,self.errors,encoding_map)[0] + +class IncrementalDecoder(codecs.IncrementalDecoder): + def decode(self, input, final=False): + return codecs.charmap_decode(input,self.errors,decoding_table)[0] + class StreamWriter(Codec,codecs.StreamWriter): pass @@ -25,8 +31,15 @@ ### encodings module API def getregentry(): - - return (Codec().encode,Codec().decode,StreamReader,StreamWriter) + return codecs.CodecInfo( + name='iso8859-10', + encode=Codec().encode, + decode=Codec().decode, + incrementalencoder=IncrementalEncoder, + incrementaldecoder=IncrementalDecoder, + streamreader=StreamReader, + streamwriter=StreamWriter, + ) ### Decoding Table Modified: python/branches/p3yk/Lib/encodings/iso8859_11.py ============================================================================== --- python/branches/p3yk/Lib/encodings/iso8859_11.py (original) +++ python/branches/p3yk/Lib/encodings/iso8859_11.py Fri Apr 21 11:43:23 2006 @@ -1,4 +1,4 @@ -""" Python Character Mapping Codec generated from 'MAPPINGS/ISO8859/8859-11.TXT' with gencodec.py. +""" Python Character Mapping Codec iso8859_11 generated from 'MAPPINGS/ISO8859/8859-11.TXT' with gencodec.py. """#" @@ -9,13 +9,19 @@ class Codec(codecs.Codec): def encode(self,input,errors='strict'): - return codecs.charmap_encode(input,errors,encoding_map) def decode(self,input,errors='strict'): - return codecs.charmap_decode(input,errors,decoding_table) +class IncrementalEncoder(codecs.IncrementalEncoder): + def encode(self, input, final=False): + return codecs.charmap_encode(input,self.errors,encoding_map)[0] + +class IncrementalDecoder(codecs.IncrementalDecoder): + def decode(self, input, final=False): + return codecs.charmap_decode(input,self.errors,decoding_table)[0] + class StreamWriter(Codec,codecs.StreamWriter): pass @@ -25,8 +31,15 @@ ### encodings module API def getregentry(): - - return (Codec().encode,Codec().decode,StreamReader,StreamWriter) + return codecs.CodecInfo( + name='iso8859-11', + encode=Codec().encode, + decode=Codec().decode, + incrementalencoder=IncrementalEncoder, + incrementaldecoder=IncrementalDecoder, + streamreader=StreamReader, + streamwriter=StreamWriter, + ) ### Decoding Table Modified: python/branches/p3yk/Lib/encodings/iso8859_13.py ============================================================================== --- python/branches/p3yk/Lib/encodings/iso8859_13.py (original) +++ python/branches/p3yk/Lib/encodings/iso8859_13.py Fri Apr 21 11:43:23 2006 @@ -1,4 +1,4 @@ -""" Python Character Mapping Codec generated from 'MAPPINGS/ISO8859/8859-13.TXT' with gencodec.py. +""" Python Character Mapping Codec iso8859_13 generated from 'MAPPINGS/ISO8859/8859-13.TXT' with gencodec.py. """#" @@ -9,13 +9,19 @@ class Codec(codecs.Codec): def encode(self,input,errors='strict'): - return codecs.charmap_encode(input,errors,encoding_map) def decode(self,input,errors='strict'): - return codecs.charmap_decode(input,errors,decoding_table) +class IncrementalEncoder(codecs.IncrementalEncoder): + def encode(self, input, final=False): + return codecs.charmap_encode(input,self.errors,encoding_map)[0] + +class IncrementalDecoder(codecs.IncrementalDecoder): + def decode(self, input, final=False): + return codecs.charmap_decode(input,self.errors,decoding_table)[0] + class StreamWriter(Codec,codecs.StreamWriter): pass @@ -25,8 +31,15 @@ ### encodings module API def getregentry(): - - return (Codec().encode,Codec().decode,StreamReader,StreamWriter) + return codecs.CodecInfo( + name='iso8859-13', + encode=Codec().encode, + decode=Codec().decode, + incrementalencoder=IncrementalEncoder, + incrementaldecoder=IncrementalDecoder, + streamreader=StreamReader, + streamwriter=StreamWriter, + ) ### Decoding Table Modified: python/branches/p3yk/Lib/encodings/iso8859_14.py ============================================================================== --- python/branches/p3yk/Lib/encodings/iso8859_14.py (original) +++ python/branches/p3yk/Lib/encodings/iso8859_14.py Fri Apr 21 11:43:23 2006 @@ -1,4 +1,4 @@ -""" Python Character Mapping Codec generated from 'MAPPINGS/ISO8859/8859-14.TXT' with gencodec.py. +""" Python Character Mapping Codec iso8859_14 generated from 'MAPPINGS/ISO8859/8859-14.TXT' with gencodec.py. """#" @@ -9,13 +9,19 @@ class Codec(codecs.Codec): def encode(self,input,errors='strict'): - return codecs.charmap_encode(input,errors,encoding_map) def decode(self,input,errors='strict'): - return codecs.charmap_decode(input,errors,decoding_table) +class IncrementalEncoder(codecs.IncrementalEncoder): + def encode(self, input, final=False): + return codecs.charmap_encode(input,self.errors,encoding_map)[0] + +class IncrementalDecoder(codecs.IncrementalDecoder): + def decode(self, input, final=False): + return codecs.charmap_decode(input,self.errors,decoding_table)[0] + class StreamWriter(Codec,codecs.StreamWriter): pass @@ -25,8 +31,15 @@ ### encodings module API def getregentry(): - - return (Codec().encode,Codec().decode,StreamReader,StreamWriter) + return codecs.CodecInfo( + name='iso8859-14', + encode=Codec().encode, + decode=Codec().decode, + incrementalencoder=IncrementalEncoder, + incrementaldecoder=IncrementalDecoder, + streamreader=StreamReader, + streamwriter=StreamWriter, + ) ### Decoding Table Modified: python/branches/p3yk/Lib/encodings/iso8859_15.py ============================================================================== --- python/branches/p3yk/Lib/encodings/iso8859_15.py (original) +++ python/branches/p3yk/Lib/encodings/iso8859_15.py Fri Apr 21 11:43:23 2006 @@ -1,4 +1,4 @@ -""" Python Character Mapping Codec generated from 'MAPPINGS/ISO8859/8859-15.TXT' with gencodec.py. +""" Python Character Mapping Codec iso8859_15 generated from 'MAPPINGS/ISO8859/8859-15.TXT' with gencodec.py. """#" @@ -9,13 +9,19 @@ class Codec(codecs.Codec): def encode(self,input,errors='strict'): - return codecs.charmap_encode(input,errors,encoding_map) def decode(self,input,errors='strict'): - return codecs.charmap_decode(input,errors,decoding_table) +class IncrementalEncoder(codecs.IncrementalEncoder): + def encode(self, input, final=False): + return codecs.charmap_encode(input,self.errors,encoding_map)[0] + +class IncrementalDecoder(codecs.IncrementalDecoder): + def decode(self, input, final=False): + return codecs.charmap_decode(input,self.errors,decoding_table)[0] + class StreamWriter(Codec,codecs.StreamWriter): pass @@ -25,8 +31,15 @@ ### encodings module API def getregentry(): - - return (Codec().encode,Codec().decode,StreamReader,StreamWriter) + return codecs.CodecInfo( + name='iso8859-15', + encode=Codec().encode, + decode=Codec().decode, + incrementalencoder=IncrementalEncoder, + incrementaldecoder=IncrementalDecoder, + streamreader=StreamReader, + streamwriter=StreamWriter, + ) ### Decoding Table Modified: python/branches/p3yk/Lib/encodings/iso8859_16.py ============================================================================== --- python/branches/p3yk/Lib/encodings/iso8859_16.py (original) +++ python/branches/p3yk/Lib/encodings/iso8859_16.py Fri Apr 21 11:43:23 2006 @@ -1,4 +1,4 @@ -""" Python Character Mapping Codec generated from 'MAPPINGS/ISO8859/8859-16.TXT' with gencodec.py. +""" Python Character Mapping Codec iso8859_16 generated from 'MAPPINGS/ISO8859/8859-16.TXT' with gencodec.py. """#" @@ -9,13 +9,19 @@ class Codec(codecs.Codec): def encode(self,input,errors='strict'): - return codecs.charmap_encode(input,errors,encoding_map) def decode(self,input,errors='strict'): - return codecs.charmap_decode(input,errors,decoding_table) +class IncrementalEncoder(codecs.IncrementalEncoder): + def encode(self, input, final=False): + return codecs.charmap_encode(input,self.errors,encoding_map)[0] + +class IncrementalDecoder(codecs.IncrementalDecoder): + def decode(self, input, final=False): + return codecs.charmap_decode(input,self.errors,decoding_table)[0] + class StreamWriter(Codec,codecs.StreamWriter): pass @@ -25,8 +31,15 @@ ### encodings module API def getregentry(): - - return (Codec().encode,Codec().decode,StreamReader,StreamWriter) + return codecs.CodecInfo( + name='iso8859-16', + encode=Codec().encode, + decode=Codec().decode, + incrementalencoder=IncrementalEncoder, + incrementaldecoder=IncrementalDecoder, + streamreader=StreamReader, + streamwriter=StreamWriter, + ) ### Decoding Table Modified: python/branches/p3yk/Lib/encodings/iso8859_2.py ============================================================================== --- python/branches/p3yk/Lib/encodings/iso8859_2.py (original) +++ python/branches/p3yk/Lib/encodings/iso8859_2.py Fri Apr 21 11:43:23 2006 @@ -1,4 +1,4 @@ -""" Python Character Mapping Codec generated from 'MAPPINGS/ISO8859/8859-2.TXT' with gencodec.py. +""" Python Character Mapping Codec iso8859_2 generated from 'MAPPINGS/ISO8859/8859-2.TXT' with gencodec.py. """#" @@ -9,13 +9,19 @@ class Codec(codecs.Codec): def encode(self,input,errors='strict'): - return codecs.charmap_encode(input,errors,encoding_map) def decode(self,input,errors='strict'): - return codecs.charmap_decode(input,errors,decoding_table) +class IncrementalEncoder(codecs.IncrementalEncoder): + def encode(self, input, final=False): + return codecs.charmap_encode(input,self.errors,encoding_map)[0] + +class IncrementalDecoder(codecs.IncrementalDecoder): + def decode(self, input, final=False): + return codecs.charmap_decode(input,self.errors,decoding_table)[0] + class StreamWriter(Codec,codecs.StreamWriter): pass @@ -25,8 +31,15 @@ ### encodings module API def getregentry(): - - return (Codec().encode,Codec().decode,StreamReader,StreamWriter) + return codecs.CodecInfo( + name='iso8859-2', + encode=Codec().encode, + decode=Codec().decode, + incrementalencoder=IncrementalEncoder, + incrementaldecoder=IncrementalDecoder, + streamreader=StreamReader, + streamwriter=StreamWriter, + ) ### Decoding Table Modified: python/branches/p3yk/Lib/encodings/iso8859_3.py ============================================================================== --- python/branches/p3yk/Lib/encodings/iso8859_3.py (original) +++ python/branches/p3yk/Lib/encodings/iso8859_3.py Fri Apr 21 11:43:23 2006 @@ -1,4 +1,4 @@ -""" Python Character Mapping Codec generated from 'MAPPINGS/ISO8859/8859-3.TXT' with gencodec.py. +""" Python Character Mapping Codec iso8859_3 generated from 'MAPPINGS/ISO8859/8859-3.TXT' with gencodec.py. """#" @@ -9,13 +9,19 @@ class Codec(codecs.Codec): def encode(self,input,errors='strict'): - return codecs.charmap_encode(input,errors,encoding_map) def decode(self,input,errors='strict'): - return codecs.charmap_decode(input,errors,decoding_table) +class IncrementalEncoder(codecs.IncrementalEncoder): + def encode(self, input, final=False): + return codecs.charmap_encode(input,self.errors,encoding_map)[0] + +class IncrementalDecoder(codecs.IncrementalDecoder): + def decode(self, input, final=False): + return codecs.charmap_decode(input,self.errors,decoding_table)[0] + class StreamWriter(Codec,codecs.StreamWriter): pass @@ -25,8 +31,15 @@ ### encodings module API def getregentry(): - - return (Codec().encode,Codec().decode,StreamReader,StreamWriter) + return codecs.CodecInfo( + name='iso8859-3', + encode=Codec().encode, + decode=Codec().decode, + incrementalencoder=IncrementalEncoder, + incrementaldecoder=IncrementalDecoder, + streamreader=StreamReader, + streamwriter=StreamWriter, + ) ### Decoding Table Modified: python/branches/p3yk/Lib/encodings/iso8859_4.py ============================================================================== --- python/branches/p3yk/Lib/encodings/iso8859_4.py (original) +++ python/branches/p3yk/Lib/encodings/iso8859_4.py Fri Apr 21 11:43:23 2006 @@ -1,4 +1,4 @@ -""" Python Character Mapping Codec generated from 'MAPPINGS/ISO8859/8859-4.TXT' with gencodec.py. +""" Python Character Mapping Codec iso8859_4 generated from 'MAPPINGS/ISO8859/8859-4.TXT' with gencodec.py. """#" @@ -9,13 +9,19 @@ class Codec(codecs.Codec): def encode(self,input,errors='strict'): - return codecs.charmap_encode(input,errors,encoding_map) def decode(self,input,errors='strict'): - return codecs.charmap_decode(input,errors,decoding_table) +class IncrementalEncoder(codecs.IncrementalEncoder): + def encode(self, input, final=False): + return codecs.charmap_encode(input,self.errors,encoding_map)[0] + +class IncrementalDecoder(codecs.IncrementalDecoder): + def decode(self, input, final=False): + return codecs.charmap_decode(input,self.errors,decoding_table)[0] + class StreamWriter(Codec,codecs.StreamWriter): pass @@ -25,8 +31,15 @@ ### encodings module API def getregentry(): - - return (Codec().encode,Codec().decode,StreamReader,StreamWriter) + return codecs.CodecInfo( + name='iso8859-4', + encode=Codec().encode, + decode=Codec().decode, + incrementalencoder=IncrementalEncoder, + incrementaldecoder=IncrementalDecoder, + streamreader=StreamReader, + streamwriter=StreamWriter, + ) ### Decoding Table Modified: python/branches/p3yk/Lib/encodings/iso8859_5.py ============================================================================== --- python/branches/p3yk/Lib/encodings/iso8859_5.py (original) +++ python/branches/p3yk/Lib/encodings/iso8859_5.py Fri Apr 21 11:43:23 2006 @@ -1,4 +1,4 @@ -""" Python Character Mapping Codec generated from 'MAPPINGS/ISO8859/8859-5.TXT' with gencodec.py. +""" Python Character Mapping Codec iso8859_5 generated from 'MAPPINGS/ISO8859/8859-5.TXT' with gencodec.py. """#" @@ -9,13 +9,19 @@ class Codec(codecs.Codec): def encode(self,input,errors='strict'): - return codecs.charmap_encode(input,errors,encoding_map) def decode(self,input,errors='strict'): - return codecs.charmap_decode(input,errors,decoding_table) +class IncrementalEncoder(codecs.IncrementalEncoder): + def encode(self, input, final=False): + return codecs.charmap_encode(input,self.errors,encoding_map)[0] + +class IncrementalDecoder(codecs.IncrementalDecoder): + def decode(self, input, final=False): + return codecs.charmap_decode(input,self.errors,decoding_table)[0] + class StreamWriter(Codec,codecs.StreamWriter): pass @@ -25,8 +31,15 @@ ### encodings module API def getregentry(): - - return (Codec().encode,Codec().decode,StreamReader,StreamWriter) + return codecs.CodecInfo( + name='iso8859-5', + encode=Codec().encode, + decode=Codec().decode, + incrementalencoder=IncrementalEncoder, + incrementaldecoder=IncrementalDecoder, + streamreader=StreamReader, + streamwriter=StreamWriter, + ) ### Decoding Table Modified: python/branches/p3yk/Lib/encodings/iso8859_6.py ============================================================================== --- python/branches/p3yk/Lib/encodings/iso8859_6.py (original) +++ python/branches/p3yk/Lib/encodings/iso8859_6.py Fri Apr 21 11:43:23 2006 @@ -1,4 +1,4 @@ -""" Python Character Mapping Codec generated from 'MAPPINGS/ISO8859/8859-6.TXT' with gencodec.py. +""" Python Character Mapping Codec iso8859_6 generated from 'MAPPINGS/ISO8859/8859-6.TXT' with gencodec.py. """#" @@ -9,13 +9,19 @@ class Codec(codecs.Codec): def encode(self,input,errors='strict'): - return codecs.charmap_encode(input,errors,encoding_map) def decode(self,input,errors='strict'): - return codecs.charmap_decode(input,errors,decoding_table) +class IncrementalEncoder(codecs.IncrementalEncoder): + def encode(self, input, final=False): + return codecs.charmap_encode(input,self.errors,encoding_map)[0] + +class IncrementalDecoder(codecs.IncrementalDecoder): + def decode(self, input, final=False): + return codecs.charmap_decode(input,self.errors,decoding_table)[0] + class StreamWriter(Codec,codecs.StreamWriter): pass @@ -25,8 +31,15 @@ ### encodings module API def getregentry(): - - return (Codec().encode,Codec().decode,StreamReader,StreamWriter) + return codecs.CodecInfo( + name='iso8859-6', + encode=Codec().encode, + decode=Codec().decode, + incrementalencoder=IncrementalEncoder, + incrementaldecoder=IncrementalDecoder, + streamreader=StreamReader, + streamwriter=StreamWriter, + ) ### Decoding Table Modified: python/branches/p3yk/Lib/encodings/iso8859_7.py ============================================================================== --- python/branches/p3yk/Lib/encodings/iso8859_7.py (original) +++ python/branches/p3yk/Lib/encodings/iso8859_7.py Fri Apr 21 11:43:23 2006 @@ -1,4 +1,4 @@ -""" Python Character Mapping Codec generated from 'MAPPINGS/ISO8859/8859-7.TXT' with gencodec.py. +""" Python Character Mapping Codec iso8859_7 generated from 'MAPPINGS/ISO8859/8859-7.TXT' with gencodec.py. """#" @@ -9,13 +9,19 @@ class Codec(codecs.Codec): def encode(self,input,errors='strict'): - return codecs.charmap_encode(input,errors,encoding_map) def decode(self,input,errors='strict'): - return codecs.charmap_decode(input,errors,decoding_table) +class IncrementalEncoder(codecs.IncrementalEncoder): + def encode(self, input, final=False): + return codecs.charmap_encode(input,self.errors,encoding_map)[0] + +class IncrementalDecoder(codecs.IncrementalDecoder): + def decode(self, input, final=False): + return codecs.charmap_decode(input,self.errors,decoding_table)[0] + class StreamWriter(Codec,codecs.StreamWriter): pass @@ -25,8 +31,15 @@ ### encodings module API def getregentry(): - - return (Codec().encode,Codec().decode,StreamReader,StreamWriter) + return codecs.CodecInfo( + name='iso8859-7', + encode=Codec().encode, + decode=Codec().decode, + incrementalencoder=IncrementalEncoder, + incrementaldecoder=IncrementalDecoder, + streamreader=StreamReader, + streamwriter=StreamWriter, + ) ### Decoding Table Modified: python/branches/p3yk/Lib/encodings/iso8859_8.py ============================================================================== --- python/branches/p3yk/Lib/encodings/iso8859_8.py (original) +++ python/branches/p3yk/Lib/encodings/iso8859_8.py Fri Apr 21 11:43:23 2006 @@ -1,4 +1,4 @@ -""" Python Character Mapping Codec generated from 'MAPPINGS/ISO8859/8859-8.TXT' with gencodec.py. +""" Python Character Mapping Codec iso8859_8 generated from 'MAPPINGS/ISO8859/8859-8.TXT' with gencodec.py. """#" @@ -9,13 +9,19 @@ class Codec(codecs.Codec): def encode(self,input,errors='strict'): - return codecs.charmap_encode(input,errors,encoding_map) def decode(self,input,errors='strict'): - return codecs.charmap_decode(input,errors,decoding_table) +class IncrementalEncoder(codecs.IncrementalEncoder): + def encode(self, input, final=False): + return codecs.charmap_encode(input,self.errors,encoding_map)[0] + +class IncrementalDecoder(codecs.IncrementalDecoder): + def decode(self, input, final=False): + return codecs.charmap_decode(input,self.errors,decoding_table)[0] + class StreamWriter(Codec,codecs.StreamWriter): pass @@ -25,8 +31,15 @@ ### encodings module API def getregentry(): - - return (Codec().encode,Codec().decode,StreamReader,StreamWriter) + return codecs.CodecInfo( + name='iso8859-8', + encode=Codec().encode, + decode=Codec().decode, + incrementalencoder=IncrementalEncoder, + incrementaldecoder=IncrementalDecoder, + streamreader=StreamReader, + streamwriter=StreamWriter, + ) ### Decoding Table Modified: python/branches/p3yk/Lib/encodings/iso8859_9.py ============================================================================== --- python/branches/p3yk/Lib/encodings/iso8859_9.py (original) +++ python/branches/p3yk/Lib/encodings/iso8859_9.py Fri Apr 21 11:43:23 2006 @@ -1,4 +1,4 @@ -""" Python Character Mapping Codec generated from 'MAPPINGS/ISO8859/8859-9.TXT' with gencodec.py. +""" Python Character Mapping Codec iso8859_9 generated from 'MAPPINGS/ISO8859/8859-9.TXT' with gencodec.py. """#" @@ -9,13 +9,19 @@ class Codec(codecs.Codec): def encode(self,input,errors='strict'): - return codecs.charmap_encode(input,errors,encoding_map) def decode(self,input,errors='strict'): - return codecs.charmap_decode(input,errors,decoding_table) +class IncrementalEncoder(codecs.IncrementalEncoder): + def encode(self, input, final=False): + return codecs.charmap_encode(input,self.errors,encoding_map)[0] + +class IncrementalDecoder(codecs.IncrementalDecoder): + def decode(self, input, final=False): + return codecs.charmap_decode(input,self.errors,decoding_table)[0] + class StreamWriter(Codec,codecs.StreamWriter): pass @@ -25,8 +31,15 @@ ### encodings module API def getregentry(): - - return (Codec().encode,Codec().decode,StreamReader,StreamWriter) + return codecs.CodecInfo( + name='iso8859-9', + encode=Codec().encode, + decode=Codec().decode, + incrementalencoder=IncrementalEncoder, + incrementaldecoder=IncrementalDecoder, + streamreader=StreamReader, + streamwriter=StreamWriter, + ) ### Decoding Table Modified: python/branches/p3yk/Lib/encodings/koi8_r.py ============================================================================== --- python/branches/p3yk/Lib/encodings/koi8_r.py (original) +++ python/branches/p3yk/Lib/encodings/koi8_r.py Fri Apr 21 11:43:23 2006 @@ -1,4 +1,4 @@ -""" Python Character Mapping Codec generated from 'MAPPINGS/VENDORS/MISC/KOI8-R.TXT' with gencodec.py. +""" Python Character Mapping Codec koi8_r generated from 'MAPPINGS/VENDORS/MISC/KOI8-R.TXT' with gencodec.py. """#" @@ -9,13 +9,19 @@ class Codec(codecs.Codec): def encode(self,input,errors='strict'): - return codecs.charmap_encode(input,errors,encoding_map) def decode(self,input,errors='strict'): - return codecs.charmap_decode(input,errors,decoding_table) +class IncrementalEncoder(codecs.IncrementalEncoder): + def encode(self, input, final=False): + return codecs.charmap_encode(input,self.errors,encoding_map)[0] + +class IncrementalDecoder(codecs.IncrementalDecoder): + def decode(self, input, final=False): + return codecs.charmap_decode(input,self.errors,decoding_table)[0] + class StreamWriter(Codec,codecs.StreamWriter): pass @@ -25,8 +31,15 @@ ### encodings module API def getregentry(): - - return (Codec().encode,Codec().decode,StreamReader,StreamWriter) + return codecs.CodecInfo( + name='koi8-r', + encode=Codec().encode, + decode=Codec().decode, + incrementalencoder=IncrementalEncoder, + incrementaldecoder=IncrementalDecoder, + streamreader=StreamReader, + streamwriter=StreamWriter, + ) ### Decoding Table Modified: python/branches/p3yk/Lib/encodings/koi8_u.py ============================================================================== --- python/branches/p3yk/Lib/encodings/koi8_u.py (original) +++ python/branches/p3yk/Lib/encodings/koi8_u.py Fri Apr 21 11:43:23 2006 @@ -1,4 +1,4 @@ -""" Python Character Mapping Codec generated from 'python-mappings/KOI8-U.TXT' with gencodec.py. +""" Python Character Mapping Codec koi8_u generated from 'python-mappings/KOI8-U.TXT' with gencodec.py. """#" @@ -9,13 +9,19 @@ class Codec(codecs.Codec): def encode(self,input,errors='strict'): - return codecs.charmap_encode(input,errors,encoding_map) def decode(self,input,errors='strict'): - return codecs.charmap_decode(input,errors,decoding_table) +class IncrementalEncoder(codecs.IncrementalEncoder): + def encode(self, input, final=False): + return codecs.charmap_encode(input,self.errors,encoding_map)[0] + +class IncrementalDecoder(codecs.IncrementalDecoder): + def decode(self, input, final=False): + return codecs.charmap_decode(input,self.errors,decoding_table)[0] + class StreamWriter(Codec,codecs.StreamWriter): pass @@ -25,8 +31,15 @@ ### encodings module API def getregentry(): - - return (Codec().encode,Codec().decode,StreamReader,StreamWriter) + return codecs.CodecInfo( + name='koi8-u', + encode=Codec().encode, + decode=Codec().decode, + incrementalencoder=IncrementalEncoder, + incrementaldecoder=IncrementalDecoder, + streamreader=StreamReader, + streamwriter=StreamWriter, + ) ### Decoding Table Modified: python/branches/p3yk/Lib/encodings/latin_1.py ============================================================================== --- python/branches/p3yk/Lib/encodings/latin_1.py (original) +++ python/branches/p3yk/Lib/encodings/latin_1.py Fri Apr 21 11:43:23 2006 @@ -17,6 +17,14 @@ encode = codecs.latin_1_encode decode = codecs.latin_1_decode +class IncrementalEncoder(codecs.IncrementalEncoder): + def encode(self, input, final=False): + return codecs.latin_1_encode(input,self.errors)[0] + +class IncrementalDecoder(codecs.IncrementalDecoder): + def decode(self, input, final=False): + return codecs.latin_1_decode(input,self.errors)[0] + class StreamWriter(Codec,codecs.StreamWriter): pass @@ -31,5 +39,12 @@ ### encodings module API def getregentry(): - - return (Codec.encode,Codec.decode,StreamReader,StreamWriter) + return codecs.CodecInfo( + name='iso8859-1', + encode=Codec.encode, + decode=Codec.decode, + incrementalencoder=IncrementalEncoder, + incrementaldecoder=IncrementalDecoder, + streamreader=StreamReader, + streamwriter=StreamWriter, + ) Modified: python/branches/p3yk/Lib/encodings/mac_arabic.py ============================================================================== --- python/branches/p3yk/Lib/encodings/mac_arabic.py (original) +++ python/branches/p3yk/Lib/encodings/mac_arabic.py Fri Apr 21 11:43:23 2006 @@ -9,13 +9,19 @@ class Codec(codecs.Codec): def encode(self,input,errors='strict'): - return codecs.charmap_encode(input,errors,encoding_map) def decode(self,input,errors='strict'): - return codecs.charmap_decode(input,errors,decoding_table) +class IncrementalEncoder(codecs.IncrementalEncoder): + def encode(self, input, final=False): + return codecs.charmap_encode(input,self.errors,encoding_map)[0] + +class IncrementalDecoder(codecs.IncrementalDecoder): + def decode(self, input, final=False): + return codecs.charmap_decode(input,self.errors,decoding_table)[0] + class StreamWriter(Codec,codecs.StreamWriter): pass @@ -25,8 +31,15 @@ ### encodings module API def getregentry(): - - return (Codec().encode,Codec().decode,StreamReader,StreamWriter) + return codecs.CodecInfo( + name='mac-arabic', + encode=Codec().encode, + decode=Codec().decode, + incrementalencoder=IncrementalEncoder, + incrementaldecoder=IncrementalDecoder, + streamreader=StreamReader, + streamwriter=StreamWriter, + ) ### Decoding Map Modified: python/branches/p3yk/Lib/encodings/mac_centeuro.py ============================================================================== --- python/branches/p3yk/Lib/encodings/mac_centeuro.py (original) +++ python/branches/p3yk/Lib/encodings/mac_centeuro.py Fri Apr 21 11:43:23 2006 @@ -1,4 +1,4 @@ -""" Python Character Mapping Codec generated from 'MAPPINGS/VENDORS/APPLE/CENTEURO.TXT' with gencodec.py. +""" Python Character Mapping Codec mac_centeuro generated from 'MAPPINGS/VENDORS/APPLE/CENTEURO.TXT' with gencodec.py. """#" @@ -9,13 +9,19 @@ class Codec(codecs.Codec): def encode(self,input,errors='strict'): - return codecs.charmap_encode(input,errors,encoding_map) def decode(self,input,errors='strict'): - return codecs.charmap_decode(input,errors,decoding_table) +class IncrementalEncoder(codecs.IncrementalEncoder): + def encode(self, input, final=False): + return codecs.charmap_encode(input,self.errors,encoding_map)[0] + +class IncrementalDecoder(codecs.IncrementalDecoder): + def decode(self, input, final=False): + return codecs.charmap_decode(input,self.errors,decoding_table)[0] + class StreamWriter(Codec,codecs.StreamWriter): pass @@ -25,8 +31,15 @@ ### encodings module API def getregentry(): - - return (Codec().encode,Codec().decode,StreamReader,StreamWriter) + return codecs.CodecInfo( + name='mac-centeuro', + encode=Codec().encode, + decode=Codec().decode, + incrementalencoder=IncrementalEncoder, + incrementaldecoder=IncrementalDecoder, + streamreader=StreamReader, + streamwriter=StreamWriter, + ) ### Decoding Table Modified: python/branches/p3yk/Lib/encodings/mac_croatian.py ============================================================================== --- python/branches/p3yk/Lib/encodings/mac_croatian.py (original) +++ python/branches/p3yk/Lib/encodings/mac_croatian.py Fri Apr 21 11:43:23 2006 @@ -1,4 +1,4 @@ -""" Python Character Mapping Codec generated from 'MAPPINGS/VENDORS/APPLE/CROATIAN.TXT' with gencodec.py. +""" Python Character Mapping Codec mac_croatian generated from 'MAPPINGS/VENDORS/APPLE/CROATIAN.TXT' with gencodec.py. """#" @@ -9,13 +9,19 @@ class Codec(codecs.Codec): def encode(self,input,errors='strict'): - return codecs.charmap_encode(input,errors,encoding_map) def decode(self,input,errors='strict'): - return codecs.charmap_decode(input,errors,decoding_table) +class IncrementalEncoder(codecs.IncrementalEncoder): + def encode(self, input, final=False): + return codecs.charmap_encode(input,self.errors,encoding_map)[0] + +class IncrementalDecoder(codecs.IncrementalDecoder): + def decode(self, input, final=False): + return codecs.charmap_decode(input,self.errors,decoding_table)[0] + class StreamWriter(Codec,codecs.StreamWriter): pass @@ -25,8 +31,15 @@ ### encodings module API def getregentry(): - - return (Codec().encode,Codec().decode,StreamReader,StreamWriter) + return codecs.CodecInfo( + name='mac-croatian', + encode=Codec().encode, + decode=Codec().decode, + incrementalencoder=IncrementalEncoder, + incrementaldecoder=IncrementalDecoder, + streamreader=StreamReader, + streamwriter=StreamWriter, + ) ### Decoding Table Modified: python/branches/p3yk/Lib/encodings/mac_cyrillic.py ============================================================================== --- python/branches/p3yk/Lib/encodings/mac_cyrillic.py (original) +++ python/branches/p3yk/Lib/encodings/mac_cyrillic.py Fri Apr 21 11:43:23 2006 @@ -1,4 +1,4 @@ -""" Python Character Mapping Codec generated from 'MAPPINGS/VENDORS/APPLE/CYRILLIC.TXT' with gencodec.py. +""" Python Character Mapping Codec mac_cyrillic generated from 'MAPPINGS/VENDORS/APPLE/CYRILLIC.TXT' with gencodec.py. """#" @@ -9,13 +9,19 @@ class Codec(codecs.Codec): def encode(self,input,errors='strict'): - return codecs.charmap_encode(input,errors,encoding_map) def decode(self,input,errors='strict'): - return codecs.charmap_decode(input,errors,decoding_table) +class IncrementalEncoder(codecs.IncrementalEncoder): + def encode(self, input, final=False): + return codecs.charmap_encode(input,self.errors,encoding_map)[0] + +class IncrementalDecoder(codecs.IncrementalDecoder): + def decode(self, input, final=False): + return codecs.charmap_decode(input,self.errors,decoding_table)[0] + class StreamWriter(Codec,codecs.StreamWriter): pass @@ -25,8 +31,15 @@ ### encodings module API def getregentry(): - - return (Codec().encode,Codec().decode,StreamReader,StreamWriter) + return codecs.CodecInfo( + name='mac-cyrillic', + encode=Codec().encode, + decode=Codec().decode, + incrementalencoder=IncrementalEncoder, + incrementaldecoder=IncrementalDecoder, + streamreader=StreamReader, + streamwriter=StreamWriter, + ) ### Decoding Table Modified: python/branches/p3yk/Lib/encodings/mac_farsi.py ============================================================================== --- python/branches/p3yk/Lib/encodings/mac_farsi.py (original) +++ python/branches/p3yk/Lib/encodings/mac_farsi.py Fri Apr 21 11:43:23 2006 @@ -1,4 +1,4 @@ -""" Python Character Mapping Codec generated from 'MAPPINGS/VENDORS/APPLE/FARSI.TXT' with gencodec.py. +""" Python Character Mapping Codec mac_farsi generated from 'MAPPINGS/VENDORS/APPLE/FARSI.TXT' with gencodec.py. """#" @@ -9,13 +9,19 @@ class Codec(codecs.Codec): def encode(self,input,errors='strict'): - return codecs.charmap_encode(input,errors,encoding_map) def decode(self,input,errors='strict'): - return codecs.charmap_decode(input,errors,decoding_table) +class IncrementalEncoder(codecs.IncrementalEncoder): + def encode(self, input, final=False): + return codecs.charmap_encode(input,self.errors,encoding_map)[0] + +class IncrementalDecoder(codecs.IncrementalDecoder): + def decode(self, input, final=False): + return codecs.charmap_decode(input,self.errors,decoding_table)[0] + class StreamWriter(Codec,codecs.StreamWriter): pass @@ -25,8 +31,15 @@ ### encodings module API def getregentry(): - - return (Codec().encode,Codec().decode,StreamReader,StreamWriter) + return codecs.CodecInfo( + name='mac-farsi', + encode=Codec().encode, + decode=Codec().decode, + incrementalencoder=IncrementalEncoder, + incrementaldecoder=IncrementalDecoder, + streamreader=StreamReader, + streamwriter=StreamWriter, + ) ### Decoding Table Modified: python/branches/p3yk/Lib/encodings/mac_greek.py ============================================================================== --- python/branches/p3yk/Lib/encodings/mac_greek.py (original) +++ python/branches/p3yk/Lib/encodings/mac_greek.py Fri Apr 21 11:43:23 2006 @@ -1,4 +1,4 @@ -""" Python Character Mapping Codec generated from 'MAPPINGS/VENDORS/APPLE/GREEK.TXT' with gencodec.py. +""" Python Character Mapping Codec mac_greek generated from 'MAPPINGS/VENDORS/APPLE/GREEK.TXT' with gencodec.py. """#" @@ -9,13 +9,19 @@ class Codec(codecs.Codec): def encode(self,input,errors='strict'): - return codecs.charmap_encode(input,errors,encoding_map) def decode(self,input,errors='strict'): - return codecs.charmap_decode(input,errors,decoding_table) +class IncrementalEncoder(codecs.IncrementalEncoder): + def encode(self, input, final=False): + return codecs.charmap_encode(input,self.errors,encoding_map)[0] + +class IncrementalDecoder(codecs.IncrementalDecoder): + def decode(self, input, final=False): + return codecs.charmap_decode(input,self.errors,decoding_table)[0] + class StreamWriter(Codec,codecs.StreamWriter): pass @@ -25,8 +31,15 @@ ### encodings module API def getregentry(): - - return (Codec().encode,Codec().decode,StreamReader,StreamWriter) + return codecs.CodecInfo( + name='mac-greek', + encode=Codec().encode, + decode=Codec().decode, + incrementalencoder=IncrementalEncoder, + incrementaldecoder=IncrementalDecoder, + streamreader=StreamReader, + streamwriter=StreamWriter, + ) ### Decoding Table Modified: python/branches/p3yk/Lib/encodings/mac_iceland.py ============================================================================== --- python/branches/p3yk/Lib/encodings/mac_iceland.py (original) +++ python/branches/p3yk/Lib/encodings/mac_iceland.py Fri Apr 21 11:43:23 2006 @@ -1,4 +1,4 @@ -""" Python Character Mapping Codec generated from 'MAPPINGS/VENDORS/APPLE/ICELAND.TXT' with gencodec.py. +""" Python Character Mapping Codec mac_iceland generated from 'MAPPINGS/VENDORS/APPLE/ICELAND.TXT' with gencodec.py. """#" @@ -9,13 +9,19 @@ class Codec(codecs.Codec): def encode(self,input,errors='strict'): - return codecs.charmap_encode(input,errors,encoding_map) def decode(self,input,errors='strict'): - return codecs.charmap_decode(input,errors,decoding_table) +class IncrementalEncoder(codecs.IncrementalEncoder): + def encode(self, input, final=False): + return codecs.charmap_encode(input,self.errors,encoding_map)[0] + +class IncrementalDecoder(codecs.IncrementalDecoder): + def decode(self, input, final=False): + return codecs.charmap_decode(input,self.errors,decoding_table)[0] + class StreamWriter(Codec,codecs.StreamWriter): pass @@ -25,8 +31,15 @@ ### encodings module API def getregentry(): - - return (Codec().encode,Codec().decode,StreamReader,StreamWriter) + return codecs.CodecInfo( + name='mac-iceland', + encode=Codec().encode, + decode=Codec().decode, + incrementalencoder=IncrementalEncoder, + incrementaldecoder=IncrementalDecoder, + streamreader=StreamReader, + streamwriter=StreamWriter, + ) ### Decoding Table Modified: python/branches/p3yk/Lib/encodings/mac_latin2.py ============================================================================== --- python/branches/p3yk/Lib/encodings/mac_latin2.py (original) +++ python/branches/p3yk/Lib/encodings/mac_latin2.py Fri Apr 21 11:43:23 2006 @@ -14,13 +14,19 @@ class Codec(codecs.Codec): def encode(self,input,errors='strict'): - return codecs.charmap_encode(input,errors,encoding_map) def decode(self,input,errors='strict'): - return codecs.charmap_decode(input,errors,decoding_map) +class IncrementalEncoder(codecs.IncrementalEncoder): + def encode(self, input, final=False): + return codecs.charmap_encode(input,self.errors,encoding_map)[0] + +class IncrementalDecoder(codecs.IncrementalDecoder): + def decode(self, input, final=False): + return codecs.charmap_decode(input,self.errors,decoding_map)[0] + class StreamWriter(Codec,codecs.StreamWriter): pass @@ -30,8 +36,15 @@ ### encodings module API def getregentry(): - - return (Codec().encode,Codec().decode,StreamReader,StreamWriter) + return codecs.CodecInfo( + name='mac-latin2', + encode=Codec().encode, + decode=Codec().decode, + incrementalencoder=IncrementalEncoder, + incrementaldecoder=IncrementalDecoder, + streamreader=StreamReader, + streamwriter=StreamWriter, + ) ### Decoding Map Modified: python/branches/p3yk/Lib/encodings/mac_roman.py ============================================================================== --- python/branches/p3yk/Lib/encodings/mac_roman.py (original) +++ python/branches/p3yk/Lib/encodings/mac_roman.py Fri Apr 21 11:43:23 2006 @@ -1,4 +1,4 @@ -""" Python Character Mapping Codec generated from 'MAPPINGS/VENDORS/APPLE/ROMAN.TXT' with gencodec.py. +""" Python Character Mapping Codec mac_roman generated from 'MAPPINGS/VENDORS/APPLE/ROMAN.TXT' with gencodec.py. """#" @@ -9,13 +9,19 @@ class Codec(codecs.Codec): def encode(self,input,errors='strict'): - return codecs.charmap_encode(input,errors,encoding_map) def decode(self,input,errors='strict'): - return codecs.charmap_decode(input,errors,decoding_table) +class IncrementalEncoder(codecs.IncrementalEncoder): + def encode(self, input, final=False): + return codecs.charmap_encode(input,self.errors,encoding_map)[0] + +class IncrementalDecoder(codecs.IncrementalDecoder): + def decode(self, input, final=False): + return codecs.charmap_decode(input,self.errors,decoding_table)[0] + class StreamWriter(Codec,codecs.StreamWriter): pass @@ -25,8 +31,15 @@ ### encodings module API def getregentry(): - - return (Codec().encode,Codec().decode,StreamReader,StreamWriter) + return codecs.CodecInfo( + name='mac-roman', + encode=Codec().encode, + decode=Codec().decode, + incrementalencoder=IncrementalEncoder, + incrementaldecoder=IncrementalDecoder, + streamreader=StreamReader, + streamwriter=StreamWriter, + ) ### Decoding Table Modified: python/branches/p3yk/Lib/encodings/mac_romanian.py ============================================================================== --- python/branches/p3yk/Lib/encodings/mac_romanian.py (original) +++ python/branches/p3yk/Lib/encodings/mac_romanian.py Fri Apr 21 11:43:23 2006 @@ -1,4 +1,4 @@ -""" Python Character Mapping Codec generated from 'MAPPINGS/VENDORS/APPLE/ROMANIAN.TXT' with gencodec.py. +""" Python Character Mapping Codec mac_romanian generated from 'MAPPINGS/VENDORS/APPLE/ROMANIAN.TXT' with gencodec.py. """#" @@ -9,13 +9,19 @@ class Codec(codecs.Codec): def encode(self,input,errors='strict'): - return codecs.charmap_encode(input,errors,encoding_map) def decode(self,input,errors='strict'): - return codecs.charmap_decode(input,errors,decoding_table) +class IncrementalEncoder(codecs.IncrementalEncoder): + def encode(self, input, final=False): + return codecs.charmap_encode(input,self.errors,encoding_map)[0] + +class IncrementalDecoder(codecs.IncrementalDecoder): + def decode(self, input, final=False): + return codecs.charmap_decode(input,self.errors,decoding_table)[0] + class StreamWriter(Codec,codecs.StreamWriter): pass @@ -25,8 +31,15 @@ ### encodings module API def getregentry(): - - return (Codec().encode,Codec().decode,StreamReader,StreamWriter) + return codecs.CodecInfo( + name='mac-romanian', + encode=Codec().encode, + decode=Codec().decode, + incrementalencoder=IncrementalEncoder, + incrementaldecoder=IncrementalDecoder, + streamreader=StreamReader, + streamwriter=StreamWriter, + ) ### Decoding Table Modified: python/branches/p3yk/Lib/encodings/mac_turkish.py ============================================================================== --- python/branches/p3yk/Lib/encodings/mac_turkish.py (original) +++ python/branches/p3yk/Lib/encodings/mac_turkish.py Fri Apr 21 11:43:23 2006 @@ -1,4 +1,4 @@ -""" Python Character Mapping Codec generated from 'MAPPINGS/VENDORS/APPLE/TURKISH.TXT' with gencodec.py. +""" Python Character Mapping Codec mac_turkish generated from 'MAPPINGS/VENDORS/APPLE/TURKISH.TXT' with gencodec.py. """#" @@ -9,13 +9,19 @@ class Codec(codecs.Codec): def encode(self,input,errors='strict'): - return codecs.charmap_encode(input,errors,encoding_map) def decode(self,input,errors='strict'): - return codecs.charmap_decode(input,errors,decoding_table) +class IncrementalEncoder(codecs.IncrementalEncoder): + def encode(self, input, final=False): + return codecs.charmap_encode(input,self.errors,encoding_map)[0] + +class IncrementalDecoder(codecs.IncrementalDecoder): + def decode(self, input, final=False): + return codecs.charmap_decode(input,self.errors,decoding_table)[0] + class StreamWriter(Codec,codecs.StreamWriter): pass @@ -25,8 +31,15 @@ ### encodings module API def getregentry(): - - return (Codec().encode,Codec().decode,StreamReader,StreamWriter) + return codecs.CodecInfo( + name='mac-turkish', + encode=Codec().encode, + decode=Codec().decode, + incrementalencoder=IncrementalEncoder, + incrementaldecoder=IncrementalDecoder, + streamreader=StreamReader, + streamwriter=StreamWriter, + ) ### Decoding Table Modified: python/branches/p3yk/Lib/encodings/mbcs.py ============================================================================== --- python/branches/p3yk/Lib/encodings/mbcs.py (original) +++ python/branches/p3yk/Lib/encodings/mbcs.py Fri Apr 21 11:43:23 2006 @@ -18,6 +18,13 @@ encode = codecs.mbcs_encode decode = codecs.mbcs_decode +class IncrementalEncoder(codecs.IncrementalEncoder): + def encode(self, input, final=False): + return codecs.mbcs_encode(input,self.errors)[0] + +class IncrementalDecoder(codecs.IncrementalDecoder): + def decode(self, input, final=False): + return codecs.mbcs_decode(input,self.errors)[0] class StreamWriter(Codec,codecs.StreamWriter): pass @@ -32,5 +39,12 @@ ### encodings module API def getregentry(): - - return (Codec.encode,Codec.decode,StreamReader,StreamWriter) + return codecs.CodecInfo( + name='mbcs', + encode=Codec.encode, + decode=Codec.decode, + incrementalencoder=IncrementalEncoder, + incrementaldecoder=IncrementalDecoder, + streamreader=StreamReader, + streamwriter=StreamWriter, + ) Modified: python/branches/p3yk/Lib/encodings/palmos.py ============================================================================== --- python/branches/p3yk/Lib/encodings/palmos.py (original) +++ python/branches/p3yk/Lib/encodings/palmos.py Fri Apr 21 11:43:23 2006 @@ -15,6 +15,14 @@ def decode(self,input,errors='strict'): return codecs.charmap_decode(input,errors,decoding_map) +class IncrementalEncoder(codecs.IncrementalEncoder): + def encode(self, input, final=False): + return codecs.charmap_encode(input,self.errors,encoding_map)[0] + +class IncrementalDecoder(codecs.IncrementalDecoder): + def decode(self, input, final=False): + return codecs.charmap_decode(input,self.errors,decoding_map)[0] + class StreamWriter(Codec,codecs.StreamWriter): pass @@ -24,7 +32,15 @@ ### encodings module API def getregentry(): - return (Codec().encode,Codec().decode,StreamReader,StreamWriter) + return codecs.CodecInfo( + name='palmos', + encode=Codec().encode, + decode=Codec().decode, + incrementalencoder=IncrementalEncoder, + incrementaldecoder=IncrementalDecoder, + streamreader=StreamReader, + streamwriter=StreamWriter, + ) ### Decoding Map Modified: python/branches/p3yk/Lib/encodings/ptcp154.py ============================================================================== --- python/branches/p3yk/Lib/encodings/ptcp154.py (original) +++ python/branches/p3yk/Lib/encodings/ptcp154.py Fri Apr 21 11:43:23 2006 @@ -14,13 +14,19 @@ class Codec(codecs.Codec): def encode(self,input,errors='strict'): - return codecs.charmap_encode(input,errors,encoding_map) def decode(self,input,errors='strict'): - return codecs.charmap_decode(input,errors,decoding_map) +class IncrementalEncoder(codecs.IncrementalEncoder): + def encode(self, input, final=False): + return codecs.charmap_encode(input,self.errors,encoding_map)[0] + +class IncrementalDecoder(codecs.IncrementalDecoder): + def decode(self, input, final=False): + return codecs.charmap_decode(input,self.errors,decoding_map)[0] + class StreamWriter(Codec,codecs.StreamWriter): pass @@ -30,8 +36,15 @@ ### encodings module API def getregentry(): - - return (Codec().encode,Codec().decode,StreamReader,StreamWriter) + return codecs.CodecInfo( + name='ptcp154', + encode=Codec().encode, + decode=Codec().decode, + incrementalencoder=IncrementalEncoder, + incrementaldecoder=IncrementalDecoder, + streamreader=StreamReader, + streamwriter=StreamWriter, + ) ### Decoding Map Modified: python/branches/p3yk/Lib/encodings/punycode.py ============================================================================== --- python/branches/p3yk/Lib/encodings/punycode.py (original) +++ python/branches/p3yk/Lib/encodings/punycode.py Fri Apr 21 11:43:23 2006 @@ -197,18 +197,27 @@ ### Codec APIs class Codec(codecs.Codec): - def encode(self,input,errors='strict'): + def encode(self,input,errors='strict'): res = punycode_encode(input) return res, len(input) def decode(self,input,errors='strict'): - if errors not in ('strict', 'replace', 'ignore'): raise UnicodeError, "Unsupported error handling "+errors res = punycode_decode(input, errors) return res, len(input) +class IncrementalEncoder(codecs.IncrementalEncoder): + def encode(self, input, final=False): + return punycode_encode(input) + +class IncrementalDecoder(codecs.IncrementalDecoder): + def decode(self, input, final=False): + if errors not in ('strict', 'replace', 'ignore'): + raise UnicodeError, "Unsupported error handling "+errors + return punycode_decode(input, errors) + class StreamWriter(Codec,codecs.StreamWriter): pass @@ -218,5 +227,12 @@ ### encodings module API def getregentry(): - - return (Codec().encode,Codec().decode,StreamReader,StreamWriter) + return codecs.CodecInfo( + name='punycode', + encode=Codec().encode, + decode=Codec().decode, + incrementalencoder=IncrementalEncoder, + incrementaldecoder=IncrementalDecoder, + streamwriter=StreamWriter, + streamreader=StreamReader, + ) Modified: python/branches/p3yk/Lib/encodings/quopri_codec.py ============================================================================== --- python/branches/p3yk/Lib/encodings/quopri_codec.py (original) +++ python/branches/p3yk/Lib/encodings/quopri_codec.py Fri Apr 21 11:43:23 2006 @@ -46,6 +46,14 @@ def decode(self, input,errors='strict'): return quopri_decode(input,errors) +class IncrementalEncoder(codecs.IncrementalEncoder): + def encode(self, input, final=False): + return quopri_encode(input, self.errors)[0] + +class IncrementalDecoder(codecs.IncrementalDecoder): + def decode(self, input, final=False): + return quopri_decode(input, self.errors)[0] + class StreamWriter(Codec, codecs.StreamWriter): pass @@ -55,4 +63,12 @@ # encodings module API def getregentry(): - return (quopri_encode, quopri_decode, StreamReader, StreamWriter) + return codecs.CodecInfo( + name='quopri', + encode=quopri_encode, + decode=quopri_decode, + incrementalencoder=IncrementalEncoder, + incrementaldecoder=IncrementalDecoder, + streamwriter=StreamWriter, + streamreader=StreamReader, + ) Modified: python/branches/p3yk/Lib/encodings/raw_unicode_escape.py ============================================================================== --- python/branches/p3yk/Lib/encodings/raw_unicode_escape.py (original) +++ python/branches/p3yk/Lib/encodings/raw_unicode_escape.py Fri Apr 21 11:43:23 2006 @@ -17,6 +17,14 @@ encode = codecs.raw_unicode_escape_encode decode = codecs.raw_unicode_escape_decode +class IncrementalEncoder(codecs.IncrementalEncoder): + def encode(self, input, final=False): + return codecs.raw_unicode_escape_encode(input, self.errors)[0] + +class IncrementalDecoder(codecs.IncrementalDecoder): + def decode(self, input, final=False): + return codecs.raw_unicode_escape_decode(input, self.errors)[0] + class StreamWriter(Codec,codecs.StreamWriter): pass @@ -26,5 +34,12 @@ ### encodings module API def getregentry(): - - return (Codec.encode,Codec.decode,StreamReader,StreamWriter) + return codecs.CodecInfo( + name='raw-unicode-escape', + encode=Codec.encode, + decode=Codec.decode, + incrementalencoder=IncrementalEncoder, + incrementaldecoder=IncrementalDecoder, + streamwriter=StreamWriter, + streamreader=StreamReader, + ) Modified: python/branches/p3yk/Lib/encodings/rot_13.py ============================================================================== --- python/branches/p3yk/Lib/encodings/rot_13.py (original) +++ python/branches/p3yk/Lib/encodings/rot_13.py Fri Apr 21 11:43:23 2006 @@ -14,13 +14,19 @@ class Codec(codecs.Codec): def encode(self,input,errors='strict'): - return codecs.charmap_encode(input,errors,encoding_map) def decode(self,input,errors='strict'): - return codecs.charmap_decode(input,errors,decoding_map) +class IncrementalEncoder(codecs.IncrementalEncoder): + def encode(self, input, final=False): + return codecs.charmap_encode(input,self.errors,encoding_map)[0] + +class IncrementalDecoder(codecs.IncrementalDecoder): + def decode(self, input, final=False): + return codecs.charmap_decode(input,self.errors,decoding_map)[0] + class StreamWriter(Codec,codecs.StreamWriter): pass @@ -30,8 +36,15 @@ ### encodings module API def getregentry(): - - return (Codec().encode,Codec().decode,StreamReader,StreamWriter) + return codecs.CodecInfo( + name='rot-13', + encode=Codec().encode, + decode=Codec().decode, + incrementalencoder=IncrementalEncoder, + incrementaldecoder=IncrementalDecoder, + streamwriter=StreamWriter, + streamreader=StreamReader, + ) ### Decoding Map Modified: python/branches/p3yk/Lib/encodings/string_escape.py ============================================================================== --- python/branches/p3yk/Lib/encodings/string_escape.py (original) +++ python/branches/p3yk/Lib/encodings/string_escape.py Fri Apr 21 11:43:23 2006 @@ -12,6 +12,14 @@ encode = codecs.escape_encode decode = codecs.escape_decode +class IncrementalEncoder(codecs.IncrementalEncoder): + def encode(self, input, final=False): + return codecs.escape_encode(input, self.errors)[0] + +class IncrementalDecoder(codecs.IncrementalDecoder): + def decode(self, input, final=False): + return codecs.escape_decode(input, self.errors)[0] + class StreamWriter(Codec,codecs.StreamWriter): pass @@ -19,5 +27,12 @@ pass def getregentry(): - - return (Codec.encode,Codec.decode,StreamReader,StreamWriter) + return codecs.CodecInfo( + name='string-escape', + encode=Codec.encode, + decode=Codec.decode, + incrementalencoder=IncrementalEncoder, + incrementaldecoder=IncrementalDecoder, + streamwriter=StreamWriter, + streamreader=StreamReader, + ) Modified: python/branches/p3yk/Lib/encodings/tis_620.py ============================================================================== --- python/branches/p3yk/Lib/encodings/tis_620.py (original) +++ python/branches/p3yk/Lib/encodings/tis_620.py Fri Apr 21 11:43:23 2006 @@ -1,4 +1,4 @@ -""" Python Character Mapping Codec generated from 'python-mappings/TIS-620.TXT' with gencodec.py. +""" Python Character Mapping Codec tis_620 generated from 'python-mappings/TIS-620.TXT' with gencodec.py. """#" @@ -9,13 +9,19 @@ class Codec(codecs.Codec): def encode(self,input,errors='strict'): - return codecs.charmap_encode(input,errors,encoding_map) def decode(self,input,errors='strict'): - return codecs.charmap_decode(input,errors,decoding_table) +class IncrementalEncoder(codecs.IncrementalEncoder): + def encode(self, input, final=False): + return codecs.charmap_encode(input,self.errors,encoding_map)[0] + +class IncrementalDecoder(codecs.IncrementalDecoder): + def decode(self, input, final=False): + return codecs.charmap_decode(input,self.errors,decoding_table)[0] + class StreamWriter(Codec,codecs.StreamWriter): pass @@ -25,8 +31,15 @@ ### encodings module API def getregentry(): - - return (Codec().encode,Codec().decode,StreamReader,StreamWriter) + return codecs.CodecInfo( + name='tis-620', + encode=Codec().encode, + decode=Codec().decode, + incrementalencoder=IncrementalEncoder, + incrementaldecoder=IncrementalDecoder, + streamreader=StreamReader, + streamwriter=StreamWriter, + ) ### Decoding Table Modified: python/branches/p3yk/Lib/encodings/undefined.py ============================================================================== --- python/branches/p3yk/Lib/encodings/undefined.py (original) +++ python/branches/p3yk/Lib/encodings/undefined.py Fri Apr 21 11:43:23 2006 @@ -16,10 +16,18 @@ class Codec(codecs.Codec): def encode(self,input,errors='strict'): - raise UnicodeError, "undefined encoding" + raise UnicodeError("undefined encoding") def decode(self,input,errors='strict'): - raise UnicodeError, "undefined encoding" + raise UnicodeError("undefined encoding") + +class IncrementalEncoder(codecs.IncrementalEncoder): + def encode(self, input, final=False): + raise UnicodeError("undefined encoding") + +class IncrementalDecoder(codecs.IncrementalDecoder): + def decode(self, input, final=False): + raise UnicodeError("undefined encoding") class StreamWriter(Codec,codecs.StreamWriter): pass @@ -30,5 +38,12 @@ ### encodings module API def getregentry(): - - return (Codec().encode,Codec().decode,StreamReader,StreamWriter) + return codecs.CodecInfo( + name='undefined', + encode=Codec().encode, + decode=Codec().decode, + incrementalencoder=IncrementalEncoder, + incrementaldecoder=IncrementalDecoder, + streamwriter=StreamWriter, + streamreader=StreamReader, + ) Modified: python/branches/p3yk/Lib/encodings/unicode_escape.py ============================================================================== --- python/branches/p3yk/Lib/encodings/unicode_escape.py (original) +++ python/branches/p3yk/Lib/encodings/unicode_escape.py Fri Apr 21 11:43:23 2006 @@ -17,6 +17,14 @@ encode = codecs.unicode_escape_encode decode = codecs.unicode_escape_decode +class IncrementalEncoder(codecs.IncrementalEncoder): + def encode(self, input, final=False): + return codecs.unicode_escape_encode(input, self.errors)[0] + +class IncrementalDecoder(codecs.IncrementalDecoder): + def decode(self, input, final=False): + return codecs.unicode_escape_decode(input, self.errors)[0] + class StreamWriter(Codec,codecs.StreamWriter): pass @@ -26,5 +34,12 @@ ### encodings module API def getregentry(): - - return (Codec.encode,Codec.decode,StreamReader,StreamWriter) + return codecs.CodecInfo( + name='unicode-escape', + encode=Codec.encode, + decode=Codec.decode, + incrementalencoder=IncrementalEncoder, + incrementaldecoder=IncrementalDecoder, + streamwriter=StreamWriter, + streamreader=StreamReader, + ) Modified: python/branches/p3yk/Lib/encodings/unicode_internal.py ============================================================================== --- python/branches/p3yk/Lib/encodings/unicode_internal.py (original) +++ python/branches/p3yk/Lib/encodings/unicode_internal.py Fri Apr 21 11:43:23 2006 @@ -17,6 +17,14 @@ encode = codecs.unicode_internal_encode decode = codecs.unicode_internal_decode +class IncrementalEncoder(codecs.IncrementalEncoder): + def encode(self, input, final=False): + return codecs.unicode_internal_encode(input, self.errors)[0] + +class IncrementalDecoder(codecs.IncrementalDecoder): + def decode(self, input, final=False): + return codecs.unicode_internal_decode(input, self.errors)[0] + class StreamWriter(Codec,codecs.StreamWriter): pass @@ -26,5 +34,12 @@ ### encodings module API def getregentry(): - - return (Codec.encode,Codec.decode,StreamReader,StreamWriter) + return codecs.CodecInfo( + name='unicode-internal', + encode=Codec.encode, + decode=Codec.decode, + incrementalencoder=IncrementalEncoder, + incrementaldecoder=IncrementalDecoder, + streamwriter=StreamWriter, + streamreader=StreamReader, + ) Modified: python/branches/p3yk/Lib/encodings/utf_16.py ============================================================================== --- python/branches/p3yk/Lib/encodings/utf_16.py (original) +++ python/branches/p3yk/Lib/encodings/utf_16.py Fri Apr 21 11:43:23 2006 @@ -15,6 +15,47 @@ def decode(input, errors='strict'): return codecs.utf_16_decode(input, errors, True) +class IncrementalEncoder(codecs.IncrementalEncoder): + def __init__(self, errors='strict'): + codecs.IncrementalEncoder.__init__(self, errors) + self.encoder = None + + def encode(self, input, final=False): + if self.encoder is None: + result = codecs.utf_16_encode(input, self.errors)[0] + if sys.byteorder == 'little': + self.encoder = codecs.utf_16_le_encode + else: + self.encoder = codecs.utf_16_be_encode + return result + return self.encoder(input, self.errors)[0] + + def reset(self): + codecs.IncrementalEncoder.reset(self) + self.encoder = None + +class IncrementalDecoder(codecs.BufferedIncrementalDecoder): + def __init__(self, errors='strict'): + codecs.BufferedIncrementalDecoder.__init__(self, errors) + self.decoder = None + + def _buffer_decode(self, input, errors, final): + if self.decoder is None: + (output, consumed, byteorder) = \ + codecs.utf_16_ex_decode(input, errors, 0, final) + if byteorder == -1: + self.decoder = codecs.utf_16_le_decode + elif byteorder == 1: + self.decoder = codecs.utf_16_be_decode + elif consumed >= 2: + raise UnicodeError("UTF-16 stream does not start with BOM") + return (output, consumed) + return self.decoder(input, self.errors, final) + + def reset(self): + codecs.BufferedIncrementalDecoder.reset(self) + self.decoder = None + class StreamWriter(codecs.StreamWriter): def __init__(self, stream, errors='strict'): self.bom_written = False @@ -52,5 +93,12 @@ ### encodings module API def getregentry(): - - return (encode,decode,StreamReader,StreamWriter) + return codecs.CodecInfo( + name='utf-16', + encode=encode, + decode=decode, + incrementalencoder=IncrementalEncoder, + incrementaldecoder=IncrementalDecoder, + streamreader=StreamReader, + streamwriter=StreamWriter, + ) Modified: python/branches/p3yk/Lib/encodings/utf_16_be.py ============================================================================== --- python/branches/p3yk/Lib/encodings/utf_16_be.py (original) +++ python/branches/p3yk/Lib/encodings/utf_16_be.py Fri Apr 21 11:43:23 2006 @@ -15,6 +15,13 @@ def decode(input, errors='strict'): return codecs.utf_16_be_decode(input, errors, True) +class IncrementalEncoder(codecs.IncrementalEncoder): + def encode(self, input, final=False): + return codecs.utf_16_be_encode(input, self.errors)[0] + +class IncrementalDecoder(codecs.BufferedIncrementalDecoder): + _buffer_decode = codecs.utf_16_be_decode + class StreamWriter(codecs.StreamWriter): encode = codecs.utf_16_be_encode @@ -24,5 +31,12 @@ ### encodings module API def getregentry(): - - return (encode,decode,StreamReader,StreamWriter) + return codecs.CodecInfo( + name='utf-16-be', + encode=encode, + decode=decode, + incrementalencoder=IncrementalEncoder, + incrementaldecoder=IncrementalDecoder, + streamreader=StreamReader, + streamwriter=StreamWriter, + ) Modified: python/branches/p3yk/Lib/encodings/utf_16_le.py ============================================================================== --- python/branches/p3yk/Lib/encodings/utf_16_le.py (original) +++ python/branches/p3yk/Lib/encodings/utf_16_le.py Fri Apr 21 11:43:23 2006 @@ -15,15 +15,28 @@ def decode(input, errors='strict'): return codecs.utf_16_le_decode(input, errors, True) +class IncrementalEncoder(codecs.IncrementalEncoder): + def encode(self, input, final=False): + return codecs.utf_16_le_encode(input, self.errors)[0] + +class IncrementalDecoder(codecs.BufferedIncrementalDecoder): + _buffer_decode = codecs.utf_16_le_decode + class StreamWriter(codecs.StreamWriter): encode = codecs.utf_16_le_encode class StreamReader(codecs.StreamReader): decode = codecs.utf_16_le_decode - ### encodings module API def getregentry(): - - return (encode,decode,StreamReader,StreamWriter) + return codecs.CodecInfo( + name='utf-16-le', + encode=encode, + decode=decode, + incrementalencoder=IncrementalEncoder, + incrementaldecoder=IncrementalDecoder, + streamreader=StreamReader, + streamwriter=StreamWriter, + ) Modified: python/branches/p3yk/Lib/encodings/utf_7.py ============================================================================== --- python/branches/p3yk/Lib/encodings/utf_7.py (original) +++ python/branches/p3yk/Lib/encodings/utf_7.py Fri Apr 21 11:43:23 2006 @@ -13,6 +13,14 @@ encode = codecs.utf_7_encode decode = codecs.utf_7_decode +class IncrementalEncoder(codecs.IncrementalEncoder): + def encode(self, input, final=False): + return codecs.utf_7_encode(input, self.errors)[0] + +class IncrementalDecoder(codecs.BufferedIncrementalDecoder): + def _buffer_decode(self, input, errors, final): + return codecs.utf_7_decode(input, self.errors) + class StreamWriter(Codec,codecs.StreamWriter): pass @@ -22,5 +30,12 @@ ### encodings module API def getregentry(): - - return (Codec.encode,Codec.decode,StreamReader,StreamWriter) + return codecs.CodecInfo( + name='utf-7', + encode=Codec.encode, + decode=Codec.decode, + incrementalencoder=IncrementalEncoder, + incrementaldecoder=IncrementalDecoder, + streamreader=StreamReader, + streamwriter=StreamWriter, + ) Modified: python/branches/p3yk/Lib/encodings/utf_8.py ============================================================================== --- python/branches/p3yk/Lib/encodings/utf_8.py (original) +++ python/branches/p3yk/Lib/encodings/utf_8.py Fri Apr 21 11:43:23 2006 @@ -15,6 +15,13 @@ def decode(input, errors='strict'): return codecs.utf_8_decode(input, errors, True) +class IncrementalEncoder(codecs.IncrementalEncoder): + def encode(self, input, final=False): + return codecs.utf_8_encode(input, self.errors)[0] + +class IncrementalDecoder(codecs.BufferedIncrementalDecoder): + _buffer_decode = codecs.utf_8_decode + class StreamWriter(codecs.StreamWriter): encode = codecs.utf_8_encode @@ -24,5 +31,12 @@ ### encodings module API def getregentry(): - - return (encode,decode,StreamReader,StreamWriter) + return codecs.CodecInfo( + name='utf-8', + encode=encode, + decode=decode, + incrementalencoder=IncrementalEncoder, + incrementaldecoder=IncrementalDecoder, + streamreader=StreamReader, + streamwriter=StreamWriter, + ) Modified: python/branches/p3yk/Lib/encodings/utf_8_sig.py ============================================================================== --- python/branches/p3yk/Lib/encodings/utf_8_sig.py (original) +++ python/branches/p3yk/Lib/encodings/utf_8_sig.py Fri Apr 21 11:43:23 2006 @@ -22,6 +22,42 @@ (output, consumed) = codecs.utf_8_decode(input, errors, True) return (output, consumed+prefix) +class IncrementalEncoder(codecs.IncrementalEncoder): + def __init__(self, errors='strict'): + codecs.IncrementalEncoder.__init__(self, errors) + self.first = True + + def encode(self, input, final=False): + if self.first: + self.first = False + return codecs.BOM_UTF8 + codecs.utf_8_encode(input, errors)[0] + else: + return codecs.utf_8_encode(input, errors)[0] + + def reset(self): + codecs.IncrementalEncoder.reset(self) + self.first = True + +class IncrementalDecoder(codecs.BufferedIncrementalDecoder): + def __init__(self, errors='strict'): + codecs.BufferedIncrementalDecoder.__init__(self, errors) + self.first = True + + def _buffer_decode(self, input, errors, final): + if self.first and codecs.BOM_UTF8.startswith(input): # might be a BOM + if len(input) < 3: + # not enough data to decide if this really is a BOM + # => try again on the next call + return (u"", 0) + (output, consumed) = codecs.utf_8_decode(input[3:], errors, final) + self.first = False + return (output, consumed+3) + return codecs.utf_8_decode(input, errors, final) + + def reset(self): + codecs.BufferedIncrementalDecoder.reset(self) + self.first = True + class StreamWriter(codecs.StreamWriter): def reset(self): codecs.StreamWriter.reset(self) @@ -53,5 +89,12 @@ ### encodings module API def getregentry(): - - return (encode,decode,StreamReader,StreamWriter) + return codecs.CodecInfo( + name='utf-8-sig', + encode=encode, + decode=decode, + incrementalencoder=IncrementalEncoder, + incrementaldecoder=IncrementalDecoder, + streamreader=StreamReader, + streamwriter=StreamWriter, + ) Modified: python/branches/p3yk/Lib/encodings/uu_codec.py ============================================================================== --- python/branches/p3yk/Lib/encodings/uu_codec.py (original) +++ python/branches/p3yk/Lib/encodings/uu_codec.py Fri Apr 21 11:43:23 2006 @@ -96,9 +96,18 @@ def encode(self,input,errors='strict'): return uu_encode(input,errors) + def decode(self,input,errors='strict'): return uu_decode(input,errors) +class IncrementalEncoder(codecs.IncrementalEncoder): + def encode(self, input, final=False): + return uu_encode(input, errors)[0] + +class IncrementalDecoder(codecs.IncrementalDecoder): + def decode(self, input, final=False): + return uu_decode(input, errors)[0] + class StreamWriter(Codec,codecs.StreamWriter): pass @@ -108,5 +117,12 @@ ### encodings module API def getregentry(): - - return (uu_encode,uu_decode,StreamReader,StreamWriter) + return codecs.CodecInfo( + name='uu', + encode=uu_encode, + decode=uu_decode, + incrementalencoder=IncrementalEncoder, + incrementaldecoder=IncrementalDecoder, + streamreader=StreamReader, + streamwriter=StreamWriter, + ) Modified: python/branches/p3yk/Lib/encodings/zlib_codec.py ============================================================================== --- python/branches/p3yk/Lib/encodings/zlib_codec.py (original) +++ python/branches/p3yk/Lib/encodings/zlib_codec.py Fri Apr 21 11:43:23 2006 @@ -50,6 +50,16 @@ def decode(self, input, errors='strict'): return zlib_decode(input, errors) +class IncrementalEncoder(codecs.IncrementalEncoder): + def encode(self, input, final=False): + assert self.errors == 'strict' + return zlib.compress(input) + +class IncrementalDecoder(codecs.IncrementalDecoder): + def decode(self, input, final=False): + assert self.errors == 'strict' + return zlib.decompress(input) + class StreamWriter(Codec,codecs.StreamWriter): pass @@ -59,5 +69,12 @@ ### encodings module API def getregentry(): - - return (zlib_encode,zlib_decode,StreamReader,StreamWriter) + return codecs.CodecInfo( + name='zlib', + encode=zlib_encode, + decode=zlib_decode, + incrementalencoder=IncrementalEncoder, + incrementaldecoder=IncrementalDecoder, + streamreader=StreamReader, + streamwriter=StreamWriter, + ) Modified: python/branches/p3yk/Lib/logging/__init__.py ============================================================================== --- python/branches/p3yk/Lib/logging/__init__.py (original) +++ python/branches/p3yk/Lib/logging/__init__.py Fri Apr 21 11:43:23 2006 @@ -1058,13 +1058,16 @@ file name, line number and function name. """ f = currentframe().f_back - while 1: + rv = "(unknown file)", 0, "(unknown function)" + while hasattr(f, "f_code"): co = f.f_code filename = os.path.normcase(co.co_filename) if filename == _srcfile: f = f.f_back continue - return filename, f.f_lineno, co.co_name + rv = (filename, f.f_lineno, co.co_name) + break + return rv def makeRecord(self, name, level, fn, lno, msg, args, exc_info, func=None, extra=None): """ Modified: python/branches/p3yk/Lib/test/regrtest.py ============================================================================== --- python/branches/p3yk/Lib/test/regrtest.py (original) +++ python/branches/p3yk/Lib/test/regrtest.py Fri Apr 21 11:43:23 2006 @@ -1136,9 +1136,6 @@ s = _expectations[sys.platform] self.expected = set(s.split()) - # this isn't a regularly run unit test, it is always skipped - self.expected.add('test_hashlib_speed') - if not os.path.supports_unicode_filenames: self.expected.add('test_pep277') Modified: python/branches/p3yk/Lib/test/test_StringIO.py ============================================================================== --- python/branches/p3yk/Lib/test/test_StringIO.py (original) +++ python/branches/p3yk/Lib/test/test_StringIO.py Fri Apr 21 11:43:23 2006 @@ -75,6 +75,13 @@ f.close() self.assertEqual(f.closed, True) + def test_isatty(self): + f = self.MODULE.StringIO() + self.assertRaises(TypeError, f.isatty, None) + self.assertEqual(f.isatty(), False) + f.close() + self.assertRaises(ValueError, f.isatty) + def test_iterator(self): eq = self.assertEqual unless = self.failUnless @@ -87,6 +94,8 @@ eq(line, self._line + '\n') i += 1 eq(i, 5) + self._fp.close() + self.assertRaises(ValueError, self._fp.next) class TestStringIO(TestGenericStringIO): MODULE = StringIO Modified: python/branches/p3yk/Lib/test/test___all__.py ============================================================================== --- python/branches/p3yk/Lib/test/test___all__.py (original) +++ python/branches/p3yk/Lib/test/test___all__.py Fri Apr 21 11:43:23 2006 @@ -5,8 +5,6 @@ import sys import warnings -warnings.filterwarnings("ignore", ".* 'pre' .*", DeprecationWarning, - r'pre$') warnings.filterwarnings("ignore", ".* regsub .*", DeprecationWarning, r'^regsub$') warnings.filterwarnings("ignore", @@ -122,7 +120,6 @@ self.check_all("poplib") self.check_all("posixpath") self.check_all("pprint") - self.check_all("pre") # deprecated self.check_all("profile") self.check_all("pstats") self.check_all("pty") Modified: python/branches/p3yk/Lib/test/test_codecs.py ============================================================================== --- python/branches/p3yk/Lib/test/test_codecs.py (original) +++ python/branches/p3yk/Lib/test/test_codecs.py Fri Apr 21 11:43:23 2006 @@ -41,6 +41,33 @@ self.assertEqual(r.bytebuffer, "") self.assertEqual(r.charbuffer, u"") + # do the check again, this time using a incremental decoder + d = codecs.getincrementaldecoder(self.encoding)() + result = u"" + for (c, partialresult) in zip(input.encode(self.encoding), partialresults): + result += d.decode(c) + self.assertEqual(result, partialresult) + # check that there's nothing left in the buffers + self.assertEqual(d.decode("", True), u"") + self.assertEqual(d.buffer, "") + + # Check whether the rest method works properly + d.reset() + result = u"" + for (c, partialresult) in zip(input.encode(self.encoding), partialresults): + result += d.decode(c) + self.assertEqual(result, partialresult) + # check that there's nothing left in the buffers + self.assertEqual(d.decode("", True), u"") + self.assertEqual(d.buffer, "") + + # check iterdecode() + encoded = input.encode(self.encoding) + self.assertEqual( + input, + u"".join(codecs.iterdecode(encoded, self.encoding)) + ) + def test_readline(self): def getreader(input): stream = StringIO.StringIO(input.encode(self.encoding)) @@ -977,6 +1004,12 @@ def test_basics(self): s = u"abc123" # all codecs should be able to encode these for encoding in all_unicode_encodings: + name = codecs.lookup(encoding).name + if encoding.endswith("_codec"): + name += "_codec" + elif encoding == "latin_1": + name = "latin_1" + self.assertEqual(encoding.replace("_", "-"), name.replace("_", "-")) (bytes, size) = codecs.getencoder(encoding)(s) if encoding != "unicode_internal": self.assertEqual(size, len(s), "%r != %r (encoding=%r)" % (size, len(s), encoding)) @@ -999,6 +1032,30 @@ decodedresult += reader.read() self.assertEqual(decodedresult, s, "%r != %r (encoding=%r)" % (decodedresult, s, encoding)) + # check incremental decoder/encoder and iterencode()/iterdecode() + try: + encoder = codecs.getincrementalencoder(encoding)() + except LookupError: # no IncrementalEncoder + pass + else: + # check incremental decoder/encoder + encodedresult = "" + for c in s: + encodedresult += encoder.encode(c) + decoder = codecs.getincrementaldecoder(encoding)() + decodedresult = u"" + for c in encodedresult: + decodedresult += decoder.decode(c) + self.assertEqual(decodedresult, s, "%r != %r (encoding=%r)" % (decodedresult, s, encoding)) + + # check iterencode()/iterdecode() + result = u"".join(codecs.iterdecode(codecs.iterencode(s, encoding), encoding)) + self.assertEqual(result, s, "%r != %r (encoding=%r)" % (result, s, encoding)) + + # check iterencode()/iterdecode() with empty string + result = u"".join(codecs.iterdecode(codecs.iterencode(u"", encoding), encoding)) + self.assertEqual(result, u"") + def test_seek(self): # all codecs should be able to encode these s = u"%s\n%s\n" % (100*u"abc123", 100*u"def456") Deleted: /python/branches/p3yk/Lib/test/test_hashlib_speed.py ============================================================================== --- /python/branches/p3yk/Lib/test/test_hashlib_speed.py Fri Apr 21 11:43:23 2006 +++ (empty file) @@ -1,92 +0,0 @@ - -import sys, time -import hashlib -from test import test_support - - -def creatorFunc(): - raise RuntimeError, "eek, creatorFunc not overridden" - - -def test_scaled_msg(scale, name): - - iterations = 106201/scale * 20 - longStr = 'Z'*scale - - localCF = creatorFunc - start = time.time() - for f in xrange(iterations): - x = localCF(longStr).digest() - end = time.time() - - print ('%2.2f' % (end-start)), "seconds", iterations, "x", len(longStr), "bytes", name - -def test_create(): - start = time.time() - for f in xrange(20000): - d = creatorFunc() - end = time.time() - - print ('%2.2f' % (end-start)), "seconds", '[20000 creations]' - -def test_zero(): - start = time.time() - for f in xrange(20000): - x = creatorFunc().digest() - end = time.time() - - print ('%2.2f' % (end-start)), "seconds", '[20000 "" digests]' - - - -### this 'test' is not normally run. skip it if the test runner finds it -if __name__ != '__main__': - raise test_support.TestSkipped, "not a unit test (stand alone benchmark)" - -hName = sys.argv[1] - -# -# setup our creatorFunc to test the requested hash -# -if hName in ('_md5', '_sha'): - exec 'import '+hName - exec 'creatorFunc = '+hName+'.new' - print "testing speed of old", hName, "legacy interface" -elif hName == '_hashlib' and len(sys.argv) > 3: - import _hashlib - exec 'creatorFunc = _hashlib.%s' % sys.argv[2] - print "testing speed of _hashlib.%s" % sys.argv[2], getattr(_hashlib, sys.argv[2]) -elif hName == '_hashlib' and len(sys.argv) == 3: - import _hashlib - exec 'creatorFunc = lambda x=_hashlib.new : x(%r)' % sys.argv[2] - print "testing speed of _hashlib.new(%r)" % sys.argv[2] -elif hasattr(hashlib, hName) and callable(getattr(hashlib, hName)): - creatorFunc = getattr(hashlib, hName) - print "testing speed of hashlib."+hName, getattr(hashlib, hName) -else: - exec "creatorFunc = lambda x=hashlib.new : x(%r)" % hName - print "testing speed of hashlib.new(%r)" % hName - -try: - test_create() -except ValueError: - print - print "pass argument(s) naming the hash to run a speed test on:" - print " '_md5' and '_sha' test the legacy builtin md5 and sha" - print " '_hashlib' 'openssl_hName' 'fast' tests the builtin _hashlib" - print " '_hashlib' 'hName' tests builtin _hashlib.new(shaFOO)" - print " 'hName' tests the hashlib.hName() implementation if it exists" - print " otherwise it uses hashlib.new(hName)." - print - raise - -test_zero() -test_scaled_msg(scale=106201, name='[huge data]') -test_scaled_msg(scale=10620, name='[large data]') -test_scaled_msg(scale=1062, name='[medium data]') -test_scaled_msg(scale=424, name='[4*small data]') -test_scaled_msg(scale=336, name='[3*small data]') -test_scaled_msg(scale=212, name='[2*small data]') -test_scaled_msg(scale=106, name='[small data]') -test_scaled_msg(scale=creatorFunc().digest_size, name='[digest_size data]') -test_scaled_msg(scale=10, name='[tiny data]') Modified: python/branches/p3yk/Makefile.pre.in ============================================================================== --- python/branches/p3yk/Makefile.pre.in (original) +++ python/branches/p3yk/Makefile.pre.in Fri Apr 21 11:43:23 2006 @@ -974,6 +974,8 @@ find . -name '*.o' -exec rm -f {} ';' find . -name '*.s[ol]' -exec rm -f {} ';' find $(srcdir) -name '*.py[co]' -exec rm -f {} ';' + find $(srcdir) -name 'fficonfig.h' -exec rm -f {} ';' + find $(srcdir) -name 'fficonfig.py' -exec rm -f {} ';' clobber: clean -rm -f $(BUILDPYTHON) $(PGEN) $(LIBRARY) $(LDLIBRARY) $(DLLLIBRARY) \ Modified: python/branches/p3yk/Modules/_ctypes/_ctypes.c ============================================================================== --- python/branches/p3yk/Modules/_ctypes/_ctypes.c (original) +++ python/branches/p3yk/Modules/_ctypes/_ctypes.c Fri Apr 21 11:43:23 2006 @@ -3674,7 +3674,11 @@ if (cache == NULL) return NULL; } +#if (PY_VERSION_HEX < 0x02050000) + key = Py_BuildValue("(Oi)", itemtype, length); +#else key = Py_BuildValue("(On)", itemtype, length); +#endif if (!key) return NULL; result = PyDict_GetItem(cache, key); @@ -3698,7 +3702,11 @@ #endif result = PyObject_CallFunction((PyObject *)&ArrayType_Type, +#if (PY_VERSION_HEX < 0x02050000) + "s(O){s:i,s:O}", +#else "s(O){s:n,s:O}", +#endif name, &Array_Type, "_length_", Modified: python/branches/p3yk/Modules/_ctypes/cfield.c ============================================================================== --- python/branches/p3yk/Modules/_ctypes/cfield.c (original) +++ python/branches/p3yk/Modules/_ctypes/cfield.c Fri Apr 21 11:43:23 2006 @@ -250,11 +250,21 @@ name = ((PyTypeObject *)self->proto)->tp_name; if (bits) - result = PyString_FromFormat("", - name, (int)self->offset, size, bits); + result = PyString_FromFormat( +#if (PY_VERSION_HEX < 0x02050000) + "", +#else + "", +#endif + name, self->offset, size, bits); else - result = PyString_FromFormat("", - name, (int)self->offset, size); + result = PyString_FromFormat( +#if (PY_VERSION_HEX < 0x02050000) + "", +#else + "", +#endif + name, self->offset, size); return result; } Modified: python/branches/p3yk/Modules/_ctypes/ctypes.h ============================================================================== --- python/branches/p3yk/Modules/_ctypes/ctypes.h (original) +++ python/branches/p3yk/Modules/_ctypes/ctypes.h Fri Apr 21 11:43:23 2006 @@ -1,5 +1,18 @@ /******************************************************************/ +#if (PY_VERSION_HEX < 0x02050000) +typedef int Py_ssize_t; +#define lenfunc inquiry +#define readbufferproc getreadbufferproc +#define writebufferproc getwritebufferproc +#define segcountproc getsegcountproc +#define charbufferproc getcharbufferproc +#define ssizeargfunc intargfunc +#define ssizessizeargfunc intintargfunc +#define ssizeobjargproc intobjargproc +#define ssizessizeobjargproc intintobjargproc +#endif + #ifndef MS_WIN32 #define max(a, b) ((a) > (b) ? (a) : (b)) #define min(a, b) ((a) < (b) ? (a) : (b)) Modified: python/branches/p3yk/Modules/_testcapimodule.c ============================================================================== --- python/branches/p3yk/Modules/_testcapimodule.c (original) +++ python/branches/p3yk/Modules/_testcapimodule.c Fri Apr 21 11:43:23 2006 @@ -522,6 +522,18 @@ return Py_None; } +/* Example passing NULLs to PyObject_Str(NULL) and PyObject_Unicode(NULL). */ + +static PyObject * +test_null_strings(PyObject *self) +{ + PyObject *o1 = PyObject_Str(NULL), *o2 = PyObject_Unicode(NULL); + PyObject *tuple = PyTuple_Pack(2, o1, o2); + Py_XDECREF(o1); + Py_XDECREF(o2); + return tuple; +} + static PyObject * raise_exception(PyObject *self, PyObject *args) { @@ -597,6 +609,7 @@ {"test_long_api", (PyCFunction)test_long_api, METH_NOARGS}, {"test_long_numbits", (PyCFunction)test_long_numbits, METH_NOARGS}, {"test_k_code", (PyCFunction)test_k_code, METH_NOARGS}, + {"test_null_strings", (PyCFunction)test_null_strings, METH_NOARGS}, {"getargs_b", (PyCFunction)getargs_b, METH_VARARGS}, {"getargs_B", (PyCFunction)getargs_B, METH_VARARGS}, Modified: python/branches/p3yk/Modules/cStringIO.c ============================================================================== --- python/branches/p3yk/Modules/cStringIO.c (original) +++ python/branches/p3yk/Modules/cStringIO.c Fri Apr 21 11:43:23 2006 @@ -144,7 +144,8 @@ static PyObject * IO_isatty(IOobject *self, PyObject *unused) { - Py_INCREF(Py_False); + if (!IO__opencheck(self)) return NULL; + Py_INCREF(Py_False); return Py_False; } Modified: python/branches/p3yk/Modules/main.c ============================================================================== --- python/branches/p3yk/Modules/main.c (original) +++ python/branches/p3yk/Modules/main.c Fri Apr 21 11:43:23 2006 @@ -130,27 +130,42 @@ } } -/* Get the path to a top-level module */ -static struct filedescr * FindModule(const char *module, - FILE **fp, char **filename) -{ - struct filedescr *fdescr = NULL; - *fp = NULL; - *filename = malloc(MAXPATHLEN); - - if (*filename == NULL) - return NULL; - /* Find the actual module source code */ - fdescr = _PyImport_FindModule(module, NULL, - *filename, MAXPATHLEN, fp, NULL); - - if (fdescr == NULL) { - free(*filename); - *filename = NULL; +static int RunModule(char *module) +{ + PyObject *runpy, *runmodule, *runargs, *result; + runpy = PyImport_ImportModule("runpy"); + if (runpy == NULL) { + fprintf(stderr, "Could not import runpy module\n"); + return -1; + } + runmodule = PyObject_GetAttrString(runpy, "run_module"); + if (runmodule == NULL) { + fprintf(stderr, "Could not access runpy.run_module\n"); + Py_DECREF(runpy); + return -1; + } + runargs = Py_BuildValue("sOsO", module, + Py_None, "__main__", Py_True); + if (runargs == NULL) { + fprintf(stderr, + "Could not create arguments for runpy.run_module\n"); + Py_DECREF(runpy); + Py_DECREF(runmodule); + return -1; + } + result = PyObject_Call(runmodule, runargs, NULL); + if (result == NULL) { + PyErr_Print(); + } + Py_DECREF(runpy); + Py_DECREF(runmodule); + Py_DECREF(runargs); + if (result == NULL) { + return -1; } - - return fdescr; + Py_DECREF(result); + return 0; } /* Main program */ @@ -410,28 +425,9 @@ } if (module != NULL) { - /* Backup _PyOS_optind and find the real file */ - struct filedescr *fdescr = NULL; + /* Backup _PyOS_optind and force sys.arv[0] = module */ _PyOS_optind--; - if ((fdescr = FindModule(module, &fp, &filename))) { - argv[_PyOS_optind] = filename; - } else { - fprintf(stderr, "%s: module %s not found\n", - argv[0], module); - return 2; - } - if (!fp) { - fprintf(stderr, - "%s: module %s has no associated file\n", - argv[0], module); - return 2; - } - if (!_PyImport_IsScript(fdescr)) { - fprintf(stderr, - "%s: module %s not usable as script\n (%s)\n", - argv[0], module, filename); - return 2; - } + argv[_PyOS_optind] = module; } PySys_SetArgv(argc-_PyOS_optind, argv+_PyOS_optind); @@ -450,9 +446,8 @@ sts = PyRun_SimpleStringFlags(command, &cf) != 0; free(command); } else if (module) { - sts = PyRun_AnyFileExFlags(fp, filename, 1, &cf) != 0; + sts = RunModule(module); free(module); - free(filename); } else { if (filename == NULL && stdin_is_interactive) { Modified: python/branches/p3yk/Modules/xxmodule.c ============================================================================== --- python/branches/p3yk/Modules/xxmodule.c (original) +++ python/branches/p3yk/Modules/xxmodule.c Fri Apr 21 11:43:23 2006 @@ -197,18 +197,6 @@ return Py_None; } -/* Example passing NULLs to PyObject_Str(NULL) and PyObject_Unicode(NULL). */ - -static PyObject * -xx_null(PyObject *self, PyObject *noargs) -{ - PyObject *o1 = PyObject_Str(NULL), *o2 = PyObject_Unicode(NULL); - PyObject *tuple = PyTuple_Pack(2, o1, o2); - Py_XDECREF(o1); - Py_XDECREF(o2); - return tuple; -} - /* Test bad format character */ static PyObject * @@ -343,8 +331,6 @@ PyDoc_STR("new() -> new Xx object")}, {"bug", xx_bug, METH_VARARGS, PyDoc_STR("bug(o) -> None")}, - {"null", xx_null, METH_NOARGS, - PyDoc_STR("null(o) -> ('NULL', u'NULL')")}, {NULL, NULL} /* sentinel */ }; Modified: python/branches/p3yk/Objects/obmalloc.c ============================================================================== --- python/branches/p3yk/Objects/obmalloc.c (original) +++ python/branches/p3yk/Objects/obmalloc.c Fri Apr 21 11:43:23 2006 @@ -217,16 +217,16 @@ * I don't care if these are defined in or elsewhere. Axiom. */ #undef uchar -#define uchar unsigned char /* assuming == 8 bits */ +#define uchar unsigned char /* assuming == 8 bits */ #undef uint -#define uint unsigned int /* assuming >= 16 bits */ +#define uint unsigned int /* assuming >= 16 bits */ #undef ulong -#define ulong unsigned long /* assuming >= 32 bits */ +#define ulong unsigned long /* assuming >= 32 bits */ #undef uptr -#define uptr Py_uintptr_t +#define uptr Py_uintptr_t /* When you say memory, my mind reasons in terms of (pointers to) blocks */ typedef uchar block; @@ -246,6 +246,47 @@ typedef struct pool_header *poolp; +/* Record keeping for arenas. */ +struct arena_object { + /* The address of the arena, as returned by malloc. Note that 0 + * will never be returned by a successful malloc, and is used + * here to mark an arena_object that doesn't correspond to an + * allocated arena. + */ + uptr address; + + /* Pool-aligned pointer to the next pool to be carved off. */ + block* pool_address; + + /* The number of available pools in the arena: free pools + never- + * allocated pools. + */ + uint nfreepools; + + /* The total number of pools in the arena, whether or not available. */ + uint ntotalpools; + + /* Singly-linked list of available pools. */ + struct pool_header* freepools; + + /* Whenever this arena_object is not associated with an allocated + * arena, the nextarena member is used to link all unassociated + * arena_objects in the singly-linked `unused_arena_objects` list. + * The prevarena member is unused in this case. + * + * When this arena_object is associated with an allocated arena + * with at least one available pool, both members are used in the + * doubly-linked `usable_arenas` list, which is maintained in + * increasing order of `nfreepools` values. + * + * Else this arena_object is associated with an allocated arena + * all of whose pools are in use. `nextarena` and `prevarena` + * are both meaningless in this case. + */ + struct arena_object* nextarena; + struct arena_object* prevarena; +}; + #undef ROUNDUP #define ROUNDUP(x) (((x) + ALIGNMENT_MASK) & ~ALIGNMENT_MASK) #define POOL_OVERHEAD ROUNDUP(sizeof(struct pool_header)) @@ -277,8 +318,9 @@ usedpools[0] corresponds to blocks of size 8, usedpools[2] to blocks of size 16, and so on: index 2*i <-> blocks of size (i+1)< 8 */ }; -/* - * Free (cached) pools +/*========================================================================== +Arena management. + +`arenas` is a vector of arena_objects. It contains maxarenas entries, some of +which may not be currently used (== they're arena_objects that aren't +currently associated with an allocated arena). Note that arenas proper are +separately malloc'ed. + +Prior to Python 2.5, arenas were never free()'ed. Starting with Python 2.5, +we do try to free() arenas, and use some mild heuristic strategies to increase +the likelihood that arenas eventually can be freed. + +unused_arena_objects + + This is a singly-linked list of the arena_objects that are currently not + being used (no arena is associated with them). Objects are taken off the + head of the list in new_arena(), and are pushed on the head of the list in + PyObject_Free() when the arena is empty. Key invariant: an arena_object + is on this list if and only if its .address member is 0. + +usable_arenas + + This is a doubly-linked list of the arena_objects associated with arenas + that have pools available. These pools are either waiting to be reused, + or have not been used before. The list is sorted to have the most- + allocated arenas first (ascending order based on the nfreepools member). + This means that the next allocation will come from a heavily used arena, + which gives the nearly empty arenas a chance to be returned to the system. + In my unscientific tests this dramatically improved the number of arenas + that could be freed. + +Note that an arena_object associated with an arena all of whose pools are +currently in use isn't on either list. +*/ + +/* Array of objects used to track chunks of memory (arenas). */ +static struct arena_object* arenas = NULL; +/* Number of slots currently allocated in the `arenas` vector. */ +static uint maxarenas = 0; + +/* The head of the singly-linked, NULL-terminated list of available + * arena_objects. */ -static poolp freepools = NULL; /* free list for cached pools */ +static struct arena_object* unused_arena_objects = NULL; -/*==========================================================================*/ -/* Arena management. */ +/* The head of the doubly-linked, NULL-terminated at each end, list of + * arena_objects associated with arenas that have pools available. + */ +static struct arena_object* usable_arenas = NULL; -/* arenas is a vector of arena base addresses, in order of allocation time. - * arenas currently contains narenas entries, and has space allocated - * for at most maxarenas entries. - * - * CAUTION: See the long comment block about thread safety in new_arena(): - * the code currently relies in deep ways on that this vector only grows, - * and only grows by appending at the end. For now we never return an arena - * to the OS. +/* How many arena_objects do we initially allocate? + * 16 = can allocate 16 arenas = 16 * ARENA_SIZE = 4MB before growing the + * `arenas` vector. */ -static uptr *volatile arenas = NULL; /* the pointer itself is volatile */ -static volatile uint narenas = 0; -static uint maxarenas = 0; +#define INITIAL_ARENA_OBJECTS 16 -/* Number of pools still available to be allocated in the current arena. */ -static uint nfreepools = 0; +/* Number of arenas allocated that haven't been free()'d. */ +static ulong narenas_currently_allocated = 0; -/* Free space start address in current arena. This is pool-aligned. */ -static block *arenabase = NULL; +#ifdef PYMALLOC_DEBUG +/* Total number of times malloc() called to allocate an arena. */ +static ulong ntimes_arena_allocated = 0; +/* High water mark (max value ever seen) for narenas_currently_allocated. */ +static ulong narenas_highwater = 0; +#endif -/* Allocate a new arena and return its base address. If we run out of - * memory, return NULL. +/* Allocate a new arena. If we run out of memory, return NULL. Else + * allocate a new arena, and return the address of an arena_object + * describing the new arena. It's expected that the caller will set + * `usable_arenas` to the return value. */ -static block * +static struct arena_object* new_arena(void) { + struct arena_object* arenaobj; uint excess; /* number of bytes above pool alignment */ - block *bp = (block *)malloc(ARENA_SIZE); - if (bp == NULL) - return NULL; #ifdef PYMALLOC_DEBUG if (Py_GETENV("PYTHONMALLOCSTATS")) _PyObject_DebugMallocStats(); #endif + if (unused_arena_objects == NULL) { + uint i; + uint numarenas; + size_t nbytes; - /* arenabase <- first pool-aligned address in the arena - nfreepools <- number of whole pools that fit after alignment */ - arenabase = bp; - nfreepools = ARENA_SIZE / POOL_SIZE; - assert(POOL_SIZE * nfreepools == ARENA_SIZE); - excess = (uint) ((Py_uintptr_t)bp & POOL_SIZE_MASK); - if (excess != 0) { - --nfreepools; - arenabase += POOL_SIZE - excess; - } + /* Double the number of arena objects on each allocation. + * Note that it's possible for `numarenas` to overflow. + */ + numarenas = maxarenas ? maxarenas << 1 : INITIAL_ARENA_OBJECTS; + if (numarenas <= maxarenas) + return NULL; /* overflow */ + nbytes = numarenas * sizeof(*arenas); + if (nbytes / sizeof(*arenas) != numarenas) + return NULL; /* overflow */ + arenaobj = realloc(arenas, nbytes); + if (arenaobj == NULL) + return NULL; + arenas = arenaobj; + + /* We might need to fix pointers that were copied. However, + * new_arena only gets called when all the pages in the + * previous arenas are full. Thus, there are *no* pointers + * into the old array. Thus, we don't have to worry about + * invalid pointers. Just to be sure, some asserts: + */ + assert(usable_arenas == NULL); + assert(unused_arena_objects == NULL); - /* Make room for a new entry in the arenas vector. */ - if (arenas == NULL) { - assert(narenas == 0 && maxarenas == 0); - arenas = (uptr *)malloc(16 * sizeof(*arenas)); - if (arenas == NULL) - goto error; - maxarenas = 16; + /* Put the new arenas on the unused_arena_objects list. */ + for (i = maxarenas; i < numarenas; ++i) { + arenas[i].address = 0; /* mark as unassociated */ + arenas[i].nextarena = i < numarenas - 1 ? + &arenas[i+1] : NULL; + } + + /* Update globals. */ + unused_arena_objects = &arenas[maxarenas]; + maxarenas = numarenas; } - else if (narenas == maxarenas) { - /* Grow arenas. - * - * Exceedingly subtle: Someone may be calling the pymalloc - * free via PyMem_{DEL, Del, FREE, Free} without holding the - *.GIL. Someone else may simultaneously be calling the - * pymalloc malloc while holding the GIL via, e.g., - * PyObject_New. Now the pymalloc free may index into arenas - * for an address check, while the pymalloc malloc calls - * new_arena and we end up here to grow a new arena *and* - * grow the arenas vector. If the value for arenas pymalloc - * free picks up "vanishes" during this resize, anything may - * happen, and it would be an incredibly rare bug. Therefore - * the code here takes great pains to make sure that, at every - * moment, arenas always points to an intact vector of - * addresses. It doesn't matter whether arenas points to a - * wholly up-to-date vector when pymalloc free checks it in - * this case, because the only legal (and that even this is - * legal is debatable) way to call PyMem_{Del, etc} while not - * holding the GIL is if the memory being released is not - * object memory, i.e. if the address check in pymalloc free - * is supposed to fail. Having an incomplete vector can't - * make a supposed-to-fail case succeed by mistake (it could - * only make a supposed-to-succeed case fail by mistake). - * - * In addition, without a lock we can't know for sure when - * an old vector is no longer referenced, so we simply let - * old vectors leak. - * - * And on top of that, since narenas and arenas can't be - * changed as-a-pair atomically without a lock, we're also - * careful to declare them volatile and ensure that we change - * arenas first. This prevents another thread from picking - * up an narenas value too large for the arenas value it - * reads up (arenas never shrinks). - * - * Read the above 50 times before changing anything in this - * block. + + /* Take the next available arena object off the head of the list. */ + assert(unused_arena_objects != NULL); + arenaobj = unused_arena_objects; + unused_arena_objects = arenaobj->nextarena; + assert(arenaobj->address == 0); + arenaobj->address = (uptr)malloc(ARENA_SIZE); + if (arenaobj->address == 0) { + /* The allocation failed: return NULL after putting the + * arenaobj back. */ - uptr *p; - uint newmax = maxarenas << 1; - if (newmax <= maxarenas) /* overflow */ - goto error; - p = (uptr *)malloc(newmax * sizeof(*arenas)); - if (p == NULL) - goto error; - memcpy(p, arenas, narenas * sizeof(*arenas)); - arenas = p; /* old arenas deliberately leaked */ - maxarenas = newmax; + arenaobj->nextarena = unused_arena_objects; + unused_arena_objects = arenaobj; + return NULL; } - /* Append the new arena address to arenas. */ - assert(narenas < maxarenas); - arenas[narenas] = (uptr)bp; - ++narenas; /* can't overflow, since narenas < maxarenas before */ - return bp; + ++narenas_currently_allocated; +#ifdef PYMALLOC_DEBUG + ++ntimes_arena_allocated; + if (narenas_currently_allocated > narenas_highwater) + narenas_highwater = narenas_currently_allocated; +#endif + arenaobj->freepools = NULL; + /* pool_address <- first pool-aligned address in the arena + nfreepools <- number of whole pools that fit after alignment */ + arenaobj->pool_address = (block*)arenaobj->address; + arenaobj->nfreepools = ARENA_SIZE / POOL_SIZE; + assert(POOL_SIZE * arenaobj->nfreepools == ARENA_SIZE); + excess = (uint)(arenaobj->address & POOL_SIZE_MASK); + if (excess != 0) { + --arenaobj->nfreepools; + arenaobj->pool_address += POOL_SIZE - excess; + } + arenaobj->ntotalpools = arenaobj->nfreepools; -error: - free(bp); - nfreepools = 0; - return NULL; + return arenaobj; } -/* Return true if and only if P is an address that was allocated by - * pymalloc. I must be the index into arenas that the address claims - * to come from. - * - * Tricky: Letting B be the arena base address in arenas[I], P belongs to the - * arena if and only if - * B <= P < B + ARENA_SIZE - * Subtracting B throughout, this is true iff - * 0 <= P-B < ARENA_SIZE - * By using unsigned arithmetic, the "0 <=" half of the test can be skipped. - * - * Obscure: A PyMem "free memory" function can call the pymalloc free or - * realloc before the first arena has been allocated. arenas is still - * NULL in that case. We're relying on that narenas is also 0 in that case, - * so the (I) < narenas must be false, saving us from trying to index into - * a NULL arenas. - */ -#define Py_ADDRESS_IN_RANGE(P, POOL) \ - ((POOL)->arenaindex < narenas && \ - (uptr)(P) - arenas[(POOL)->arenaindex] < (uptr)ARENA_SIZE) +/* +Py_ADDRESS_IN_RANGE(P, POOL) + +Return true if and only if P is an address that was allocated by pymalloc. +POOL must be the pool address associated with P, i.e., POOL = POOL_ADDR(P) +(the caller is asked to compute this because the macro expands POOL more than +once, and for efficiency it's best for the caller to assign POOL_ADDR(P) to a +variable and pass the latter to the macro; because Py_ADDRESS_IN_RANGE is +called on every alloc/realloc/free, micro-efficiency is important here). + +Tricky: Let B be the arena base address associated with the pool, B = +arenas[(POOL)->arenaindex].address. Then P belongs to the arena if and only if + + B <= P < B + ARENA_SIZE + +Subtracting B throughout, this is true iff + + 0 <= P-B < ARENA_SIZE + +By using unsigned arithmetic, the "0 <=" half of the test can be skipped. + +Obscure: A PyMem "free memory" function can call the pymalloc free or realloc +before the first arena has been allocated. `arenas` is still NULL in that +case. We're relying on that maxarenas is also 0 in that case, so that +(POOL)->arenaindex < maxarenas must be false, saving us from trying to index +into a NULL arenas. + +Details: given P and POOL, the arena_object corresponding to P is AO = +arenas[(POOL)->arenaindex]. Suppose obmalloc controls P. Then (barring wild +stores, etc), POOL is the correct address of P's pool, AO.address is the +correct base address of the pool's arena, and P must be within ARENA_SIZE of +AO.address. In addition, AO.address is not 0 (no arena can start at address 0 +(NULL)). Therefore Py_ADDRESS_IN_RANGE correctly reports that obmalloc +controls P. + +Now suppose obmalloc does not control P (e.g., P was obtained via a direct +call to the system malloc() or realloc()). (POOL)->arenaindex may be anything +in this case -- it may even be uninitialized trash. If the trash arenaindex +is >= maxarenas, the macro correctly concludes at once that obmalloc doesn't +control P. + +Else arenaindex is < maxarena, and AO is read up. If AO corresponds to an +allocated arena, obmalloc controls all the memory in slice AO.address : +AO.address+ARENA_SIZE. By case assumption, P is not controlled by obmalloc, +so P doesn't lie in that slice, so the macro correctly reports that P is not +controlled by obmalloc. + +Finally, if P is not controlled by obmalloc and AO corresponds to an unused +arena_object (one not currently associated with an allocated arena), +AO.address is 0, and the second test in the macro reduces to: + + P < ARENA_SIZE + +If P >= ARENA_SIZE (extremely likely), the macro again correctly concludes +that P is not controlled by obmalloc. However, if P < ARENA_SIZE, this part +of the test still passes, and the third clause (AO.address != 0) is necessary +to get the correct result: AO.address is 0 in this case, so the macro +correctly reports that P is not controlled by obmalloc (despite that P lies in +slice AO.address : AO.address + ARENA_SIZE). + +Note: The third (AO.address != 0) clause was added in Python 2.5. Before +2.5, arenas were never free()'ed, and an arenaindex < maxarena always +corresponded to a currently-allocated arena, so the "P is not controlled by +obmalloc, AO corresponds to an unused arena_object, and P < ARENA_SIZE" case +was impossible. + +Note that the logic is excruciating, and reading up possibly uninitialized +memory when P is not controlled by obmalloc (to get at (POOL)->arenaindex) +creates problems for some memory debuggers. The overwhelming advantage is +that this test determines whether an arbitrary address is controlled by +obmalloc in a small constant time, independent of the number of arenas +obmalloc controls. Since this test is needed at every entry point, it's +extremely desirable that it be this fast. +*/ +#define Py_ADDRESS_IN_RANGE(P, POOL) \ + ((POOL)->arenaindex < maxarenas && \ + (uptr)(P) - arenas[(POOL)->arenaindex].address < (uptr)ARENA_SIZE && \ + arenas[(POOL)->arenaindex].address != 0) + /* This is only useful when running memory debuggers such as * Purify or Valgrind. Uncomment to use. @@ -599,7 +733,7 @@ /* * Most frequent paths first */ - size = (uint )(nbytes - 1) >> ALIGNMENT_SHIFT; + size = (uint)(nbytes - 1) >> ALIGNMENT_SHIFT; pool = usedpools[size + size]; if (pool != pool->nextpool) { /* @@ -614,22 +748,18 @@ return (void *)bp; } /* - * Reached the end of the free list, try to extend it + * Reached the end of the free list, try to extend it. */ if (pool->nextoffset <= pool->maxnextoffset) { - /* - * There is room for another block - */ - pool->freeblock = (block *)pool + + /* There is room for another block. */ + pool->freeblock = (block*)pool + pool->nextoffset; pool->nextoffset += INDEX2SIZE(size); *(block **)(pool->freeblock) = NULL; UNLOCK(); return (void *)bp; } - /* - * Pool is full, unlink from used pools - */ + /* Pool is full, unlink from used pools. */ next = pool->nextpool; pool = pool->prevpool; next->prevpool = pool; @@ -637,19 +767,68 @@ UNLOCK(); return (void *)bp; } - /* - * Try to get a cached free pool + + /* There isn't a pool of the right size class immediately + * available: use a free pool. */ - pool = freepools; + if (usable_arenas == NULL) { + /* No arena has a free pool: allocate a new arena. */ +#ifdef WITH_MEMORY_LIMITS + if (narenas_currently_allocated >= MAX_ARENAS) { + UNLOCK(); + goto redirect; + } +#endif + usable_arenas = new_arena(); + if (usable_arenas == NULL) { + UNLOCK(); + goto redirect; + } + usable_arenas->nextarena = + usable_arenas->prevarena = NULL; + } + assert(usable_arenas->address != 0); + + /* Try to get a cached free pool. */ + pool = usable_arenas->freepools; if (pool != NULL) { - /* - * Unlink from cached pools + /* Unlink from cached pools. */ + usable_arenas->freepools = pool->nextpool; + + /* This arena already had the smallest nfreepools + * value, so decreasing nfreepools doesn't change + * that, and we don't need to rearrange the + * usable_arenas list. However, if the arena has + * become wholly allocated, we need to remove its + * arena_object from usable_arenas. */ - freepools = pool->nextpool; + --usable_arenas->nfreepools; + if (usable_arenas->nfreepools == 0) { + /* Wholly allocated: remove. */ + assert(usable_arenas->freepools == NULL); + assert(usable_arenas->nextarena == NULL || + usable_arenas->nextarena->prevarena == + usable_arenas); + + usable_arenas = usable_arenas->nextarena; + if (usable_arenas != NULL) { + usable_arenas->prevarena = NULL; + assert(usable_arenas->address != 0); + } + } + else { + /* nfreepools > 0: it must be that freepools + * isn't NULL, or that we haven't yet carved + * off all the arena's pools for the first + * time. + */ + assert(usable_arenas->freepools != NULL || + usable_arenas->pool_address <= + (block*)usable_arenas->address + + ARENA_SIZE - POOL_SIZE); + } init_pool: - /* - * Frontlink to used pools - */ + /* Frontlink to used pools. */ next = usedpools[size + size]; /* == prev */ pool->nextpool = next; pool->prevpool = next; @@ -657,8 +836,7 @@ next->prevpool = pool; pool->ref.count = 1; if (pool->szidx == size) { - /* - * Luckily, this pool last contained blocks + /* Luckily, this pool last contained blocks * of the same size class, so its header * and free list are already initialized. */ @@ -682,39 +860,38 @@ UNLOCK(); return (void *)bp; } - /* - * Allocate new pool - */ - if (nfreepools) { - commit_pool: - --nfreepools; - pool = (poolp)arenabase; - arenabase += POOL_SIZE; - pool->arenaindex = narenas - 1; - pool->szidx = DUMMY_SIZE_IDX; - goto init_pool; - } - /* - * Allocate new arena - */ -#ifdef WITH_MEMORY_LIMITS - if (!(narenas < MAX_ARENAS)) { - UNLOCK(); - goto redirect; + + /* Carve off a new pool. */ + assert(usable_arenas->nfreepools > 0); + assert(usable_arenas->freepools == NULL); + pool = (poolp)usable_arenas->pool_address; + assert((block*)pool <= (block*)usable_arenas->address + + ARENA_SIZE - POOL_SIZE); + pool->arenaindex = usable_arenas - arenas; + assert(&arenas[pool->arenaindex] == usable_arenas); + pool->szidx = DUMMY_SIZE_IDX; + usable_arenas->pool_address += POOL_SIZE; + --usable_arenas->nfreepools; + + if (usable_arenas->nfreepools == 0) { + assert(usable_arenas->nextarena == NULL || + usable_arenas->nextarena->prevarena == + usable_arenas); + /* Unlink the arena: it is completely allocated. */ + usable_arenas = usable_arenas->nextarena; + if (usable_arenas != NULL) { + usable_arenas->prevarena = NULL; + assert(usable_arenas->address != 0); + } } -#endif - bp = new_arena(); - if (bp != NULL) - goto commit_pool; - UNLOCK(); - goto redirect; + + goto init_pool; } /* The small block allocator ends here. */ redirect: - /* - * Redirect the original request to the underlying (libc) allocator. + /* Redirect the original request to the underlying (libc) allocator. * We jump here on bigger requests, on error in the code above (as a * last chance to serve the request) or when the max memory limit * has been reached. @@ -742,8 +919,7 @@ if (Py_ADDRESS_IN_RANGE(p, pool)) { /* We allocated this address. */ LOCK(); - /* - * Link p to the start of the pool's freeblock list. Since + /* Link p to the start of the pool's freeblock list. Since * the pool had at least the p block outstanding, the pool * wasn't empty (so it's already in a usedpools[] list, or * was full and is in no list -- it's not in the freeblocks @@ -753,8 +929,10 @@ *(block **)p = lastfree = pool->freeblock; pool->freeblock = (block *)p; if (lastfree) { - /* - * freeblock wasn't NULL, so the pool wasn't full, + struct arena_object* ao; + uint nf; /* ao->nfreepools */ + + /* freeblock wasn't NULL, so the pool wasn't full, * and the pool is in a usedpools[] list. */ if (--pool->ref.count != 0) { @@ -762,8 +940,7 @@ UNLOCK(); return; } - /* - * Pool is now empty: unlink from usedpools, and + /* Pool is now empty: unlink from usedpools, and * link to the front of freepools. This ensures that * previously freed pools will be allocated later * (being not referenced, they are perhaps paged out). @@ -772,16 +949,147 @@ prev = pool->prevpool; next->prevpool = prev; prev->nextpool = next; - /* Link to freepools. This is a singly-linked list, - * and pool->prevpool isn't used there. + + /* Link the pool to freepools. This is a singly-linked + * list, and pool->prevpool isn't used there. */ - pool->nextpool = freepools; - freepools = pool; + ao = &arenas[pool->arenaindex]; + pool->nextpool = ao->freepools; + ao->freepools = pool; + nf = ++ao->nfreepools; + + /* All the rest is arena management. We just freed + * a pool, and there are 4 cases for arena mgmt: + * 1. If all the pools are free, return the arena to + * the system free(). + * 2. If this is the only free pool in the arena, + * add the arena back to the `usable_arenas` list. + * 3. If the "next" arena has a smaller count of free + * pools, we have to "slide this arena right" to + * restore that usable_arenas is sorted in order of + * nfreepools. + * 4. Else there's nothing more to do. + */ + if (nf == ao->ntotalpools) { + /* Case 1. First unlink ao from usable_arenas. + */ + assert(ao->prevarena == NULL || + ao->prevarena->address != 0); + assert(ao ->nextarena == NULL || + ao->nextarena->address != 0); + + /* Fix the pointer in the prevarena, or the + * usable_arenas pointer. + */ + if (ao->prevarena == NULL) { + usable_arenas = ao->nextarena; + assert(usable_arenas == NULL || + usable_arenas->address != 0); + } + else { + assert(ao->prevarena->nextarena == ao); + ao->prevarena->nextarena = + ao->nextarena; + } + /* Fix the pointer in the nextarena. */ + if (ao->nextarena != NULL) { + assert(ao->nextarena->prevarena == ao); + ao->nextarena->prevarena = + ao->prevarena; + } + /* Record that this arena_object slot is + * available to be reused. + */ + ao->nextarena = unused_arena_objects; + unused_arena_objects = ao; + + /* Free the entire arena. */ + free((void *)ao->address); + ao->address = 0; /* mark unassociated */ + --narenas_currently_allocated; + + UNLOCK(); + return; + } + if (nf == 1) { + /* Case 2. Put ao at the head of + * usable_arenas. Note that because + * ao->nfreepools was 0 before, ao isn't + * currently on the usable_arenas list. + */ + ao->nextarena = usable_arenas; + ao->prevarena = NULL; + if (usable_arenas) + usable_arenas->prevarena = ao; + usable_arenas = ao; + assert(usable_arenas->address != 0); + + UNLOCK(); + return; + } + /* If this arena is now out of order, we need to keep + * the list sorted. The list is kept sorted so that + * the "most full" arenas are used first, which allows + * the nearly empty arenas to be completely freed. In + * a few un-scientific tests, it seems like this + * approach allowed a lot more memory to be freed. + */ + if (ao->nextarena == NULL || + nf <= ao->nextarena->nfreepools) { + /* Case 4. Nothing to do. */ + UNLOCK(); + return; + } + /* Case 3: We have to move the arena towards the end + * of the list, because it has more free pools than + * the arena to its right. + * First unlink ao from usable_arenas. + */ + if (ao->prevarena != NULL) { + /* ao isn't at the head of the list */ + assert(ao->prevarena->nextarena == ao); + ao->prevarena->nextarena = ao->nextarena; + } + else { + /* ao is at the head of the list */ + assert(usable_arenas == ao); + usable_arenas = ao->nextarena; + } + ao->nextarena->prevarena = ao->prevarena; + + /* Locate the new insertion point by iterating over + * the list, using our nextarena pointer. + */ + while (ao->nextarena != NULL && + nf > ao->nextarena->nfreepools) { + ao->prevarena = ao->nextarena; + ao->nextarena = ao->nextarena->nextarena; + } + + /* Insert ao at this point. */ + assert(ao->nextarena == NULL || + ao->prevarena == ao->nextarena->prevarena); + assert(ao->prevarena->nextarena == ao->nextarena); + + ao->prevarena->nextarena = ao; + if (ao->nextarena != NULL) + ao->nextarena->prevarena = ao; + + /* Verify that the swaps worked. */ + assert(ao->nextarena == NULL || + nf <= ao->nextarena->nfreepools); + assert(ao->prevarena == NULL || + nf > ao->prevarena->nfreepools); + assert(ao->nextarena == NULL || + ao->nextarena->prevarena == ao); + assert((usable_arenas == ao && + ao->prevarena == NULL) || + ao->prevarena->nextarena == ao); + UNLOCK(); return; } - /* - * Pool was full, so doesn't currently live in any list: + /* Pool was full, so doesn't currently live in any list: * link it to the front of the appropriate usedpools[] list. * This mimics LRU pool usage for new allocations and * targets optimal filling when several pools contain @@ -1302,6 +1610,8 @@ * full pools. */ ulong quantization = 0; + /* # of arenas actually allocated. */ + ulong narenas = 0; /* running total -- should equal narenas * ARENA_SIZE */ ulong total; char buf[128]; @@ -1316,36 +1626,38 @@ * to march over all the arenas. If we're lucky, most of the memory * will be living in full pools -- would be a shame to miss them. */ - for (i = 0; i < narenas; ++i) { + for (i = 0; i < maxarenas; ++i) { uint poolsinarena; uint j; - uptr base = arenas[i]; + uptr base = arenas[i].address; + + /* Skip arenas which are not allocated. */ + if (arenas[i].address == (uptr)NULL) + continue; + narenas += 1; + + poolsinarena = arenas[i].ntotalpools; + numfreepools += arenas[i].nfreepools; /* round up to pool alignment */ - poolsinarena = ARENA_SIZE / POOL_SIZE; if (base & (uptr)POOL_SIZE_MASK) { - --poolsinarena; arena_alignment += POOL_SIZE; base &= ~(uptr)POOL_SIZE_MASK; base += POOL_SIZE; } - if (i == narenas - 1) { - /* current arena may have raw memory at the end */ - numfreepools += nfreepools; - poolsinarena -= nfreepools; - } - /* visit every pool in the arena */ - for (j = 0; j < poolsinarena; ++j, base += POOL_SIZE) { + assert(base <= (uptr) arenas[i].pool_address); + for (j = 0; + base < (uptr) arenas[i].pool_address; + ++j, base += POOL_SIZE) { poolp p = (poolp)base; const uint sz = p->szidx; uint freeblocks; if (p->ref.count == 0) { /* currently unused */ - ++numfreepools; - assert(pool_is_in_list(p, freepools)); + assert(pool_is_in_list(p, arenas[i].freepools)); continue; } ++numpools[sz]; @@ -1358,6 +1670,7 @@ #endif } } + assert(narenas == narenas_currently_allocated); fputc('\n', stderr); fputs("class size num pools blocks in use avail blocks\n" @@ -1383,9 +1696,14 @@ fputc('\n', stderr); (void)printone("# times object malloc called", serialno); + (void)printone("# arenas allocated total", ntimes_arena_allocated); + (void)printone("# arenas reclaimed", ntimes_arena_allocated - narenas); + (void)printone("# arenas highwater mark", narenas_highwater); + (void)printone("# arenas allocated current", narenas); + PyOS_snprintf(buf, sizeof(buf), - "%u arenas * %d bytes/arena", narenas, ARENA_SIZE); - (void)printone(buf, (ulong)narenas * ARENA_SIZE); + "%lu arenas * %d bytes/arena", narenas, ARENA_SIZE); + (void)printone(buf, narenas * ARENA_SIZE); fputc('\n', stderr); @@ -1405,12 +1723,14 @@ #endif /* PYMALLOC_DEBUG */ #ifdef Py_USING_MEMORY_DEBUGGER -/* Make this function last so gcc won't inline it - since the definition is after the reference. */ +/* Make this function last so gcc won't inline it since the definition is + * after the reference. + */ int Py_ADDRESS_IN_RANGE(void *P, poolp pool) { - return ((pool->arenaindex) < narenas && - (uptr)(P) - arenas[pool->arenaindex] < (uptr)ARENA_SIZE); + return pool->arenaindex < maxarenas && + (uptr)P - arenas[pool->arenaindex].address < (uptr)ARENA_SIZE && + arenas[pool->arenaindex].address != 0; } #endif Modified: python/branches/p3yk/Python/codecs.c ============================================================================== --- python/branches/p3yk/Python/codecs.c (original) +++ python/branches/p3yk/Python/codecs.c Fri Apr 21 11:43:23 2006 @@ -260,6 +260,56 @@ return NULL; } +PyObject *PyCodec_IncrementalEncoder(const char *encoding, + const char *errors) +{ + PyObject *codecs, *ret, *encoder; + + codecs = _PyCodec_Lookup(encoding); + if (codecs == NULL) + goto onError; + encoder = PyObject_GetAttrString(codecs, "incrementalencoder"); + if (encoder == NULL) { + Py_DECREF(codecs); + return NULL; + } + if (errors) + ret = PyObject_CallFunction(encoder, "O", errors); + else + ret = PyObject_CallFunction(encoder, NULL); + Py_DECREF(encoder); + Py_DECREF(codecs); + return ret; + + onError: + return NULL; +} + +PyObject *PyCodec_IncrementalDecoder(const char *encoding, + const char *errors) +{ + PyObject *codecs, *ret, *decoder; + + codecs = _PyCodec_Lookup(encoding); + if (codecs == NULL) + goto onError; + decoder = PyObject_GetAttrString(codecs, "incrementaldecoder"); + if (decoder == NULL) { + Py_DECREF(codecs); + return NULL; + } + if (errors) + ret = PyObject_CallFunction(decoder, "O", errors); + else + ret = PyObject_CallFunction(decoder, NULL); + Py_DECREF(decoder); + Py_DECREF(codecs); + return ret; + + onError: + return NULL; +} + PyObject *PyCodec_StreamReader(const char *encoding, PyObject *stream, const char *errors) Modified: python/branches/p3yk/Tools/buildbot/clean.bat ============================================================================== --- python/branches/p3yk/Tools/buildbot/clean.bat (original) +++ python/branches/p3yk/Tools/buildbot/clean.bat Fri Apr 21 11:43:23 2006 @@ -1,3 +1,6 @@ @rem Used by the buildbot "clean" step. call "%VS71COMNTOOLS%vsvars32.bat" -devenv.com /clean Debug PCbuild\pcbuild.sln +cd PCbuild + at echo Deleting .pyc/.pyo files ... +python_d.exe rmpyc.py +devenv.com /clean Debug pcbuild.sln Modified: python/branches/p3yk/Tools/unicode/Makefile ============================================================================== --- python/branches/p3yk/Tools/unicode/Makefile (original) +++ python/branches/p3yk/Tools/unicode/Makefile Fri Apr 21 11:43:23 2006 @@ -44,11 +44,11 @@ $(RM) -f build/readme.* iso: build/ - $(PYTHON) gencodec.py MAPPINGS/ISO8859/ build/iso + $(PYTHON) gencodec.py MAPPINGS/ISO8859/ build/ iso $(RM) -f build/isoreadme.* apple: build/ - $(PYTHON) gencodec.py MAPPINGS/VENDORS/APPLE/ build/mac_ + $(PYTHON) gencodec.py MAPPINGS/VENDORS/APPLE/ build/ mac_ $(RM) build/mac_dingbats.* $(RM) build/mac_japanese.* $(RM) build/mac_chin* Modified: python/branches/p3yk/Tools/unicode/gencodec.py ============================================================================== --- python/branches/p3yk/Tools/unicode/gencodec.py (original) +++ python/branches/p3yk/Tools/unicode/gencodec.py Fri Apr 21 11:43:23 2006 @@ -248,7 +248,7 @@ append(')') return l -def codegen(name, map, comments=1): +def codegen(name, map, encodingname, comments=1): """ Returns Python source for the given map. @@ -272,7 +272,7 @@ l = [ '''\ -""" Python Character Mapping Codec generated from '%s' with gencodec.py. +""" Python Character Mapping Codec %s generated from '%s' with gencodec.py. """#" @@ -283,11 +283,9 @@ class Codec(codecs.Codec): def encode(self,input,errors='strict'): - return codecs.charmap_encode(input,errors,encoding_map) - def decode(self,input,errors='strict'): -''' % name + def decode(self,input,errors='strict'):''' % (encodingname, name) ] if decoding_table_code: l.append('''\ @@ -297,6 +295,20 @@ return codecs.charmap_decode(input,errors,decoding_map)''') l.append(''' +class IncrementalEncoder(codecs.IncrementalEncoder): + def encode(self, input, final=False): + return codecs.charmap_encode(input,self.errors,encoding_map)[0] + +class IncrementalDecoder(codecs.IncrementalDecoder): + def decode(self, input, final=False):''') + if decoding_table_code: + l.append('''\ + return codecs.charmap_decode(input,self.errors,decoding_table)[0]''') + else: + l.append('''\ + return codecs.charmap_decode(input,self.errors,decoding_map)[0]''') + + l.append(''' class StreamWriter(Codec,codecs.StreamWriter): pass @@ -306,9 +318,16 @@ ### encodings module API def getregentry(): - - return (Codec().encode,Codec().decode,StreamReader,StreamWriter) -''') + return codecs.CodecInfo(( + name=%r, + Codec().encode, + Codec().decode, + streamwriter=StreamWriter, + streamreader=StreamReader, + incrementalencoder=IncrementalEncoder, + incrementaldecoder=IncrementalDecoder, + )) +''' % encodingname.replace('_', '-')) # Add decoding table or map (with preference to the table) if not decoding_table_code: @@ -331,11 +350,11 @@ # Final new-line l.append('\n') - return '\n'.join(l) + return '\n'.join(l).expandtabs() -def pymap(name,map,pyfile,comments=1): +def pymap(name,map,pyfile,encodingname,comments=1): - code = codegen(name,map,comments) + code = codegen(name,map,encodingname,comments) f = open(pyfile,'w') f.write(code) f.close() @@ -349,7 +368,7 @@ marshal.dump(d,f) f.close() -def convertdir(dir,prefix='',comments=1): +def convertdir(dir, dirprefix='', nameprefix='', comments=1): mapnames = os.listdir(dir) for mapname in mapnames: @@ -360,38 +379,40 @@ name = name.replace('-','_') name = name.split('.')[0] name = name.lower() + name = nameprefix + name codefile = name + '.py' marshalfile = name + '.mapping' print 'converting %s to %s and %s' % (mapname, - prefix + codefile, - prefix + marshalfile) + dirprefix + codefile, + dirprefix + marshalfile) try: map = readmap(os.path.join(dir,mapname)) if not map: print '* map is empty; skipping' else: - pymap(mappathname, map, prefix + codefile,comments) - marshalmap(mappathname, map, prefix + marshalfile) + pymap(mappathname, map, dirprefix + codefile,name,comments) + marshalmap(mappathname, map, dirprefix + marshalfile) except ValueError, why: print '* conversion failed: %s' % why raise -def rewritepythondir(dir,prefix='',comments=1): +def rewritepythondir(dir, dirprefix='', comments=1): mapnames = os.listdir(dir) for mapname in mapnames: if not mapname.endswith('.mapping'): continue - codefile = mapname[:-len('.mapping')] + '.py' + name = mapname[:-len('.mapping')] + codefile = name + '.py' print 'converting %s to %s' % (mapname, - prefix + codefile) + dirprefix + codefile) try: map = marshal.load(open(os.path.join(dir,mapname), 'rb')) if not map: print '* map is empty; skipping' else: - pymap(mapname, map, prefix + codefile,comments) + pymap(mapname, map, dirprefix + codefile,name,comments) except ValueError, why: print '* conversion failed: %s' % why From python-checkins at python.org Fri Apr 21 11:46:13 2006 From: python-checkins at python.org (thomas.wouters) Date: Fri, 21 Apr 2006 11:46:13 +0200 (CEST) Subject: [Python-checkins] r45594 - in python/branches/p3yk: Lib/re.py Message-ID: <20060421094613.2D7371E400B@bag.python.org> Author: thomas.wouters Date: Fri Apr 21 11:46:12 2006 New Revision: 45594 Removed: python/branches/p3yk/Lib/re.py Modified: python/branches/p3yk/ (props changed) Log: Merge trunk's 43068, deleting re.py. This leaves the branch in a broken state (unless you happen to have a re.pyc lying around), but it'll be fixed in the next merge. Deleted: /python/branches/p3yk/Lib/re.py ============================================================================== --- /python/branches/p3yk/Lib/re.py Fri Apr 21 11:46:12 2006 +++ (empty file) @@ -1,6 +0,0 @@ -"""Minimal "re" compatibility wrapper. See "sre" for documentation.""" - -engine = "sre" # Some apps might use this undocumented variable - -from sre import * -from sre import __all__ From python-checkins at python.org Fri Apr 21 11:47:09 2006 From: python-checkins at python.org (thomas.wouters) Date: Fri, 21 Apr 2006 11:47:09 +0200 (CEST) Subject: [Python-checkins] r45595 - in python/branches/p3yk: Lib/re.py Lib/sre.py Lib/test/regrtest.py Lib/test/test___all__.py Lib/test/test_re.py Modules/_sre.c Message-ID: <20060421094709.DF30C1E400B@bag.python.org> Author: thomas.wouters Date: Fri Apr 21 11:47:09 2006 New Revision: 45595 Added: python/branches/p3yk/Lib/re.py - copied unchanged from r43069, python/trunk/Lib/re.py Removed: python/branches/p3yk/Lib/sre.py Modified: python/branches/p3yk/ (props changed) python/branches/p3yk/Lib/test/regrtest.py python/branches/p3yk/Lib/test/test___all__.py python/branches/p3yk/Lib/test/test_re.py python/branches/p3yk/Modules/_sre.c Log: Merge trunk up to 43069, putting re.py back and hopefully making the branch usable again. Deleted: /python/branches/p3yk/Lib/sre.py ============================================================================== --- /python/branches/p3yk/Lib/sre.py Fri Apr 21 11:47:09 2006 +++ (empty file) @@ -1,315 +0,0 @@ -# -# Secret Labs' Regular Expression Engine -# -# re-compatible interface for the sre matching engine -# -# Copyright (c) 1998-2001 by Secret Labs AB. All rights reserved. -# -# This version of the SRE library can be redistributed under CNRI's -# Python 1.6 license. For any other use, please contact Secret Labs -# AB (info at pythonware.com). -# -# Portions of this engine have been developed in cooperation with -# CNRI. Hewlett-Packard provided funding for 1.6 integration and -# other compatibility work. -# - -r"""Support for regular expressions (RE). - -This module provides regular expression matching operations similar to -those found in Perl. It supports both 8-bit and Unicode strings; both -the pattern and the strings being processed can contain null bytes and -characters outside the US ASCII range. - -Regular expressions can contain both special and ordinary characters. -Most ordinary characters, like "A", "a", or "0", are the simplest -regular expressions; they simply match themselves. You can -concatenate ordinary characters, so last matches the string 'last'. - -The special characters are: - "." Matches any character except a newline. - "^" Matches the start of the string. - "$" Matches the end of the string. - "*" Matches 0 or more (greedy) repetitions of the preceding RE. - Greedy means that it will match as many repetitions as possible. - "+" Matches 1 or more (greedy) repetitions of the preceding RE. - "?" Matches 0 or 1 (greedy) of the preceding RE. - *?,+?,?? Non-greedy versions of the previous three special characters. - {m,n} Matches from m to n repetitions of the preceding RE. - {m,n}? Non-greedy version of the above. - "\\" Either escapes special characters or signals a special sequence. - [] Indicates a set of characters. - A "^" as the first character indicates a complementing set. - "|" A|B, creates an RE that will match either A or B. - (...) Matches the RE inside the parentheses. - The contents can be retrieved or matched later in the string. - (?iLmsux) Set the I, L, M, S, U, or X flag for the RE (see below). - (?:...) Non-grouping version of regular parentheses. - (?P...) The substring matched by the group is accessible by name. - (?P=name) Matches the text matched earlier by the group named name. - (?#...) A comment; ignored. - (?=...) Matches if ... matches next, but doesn't consume the string. - (?!...) Matches if ... doesn't match next. - -The special sequences consist of "\\" and a character from the list -below. If the ordinary character is not on the list, then the -resulting RE will match the second character. - \number Matches the contents of the group of the same number. - \A Matches only at the start of the string. - \Z Matches only at the end of the string. - \b Matches the empty string, but only at the start or end of a word. - \B Matches the empty string, but not at the start or end of a word. - \d Matches any decimal digit; equivalent to the set [0-9]. - \D Matches any non-digit character; equivalent to the set [^0-9]. - \s Matches any whitespace character; equivalent to [ \t\n\r\f\v]. - \S Matches any non-whitespace character; equiv. to [^ \t\n\r\f\v]. - \w Matches any alphanumeric character; equivalent to [a-zA-Z0-9_]. - With LOCALE, it will match the set [0-9_] plus characters defined - as letters for the current locale. - \W Matches the complement of \w. - \\ Matches a literal backslash. - -This module exports the following functions: - match Match a regular expression pattern to the beginning of a string. - search Search a string for the presence of a pattern. - sub Substitute occurrences of a pattern found in a string. - subn Same as sub, but also return the number of substitutions made. - split Split a string by the occurrences of a pattern. - findall Find all occurrences of a pattern in a string. - compile Compile a pattern into a RegexObject. - purge Clear the regular expression cache. - escape Backslash all non-alphanumerics in a string. - -Some of the functions in this module takes flags as optional parameters: - I IGNORECASE Perform case-insensitive matching. - L LOCALE Make \w, \W, \b, \B, dependent on the current locale. - M MULTILINE "^" matches the beginning of lines as well as the string. - "$" matches the end of lines as well as the string. - S DOTALL "." matches any character at all, including the newline. - X VERBOSE Ignore whitespace and comments for nicer looking RE's. - U UNICODE Make \w, \W, \b, \B, dependent on the Unicode locale. - -This module also defines an exception 'error'. - -""" - -import sys -import sre_compile -import sre_parse - -# public symbols -__all__ = [ "match", "search", "sub", "subn", "split", "findall", - "compile", "purge", "template", "escape", "I", "L", "M", "S", "X", - "U", "IGNORECASE", "LOCALE", "MULTILINE", "DOTALL", "VERBOSE", - "UNICODE", "error" ] - -__version__ = "2.2.1" - -# flags -I = IGNORECASE = sre_compile.SRE_FLAG_IGNORECASE # ignore case -L = LOCALE = sre_compile.SRE_FLAG_LOCALE # assume current 8-bit locale -U = UNICODE = sre_compile.SRE_FLAG_UNICODE # assume unicode locale -M = MULTILINE = sre_compile.SRE_FLAG_MULTILINE # make anchors look for newline -S = DOTALL = sre_compile.SRE_FLAG_DOTALL # make dot match newline -X = VERBOSE = sre_compile.SRE_FLAG_VERBOSE # ignore whitespace and comments - -# sre extensions (experimental, don't rely on these) -T = TEMPLATE = sre_compile.SRE_FLAG_TEMPLATE # disable backtracking -DEBUG = sre_compile.SRE_FLAG_DEBUG # dump pattern after compilation - -# sre exception -error = sre_compile.error - -# -------------------------------------------------------------------- -# public interface - -def match(pattern, string, flags=0): - """Try to apply the pattern at the start of the string, returning - a match object, or None if no match was found.""" - return _compile(pattern, flags).match(string) - -def search(pattern, string, flags=0): - """Scan through string looking for a match to the pattern, returning - a match object, or None if no match was found.""" - return _compile(pattern, flags).search(string) - -def sub(pattern, repl, string, count=0): - """Return the string obtained by replacing the leftmost - non-overlapping occurrences of the pattern in string by the - replacement repl. repl can be either a string or a callable; - if a callable, it's passed the match object and must return - a replacement string to be used.""" - return _compile(pattern, 0).sub(repl, string, count) - -def subn(pattern, repl, string, count=0): - """Return a 2-tuple containing (new_string, number). - new_string is the string obtained by replacing the leftmost - non-overlapping occurrences of the pattern in the source - string by the replacement repl. number is the number of - substitutions that were made. repl can be either a string or a - callable; if a callable, it's passed the match object and must - return a replacement string to be used.""" - return _compile(pattern, 0).subn(repl, string, count) - -def split(pattern, string, maxsplit=0): - """Split the source string by the occurrences of the pattern, - returning a list containing the resulting substrings.""" - return _compile(pattern, 0).split(string, maxsplit) - -def findall(pattern, string, flags=0): - """Return a list of all non-overlapping matches in the string. - - If one or more groups are present in the pattern, return a - list of groups; this will be a list of tuples if the pattern - has more than one group. - - Empty matches are included in the result.""" - return _compile(pattern, flags).findall(string) - -if sys.hexversion >= 0x02020000: - __all__.append("finditer") - def finditer(pattern, string, flags=0): - """Return an iterator over all non-overlapping matches in the - string. For each match, the iterator returns a match object. - - Empty matches are included in the result.""" - return _compile(pattern, flags).finditer(string) - -def compile(pattern, flags=0): - "Compile a regular expression pattern, returning a pattern object." - return _compile(pattern, flags) - -def purge(): - "Clear the regular expression cache" - _cache.clear() - _cache_repl.clear() - -def template(pattern, flags=0): - "Compile a template pattern, returning a pattern object" - return _compile(pattern, flags|T) - -_alphanum = {} -for c in 'abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ01234567890': - _alphanum[c] = 1 -del c - -def escape(pattern): - "Escape all non-alphanumeric characters in pattern." - s = list(pattern) - alphanum = _alphanum - for i in range(len(pattern)): - c = pattern[i] - if c not in alphanum: - if c == "\000": - s[i] = "\\000" - else: - s[i] = "\\" + c - return pattern[:0].join(s) - -# -------------------------------------------------------------------- -# internals - -_cache = {} -_cache_repl = {} - -_pattern_type = type(sre_compile.compile("", 0)) - -_MAXCACHE = 100 - -def _compile(*key): - # internal: compile pattern - cachekey = (type(key[0]),) + key - p = _cache.get(cachekey) - if p is not None: - return p - pattern, flags = key - if isinstance(pattern, _pattern_type): - return pattern - if not sre_compile.isstring(pattern): - raise TypeError, "first argument must be string or compiled pattern" - try: - p = sre_compile.compile(pattern, flags) - except error, v: - raise error, v # invalid expression - if len(_cache) >= _MAXCACHE: - _cache.clear() - _cache[cachekey] = p - return p - -def _compile_repl(*key): - # internal: compile replacement pattern - p = _cache_repl.get(key) - if p is not None: - return p - repl, pattern = key - try: - p = sre_parse.parse_template(repl, pattern) - except error, v: - raise error, v # invalid expression - if len(_cache_repl) >= _MAXCACHE: - _cache_repl.clear() - _cache_repl[key] = p - return p - -def _expand(pattern, match, template): - # internal: match.expand implementation hook - template = sre_parse.parse_template(template, pattern) - return sre_parse.expand_template(template, match) - -def _subx(pattern, template): - # internal: pattern.sub/subn implementation helper - template = _compile_repl(template, pattern) - if not template[0] and len(template[1]) == 1: - # literal replacement - return template[1][0] - def filter(match, template=template): - return sre_parse.expand_template(template, match) - return filter - -# register myself for pickling - -import copy_reg - -def _pickle(p): - return _compile, (p.pattern, p.flags) - -copy_reg.pickle(_pattern_type, _pickle, _compile) - -# -------------------------------------------------------------------- -# experimental stuff (see python-dev discussions for details) - -class Scanner: - def __init__(self, lexicon, flags=0): - from sre_constants import BRANCH, SUBPATTERN - self.lexicon = lexicon - # combine phrases into a compound pattern - p = [] - s = sre_parse.Pattern() - s.flags = flags - for phrase, action in lexicon: - p.append(sre_parse.SubPattern(s, [ - (SUBPATTERN, (len(p)+1, sre_parse.parse(phrase, flags))), - ])) - p = sre_parse.SubPattern(s, [(BRANCH, (None, p))]) - s.groups = len(p) - self.scanner = sre_compile.compile(p) - def scan(self, string): - result = [] - append = result.append - match = self.scanner.scanner(string).match - i = 0 - while 1: - m = match() - if not m: - break - j = m.end() - if i == j: - break - action = self.lexicon[m.lastindex-1][1] - if callable(action): - self.match = m - action = action(self, m.group()) - if action is not None: - append(action) - i = j - return result, string[i:] Modified: python/branches/p3yk/Lib/test/regrtest.py ============================================================================== --- python/branches/p3yk/Lib/test/regrtest.py (original) +++ python/branches/p3yk/Lib/test/regrtest.py Fri Apr 21 11:47:09 2006 @@ -110,7 +110,7 @@ import getopt import random import warnings -import sre +import re import cStringIO import traceback @@ -525,7 +525,7 @@ _path_created.clear() warnings.filters[:] = fs gc.collect() - sre.purge() + re.purge() _strptime._regex_cache.clear() urlparse.clear_cache() urllib.urlcleanup() Modified: python/branches/p3yk/Lib/test/test___all__.py ============================================================================== --- python/branches/p3yk/Lib/test/test___all__.py (original) +++ python/branches/p3yk/Lib/test/test___all__.py Fri Apr 21 11:47:09 2006 @@ -145,7 +145,6 @@ self.check_all("smtplib") self.check_all("sndhdr") self.check_all("socket") - self.check_all("sre") self.check_all("_strptime") self.check_all("symtable") self.check_all("tabnanny") Modified: python/branches/p3yk/Lib/test/test_re.py ============================================================================== --- python/branches/p3yk/Lib/test/test_re.py (original) +++ python/branches/p3yk/Lib/test/test_re.py Fri Apr 21 11:47:09 2006 @@ -3,7 +3,7 @@ from test.test_support import verbose, run_unittest import re -from sre import Scanner +from re import Scanner import sys, os, traceback from weakref import proxy Modified: python/branches/p3yk/Modules/_sre.c ============================================================================== --- python/branches/p3yk/Modules/_sre.c (original) +++ python/branches/p3yk/Modules/_sre.c Fri Apr 21 11:47:09 2006 @@ -51,6 +51,8 @@ #define SRE_MODULE "sre" #endif +#define SRE_PY_MODULE "re" + /* defining this one enables tracing */ #undef VERBOSE @@ -2455,7 +2457,7 @@ } else { /* not a literal; hand it over to the template compiler */ filter = call( - SRE_MODULE, "_subx", + SRE_PY_MODULE, "_subx", PyTuple_Pack(2, self, template) ); if (!filter) @@ -2872,7 +2874,7 @@ /* delegate to Python code */ return call( - SRE_MODULE, "_expand", + SRE_PY_MODULE, "_expand", PyTuple_Pack(3, self->pattern, self, template) ); } From python-checkins at python.org Fri Apr 21 12:00:46 2006 From: python-checkins at python.org (martin.v.loewis) Date: Fri, 21 Apr 2006 12:00:46 +0200 (CEST) Subject: [Python-checkins] r45596 - python/trunk/Tools/msi/msi.py Message-ID: <20060421100046.BB7E41E4065@bag.python.org> Author: martin.v.loewis Date: Fri Apr 21 12:00:46 2006 New Revision: 45596 Modified: python/trunk/Tools/msi/msi.py Log: Package setuptools Modified: python/trunk/Tools/msi/msi.py ============================================================================== --- python/trunk/Tools/msi/msi.py (original) +++ python/trunk/Tools/msi/msi.py Fri Apr 21 12:00:46 2006 @@ -908,6 +908,12 @@ if files: # Add an entry to the RemoveFile table to remove bytecode files. lib.remove_pyc() + if dir.endswith('.egg-info'): + lib.add_file('entry_points.txt') + lib.add_file('PKG-INFO') + lib.add_file('top_level.txt') + lib.add_file('zip-safe') + continue if dir=='test' and parent.physical=='Lib': lib.add_file("185test.db") lib.add_file("audiotest.au") @@ -930,9 +936,12 @@ if dir=="Icons": lib.glob("*.gif") lib.add_file("idle.icns") - if dir=="command": + if dir=="command" and parent.physical=="distutils": lib.add_file("wininst-6.exe") lib.add_file("wininst-7.1.exe") + if dir=="setuptools": + lib.add_file("cli.exe") + lib.add_file("gui.exe") if dir=="data" and parent.physical=="test" and parent.basedir.physical=="email": # This should contain all non-.svn files listed in subversion for f in os.listdir(lib.absolute): From theller at python.net Fri Apr 21 12:07:39 2006 From: theller at python.net (Thomas Heller) Date: Fri, 21 Apr 2006 12:07:39 +0200 Subject: [Python-checkins] r45585 - in external/ctypes: .cvsignore ACKS ANNOUNCE ChangeLog LICENSE.txt MANIFEST.in README.CVS README.txt ctypes-dev.el ctypes/.CTYPES_DEVEL ctypes/.cvsignore ctypes/__init__.py ctypes/_endian.py ctypes/_loader.py ctypes/mac In-Reply-To: References: Message-ID: <4448AEEB.5020904@python.net> Neal Norwitz wrote: > On 4/20/06, thomas.heller wrote: >> Author: thomas.heller >> Date: Thu Apr 20 21:23:42 2006 >> New Revision: 45585 > > I tried to diff this to see what changed, but I don't get anything > returned. Does anyone know how to view the full diff of this checkin? > > n I didn't receive the checkin message (via gmane) at all, probably because it was too large. I wonder if there's something broken, maybe the files have been checked in with wrong line endings or something like that. The diffs always have both complete files, for example: c:\svn\external\ctypes>svn -r 45584:45585 diff .cvsignore Index: .cvsignore =================================================================== --- .cvsignore (Revision 45584) +++ .cvsignore (Revision 45585) @@ -1,18 +1,19 @@ -*.c -*.h -*.ilk -*.log -*.pdb -*.pyc -*.pyo -*.tlb -*.zip -MANIFEST -_ctypes.pyd -_ctypes_d.pyd -archive -build -dist -extensions -test.output -comtypes \ No newline at end of file +*.c +*.h +*.ilk +*.log +*.pdb +*.pyc +*.pyo +*.tlb +*.zip +MANIFEST +_ctypes.pyd +_ctypes_d.pyd +archive +build +comtypes +dist +extensions +others.py +test.output c:\svn\external\ctypes> For merging the changes into the trunk, the 'svn merge' command that I tried to use also had the same problem, so I merged manually. This is not yet committed, but you will be able to see the changes then. Thomas From buildbot at python.org Fri Apr 21 12:41:19 2006 From: buildbot at python.org (buildbot at python.org) Date: Fri, 21 Apr 2006 10:41:19 +0000 Subject: [Python-checkins] buildbot warnings in sparc solaris10 gcc trunk Message-ID: <20060421104119.DE33D1E400B@bag.python.org> The Buildbot has detected a new failure of sparc solaris10 gcc trunk. Full details are available at: http://www.python.org/dev/buildbot/all/sparc%2520solaris10%2520gcc%2520trunk/builds/502 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: martin.v.loewis Build Had Warnings: warnings test sincerely, -The Buildbot From python-checkins at python.org Fri Apr 21 13:26:57 2006 From: python-checkins at python.org (thomas.wouters) Date: Fri, 21 Apr 2006 13:26:57 +0200 (CEST) Subject: [Python-checkins] r45598 - python/trunk/Objects/typeobject.c Message-ID: <20060421112657.B987B1E400B@bag.python.org> Author: thomas.wouters Date: Fri Apr 21 13:26:56 2006 New Revision: 45598 Modified: python/trunk/Objects/typeobject.c Log: Fix variable/format-char discrepancy in new-style class __getitem__, __delitem__, __setslice__ and __delslice__ hooks. This caused test_weakref and test_userlist to fail in the p3yk branch (where UserList, like all classes, is new-style) on amd64 systems, with open-ended slices: the sys.maxint value for empty-endpoint was transformed into -1. Modified: python/trunk/Objects/typeobject.c ============================================================================== --- python/trunk/Objects/typeobject.c (original) +++ python/trunk/Objects/typeobject.c Fri Apr 21 13:26:56 2006 @@ -4186,10 +4186,10 @@ if (value == NULL) res = call_method(self, "__delitem__", &delitem_str, - "(i)", index); + "(n)", index); else res = call_method(self, "__setitem__", &setitem_str, - "(iO)", index, value); + "(nO)", index, value); if (res == NULL) return -1; Py_DECREF(res); @@ -4204,10 +4204,10 @@ if (value == NULL) res = call_method(self, "__delslice__", &delslice_str, - "(ii)", i, j); + "(nn)", i, j); else res = call_method(self, "__setslice__", &setslice_str, - "(iiO)", i, j, value); + "(nnO)", i, j, value); if (res == NULL) return -1; Py_DECREF(res); From python-checkins at python.org Fri Apr 21 13:28:17 2006 From: python-checkins at python.org (thomas.wouters) Date: Fri, 21 Apr 2006 13:28:17 +0200 (CEST) Subject: [Python-checkins] r45599 - in python/branches/p3yk: Objects/typeobject.c Tools/msi/msi.py Message-ID: <20060421112817.CE1F51E400B@bag.python.org> Author: thomas.wouters Date: Fri Apr 21 13:28:17 2006 New Revision: 45599 Modified: python/branches/p3yk/ (props changed) python/branches/p3yk/Objects/typeobject.c python/branches/p3yk/Tools/msi/msi.py Log: Merge trunk up to 45598. Modified: python/branches/p3yk/Objects/typeobject.c ============================================================================== --- python/branches/p3yk/Objects/typeobject.c (original) +++ python/branches/p3yk/Objects/typeobject.c Fri Apr 21 13:28:17 2006 @@ -4183,10 +4183,10 @@ if (value == NULL) res = call_method(self, "__delitem__", &delitem_str, - "(i)", index); + "(n)", index); else res = call_method(self, "__setitem__", &setitem_str, - "(iO)", index, value); + "(nO)", index, value); if (res == NULL) return -1; Py_DECREF(res); @@ -4201,10 +4201,10 @@ if (value == NULL) res = call_method(self, "__delslice__", &delslice_str, - "(ii)", i, j); + "(nn)", i, j); else res = call_method(self, "__setslice__", &setslice_str, - "(iiO)", i, j, value); + "(nnO)", i, j, value); if (res == NULL) return -1; Py_DECREF(res); Modified: python/branches/p3yk/Tools/msi/msi.py ============================================================================== --- python/branches/p3yk/Tools/msi/msi.py (original) +++ python/branches/p3yk/Tools/msi/msi.py Fri Apr 21 13:28:17 2006 @@ -908,6 +908,12 @@ if files: # Add an entry to the RemoveFile table to remove bytecode files. lib.remove_pyc() + if dir.endswith('.egg-info'): + lib.add_file('entry_points.txt') + lib.add_file('PKG-INFO') + lib.add_file('top_level.txt') + lib.add_file('zip-safe') + continue if dir=='test' and parent.physical=='Lib': lib.add_file("185test.db") lib.add_file("audiotest.au") @@ -930,9 +936,12 @@ if dir=="Icons": lib.glob("*.gif") lib.add_file("idle.icns") - if dir=="command": + if dir=="command" and parent.physical=="distutils": lib.add_file("wininst-6.exe") lib.add_file("wininst-7.1.exe") + if dir=="setuptools": + lib.add_file("cli.exe") + lib.add_file("gui.exe") if dir=="data" and parent.physical=="test" and parent.basedir.physical=="email": # This should contain all non-.svn files listed in subversion for f in os.listdir(lib.absolute): From python-checkins at python.org Fri Apr 21 14:38:42 2006 From: python-checkins at python.org (andrew.kuchling) Date: Fri, 21 Apr 2006 14:38:42 +0200 (CEST) Subject: [Python-checkins] r45602 - python/trunk/Doc/lib/libcodecs.tex Message-ID: <20060421123842.088971E400B@bag.python.org> Author: andrew.kuchling Date: Fri Apr 21 14:38:41 2006 New Revision: 45602 Modified: python/trunk/Doc/lib/libcodecs.tex Log: Typo fixes Modified: python/trunk/Doc/lib/libcodecs.tex ============================================================================== --- python/trunk/Doc/lib/libcodecs.tex (original) +++ python/trunk/Doc/lib/libcodecs.tex Fri Apr 21 14:38:41 2006 @@ -93,21 +93,21 @@ lookup: \begin{funcdesc}{getencoder}{encoding} -Lookup up the codec for the given encoding and return its encoder +Look up the codec for the given encoding and return its encoder function. Raises a \exception{LookupError} in case the encoding cannot be found. \end{funcdesc} \begin{funcdesc}{getdecoder}{encoding} -Lookup up the codec for the given encoding and return its decoder +Look up the codec for the given encoding and return its decoder function. Raises a \exception{LookupError} in case the encoding cannot be found. \end{funcdesc} \begin{funcdesc}{getincrementalencoder}{encoding} -Lookup up the codec for the given encoding and return its incremental encoder +Look up the codec for the given encoding and return its incremental encoder class or factory function. Raises a \exception{LookupError} in case the encoding cannot be found or the @@ -116,7 +116,7 @@ \end{funcdesc} \begin{funcdesc}{getincrementaldecoder}{encoding} -Lookup up the codec for the given encoding and return its incremental decoder +Look up the codec for the given encoding and return its incremental decoder class or factory function. Raises a \exception{LookupError} in case the encoding cannot be found or the @@ -125,14 +125,14 @@ \end{funcdesc} \begin{funcdesc}{getreader}{encoding} -Lookup up the codec for the given encoding and return its StreamReader +Look up the codec for the given encoding and return its StreamReader class or factory function. Raises a \exception{LookupError} in case the encoding cannot be found. \end{funcdesc} \begin{funcdesc}{getwriter}{encoding} -Lookup up the codec for the given encoding and return its StreamWriter +Look up the codec for the given encoding and return its StreamWriter class or factory function. Raises a \exception{LookupError} in case the encoding cannot be found. From python-checkins at python.org Fri Apr 21 14:40:03 2006 From: python-checkins at python.org (andrew.kuchling) Date: Fri, 21 Apr 2006 14:40:03 +0200 (CEST) Subject: [Python-checkins] r45603 - python/branches/release24-maint/Doc/lib/libcodecs.tex Message-ID: <20060421124003.B48411E400B@bag.python.org> Author: andrew.kuchling Date: Fri Apr 21 14:40:03 2006 New Revision: 45603 Modified: python/branches/release24-maint/Doc/lib/libcodecs.tex Log: Typo fixes Modified: python/branches/release24-maint/Doc/lib/libcodecs.tex ============================================================================== --- python/branches/release24-maint/Doc/lib/libcodecs.tex (original) +++ python/branches/release24-maint/Doc/lib/libcodecs.tex Fri Apr 21 14:40:03 2006 @@ -72,28 +72,28 @@ lookup: \begin{funcdesc}{getencoder}{encoding} -Lookup up the codec for the given encoding and return its encoder +Look up the codec for the given encoding and return its encoder function. Raises a \exception{LookupError} in case the encoding cannot be found. \end{funcdesc} \begin{funcdesc}{getdecoder}{encoding} -Lookup up the codec for the given encoding and return its decoder +Look up the codec for the given encoding and return its decoder function. Raises a \exception{LookupError} in case the encoding cannot be found. \end{funcdesc} \begin{funcdesc}{getreader}{encoding} -Lookup up the codec for the given encoding and return its StreamReader +Look up the codec for the given encoding and return its StreamReader class or factory function. Raises a \exception{LookupError} in case the encoding cannot be found. \end{funcdesc} \begin{funcdesc}{getwriter}{encoding} -Lookup up the codec for the given encoding and return its StreamWriter +Look up the codec for the given encoding and return its StreamWriter class or factory function. Raises a \exception{LookupError} in case the encoding cannot be found. From python-checkins at python.org Fri Apr 21 14:57:35 2006 From: python-checkins at python.org (andrew.kuchling) Date: Fri, 21 Apr 2006 14:57:35 +0200 (CEST) Subject: [Python-checkins] r45604 - python/trunk/Lib/SimpleXMLRPCServer.py Message-ID: <20060421125735.009591E400C@bag.python.org> Author: andrew.kuchling Date: Fri Apr 21 14:57:35 2006 New Revision: 45604 Modified: python/trunk/Lib/SimpleXMLRPCServer.py Log: Add explanatory message Modified: python/trunk/Lib/SimpleXMLRPCServer.py ============================================================================== --- python/trunk/Lib/SimpleXMLRPCServer.py (original) +++ python/trunk/Lib/SimpleXMLRPCServer.py Fri Apr 21 14:57:35 2006 @@ -560,6 +560,7 @@ self.handle_xmlrpc(request_text) if __name__ == '__main__': + print 'Running XML-RPC server on port 8000' server = SimpleXMLRPCServer(("localhost", 8000)) server.register_function(pow) server.register_function(lambda x,y: x+y, 'add') From python-checkins at python.org Fri Apr 21 14:57:59 2006 From: python-checkins at python.org (andrew.kuchling) Date: Fri, 21 Apr 2006 14:57:59 +0200 (CEST) Subject: [Python-checkins] r45605 - python/trunk/Doc/howto/unicode.rst Message-ID: <20060421125759.1CC4A1E400D@bag.python.org> Author: andrew.kuchling Date: Fri Apr 21 14:57:58 2006 New Revision: 45605 Modified: python/trunk/Doc/howto/unicode.rst Log: Add TODO item Modified: python/trunk/Doc/howto/unicode.rst ============================================================================== --- python/trunk/Doc/howto/unicode.rst (original) +++ python/trunk/Doc/howto/unicode.rst Fri Apr 21 14:57:58 2006 @@ -733,6 +733,7 @@ .. comment Additional topic: building Python w/ UCS2 or UCS4 support .. comment Describe obscure -U switch somewhere? +.. comment Describe use of codecs.StreamRecoder and StreamReaderWriter .. comment Original outline: From python-checkins at python.org Fri Apr 21 14:58:30 2006 From: python-checkins at python.org (andrew.kuchling) Date: Fri, 21 Apr 2006 14:58:30 +0200 (CEST) Subject: [Python-checkins] r45606 - python/trunk/Doc/lib/libcodecs.tex Message-ID: <20060421125830.8DC251E400B@bag.python.org> Author: andrew.kuchling Date: Fri Apr 21 14:58:30 2006 New Revision: 45606 Modified: python/trunk/Doc/lib/libcodecs.tex Log: Typo, grammar fixes. This file could use another proofreading pass. Modified: python/trunk/Doc/lib/libcodecs.tex ============================================================================== --- python/trunk/Doc/lib/libcodecs.tex (original) +++ python/trunk/Doc/lib/libcodecs.tex Fri Apr 21 14:58:30 2006 @@ -353,7 +353,7 @@ the encoding/decoding process during method calls. The joined output of calls to the \method{encode}/\method{decode} method is the -same as if the all single inputs where joined into one, and this input was +same as if all the single inputs were joined into one, and this input was encoded/decoded with the stateless encoder/decoder. @@ -363,7 +363,7 @@ The \class{IncrementalEncoder} class is used for encoding an input in multiple steps. It defines the following methods which every incremental encoder must -define in order to be compatible to the Python codec registry. +define in order to be compatible with the Python codec registry. \begin{classdesc}{IncrementalEncoder}{\optional{errors}} Constructor for a \class{IncrementalEncoder} instance. @@ -410,7 +410,7 @@ The \class{IncrementalDecoder} class is used for decoding an input in multiple steps. It defines the following methods which every incremental decoder must -define in order to be compatible to the Python codec registry. +define in order to be compatible with the Python codec registry. \begin{classdesc}{IncrementalDecoder}{\optional{errors}} Constructor for a \class{IncrementalDecoder} instance. @@ -456,15 +456,15 @@ The \class{StreamWriter} and \class{StreamReader} classes provide generic working interfaces which can be used to implement new -encodings submodules very easily. See \module{encodings.utf_8} for an -example on how this is done. +encoding submodules very easily. See \module{encodings.utf_8} for an +example of how this is done. \subsubsection{StreamWriter Objects \label{stream-writer-objects}} The \class{StreamWriter} class is a subclass of \class{Codec} and defines the following methods which every stream writer must define in -order to be compatible to the Python codec registry. +order to be compatible with the Python codec registry. \begin{classdesc}{StreamWriter}{stream\optional{, errors}} Constructor for a \class{StreamWriter} instance. @@ -473,7 +473,7 @@ free to add additional keyword arguments, but only the ones defined here are used by the Python codec registry. - \var{stream} must be a file-like object open for writing (binary) + \var{stream} must be a file-like object open for writing binary data. The \class{StreamWriter} may implement different error handling @@ -512,19 +512,19 @@ Flushes and resets the codec buffers used for keeping state. Calling this method should ensure that the data on the output is put - into a clean state, that allows appending of new fresh data without + into a clean state that allows appending of new fresh data without having to rescan the whole stream to recover state. \end{methoddesc} In addition to the above methods, the \class{StreamWriter} must also -inherit all other methods and attribute from the underlying stream. +inherit all other methods and attributes from the underlying stream. \subsubsection{StreamReader Objects \label{stream-reader-objects}} The \class{StreamReader} class is a subclass of \class{Codec} and defines the following methods which every stream reader must define in -order to be compatible to the Python codec registry. +order to be compatible with the Python codec registry. \begin{classdesc}{StreamReader}{stream\optional{, errors}} Constructor for a \class{StreamReader} instance. @@ -589,20 +589,20 @@ \var{size}, if given, is passed as size argument to the stream's \method{readline()} method. - If \var{keepends} is false lineends will be stripped from the + If \var{keepends} is false line-endings will be stripped from the lines returned. \versionchanged[\var{keepends} argument added]{2.4} \end{methoddesc} \begin{methoddesc}{readlines}{\optional{sizehint\optional{, keepends}}} - Read all lines available on the input stream and return them as list + Read all lines available on the input stream and return them as a list of lines. - Line breaks are implemented using the codec's decoder method and are + Line-endings are implemented using the codec's decoder method and are included in the list entries if \var{keepends} is true. - \var{sizehint}, if given, is passed as \var{size} argument to the + \var{sizehint}, if given, is passed as the \var{size} argument to the stream's \method{read()} method. \end{methoddesc} @@ -614,7 +614,7 @@ \end{methoddesc} In addition to the above methods, the \class{StreamReader} must also -inherit all other methods and attribute from the underlying stream. +inherit all other methods and attributes from the underlying stream. The next two base classes are included for convenience. They are not needed by the codec registry, but may provide useful in practice. @@ -640,7 +640,7 @@ \class{StreamReaderWriter} instances define the combined interfaces of \class{StreamReader} and \class{StreamWriter} classes. They inherit -all other methods and attribute from the underlying stream. +all other methods and attributes from the underlying stream. \subsubsection{StreamRecoder Objects \label{stream-recoder-objects}} @@ -666,14 +666,14 @@ \var{stream} must be a file-like object. \var{encode}, \var{decode} must adhere to the \class{Codec} - interface, \var{Reader}, \var{Writer} must be factory functions or + interface. \var{Reader}, \var{Writer} must be factory functions or classes providing objects of the \class{StreamReader} and \class{StreamWriter} interface respectively. \var{encode} and \var{decode} are needed for the frontend translation, \var{Reader} and \var{Writer} for the backend translation. The intermediate format used is determined by the two - sets of codecs, e.g. the Unicode codecs will use Unicode as + sets of codecs, e.g. the Unicode codecs will use Unicode as the intermediate encoding. Error handling is done in the same way as defined for the @@ -682,7 +682,7 @@ \class{StreamRecoder} instances define the combined interfaces of \class{StreamReader} and \class{StreamWriter} classes. They inherit -all other methods and attribute from the underlying stream. +all other methods and attributes from the underlying stream. \subsection{Encodings and Unicode\label{encodings-overview}} @@ -695,7 +695,7 @@ memory, CPU endianness and how these arrays are stored as bytes become an issue. Transforming a unicode object into a sequence of bytes is called encoding and recreating the unicode object from the sequence of -bytes is known as decoding. There are many different methods how this +bytes is known as decoding. There are many different methods for how this transformation can be done (these methods are also called encodings). The simplest method is to map the codepoints 0-255 to the bytes \code{0x0}-\code{0xff}. This means that a unicode object that contains @@ -742,7 +742,7 @@ it's a normal character that will be decoded like any other. There's another encoding that is able to encoding the full range of -Unicode characters: UTF-8. UTF-8 is an 8bit encoding, which means +Unicode characters: UTF-8. UTF-8 is an 8-bit encoding, which means there are no issues with byte order in UTF-8. Each byte in a UTF-8 byte sequence consists of two parts: Marker bits (the most significant bits) and payload bits. The marker bits are a sequence of zero to six @@ -762,7 +762,7 @@ The least significant bit of the Unicode character is the rightmost x bit. -As UTF-8 is an 8bit encoding no BOM is required and any \code{U+FEFF} +As UTF-8 is an 8-bit encoding no BOM is required and any \code{U+FEFF} character in the decoded Unicode string (even if it's the first character) is treated as a \samp{ZERO WIDTH NO-BREAK SPACE}. @@ -775,7 +775,7 @@ variant of UTF-8 (that Python 2.5 calls \code{"utf-8-sig"}) for its Notepad program: Before any of the Unicode characters is written to the file, a UTF-8 encoded BOM (which looks like this as a byte sequence: \code{0xef}, -\code{0xbb}, \code{0xbf}) is written. As it's rather improbably that any +\code{0xbb}, \code{0xbf}) is written. As it's rather improbable that any charmap encoded file starts with these byte values (which would e.g. map to LATIN SMALL LETTER I WITH DIAERESIS \\ @@ -794,8 +794,8 @@ \subsection{Standard Encodings\label{standard-encodings}} -Python comes with a number of codecs builtin, either implemented as C -functions, or with dictionaries as mapping tables. The following table +Python comes with a number of codecs built-in, either implemented as C +functions or with dictionaries as mapping tables. The following table lists the codecs by name, together with a few common aliases, and the languages for which the encoding is likely used. Neither the list of aliases nor the list of languages is meant to be exhaustive. Notice From python-checkins at python.org Fri Apr 21 14:59:13 2006 From: python-checkins at python.org (andrew.kuchling) Date: Fri, 21 Apr 2006 14:59:13 +0200 (CEST) Subject: [Python-checkins] r45607 - python/branches/release24-maint/Doc/lib/libcodecs.tex Message-ID: <20060421125913.0345A1E400B@bag.python.org> Author: andrew.kuchling Date: Fri Apr 21 14:59:12 2006 New Revision: 45607 Modified: python/branches/release24-maint/Doc/lib/libcodecs.tex Log: Typo, grammar fixes Modified: python/branches/release24-maint/Doc/lib/libcodecs.tex ============================================================================== --- python/branches/release24-maint/Doc/lib/libcodecs.tex (original) +++ python/branches/release24-maint/Doc/lib/libcodecs.tex Fri Apr 21 14:59:12 2006 @@ -295,14 +295,14 @@ The \class{StreamWriter} and \class{StreamReader} classes provide generic working interfaces which can be used to implement new encodings submodules very easily. See \module{encodings.utf_8} for an -example on how this is done. +example of how this is done. \subsubsection{StreamWriter Objects \label{stream-writer-objects}} The \class{StreamWriter} class is a subclass of \class{Codec} and defines the following methods which every stream writer must define in -order to be compatible to the Python codec registry. +order to be compatible with the Python codec registry. \begin{classdesc}{StreamWriter}{stream\optional{, errors}} Constructor for a \class{StreamWriter} instance. @@ -355,14 +355,14 @@ \end{methoddesc} In addition to the above methods, the \class{StreamWriter} must also -inherit all other methods and attribute from the underlying stream. +inherit all other methods and attributes from the underlying stream. \subsubsection{StreamReader Objects \label{stream-reader-objects}} The \class{StreamReader} class is a subclass of \class{Codec} and defines the following methods which every stream reader must define in -order to be compatible to the Python codec registry. +order to be compatible with the Python codec registry. \begin{classdesc}{StreamReader}{stream\optional{, errors}} Constructor for a \class{StreamReader} instance. @@ -371,7 +371,7 @@ free to add additional keyword arguments, but only the ones defined here are used by the Python codec registry. - \var{stream} must be a file-like object open for reading (binary) + \var{stream} must be a file-like object open for reading binary data. The \class{StreamReader} may implement different error handling @@ -427,20 +427,20 @@ \var{size}, if given, is passed as size argument to the stream's \method{readline()} method. - If \var{keepends} is false lineends will be stripped from the + If \var{keepends} is false line-endings will be stripped from the lines returned. \versionchanged[\var{keepends} argument added]{2.4} \end{methoddesc} \begin{methoddesc}{readlines}{\optional{sizehint\optional{, keepends}}} - Read all lines available on the input stream and return them as list + Read all lines available on the input stream and return them as a list of lines. - Line breaks are implemented using the codec's decoder method and are + Line-endings are implemented using the codec's decoder method and are included in the list entries if \var{keepends} is true. - \var{sizehint}, if given, is passed as \var{size} argument to the + \var{sizehint}, if given, is passed as the \var{size} argument to the stream's \method{read()} method. \end{methoddesc} @@ -452,7 +452,7 @@ \end{methoddesc} In addition to the above methods, the \class{StreamReader} must also -inherit all other methods and attribute from the underlying stream. +inherit all other methods and attributes from the underlying stream. The next two base classes are included for convenience. They are not needed by the codec registry, but may provide useful in practice. @@ -478,7 +478,7 @@ \class{StreamReaderWriter} instances define the combined interfaces of \class{StreamReader} and \class{StreamWriter} classes. They inherit -all other methods and attribute from the underlying stream. +all other methods and attributes from the underlying stream. \subsubsection{StreamRecoder Objects \label{stream-recoder-objects}} @@ -504,14 +504,14 @@ \var{stream} must be a file-like object. \var{encode}, \var{decode} must adhere to the \class{Codec} - interface, \var{Reader}, \var{Writer} must be factory functions or + interface. \var{Reader}, \var{Writer} must be factory functions or classes providing objects of the \class{StreamReader} and \class{StreamWriter} interface respectively. \var{encode} and \var{decode} are needed for the frontend translation, \var{Reader} and \var{Writer} for the backend translation. The intermediate format used is determined by the two - sets of codecs, e.g. the Unicode codecs will use Unicode as + sets of codecs, e.g. the Unicode codecs will use Unicode as the intermediate encoding. Error handling is done in the same way as defined for the @@ -520,12 +520,12 @@ \class{StreamRecoder} instances define the combined interfaces of \class{StreamReader} and \class{StreamWriter} classes. They inherit -all other methods and attribute from the underlying stream. +all other methods and attributes from the underlying stream. \subsection{Standard Encodings\label{standard-encodings}} -Python comes with a number of codecs builtin, either implemented as C -functions, or with dictionaries as mapping tables. The following table +Python comes with a number of codecs built-in, either implemented as C +functions or with dictionaries as mapping tables. The following table lists the codecs by name, together with a few common aliases, and the languages for which the encoding is likely used. Neither the list of aliases nor the list of languages is meant to be exhaustive. Notice From python-checkins at python.org Fri Apr 21 15:01:47 2006 From: python-checkins at python.org (andrew.kuchling) Date: Fri, 21 Apr 2006 15:01:47 +0200 (CEST) Subject: [Python-checkins] r45608 - python/trunk/Doc/whatsnew/whatsnew25.tex Message-ID: <20060421130147.2F9211E400B@bag.python.org> Author: andrew.kuchling Date: Fri Apr 21 15:01:45 2006 New Revision: 45608 Modified: python/trunk/Doc/whatsnew/whatsnew25.tex Log: Add two items; typographical improvement for the 'with' statement; minor edits Modified: python/trunk/Doc/whatsnew/whatsnew25.tex ============================================================================== --- python/trunk/Doc/whatsnew/whatsnew25.tex (original) +++ python/trunk/Doc/whatsnew/whatsnew25.tex Fri Apr 21 15:01:45 2006 @@ -4,7 +4,6 @@ % The easy_install stuff % Describe the pkgutil module -% Stateful codec changes % Fix XXX comments % Count up the patches and bugs @@ -580,7 +579,7 @@ %====================================================================== \section{PEP 343: The 'with' statement} -The \keyword{with} statement allows a clearer version of code that +The '\keyword{with}' statement allows a clearer version of code that uses \code{try...finally} blocks to ensure that clean-up code is executed. @@ -589,7 +588,7 @@ and show how to write objects called ``context managers'' and ``contexts'' for use with this statement. -The \keyword{with} statement is a new control-flow structure whose +The '\keyword{with}' statement is a new control-flow structure whose basic structure is: \begin{verbatim} @@ -625,11 +624,11 @@ \end{verbatim} After this statement has executed, the file object in \var{f} will -have been automatically closed at this point, even if the 'for' loop +have been automatically closed, even if the 'for' loop raised an exception part-way through the block. The \module{threading} module's locks and condition variables -also support the \keyword{with} statement: +also support the '\keyword{with}' statement: \begin{verbatim} lock = threading.Lock() @@ -660,8 +659,8 @@ \subsection{Writing Context Managers} -Under the hood, the \keyword{with} statement is fairly complicated. -Most people will only use \keyword{with} in company with +Under the hood, the '\keyword{with}' statement is fairly complicated. +Most people will only use '\keyword{with}' in company with existing objects that are documented to work as context managers, and don't need to know these details, so you can skip the following section if you like. Authors of new context managers will need to understand the @@ -678,7 +677,7 @@ return a context object. \item The context's \method{__enter__()} method is called. -The value returned is assigned to \var{VAR}. If no \code{as \var{VAR}} +The value returned is assigned to \var{VAR}. If no \code{'as \var{VAR}'} clause is present, the value is simply discarded. \item The code in \var{BLOCK} is executed. @@ -690,7 +689,7 @@ controls whether the exception is re-raised: any false value re-raises the exception, and \code{True} will result in suppressing it. You'll only rarely want to suppress the exception; the -author of the code containing the \keyword{with} statement will +author of the code containing the '\keyword{with}' statement will never realize anything went wrong. \item If \var{BLOCK} didn't raise an exception, @@ -761,7 +760,7 @@ to start a new transaction. In this example, the resulting cursor object would be a useful result, so the method will return it. The user can -then add \code{as cursor} to their \keyword{with} statement +then add \code{as cursor} to their '\keyword{with}' statement to bind the cursor to a variable name. \begin{verbatim} @@ -806,7 +805,7 @@ exactly one value. The code up to the \keyword{yield} will be executed as the \method{__enter__()} method, and the value yielded will be the method's return value that will get bound to the variable -in the \keyword{with} statement's \keyword{as} clause, if any. The +in the '\keyword{with}' statement's \keyword{as} clause, if any. The code after the \keyword{yield} will be executed in the \method{__exit__()} method. Any exception raised in the block will be raised by the \keyword{yield} statement. @@ -854,7 +853,7 @@ There's a \function{nested(\var{mgr1}, \var{mgr2}, ...)} manager that combines a number of context managers so you don't need to write -nested \keyword{with} statements. This example statement does two +nested '\keyword{with}' statements. This example statement does two things, starting a database transaction and acquiring a thread lock: \begin{verbatim} @@ -880,7 +879,7 @@ \seepep{343}{The ``with'' statement}{PEP written by Guido van~Rossum and Nick Coghlan; implemented by Mike Bland, Guido van~Rossum, and -Neal Norwitz. The PEP shows the code generated for a \keyword{with} +Neal Norwitz. The PEP shows the code generated for a '\keyword{with}' statement, which can be helpful in learning how context managers work.} @@ -1092,8 +1091,8 @@ \end{verbatim} \item The \function{min()} and \function{max()} built-in functions -gained a \code{key} keyword argument analogous to the \code{key} -argument for \method{sort()}. This argument supplies a function that +gained a \code{key} keyword parameter analogous to the \code{key} +argument for \method{sort()}. This parameter supplies a function that takes a single argument and is called for every value in the list; \function{min()}/\function{max()} will return the element with the smallest/largest return value from this function. @@ -1186,7 +1185,7 @@ %====================================================================== -\section{New, Improved, and Deprecated Modules} +\section{New, Improved, and Removed Modules} The standard library received many enhancements and bug fixes in Python 2.5. Here's a partial list of the most notable changes, sorted @@ -1196,13 +1195,23 @@ \begin{itemize} -% the cPickle module no longer accepts the deprecated None option in the -% args tuple returned by __reduce__(). - \item The \module{audioop} module now supports the a-LAW encoding, and the code for u-LAW encoding has been improved. (Contributed by Lars Immisch.) +\item The \module{codecs} module gained support for incremental +codecs. The \function{codec.lookup()} function now +returns a \class{CodecInfo} instance instead of a tuple. +\class{CodecInfo} instances behave like a 4-tuple to preserve backward +compatibility but also have the attributes \member{encode}, +\member{decode}, \member{incrementalencoder}, \member{incrementaldecoder}, +\member{streamwriter}, and \member{streamreader}. Incremental codecs +can receive input and produce output in multiple chunks; the output is +the same as if the entire input was fed to the non-incremental codec. +See the \module{codecs} module documentation for details. +(Designed and implemented by Walter D\"orwald.) +% Patch 1436130 + \item The \module{collections} module gained a new type, \class{defaultdict}, that subclasses the standard \class{dict} type. The new type mostly behaves like a dictionary but constructs a @@ -1244,7 +1253,7 @@ raising \exception{ValueError} if the value isn't found. \item New module: The \module{contextlib} module contains helper functions for use -with the new \keyword{with} statement. See +with the new '\keyword{with}' statement. See section~\ref{module-contextlib} for more about this module. (Contributed by Phillip J. Eby.) @@ -1302,7 +1311,7 @@ \item The \function{nsmallest()} and \function{nlargest()} functions in the \module{heapq} module -now support a \code{key} keyword argument similar to the one +now support a \code{key} keyword parameter similar to the one provided by the \function{min()}/\function{max()} functions and the \method{sort()} methods. For example: Example: @@ -1375,14 +1384,20 @@ (Contributed by Antti Louko and Diego Petten\`o.) % (Patch 1180695, 1212117) +\item The \module{pickle} and \module{cPickle} modules no +longer accept a return value of \code{None} from the +\method{__reduce__()} method; the method must return a tuple of +arguments instead. The ability to return \code{None} was deprecated +in Python 2.4, so this completes the removal of the feature. + \item The old \module{regex} and \module{regsub} modules, which have been deprecated ever since Python 2.0, have finally been deleted. Other deleted modules: \module{statcache}, \module{tzparse}, \module{whrandom}. -\item The \file{lib-old} directory, +\item Also deleted: the \file{lib-old} directory, which includes ancient modules such as \module{dircmp} and -\module{ni}, was also deleted. \file{lib-old} wasn't on the default +\module{ni}, was removed. \file{lib-old} wasn't on the default \code{sys.path}, so unless your programs explicitly added the directory to \code{sys.path}, this removal shouldn't affect your code. @@ -1969,18 +1984,22 @@ characters but doesn't have an encoding declaration. In Python 2.4 this triggered a warning, not a syntax error. -\item The \module{pickle} module no longer uses the deprecated \var{bin} parameter. - \item Previously, the \member{gi_frame} attribute of a generator was always a frame object. Because of the \pep{342} changes described in section~\ref{section-generators}, it's now possible for \member{gi_frame} to be \code{None}. + +\item Library: The \module{pickle} and \module{cPickle} modules no +longer accept a return value of \code{None} from the +\method{__reduce__()} method; the method must return a tuple of +arguments instead. The modules also no longer accept the deprecated +\var{bin} keyword parameter. + \item C API: Many functions now use \ctype{Py_ssize_t} -instead of \ctype{int} to allow processing more data -on 64-bit machines. Extension code may need to make -the same change to avoid warnings and to support 64-bit machines. -See the earlier +instead of \ctype{int} to allow processing more data on 64-bit +machines. Extension code may need to make the same change to avoid +warnings and to support 64-bit machines. See the earlier section~\ref{section-353} for a discussion of this change. \item C API: From python-checkins at python.org Fri Apr 21 15:08:03 2006 From: python-checkins at python.org (andrew.kuchling) Date: Fri, 21 Apr 2006 15:08:03 +0200 (CEST) Subject: [Python-checkins] r45609 - python/trunk/Doc/whatsnew/whatsnew25.tex Message-ID: <20060421130803.7A1F01E400B@bag.python.org> Author: andrew.kuchling Date: Fri Apr 21 15:08:02 2006 New Revision: 45609 Modified: python/trunk/Doc/whatsnew/whatsnew25.tex Log: Add \label to make better HTML filenames Modified: python/trunk/Doc/whatsnew/whatsnew25.tex ============================================================================== --- python/trunk/Doc/whatsnew/whatsnew25.tex (original) +++ python/trunk/Doc/whatsnew/whatsnew25.tex Fri Apr 21 15:08:02 2006 @@ -34,7 +34,7 @@ %====================================================================== -\section{PEP 243: Uploading Modules to PyPI} +\section{PEP 243: Uploading Modules to PyPI\label{pep-243}} PEP 243 describes an HTTP-based protocol for submitting software packages to a central archive. The Python package index at @@ -60,7 +60,7 @@ %====================================================================== -\section{PEP 308: Conditional Expressions} +\section{PEP 308: Conditional Expressions\label{pep-308}} For a long time, people have been requesting a way to write conditional expressions, expressions that return value A or value B @@ -151,7 +151,7 @@ %====================================================================== -\section{PEP 309: Partial Function Application} +\section{PEP 309: Partial Function Application\label{pep-309}} The \module{functional} module is intended to contain tools for functional-style programming. Currently it only contains a @@ -213,7 +213,7 @@ %====================================================================== -\section{PEP 314: Metadata for Python Software Packages v1.1} +\section{PEP 314: Metadata for Python Software Packages v1.1\label{pep-314}} Some simple dependency support was added to Distutils. The \function{setup()} function now has \code{requires}, \code{provides}, @@ -247,7 +247,7 @@ %====================================================================== -\section{PEP 328: Absolute and Relative Imports} +\section{PEP 328: Absolute and Relative Imports\label{pep-328}} The simpler part of PEP 328 was implemented in Python 2.4: parentheses could now be used to enclose the names imported from a module using @@ -341,7 +341,7 @@ %====================================================================== -\section{PEP 338: Executing Modules as Scripts} +\section{PEP 338: Executing Modules as Scripts\label{pep-338}} The \programopt{-m} switch added in Python 2.4 to execute a module as a script gained a few more abilities. Instead of being implemented in @@ -365,7 +365,7 @@ %====================================================================== -\section{PEP 341: Unified try/except/finally} +\section{PEP 341: Unified try/except/finally\label{pep-341}} Until Python 2.5, the \keyword{try} statement came in two flavours. You could use a \keyword{finally} block to ensure that code @@ -411,7 +411,7 @@ %====================================================================== -\section{PEP 342: New Generator Features\label{section-generators}} +\section{PEP 342: New Generator Features\label{pep-342}} Python 2.5 adds a simple way to pass values \emph{into} a generator. As introduced in Python 2.3, generators only produce output; once a @@ -577,7 +577,7 @@ %====================================================================== -\section{PEP 343: The 'with' statement} +\section{PEP 343: The 'with' statement\label{pep-343}} The '\keyword{with}' statement allows a clearer version of code that uses \code{try...finally} blocks to ensure that clean-up code is @@ -657,7 +657,7 @@ print v1.sqrt() \end{verbatim} -\subsection{Writing Context Managers} +\subsection{Writing Context Managers\label{context-managers}} Under the hood, the '\keyword{with}' statement is fairly complicated. Most people will only use '\keyword{with}' in company with @@ -890,7 +890,7 @@ %====================================================================== -\section{PEP 352: Exceptions as New-Style Classes} +\section{PEP 352: Exceptions as New-Style Classes\label{pep-352}} Exception classes can now be new-style classes, not just classic classes, and the built-in \exception{Exception} class and all the @@ -956,7 +956,7 @@ %====================================================================== -\section{PEP 353: Using ssize_t as the index type\label{section-353}} +\section{PEP 353: Using ssize_t as the index type\label{pep-353}} A wide-ranging change to Python's C API, using a new \ctype{Py_ssize_t} type definition instead of \ctype{int}, @@ -1018,7 +1018,7 @@ %====================================================================== -\section{PEP 357: The '__index__' method} +\section{PEP 357: The '__index__' method\label{pep-357}} The NumPy developers had a problem that could only be solved by adding a new special method, \method{__index__}. When using slice notation, @@ -1839,7 +1839,7 @@ \item The largest change to the C API came from \pep{353}, which modifies the interpreter to use a \ctype{Py_ssize_t} type definition instead of \ctype{int}. See the earlier -section~\ref{section-353} for a discussion of this change. +section~\ref{pep-353} for a discussion of this change. \item The design of the bytecode compiler has changed a great deal, to no longer generate bytecode by traversing the parse tree. Instead @@ -1986,7 +1986,7 @@ \item Previously, the \member{gi_frame} attribute of a generator was always a frame object. Because of the \pep{342} changes -described in section~\ref{section-generators}, it's now possible +described in section~\ref{pep-342}, it's now possible for \member{gi_frame} to be \code{None}. @@ -2000,7 +2000,7 @@ instead of \ctype{int} to allow processing more data on 64-bit machines. Extension code may need to make the same change to avoid warnings and to support 64-bit machines. See the earlier -section~\ref{section-353} for a discussion of this change. +section~\ref{pep-353} for a discussion of this change. \item C API: The obmalloc changes mean that From python-checkins at python.org Fri Apr 21 15:26:43 2006 From: python-checkins at python.org (andrew.kuchling) Date: Fri, 21 Apr 2006 15:26:43 +0200 (CEST) Subject: [Python-checkins] r45610 - python/trunk/Doc/whatsnew/whatsnew25.tex Message-ID: <20060421132643.15FF41E400C@bag.python.org> Author: andrew.kuchling Date: Fri Apr 21 15:26:42 2006 New Revision: 45610 Modified: python/trunk/Doc/whatsnew/whatsnew25.tex Log: Add item Modified: python/trunk/Doc/whatsnew/whatsnew25.tex ============================================================================== --- python/trunk/Doc/whatsnew/whatsnew25.tex (original) +++ python/trunk/Doc/whatsnew/whatsnew25.tex Fri Apr 21 15:26:42 2006 @@ -1266,6 +1266,11 @@ module's interface, will continue to be maintained in future versions of Python. (Contributed by Armin Rigo.) +Also, the \module{pstats} module used to analyze the data measured by +the profiler now supports directing the output to any file stream +by supplying a \var{stream} argument to the \class{Stats} constructor. +(Contributed by Skip Montanaro.) + \item The \module{csv} module, which parses files in comma-separated value format, received several enhancements and a number of bugfixes. You can now set the maximum size in bytes of a From python-checkins at python.org Fri Apr 21 15:48:27 2006 From: python-checkins at python.org (martin.v.loewis) Date: Fri, 21 Apr 2006 15:48:27 +0200 (CEST) Subject: [Python-checkins] r45612 - python/trunk/PCbuild/_bsddb.vcproj python/trunk/PCbuild/_ctypes.vcproj python/trunk/PCbuild/_ctypes_test.vcproj python/trunk/PCbuild/_elementtree.vcproj python/trunk/PCbuild/_msi.vcproj python/trunk/PCbuild/_socket.vcproj python/trunk/PCbuild/_sqlite3.vcproj python/trunk/PCbuild/_testcapi.vcproj python/trunk/PCbuild/_tkinter.vcproj python/trunk/PCbuild/bz2.vcproj python/trunk/PCbuild/pyexpat.vcproj python/trunk/PCbuild/python.vcproj python/trunk/PCbuild/pythoncore.vcproj python/trunk/PCbuild/pythonw.vcproj python/trunk/PCbuild/readme.txt python/trunk/PCbuild/select.vcproj python/trunk/PCbuild/unicodedata.vcproj python/trunk/PCbuild/winsound.vcproj Message-ID: <20060421134827.8D74B1E400B@bag.python.org> Author: martin.v.loewis Date: Fri Apr 21 15:48:26 2006 New Revision: 45612 Modified: python/trunk/PCbuild/_bsddb.vcproj python/trunk/PCbuild/_ctypes.vcproj python/trunk/PCbuild/_ctypes_test.vcproj python/trunk/PCbuild/_elementtree.vcproj python/trunk/PCbuild/_msi.vcproj python/trunk/PCbuild/_socket.vcproj python/trunk/PCbuild/_sqlite3.vcproj python/trunk/PCbuild/_testcapi.vcproj python/trunk/PCbuild/_tkinter.vcproj python/trunk/PCbuild/bz2.vcproj python/trunk/PCbuild/pyexpat.vcproj python/trunk/PCbuild/python.vcproj python/trunk/PCbuild/pythoncore.vcproj python/trunk/PCbuild/pythonw.vcproj python/trunk/PCbuild/readme.txt python/trunk/PCbuild/select.vcproj python/trunk/PCbuild/unicodedata.vcproj python/trunk/PCbuild/winsound.vcproj Log: Upgrade to vsextcomp 0.8 (and thus the SDK for W2k3SP1) Modified: python/trunk/PCbuild/_bsddb.vcproj ============================================================================== --- python/trunk/PCbuild/_bsddb.vcproj (original) +++ python/trunk/PCbuild/_bsddb.vcproj Fri Apr 21 15:48:26 2006 @@ -133,7 +133,7 @@ AdditionalOptions=" /USECL:MS_ITANIUM" Optimization="2" InlineFunctionExpansion="1" - AdditionalIncludeDirectories="{MSSDKPATH}\include\Win64\atl;{MSSDKPATH}\include\Win64\crt;{MSSDKPATH}\include\Win64\crt\sys;{MSSDKPATH}\include\Win64\mfc;..\Include;..\PC;"..\..\db-4.4.20\build_win32"" + AdditionalIncludeDirectories="..\Include;..\PC;"..\..\db-4.4.20\build_win32"" PreprocessorDefinitions="NDEBUG;WIN32;_WINDOWS" StringPooling="TRUE" BasicRuntimeChecks="0" @@ -195,7 +195,7 @@ AdditionalOptions=" /USECL:MS_OPTERON" Optimization="2" InlineFunctionExpansion="1" - AdditionalIncludeDirectories="{MSSDKPATH}\include\Win64\atl\amd64;{MSSDKPATH}\include\Win64\crt\amd64;{MSSDKPATH}\include\Win64\crt\amd64\sys;{MSSDKPATH}\include\Win64\mfc\amd64;..\Include;..\PC;"..\..\db-4.4.20\build_win32"" + AdditionalIncludeDirectories="..\Include;..\PC;"..\..\db-4.4.20\build_win32"" PreprocessorDefinitions="NDEBUG;WIN32;_WINDOWS" StringPooling="TRUE" BasicRuntimeChecks="0" Modified: python/trunk/PCbuild/_ctypes.vcproj ============================================================================== --- python/trunk/PCbuild/_ctypes.vcproj (original) +++ python/trunk/PCbuild/_ctypes.vcproj Fri Apr 21 15:48:26 2006 @@ -130,7 +130,7 @@ Name="VCCLCompilerTool" AdditionalOptions=" /USECL:MS_OPTERON" InlineFunctionExpansion="1" - AdditionalIncludeDirectories="{MSSDKPATH}\include\Win64\atl\amd64;{MSSDKPATH}\include\Win64\crt\amd64;{MSSDKPATH}\include\Win64\crt\amd64\sys;{MSSDKPATH}\include\Win64\mfc\amd64;..\Include,..\PC,..\Modules\_ctypes\libffi_msvc" + AdditionalIncludeDirectories="..\Include,..\PC,..\Modules\_ctypes\libffi_msvc" PreprocessorDefinitions="WIN32;NDEBUG;_WINDOWS" StringPooling="TRUE" BasicRuntimeChecks="0" @@ -189,7 +189,7 @@ Name="VCCLCompilerTool" AdditionalOptions=" /USECL:MS_ITANIUM" InlineFunctionExpansion="1" - AdditionalIncludeDirectories="{MSSDKPATH}\include\Win64\atl;{MSSDKPATH}\include\Win64\crt;{MSSDKPATH}\include\Win64\crt\sys;{MSSDKPATH}\include\Win64\mfc;..\Include,..\PC,..\Modules\_ctypes\libffi_msvc" + AdditionalIncludeDirectories="..\Include,..\PC,..\Modules\_ctypes\libffi_msvc" PreprocessorDefinitions="WIN32;NDEBUG;_WINDOWS" StringPooling="TRUE" BasicRuntimeChecks="0" Modified: python/trunk/PCbuild/_ctypes_test.vcproj ============================================================================== --- python/trunk/PCbuild/_ctypes_test.vcproj (original) +++ python/trunk/PCbuild/_ctypes_test.vcproj Fri Apr 21 15:48:26 2006 @@ -126,7 +126,7 @@ Name="VCCLCompilerTool" AdditionalOptions=" /USECL:MS_ITANIUM" Optimization="0" - AdditionalIncludeDirectories="{MSSDKPATH}\include\Win64\atl;{MSSDKPATH}\include\Win64\crt;{MSSDKPATH}\include\Win64\crt\sys;{MSSDKPATH}\include\Win64\mfc;..\Include,..\PC" + AdditionalIncludeDirectories="..\Include,..\PC" PreprocessorDefinitions="WIN32;_DEBUG;_WINDOWS" MinimalRebuild="FALSE" BasicRuntimeChecks="0" @@ -181,7 +181,7 @@ Name="VCCLCompilerTool" AdditionalOptions=" /USECL:MS_OPTERON" InlineFunctionExpansion="1" - AdditionalIncludeDirectories="{MSSDKPATH}\include\Win64\atl\amd64;{MSSDKPATH}\include\Win64\crt\amd64;{MSSDKPATH}\include\Win64\crt\amd64\sys;{MSSDKPATH}\include\Win64\mfc\amd64;..\Include,..\PC" + AdditionalIncludeDirectories="..\Include,..\PC" PreprocessorDefinitions="WIN32;NDEBUG;_WINDOWS" StringPooling="TRUE" BasicRuntimeChecks="0" Modified: python/trunk/PCbuild/_elementtree.vcproj ============================================================================== --- python/trunk/PCbuild/_elementtree.vcproj (original) +++ python/trunk/PCbuild/_elementtree.vcproj Fri Apr 21 15:48:26 2006 @@ -132,7 +132,7 @@ AdditionalOptions=" /USECL:MS_ITANIUM" Optimization="2" InlineFunctionExpansion="1" - AdditionalIncludeDirectories="{MSSDKPATH}\include\Win64\atl;{MSSDKPATH}\include\Win64\crt;{MSSDKPATH}\include\Win64\crt\sys;{MSSDKPATH}\include\Win64\mfc;..\Include,..\PC,..\Modules\expat" + AdditionalIncludeDirectories="..\Include,..\PC,..\Modules\expat" PreprocessorDefinitions="NDEBUG;WIN32;_WINDOWS;XML_NS;XML_DTD;BYTEORDER=1234;XML_CONTEXT_BYTES=1024;USE_PYEXPAT_CAPI;XML_STATIC;HAVE_MEMMOVE" StringPooling="TRUE" BasicRuntimeChecks="0" @@ -193,7 +193,7 @@ AdditionalOptions=" /USECL:MS_OPTERON" Optimization="2" InlineFunctionExpansion="1" - AdditionalIncludeDirectories="{MSSDKPATH}\include\Win64\atl\amd64;{MSSDKPATH}\include\Win64\crt\amd64;{MSSDKPATH}\include\Win64\crt\amd64\sys;{MSSDKPATH}\include\Win64\mfc\amd64;..\Include,..\PC,..\Modules\expat" + AdditionalIncludeDirectories="..\Include,..\PC,..\Modules\expat" PreprocessorDefinitions="NDEBUG;WIN32;_WINDOWS;XML_NS;XML_DTD;BYTEORDER=1234;XML_CONTEXT_BYTES=1024;USE_PYEXPAT_CAPI;XML_STATIC;HAVE_MEMMOVE" StringPooling="TRUE" BasicRuntimeChecks="0" Modified: python/trunk/PCbuild/_msi.vcproj ============================================================================== --- python/trunk/PCbuild/_msi.vcproj (original) +++ python/trunk/PCbuild/_msi.vcproj Fri Apr 21 15:48:26 2006 @@ -132,7 +132,7 @@ AdditionalOptions=" /USECL:MS_ITANIUM" Optimization="2" InlineFunctionExpansion="1" - AdditionalIncludeDirectories="{MSSDKPATH}\include\Win64\atl;{MSSDKPATH}\include\Win64\crt;{MSSDKPATH}\include\Win64\crt\sys;{MSSDKPATH}\include\Win64\mfc;..\Include,..\PC" + AdditionalIncludeDirectories="..\Include,..\PC" PreprocessorDefinitions="WIN32;NDEBUG;_WINDOWS;_USRDLL" StringPooling="TRUE" BasicRuntimeChecks="0" @@ -192,7 +192,7 @@ AdditionalOptions=" /USECL:MS_OPTERON" Optimization="2" InlineFunctionExpansion="1" - AdditionalIncludeDirectories="{MSSDKPATH}\include\Win64\atl\amd64;{MSSDKPATH}\include\Win64\crt\amd64;{MSSDKPATH}\include\Win64\crt\amd64\sys;{MSSDKPATH}\include\Win64\mfc\amd64;..\Include,..\PC" + AdditionalIncludeDirectories="..\Include,..\PC" PreprocessorDefinitions="WIN32;NDEBUG;_WINDOWS;_USRDLL" StringPooling="TRUE" BasicRuntimeChecks="0" Modified: python/trunk/PCbuild/_socket.vcproj ============================================================================== --- python/trunk/PCbuild/_socket.vcproj (original) +++ python/trunk/PCbuild/_socket.vcproj Fri Apr 21 15:48:26 2006 @@ -131,7 +131,7 @@ AdditionalOptions=" /USECL:MS_ITANIUM" Optimization="2" InlineFunctionExpansion="1" - AdditionalIncludeDirectories="{MSSDKPATH}\include\Win64\atl;{MSSDKPATH}\include\Win64\crt;{MSSDKPATH}\include\Win64\crt\sys;{MSSDKPATH}\include\Win64\mfc;..\Include,..\PC" + AdditionalIncludeDirectories="..\Include,..\PC" PreprocessorDefinitions="NDEBUG;WIN32;_WINDOWS" StringPooling="TRUE" BasicRuntimeChecks="0" @@ -192,7 +192,7 @@ AdditionalOptions=" /USECL:MS_OPTERON" Optimization="2" InlineFunctionExpansion="1" - AdditionalIncludeDirectories="{MSSDKPATH}\include\Win64\atl\amd64;{MSSDKPATH}\include\Win64\crt\amd64;{MSSDKPATH}\include\Win64\crt\amd64\sys;{MSSDKPATH}\include\Win64\mfc\amd64;..\Include,..\PC" + AdditionalIncludeDirectories="..\Include,..\PC" PreprocessorDefinitions="NDEBUG;WIN32;_WINDOWS" StringPooling="TRUE" BasicRuntimeChecks="0" Modified: python/trunk/PCbuild/_sqlite3.vcproj ============================================================================== --- python/trunk/PCbuild/_sqlite3.vcproj (original) +++ python/trunk/PCbuild/_sqlite3.vcproj Fri Apr 21 15:48:26 2006 @@ -134,7 +134,7 @@ AdditionalOptions=" /USECL:MS_ITANIUM" Optimization="2" InlineFunctionExpansion="1" - AdditionalIncludeDirectories="{MSSDKPATH}\include\Win64\atl;{MSSDKPATH}\include\Win64\crt;{MSSDKPATH}\include\Win64\crt\sys;{MSSDKPATH}\include\Win64\mfc;..\Include;..\PC;..\..\sqlite-source-3.3.4" + AdditionalIncludeDirectories="..\Include;..\PC;..\..\sqlite-source-3.3.4" PreprocessorDefinitions="NDEBUG;WIN32;_WINDOWS;MODULE_NAME=\"sqlite3\"" StringPooling="TRUE" BasicRuntimeChecks="0" @@ -196,7 +196,7 @@ AdditionalOptions=" /USECL:MS_OPTERON" Optimization="2" InlineFunctionExpansion="1" - AdditionalIncludeDirectories="{MSSDKPATH}\include\Win64\atl\amd64;{MSSDKPATH}\include\Win64\crt\amd64;{MSSDKPATH}\include\Win64\crt\amd64\sys;{MSSDKPATH}\include\Win64\mfc\amd64;..\Include;..\PC;..\..\sqlite-source-3.3.4" + AdditionalIncludeDirectories="..\Include;..\PC;..\..\sqlite-source-3.3.4" PreprocessorDefinitions="NDEBUG;WIN32;_WINDOWS;MODULE_NAME=\"sqlite3\"" StringPooling="TRUE" BasicRuntimeChecks="0" Modified: python/trunk/PCbuild/_testcapi.vcproj ============================================================================== --- python/trunk/PCbuild/_testcapi.vcproj (original) +++ python/trunk/PCbuild/_testcapi.vcproj Fri Apr 21 15:48:26 2006 @@ -129,7 +129,7 @@ AdditionalOptions=" /USECL:MS_ITANIUM" Optimization="2" InlineFunctionExpansion="1" - AdditionalIncludeDirectories="{MSSDKPATH}\include\Win64\atl;{MSSDKPATH}\include\Win64\crt;{MSSDKPATH}\include\Win64\crt\sys;{MSSDKPATH}\include\Win64\mfc;..\Include,..\PC" + AdditionalIncludeDirectories="..\Include,..\PC" PreprocessorDefinitions="WIN32;NDEBUG;_WINDOWS;_USRDLL;MMAP_EXPORTS" StringPooling="TRUE" BasicRuntimeChecks="0" @@ -188,7 +188,7 @@ AdditionalOptions=" /USECL:MS_OPTERON" Optimization="2" InlineFunctionExpansion="1" - AdditionalIncludeDirectories="{MSSDKPATH}\include\Win64\atl\amd64;{MSSDKPATH}\include\Win64\crt\amd64;{MSSDKPATH}\include\Win64\crt\amd64\sys;{MSSDKPATH}\include\Win64\mfc\amd64;..\Include,..\PC" + AdditionalIncludeDirectories="..\Include,..\PC" PreprocessorDefinitions="WIN32;NDEBUG;_WINDOWS;_USRDLL;MMAP_EXPORTS" StringPooling="TRUE" BasicRuntimeChecks="0" Modified: python/trunk/PCbuild/_tkinter.vcproj ============================================================================== --- python/trunk/PCbuild/_tkinter.vcproj (original) +++ python/trunk/PCbuild/_tkinter.vcproj Fri Apr 21 15:48:26 2006 @@ -133,7 +133,7 @@ AdditionalOptions=" /USECL:MS_ITANIUM" Optimization="2" InlineFunctionExpansion="1" - AdditionalIncludeDirectories="{MSSDKPATH}\include\Win64\atl;{MSSDKPATH}\include\Win64\crt;{MSSDKPATH}\include\Win64\crt\sys;{MSSDKPATH}\include\Win64\mfc;..\..\tcltk\include,..\Include,..\PC" + AdditionalIncludeDirectories="..\..\tcltk\include,..\Include,..\PC" PreprocessorDefinitions="NDEBUG;WIN32;_WINDOWS;WITH_APPINIT" StringPooling="TRUE" BasicRuntimeChecks="0" @@ -195,7 +195,7 @@ AdditionalOptions=" /USECL:MS_OPTERON" Optimization="2" InlineFunctionExpansion="1" - AdditionalIncludeDirectories="{MSSDKPATH}\include\Win64\atl\amd64;{MSSDKPATH}\include\Win64\crt\amd64;{MSSDKPATH}\include\Win64\crt\amd64\sys;{MSSDKPATH}\include\Win64\mfc\amd64;..\..\tcltk\include,..\Include,..\PC" + AdditionalIncludeDirectories="..\..\tcltk\include,..\Include,..\PC" PreprocessorDefinitions="NDEBUG;WIN32;_WINDOWS;WITH_APPINIT" StringPooling="TRUE" BasicRuntimeChecks="0" Modified: python/trunk/PCbuild/bz2.vcproj ============================================================================== --- python/trunk/PCbuild/bz2.vcproj (original) +++ python/trunk/PCbuild/bz2.vcproj Fri Apr 21 15:48:26 2006 @@ -140,7 +140,7 @@ AdditionalOptions=" /USECL:MS_ITANIUM" Optimization="2" InlineFunctionExpansion="1" - AdditionalIncludeDirectories="{MSSDKPATH}\include\Win64\atl;{MSSDKPATH}\include\Win64\crt;{MSSDKPATH}\include\Win64\crt\sys;{MSSDKPATH}\include\Win64\mfc;..\Include,..\PC,..\..\bzip2-1.0.3" + AdditionalIncludeDirectories="..\Include,..\PC,..\..\bzip2-1.0.3" PreprocessorDefinitions="NDEBUG;WIN32;_WINDOWS" StringPooling="TRUE" BasicRuntimeChecks="0" @@ -205,7 +205,7 @@ AdditionalOptions=" /USECL:MS_OPTERON" Optimization="2" InlineFunctionExpansion="1" - AdditionalIncludeDirectories="{MSSDKPATH}\include\Win64\atl\amd64;{MSSDKPATH}\include\Win64\crt\amd64;{MSSDKPATH}\include\Win64\crt\amd64\sys;{MSSDKPATH}\include\Win64\mfc\amd64;..\Include,..\PC,..\..\bzip2-1.0.3" + AdditionalIncludeDirectories="..\Include,..\PC,..\..\bzip2-1.0.3" PreprocessorDefinitions="NDEBUG;WIN32;_WINDOWS" StringPooling="TRUE" BasicRuntimeChecks="0" Modified: python/trunk/PCbuild/pyexpat.vcproj ============================================================================== --- python/trunk/PCbuild/pyexpat.vcproj (original) +++ python/trunk/PCbuild/pyexpat.vcproj Fri Apr 21 15:48:26 2006 @@ -131,7 +131,7 @@ AdditionalOptions=" /USECL:MS_ITANIUM" Optimization="2" InlineFunctionExpansion="1" - AdditionalIncludeDirectories="{MSSDKPATH}\include\Win64\atl;{MSSDKPATH}\include\Win64\crt;{MSSDKPATH}\include\Win64\crt\sys;{MSSDKPATH}\include\Win64\mfc;..\Include,..\PC,..\Modules\expat" + AdditionalIncludeDirectories="..\Include,..\PC,..\Modules\expat" PreprocessorDefinitions="NDEBUG;WIN32;_WINDOWS;XML_NS;XML_DTD;BYTEORDER=1234;XML_CONTEXT_BYTES=1024;XML_STATIC;HAVE_MEMMOVE" StringPooling="TRUE" BasicRuntimeChecks="0" @@ -192,7 +192,7 @@ AdditionalOptions=" /USECL:MS_OPTERON" Optimization="2" InlineFunctionExpansion="1" - AdditionalIncludeDirectories="{MSSDKPATH}\include\Win64\atl\amd64;{MSSDKPATH}\include\Win64\crt\amd64;{MSSDKPATH}\include\Win64\crt\amd64\sys;{MSSDKPATH}\include\Win64\mfc\amd64;..\Include,..\PC,..\Modules\expat" + AdditionalIncludeDirectories="..\Include,..\PC,..\Modules\expat" PreprocessorDefinitions="NDEBUG;WIN32;_WINDOWS;XML_NS;XML_DTD;BYTEORDER=1234;XML_CONTEXT_BYTES=1024;XML_STATIC;HAVE_MEMMOVE" StringPooling="TRUE" BasicRuntimeChecks="0" Modified: python/trunk/PCbuild/python.vcproj ============================================================================== --- python/trunk/PCbuild/python.vcproj (original) +++ python/trunk/PCbuild/python.vcproj Fri Apr 21 15:48:26 2006 @@ -140,7 +140,7 @@ AdditionalOptions=" /USECL:MS_ITANIUM /VSEXTCOMP_VERBOSE" Optimization="2" InlineFunctionExpansion="1" - AdditionalIncludeDirectories="{MSSDKPATH}\include\Win64\atl;{MSSDKPATH}\include\Win64\crt;{MSSDKPATH}\include\Win64\crt\sys;{MSSDKPATH}\include\Win64\mfc;..\Include,..\PC" + AdditionalIncludeDirectories="..\Include,..\PC" PreprocessorDefinitions="WIN32;NDEBUG;_CONSOLE" StringPooling="TRUE" BasicRuntimeChecks="0" @@ -204,7 +204,7 @@ AdditionalOptions=" /USECL:MS_OPTERON" Optimization="2" InlineFunctionExpansion="1" - AdditionalIncludeDirectories="{MSSDKPATH}\include\Win64\atl\amd64;{MSSDKPATH}\include\Win64\crt\amd64;{MSSDKPATH}\include\Win64\crt\amd64\sys;{MSSDKPATH}\include\Win64\mfc\amd64;..\Include,..\PC" + AdditionalIncludeDirectories="..\Include,..\PC" PreprocessorDefinitions="WIN32;NDEBUG;_CONSOLE" StringPooling="TRUE" BasicRuntimeChecks="0" Modified: python/trunk/PCbuild/pythoncore.vcproj ============================================================================== --- python/trunk/PCbuild/pythoncore.vcproj (original) +++ python/trunk/PCbuild/pythoncore.vcproj Fri Apr 21 15:48:26 2006 @@ -147,7 +147,7 @@ AdditionalOptions=" /USECL:MS_ITANIUM" Optimization="2" InlineFunctionExpansion="1" - AdditionalIncludeDirectories="{MSSDKPATH}\include\Win64\atl;{MSSDKPATH}\include\Win64\crt;{MSSDKPATH}\include\Win64\crt\sys;{MSSDKPATH}\include\Win64\mfc;..\Include,..\PC" + AdditionalIncludeDirectories="..\Include,..\PC" PreprocessorDefinitions="NDEBUG;WIN32;_WINDOWS;USE_DL_EXPORT" StringPooling="TRUE" BasicRuntimeChecks="0" @@ -214,7 +214,7 @@ AdditionalOptions="/Zm200 /USECL:MS_OPTERON" Optimization="2" InlineFunctionExpansion="1" - AdditionalIncludeDirectories="{MSSDKPATH}\include\Win64\atl\amd64;{MSSDKPATH}\include\Win64\crt\amd64;{MSSDKPATH}\include\Win64\crt\amd64\sys;{MSSDKPATH}\include\Win64\mfc\amd64;..\Include,..\PC" + AdditionalIncludeDirectories="..\Include,..\PC" PreprocessorDefinitions="NDEBUG;WIN32;_WINDOWS;USE_DL_EXPORT" StringPooling="TRUE" BasicRuntimeChecks="0" Modified: python/trunk/PCbuild/pythonw.vcproj ============================================================================== --- python/trunk/PCbuild/pythonw.vcproj (original) +++ python/trunk/PCbuild/pythonw.vcproj Fri Apr 21 15:48:26 2006 @@ -133,7 +133,7 @@ AdditionalOptions=" /USECL:MS_ITANIUM" Optimization="2" InlineFunctionExpansion="1" - AdditionalIncludeDirectories="{MSSDKPATH}\include\Win64\atl;{MSSDKPATH}\include\Win64\crt;{MSSDKPATH}\include\Win64\crt\sys;{MSSDKPATH}\include\Win64\mfc;..\Include,..\PC" + AdditionalIncludeDirectories="..\Include,..\PC" PreprocessorDefinitions="WIN32;NDEBUG;_WINDOWS" StringPooling="TRUE" BasicRuntimeChecks="0" @@ -195,7 +195,7 @@ AdditionalOptions=" /USECL:MS_OPTERON" Optimization="2" InlineFunctionExpansion="1" - AdditionalIncludeDirectories="{MSSDKPATH}\include\Win64\atl\amd64;{MSSDKPATH}\include\Win64\crt\amd64;{MSSDKPATH}\include\Win64\crt\amd64\sys;{MSSDKPATH}\include\Win64\mfc\amd64;..\Include,..\PC" + AdditionalIncludeDirectories="..\Include,..\PC" PreprocessorDefinitions="WIN32;NDEBUG;_WINDOWS" StringPooling="TRUE" BasicRuntimeChecks="0" Modified: python/trunk/PCbuild/readme.txt ============================================================================== --- python/trunk/PCbuild/readme.txt (original) +++ python/trunk/PCbuild/readme.txt Fri Apr 21 15:48:26 2006 @@ -264,7 +264,7 @@ In addition, you need the Visual Studio plugin for external C compilers, from http://sf.net/projects/vsextcomp. The plugin will wrap cl.exe, to locate the proper target compiler, and convert compiler options -accordingly. +accordingly. The project files require atleast version 0.8. Building for AMD64 ------------------ Modified: python/trunk/PCbuild/select.vcproj ============================================================================== --- python/trunk/PCbuild/select.vcproj (original) +++ python/trunk/PCbuild/select.vcproj Fri Apr 21 15:48:26 2006 @@ -21,7 +21,7 @@ Name="VCCLCompilerTool" Optimization="2" InlineFunctionExpansion="1" - AdditionalIncludeDirectories="..\Include,..\PC,..\..\select113" + AdditionalIncludeDirectories="..\Include,..\PC" PreprocessorDefinitions="NDEBUG;WIN32;_WINDOWS" StringPooling="TRUE" RuntimeLibrary="2" @@ -77,7 +77,7 @@ Author: thomas.wouters Date: Fri Apr 21 15:54:43 2006 New Revision: 45614 Modified: python/trunk/Objects/stringobject.c Log: Py_ssize_t issue; repr()'ing a very large string would result in a teensy string, because of a cast to int. Modified: python/trunk/Objects/stringobject.c ============================================================================== --- python/trunk/Objects/stringobject.c (original) +++ python/trunk/Objects/stringobject.c Fri Apr 21 15:54:43 2006 @@ -865,7 +865,7 @@ *p++ = quote; *p = '\0'; _PyString_Resize( - &v, (int) (p - PyString_AS_STRING(v))); + &v, (p - PyString_AS_STRING(v))); return v; } } From jimjjewett at gmail.com Fri Apr 21 15:56:09 2006 From: jimjjewett at gmail.com (Jim Jewett) Date: Fri, 21 Apr 2006 09:56:09 -0400 Subject: [Python-checkins] r45590 - python/trunk/Modules/gcmodule.c In-Reply-To: <20060421013341.29E681E400B@bag.python.org> References: <20060421013341.29E681E400B@bag.python.org> Message-ID: Is there some reason the import is not also inside the if (debug & DEBUG_STATS) { guard clause? For example: if (debug & DEBUG_STATS) { + if (tmod == NULL) { + tmod = PyImport_ImportModule("time"); + if (tmod == NULL) + PyErr_Clear(); + } + if (tmod != NULL) { + PyObject *f = PyObject_CallMethod(tmod, "time", NULL); + if (f == NULL) { + PyErr_Clear(); + } + else { + t1 = PyFloat_AsDouble(f); + Py_DECREF(f); + } + } PySys_WriteStderr("gc: collecting generation %d...\n", generation); On 4/20/06, skip.montanaro wrote: > Author: skip.montanaro > Date: Fri Apr 21 03:33:40 2006 > New Revision: 45590 > > Modified: > python/trunk/Modules/gcmodule.c > Log: > This is a long-ago patch I submitted to SF (1100924) to time the gc passes. > Barry approved it awhile ago. Been sitting in my sandbox for awhile as > well. > > > Modified: python/trunk/Modules/gcmodule.c > ============================================================================== > --- python/trunk/Modules/gcmodule.c (original) > +++ python/trunk/Modules/gcmodule.c Fri Apr 21 03:33:40 2006 > @@ -734,6 +734,8 @@ > PyGC_Head unreachable; /* non-problematic unreachable trash */ > PyGC_Head finalizers; /* objects with, & reachable from, __del__ */ > PyGC_Head *gc; > + static PyObject *tmod = NULL; > + double t1 = 0.0; > > if (delstr == NULL) { > delstr = PyString_InternFromString("__del__"); > @@ -741,7 +743,23 @@ > Py_FatalError("gc couldn't allocate \"__del__\""); > } > > + if (tmod == NULL) { > + tmod = PyImport_ImportModule("time"); > + if (tmod == NULL) > + PyErr_Clear(); > + } > + > if (debug & DEBUG_STATS) { > + if (tmod != NULL) { > + PyObject *f = PyObject_CallMethod(tmod, "time", NULL); > + if (f == NULL) { > + PyErr_Clear(); > + } > + else { > + t1 = PyFloat_AsDouble(f); > + Py_DECREF(f); > + } > + } > PySys_WriteStderr("gc: collecting generation %d...\n", > generation); > PySys_WriteStderr("gc: objects in each generation:"); > @@ -814,6 +832,17 @@ > if (debug & DEBUG_COLLECTABLE) { > debug_cycle("collectable", FROM_GC(gc)); > } > + if (tmod != NULL && (debug & DEBUG_STATS)) { > + PyObject *f = PyObject_CallMethod(tmod, "time", NULL); > + if (f == NULL) { > + PyErr_Clear(); > + } > + else { > + t1 = PyFloat_AsDouble(f)-t1; > + Py_DECREF(f); > + PySys_WriteStderr("gc: %.4fs elapsed.\n", t1); > + } > + } > } > > /* Clear weakrefs and invoke callbacks as necessary. */ > _______________________________________________ > Python-checkins mailing list > Python-checkins at python.org > http://mail.python.org/mailman/listinfo/python-checkins > From buildbot at python.org Fri Apr 21 16:06:18 2006 From: buildbot at python.org (buildbot at python.org) Date: Fri, 21 Apr 2006 14:06:18 +0000 Subject: [Python-checkins] buildbot warnings in x86 gentoo trunk Message-ID: <20060421140618.946CB1E400B@bag.python.org> The Buildbot has detected a new failure of x86 gentoo trunk. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520gentoo%2520trunk/builds/568 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: andrew.kuchling Build Had Warnings: warnings test sincerely, -The Buildbot From buildbot at python.org Fri Apr 21 16:19:00 2006 From: buildbot at python.org (buildbot at python.org) Date: Fri, 21 Apr 2006 14:19:00 +0000 Subject: [Python-checkins] buildbot warnings in x86 cygwin 2.4 Message-ID: <20060421141901.025DA1E400B@bag.python.org> The Buildbot has detected a new failure of x86 cygwin 2.4. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520cygwin%25202.4/builds/37 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch branches/release24-maint] HEAD Blamelist: andrew.kuchling Build Had Warnings: warnings test sincerely, -The Buildbot From python-checkins at python.org Fri Apr 21 17:22:46 2006 From: python-checkins at python.org (george.yoshida) Date: Fri, 21 Apr 2006 17:22:46 +0200 (CEST) Subject: [Python-checkins] r45616 - peps/trunk/pep-0319.txt peps/trunk/pep-0341.txt peps/trunk/pep-0342.txt Message-ID: <20060421152246.B59211E400B@bag.python.org> Author: george.yoshida Date: Fri Apr 21 17:22:45 2006 New Revision: 45616 Modified: peps/trunk/pep-0319.txt peps/trunk/pep-0341.txt peps/trunk/pep-0342.txt Log: typo fixes Modified: peps/trunk/pep-0319.txt ============================================================================== --- peps/trunk/pep-0319.txt (original) +++ peps/trunk/pep-0319.txt Fri Apr 21 17:22:45 2006 @@ -416,7 +416,7 @@ with my_transaction: do_in_transaction() - # when the block terminates, the transaction is commited. + # when the block terminates, the transaction is committed. The 'synchronize' and 'asynchronize' keywords cannot serve this or any other general acquire/release pattern other than thread Modified: peps/trunk/pep-0341.txt ============================================================================== --- peps/trunk/pep-0341.txt (original) +++ peps/trunk/pep-0341.txt Fri Apr 21 17:22:45 2006 @@ -38,7 +38,7 @@ f = open(filename) text = f.read() except IOError: - print 'An error occured' + print 'An error occurred' finally: if f: f.close() @@ -99,7 +99,7 @@ However, according to Guido, it should be a piece of cake to implement[1] -- at least for a core hacker. - This patch was commited 17 December 2005, SVN revision 41740 [3]. + This patch was committed 17 December 2005, SVN revision 41740 [3]. References Modified: peps/trunk/pep-0342.txt ============================================================================== --- peps/trunk/pep-0342.txt (original) +++ peps/trunk/pep-0342.txt Fri Apr 21 17:22:45 2006 @@ -572,7 +572,7 @@ PEP is available as SourceForge patch #1223381 (http://python.org/sf/1223381). - This patch was commited to CVS 01-02 August 2005. + This patch was committed to CVS 01-02 August 2005. Acknowledgements From buildbot at python.org Fri Apr 21 17:46:39 2006 From: buildbot at python.org (buildbot at python.org) Date: Fri, 21 Apr 2006 15:46:39 +0000 Subject: [Python-checkins] buildbot warnings in x86 XP-2 trunk Message-ID: <20060421154639.7F3BF1E400B@bag.python.org> The Buildbot has detected a new failure of x86 XP-2 trunk. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520XP-2%2520trunk/builds/338 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: andrew.kuchling,martin.v.loewis,thomas.wouters Build Had Warnings: warnings failed slave lost sincerely, -The Buildbot From python-checkins at python.org Fri Apr 21 18:21:13 2006 From: python-checkins at python.org (george.yoshida) Date: Fri, 21 Apr 2006 18:21:13 +0200 (CEST) Subject: [Python-checkins] r45618 - python/trunk/Doc/lib/libcodecs.tex Message-ID: <20060421162113.508691E401F@bag.python.org> Author: george.yoshida Date: Fri Apr 21 18:21:12 2006 New Revision: 45618 Modified: python/trunk/Doc/lib/libcodecs.tex Log: Fix a markup. This generated a ugly code, "P>". Modified: python/trunk/Doc/lib/libcodecs.tex ============================================================================== --- python/trunk/Doc/lib/libcodecs.tex (original) +++ python/trunk/Doc/lib/libcodecs.tex Fri Apr 21 18:21:12 2006 @@ -1337,7 +1337,7 @@ UTF-8 codec with BOM signature} \declaremodule{standard}{encodings.utf-8-sig} % XXX utf_8_sig gives TeX errors \modulesynopsis{UTF-8 codec with BOM signature} -\moduleauthor{Walter D\"orwald} +\moduleauthor{Walter D\"orwald}{} \versionadded{2.5} From python-checkins at python.org Fri Apr 21 18:28:35 2006 From: python-checkins at python.org (hyeshik.chang) Date: Fri, 21 Apr 2006 18:28:35 +0200 (CEST) Subject: [Python-checkins] r45620 - python/trunk/Modules/cjkcodecs/multibytecodec.c Message-ID: <20060421162835.0ADFB1E400B@bag.python.org> Author: hyeshik.chang Date: Fri Apr 21 18:28:34 2006 New Revision: 45620 Modified: python/trunk/Modules/cjkcodecs/multibytecodec.c Log: Backport p3yk r45619: Add empty __init__ methods for stateful multibytecodec instances. This resolves a problem found by Thomas Wouters: http://mail.python.org/pipermail/python-dev/2006-April/064051.html Modified: python/trunk/Modules/cjkcodecs/multibytecodec.c ============================================================================== --- python/trunk/Modules/cjkcodecs/multibytecodec.c (original) +++ python/trunk/Modules/cjkcodecs/multibytecodec.c Fri Apr 21 18:28:34 2006 @@ -927,6 +927,12 @@ } static int +mbiencoder_init(PyObject *self, PyObject *args, PyObject *kwds) +{ + return 0; +} + +static int mbiencoder_traverse(MultibyteIncrementalEncoderObject *self, visitproc visit, void *arg) { @@ -982,7 +988,7 @@ 0, /* tp_descr_get */ 0, /* tp_descr_set */ 0, /* tp_dictoffset */ - 0, /* tp_init */ + mbiencoder_init, /* tp_init */ 0, /* tp_alloc */ mbiencoder_new, /* tp_new */ }; @@ -1122,6 +1128,12 @@ } static int +mbidecoder_init(PyObject *self, PyObject *args, PyObject *kwds) +{ + return 0; +} + +static int mbidecoder_traverse(MultibyteIncrementalDecoderObject *self, visitproc visit, void *arg) { @@ -1177,7 +1189,7 @@ 0, /* tp_descr_get */ 0, /* tp_descr_set */ 0, /* tp_dictoffset */ - 0, /* tp_init */ + mbidecoder_init, /* tp_init */ 0, /* tp_alloc */ mbidecoder_new, /* tp_new */ }; @@ -1427,6 +1439,12 @@ } static int +mbstreamreader_init(PyObject *self, PyObject *args, PyObject *kwds) +{ + return 0; +} + +static int mbstreamreader_traverse(MultibyteStreamReaderObject *self, visitproc visit, void *arg) { @@ -1484,7 +1502,7 @@ 0, /* tp_descr_get */ 0, /* tp_descr_set */ 0, /* tp_dictoffset */ - 0, /* tp_init */ + mbstreamreader_init, /* tp_init */ 0, /* tp_alloc */ mbstreamreader_new, /* tp_new */ }; @@ -1632,6 +1650,12 @@ } static int +mbstreamwriter_init(PyObject *self, PyObject *args, PyObject *kwds) +{ + return 0; +} + +static int mbstreamwriter_traverse(MultibyteStreamWriterObject *self, visitproc visit, void *arg) { @@ -1706,7 +1730,7 @@ 0, /* tp_descr_get */ 0, /* tp_descr_set */ 0, /* tp_dictoffset */ - 0, /* tp_init */ + mbstreamwriter_init, /* tp_init */ 0, /* tp_alloc */ mbstreamwriter_new, /* tp_new */ }; From python-checkins at python.org Fri Apr 21 18:34:18 2006 From: python-checkins at python.org (george.yoshida) Date: Fri, 21 Apr 2006 18:34:18 +0200 (CEST) Subject: [Python-checkins] r45621 - python/trunk/Doc/lib/libcodecs.tex Message-ID: <20060421163418.772BD1E400B@bag.python.org> Author: george.yoshida Date: Fri Apr 21 18:34:17 2006 New Revision: 45621 Modified: python/trunk/Doc/lib/libcodecs.tex Log: Correct the grammar Modified: python/trunk/Doc/lib/libcodecs.tex ============================================================================== --- python/trunk/Doc/lib/libcodecs.tex (original) +++ python/trunk/Doc/lib/libcodecs.tex Fri Apr 21 18:34:17 2006 @@ -161,7 +161,7 @@ \end{funcdesc} \begin{funcdesc}{lookup_error}{name} -Return the error handler previously register under the name \var{name}. +Return the error handler previously registered under the name \var{name}. Raises a \exception{LookupError} in case the handler cannot be found. \end{funcdesc} @@ -366,7 +366,7 @@ define in order to be compatible with the Python codec registry. \begin{classdesc}{IncrementalEncoder}{\optional{errors}} - Constructor for a \class{IncrementalEncoder} instance. + Constructor for an \class{IncrementalEncoder} instance. All incremental encoders must provide this constructor interface. They are free to add additional keyword arguments, but only the ones defined @@ -413,7 +413,7 @@ define in order to be compatible with the Python codec registry. \begin{classdesc}{IncrementalDecoder}{\optional{errors}} - Constructor for a \class{IncrementalDecoder} instance. + Constructor for an \class{IncrementalDecoder} instance. All incremental decoders must provide this constructor interface. They are free to add additional keyword arguments, but only the ones defined From python-checkins at python.org Fri Apr 21 18:34:55 2006 From: python-checkins at python.org (tim.peters) Date: Fri, 21 Apr 2006 18:34:55 +0200 (CEST) Subject: [Python-checkins] r45622 - python/trunk/Tools/pybench/Arithmetic.py python/trunk/Tools/pybench/Calls.py python/trunk/Tools/pybench/CommandLine.py python/trunk/Tools/pybench/Constructs.py python/trunk/Tools/pybench/Dict.py python/trunk/Tools/pybench/Exceptions.py python/trunk/Tools/pybench/Imports.py python/trunk/Tools/pybench/Instances.py python/trunk/Tools/pybench/Lists.py python/trunk/Tools/pybench/Lookups.py python/trunk/Tools/pybench/Numbers.py python/trunk/Tools/pybench/Strings.py python/trunk/Tools/pybench/Tuples.py python/trunk/Tools/pybench/Unicode.py python/trunk/Tools/pybench/pybench.py Message-ID: <20060421163455.4D5C01E400C@bag.python.org> Author: tim.peters Date: Fri Apr 21 18:34:54 2006 New Revision: 45622 Modified: python/trunk/Tools/pybench/Arithmetic.py python/trunk/Tools/pybench/Calls.py python/trunk/Tools/pybench/CommandLine.py python/trunk/Tools/pybench/Constructs.py python/trunk/Tools/pybench/Dict.py python/trunk/Tools/pybench/Exceptions.py python/trunk/Tools/pybench/Imports.py python/trunk/Tools/pybench/Instances.py python/trunk/Tools/pybench/Lists.py python/trunk/Tools/pybench/Lookups.py python/trunk/Tools/pybench/Numbers.py python/trunk/Tools/pybench/Strings.py python/trunk/Tools/pybench/Tuples.py python/trunk/Tools/pybench/Unicode.py python/trunk/Tools/pybench/pybench.py Log: Whitespace normalization. Modified: python/trunk/Tools/pybench/Arithmetic.py ============================================================================== --- python/trunk/Tools/pybench/Arithmetic.py (original) +++ python/trunk/Tools/pybench/Arithmetic.py Fri Apr 21 18:34:54 2006 @@ -775,4 +775,3 @@ for i in xrange(self.rounds): pass - Modified: python/trunk/Tools/pybench/Calls.py ============================================================================== --- python/trunk/Tools/pybench/Calls.py (original) +++ python/trunk/Tools/pybench/Calls.py Fri Apr 21 18:34:54 2006 @@ -407,4 +407,3 @@ for i in xrange(self.rounds): pass - Modified: python/trunk/Tools/pybench/CommandLine.py ============================================================================== --- python/trunk/Tools/pybench/CommandLine.py (original) +++ python/trunk/Tools/pybench/CommandLine.py Fri Apr 21 18:34:54 2006 @@ -7,7 +7,7 @@ TODO: * Incorporate the changes made by (see Inbox) - * Add number range option using srange() + * Add number range option using srange() """ @@ -194,7 +194,7 @@ """ Option that takes an argument. An optional default argument can be given. - + """ def __init__(self,name,help=None,default=None): @@ -299,7 +299,7 @@ values = None # Dictionary of passed options (or default values) # indexed by the options name, e.g. '-h' files = None # List of passed filenames - optionlist = None # List of passed options + optionlist = None # List of passed options def __init__(self,argv=None): @@ -318,15 +318,15 @@ # Init .arguments list self.arguments = argv[1:] - + # Setup Option mapping self.option_map = option_dict(self.options) - + # Append preset options for option in self.preset_options: if not self.option_map.has_key(option.name): self.add_option(option) - + # Init .files list self.files = [] @@ -336,12 +336,12 @@ rc = self.startup() if rc is not None: raise SystemExit,rc - + # Parse command line rc = self.parse() if rc is not None: raise SystemExit,rc - + # Start application rc = self.main() if rc is None: @@ -375,7 +375,7 @@ Note that this has to be done *before* .parse() is being executed. - + """ self.options.append(option) self.option_map[option.name] = option @@ -481,10 +481,10 @@ This may modify filelist in place. A typical application is checking that at least n files are given. - + If this method returns anything other than None, the process is terminated with the return value as exit code. - + """ return None @@ -554,19 +554,19 @@ """ This may process the files list in place. """ return None - + # Short option handler def handle_h(self,arg): self.help() return 0 - + def handle_v(self, value): """ Turn on verbose output. """ self.verbose = 1 - + # Handlers for long options have two underscores in their name def handle__help(self,arg): @@ -607,7 +607,7 @@ it is None, 0 is assumed (meaning OK). Unhandled exceptions are reported with exit status code 1 (see __init__ for further details). - + """ return None @@ -620,7 +620,7 @@ header = 'Test Application' version = __version__ options = [Option('-v','verbose')] - + def handle_v(self,arg): print 'VERBOSE, Yeah !' Modified: python/trunk/Tools/pybench/Constructs.py ============================================================================== --- python/trunk/Tools/pybench/Constructs.py (original) +++ python/trunk/Tools/pybench/Constructs.py Fri Apr 21 18:34:54 2006 @@ -562,4 +562,3 @@ l1 = range(1000) for i in xrange(self.rounds): pass - Modified: python/trunk/Tools/pybench/Dict.py ============================================================================== --- python/trunk/Tools/pybench/Dict.py (original) +++ python/trunk/Tools/pybench/Dict.py Fri Apr 21 18:34:54 2006 @@ -93,70 +93,70 @@ d['jkl'] = 4 d['mno'] = 5 d['pqr'] = 6 - + d['abc'] d['def'] d['ghi'] d['jkl'] d['mno'] d['pqr'] - + d['abc'] = 1 d['def'] = 2 d['ghi'] = 3 d['jkl'] = 4 d['mno'] = 5 d['pqr'] = 6 - + d['abc'] d['def'] d['ghi'] d['jkl'] d['mno'] d['pqr'] - + d['abc'] = 1 d['def'] = 2 d['ghi'] = 3 d['jkl'] = 4 d['mno'] = 5 d['pqr'] = 6 - + d['abc'] d['def'] d['ghi'] d['jkl'] d['mno'] d['pqr'] - + d['abc'] = 1 d['def'] = 2 d['ghi'] = 3 d['jkl'] = 4 d['mno'] = 5 d['pqr'] = 6 - + d['abc'] d['def'] d['ghi'] d['jkl'] d['mno'] d['pqr'] - + d['abc'] = 1 d['def'] = 2 d['ghi'] = 3 d['jkl'] = 4 d['mno'] = 5 d['pqr'] = 6 - + d['abc'] d['def'] d['ghi'] d['jkl'] d['mno'] d['pqr'] - + def calibrate(self): d = {} @@ -182,70 +182,70 @@ d[4.567] = 4 d[5.678] = 5 d[6.789] = 6 - + d[1.234] d[2.345] d[3.456] d[4.567] d[5.678] d[6.789] - + d[1.234] = 1 d[2.345] = 2 d[3.456] = 3 d[4.567] = 4 d[5.678] = 5 d[6.789] = 6 - + d[1.234] d[2.345] d[3.456] d[4.567] d[5.678] d[6.789] - + d[1.234] = 1 d[2.345] = 2 d[3.456] = 3 d[4.567] = 4 d[5.678] = 5 d[6.789] = 6 - + d[1.234] d[2.345] d[3.456] d[4.567] d[5.678] d[6.789] - + d[1.234] = 1 d[2.345] = 2 d[3.456] = 3 d[4.567] = 4 d[5.678] = 5 d[6.789] = 6 - + d[1.234] d[2.345] d[3.456] d[4.567] d[5.678] d[6.789] - + d[1.234] = 1 d[2.345] = 2 d[3.456] = 3 d[4.567] = 4 d[5.678] = 5 d[6.789] = 6 - + d[1.234] d[2.345] d[3.456] d[4.567] d[5.678] d[6.789] - + def calibrate(self): d = {} @@ -271,70 +271,70 @@ d[4] = 4 d[5] = 5 d[6] = 6 - + d[1] d[2] d[3] d[4] d[5] d[6] - + d[1] = 1 d[2] = 2 d[3] = 3 d[4] = 4 d[5] = 5 d[6] = 6 - + d[1] d[2] d[3] d[4] d[5] d[6] - + d[1] = 1 d[2] = 2 d[3] = 3 d[4] = 4 d[5] = 5 d[6] = 6 - + d[1] d[2] d[3] d[4] d[5] d[6] - + d[1] = 1 d[2] = 2 d[3] = 3 d[4] = 4 d[5] = 5 d[6] = 6 - + d[1] d[2] d[3] d[4] d[5] d[6] - + d[1] = 1 d[2] = 2 d[3] = 3 d[4] = 4 d[5] = 5 d[6] = 6 - + d[1] d[2] d[3] d[4] d[5] d[6] - + def calibrate(self): d = {} @@ -360,7 +360,7 @@ d[3] = 3 d[4] = 4 d[5] = 5 - + x = d[0] x = d[1] x = d[2] @@ -388,7 +388,7 @@ d[3] = 3 d[4] = 4 d[5] = 5 - + x = d[0] x = d[1] x = d[2] @@ -416,7 +416,7 @@ d[3] = 3 d[4] = 4 d[5] = 5 - + x = d[0] x = d[1] x = d[2] @@ -444,7 +444,7 @@ d[3] = 3 d[4] = 4 d[5] = 5 - + x = d[0] x = d[1] x = d[2] @@ -472,7 +472,7 @@ d[3] = 3 d[4] = 4 d[5] = 5 - + x = d[0] x = d[1] x = d[2] @@ -500,4 +500,3 @@ for i in xrange(self.rounds): pass - Modified: python/trunk/Tools/pybench/Exceptions.py ============================================================================== --- python/trunk/Tools/pybench/Exceptions.py (original) +++ python/trunk/Tools/pybench/Exceptions.py Fri Apr 21 18:34:54 2006 @@ -38,7 +38,7 @@ for i in xrange(self.rounds): pass - + class TryExcept(Test): @@ -677,5 +677,3 @@ for i in xrange(self.rounds): pass - - Modified: python/trunk/Tools/pybench/Imports.py ============================================================================== --- python/trunk/Tools/pybench/Imports.py (original) +++ python/trunk/Tools/pybench/Imports.py Fri Apr 21 18:34:54 2006 @@ -47,7 +47,7 @@ for i in xrange(self.rounds): pass - + class SecondPackageImport(Test): @@ -92,7 +92,7 @@ for i in xrange(self.rounds): pass - + class SecondSubmoduleImport(Test): version = 0.1 @@ -136,4 +136,3 @@ for i in xrange(self.rounds): pass - Modified: python/trunk/Tools/pybench/Instances.py ============================================================================== --- python/trunk/Tools/pybench/Instances.py (original) +++ python/trunk/Tools/pybench/Instances.py Fri Apr 21 18:34:54 2006 @@ -64,5 +64,3 @@ for i in xrange(self.rounds): pass - - Modified: python/trunk/Tools/pybench/Lists.py ============================================================================== --- python/trunk/Tools/pybench/Lists.py (original) +++ python/trunk/Tools/pybench/Lists.py Fri Apr 21 18:34:54 2006 @@ -25,7 +25,7 @@ l[3] = 3 l[4] = 4 l[5] = 5 - + x = l[0] x = l[1] x = l[2] @@ -46,7 +46,7 @@ l[3] = 3 l[4] = 4 l[5] = 5 - + x = l[0] x = l[1] x = l[2] @@ -67,7 +67,7 @@ l[3] = 3 l[4] = 4 l[5] = 5 - + x = l[0] x = l[1] x = l[2] @@ -88,7 +88,7 @@ l[3] = 3 l[4] = 4 l[5] = 5 - + x = l[0] x = l[1] x = l[2] @@ -109,7 +109,7 @@ l[3] = 3 l[4] = 4 l[5] = 5 - + x = l[0] x = l[1] x = l[2] @@ -190,11 +190,11 @@ l[3] = 3 l[4] = 4 l[5] = 5 - + l[:3] = [1,2,3] m = l[:-1] m = l[1:] - + l[-1:] = [4,5,6] l = [] @@ -212,11 +212,11 @@ l[3] = 3 l[4] = 4 l[5] = 5 - + l[:3] = [1,2,3] m = l[:-1] m = l[1:] - + l[-1:] = [4,5,6] l = [] @@ -234,11 +234,11 @@ l[3] = 3 l[4] = 4 l[5] = 5 - + l[:3] = [1,2,3] m = l[:-1] m = l[1:] - + l[-1:] = [4,5,6] l = [] @@ -256,11 +256,11 @@ l[3] = 3 l[4] = 4 l[5] = 5 - + l[:3] = [1,2,3] m = l[:-1] m = l[1:] - + l[-1:] = [4,5,6] l = [] @@ -278,15 +278,14 @@ l[3] = 3 l[4] = 4 l[5] = 5 - + l[:3] = [1,2,3] m = l[:-1] m = l[1:] - + l[-1:] = [4,5,6] def calibrate(self): for i in xrange(self.rounds): l = [] - Modified: python/trunk/Tools/pybench/Lookups.py ============================================================================== --- python/trunk/Tools/pybench/Lookups.py (original) +++ python/trunk/Tools/pybench/Lookups.py Fri Apr 21 18:34:54 2006 @@ -943,4 +943,3 @@ for i in xrange(self.rounds): pass - Modified: python/trunk/Tools/pybench/Numbers.py ============================================================================== --- python/trunk/Tools/pybench/Numbers.py (original) +++ python/trunk/Tools/pybench/Numbers.py Fri Apr 21 18:34:54 2006 @@ -15,55 +15,55 @@ 2 == 3 2 > 3 2 < 3 - + 2 < 3 2 > 3 2 == 3 2 > 3 2 < 3 - + 2 < 3 2 > 3 2 == 3 2 > 3 2 < 3 - + 2 < 3 2 > 3 2 == 3 2 > 3 2 < 3 - + 2 < 3 2 > 3 2 == 3 2 > 3 2 < 3 - + 2 < 3 2 > 3 2 == 3 2 > 3 2 < 3 - + 2 < 3 2 > 3 2 == 3 2 > 3 2 < 3 - + 2 < 3 2 > 3 2 == 3 2 > 3 2 < 3 - + 2 < 3 2 > 3 2 == 3 2 > 3 2 < 3 - + 2 < 3 2 > 3 2 == 3 @@ -75,55 +75,55 @@ 2 == 3 2 > 3 2 < 3 - + 2 < 3 2 > 3 2 == 3 2 > 3 2 < 3 - + 2 < 3 2 > 3 2 == 3 2 > 3 2 < 3 - + 2 < 3 2 > 3 2 == 3 2 > 3 2 < 3 - + 2 < 3 2 > 3 2 == 3 2 > 3 2 < 3 - + 2 < 3 2 > 3 2 == 3 2 > 3 2 < 3 - + 2 < 3 2 > 3 2 == 3 2 > 3 2 < 3 - + 2 < 3 2 > 3 2 == 3 2 > 3 2 < 3 - + 2 < 3 2 > 3 2 == 3 2 > 3 2 < 3 - + 2 < 3 2 > 3 2 == 3 @@ -135,55 +135,55 @@ 2 == 3 2 > 3 2 < 3 - + 2 < 3 2 > 3 2 == 3 2 > 3 2 < 3 - + 2 < 3 2 > 3 2 == 3 2 > 3 2 < 3 - + 2 < 3 2 > 3 2 == 3 2 > 3 2 < 3 - + 2 < 3 2 > 3 2 == 3 2 > 3 2 < 3 - + 2 < 3 2 > 3 2 == 3 2 > 3 2 < 3 - + 2 < 3 2 > 3 2 == 3 2 > 3 2 < 3 - + 2 < 3 2 > 3 2 == 3 2 > 3 2 < 3 - + 2 < 3 2 > 3 2 == 3 2 > 3 2 < 3 - + 2 < 3 2 > 3 2 == 3 @@ -211,55 +211,55 @@ 2.1 == 3.31 2.1 > 3.31 2.1 < 3.31 - + 2.1 < 3.31 2.1 > 3.31 2.1 == 3.31 2.1 > 3.31 2.1 < 3.31 - + 2.1 < 3.31 2.1 > 3.31 2.1 == 3.31 2.1 > 3.31 2.1 < 3.31 - + 2.1 < 3.31 2.1 > 3.31 2.1 == 3.31 2.1 > 3.31 2.1 < 3.31 - + 2.1 < 3.31 2.1 > 3.31 2.1 == 3.31 2.1 > 3.31 2.1 < 3.31 - + 2.1 < 3.31 2.1 > 3.31 2.1 == 3.31 2.1 > 3.31 2.1 < 3.31 - + 2.1 < 3.31 2.1 > 3.31 2.1 == 3.31 2.1 > 3.31 2.1 < 3.31 - + 2.1 < 3.31 2.1 > 3.31 2.1 == 3.31 2.1 > 3.31 2.1 < 3.31 - + 2.1 < 3.31 2.1 > 3.31 2.1 == 3.31 2.1 > 3.31 2.1 < 3.31 - + 2.1 < 3.31 2.1 > 3.31 2.1 == 3.31 @@ -271,55 +271,55 @@ 2.1 == 3.31 2.1 > 3.31 2.1 < 3.31 - + 2.1 < 3.31 2.1 > 3.31 2.1 == 3.31 2.1 > 3.31 2.1 < 3.31 - + 2.1 < 3.31 2.1 > 3.31 2.1 == 3.31 2.1 > 3.31 2.1 < 3.31 - + 2.1 < 3.31 2.1 > 3.31 2.1 == 3.31 2.1 > 3.31 2.1 < 3.31 - + 2.1 < 3.31 2.1 > 3.31 2.1 == 3.31 2.1 > 3.31 2.1 < 3.31 - + 2.1 < 3.31 2.1 > 3.31 2.1 == 3.31 2.1 > 3.31 2.1 < 3.31 - + 2.1 < 3.31 2.1 > 3.31 2.1 == 3.31 2.1 > 3.31 2.1 < 3.31 - + 2.1 < 3.31 2.1 > 3.31 2.1 == 3.31 2.1 > 3.31 2.1 < 3.31 - + 2.1 < 3.31 2.1 > 3.31 2.1 == 3.31 2.1 > 3.31 2.1 < 3.31 - + 2.1 < 3.31 2.1 > 3.31 2.1 == 3.31 @@ -331,55 +331,55 @@ 2.1 == 3.31 2.1 > 3.31 2.1 < 3.31 - + 2.1 < 3.31 2.1 > 3.31 2.1 == 3.31 2.1 > 3.31 2.1 < 3.31 - + 2.1 < 3.31 2.1 > 3.31 2.1 == 3.31 2.1 > 3.31 2.1 < 3.31 - + 2.1 < 3.31 2.1 > 3.31 2.1 == 3.31 2.1 > 3.31 2.1 < 3.31 - + 2.1 < 3.31 2.1 > 3.31 2.1 == 3.31 2.1 > 3.31 2.1 < 3.31 - + 2.1 < 3.31 2.1 > 3.31 2.1 == 3.31 2.1 > 3.31 2.1 < 3.31 - + 2.1 < 3.31 2.1 > 3.31 2.1 == 3.31 2.1 > 3.31 2.1 < 3.31 - + 2.1 < 3.31 2.1 > 3.31 2.1 == 3.31 2.1 > 3.31 2.1 < 3.31 - + 2.1 < 3.31 2.1 > 3.31 2.1 == 3.31 2.1 > 3.31 2.1 < 3.31 - + 2.1 < 3.31 2.1 > 3.31 2.1 == 3.31 @@ -407,55 +407,55 @@ 2.1 == 4 2.1 > 4 2.1 < 4 - + 2.1 < 4 2.1 > 4 2.1 == 4 2.1 > 4 2.1 < 4 - + 2.1 < 4 2.1 > 4 2.1 == 4 2.1 > 4 2.1 < 4 - + 2.1 < 4 2.1 > 4 2.1 == 4 2.1 > 4 2.1 < 4 - + 2.1 < 4 2.1 > 4 2.1 == 4 2.1 > 4 2.1 < 4 - + 2.1 < 4 2.1 > 4 2.1 == 4 2.1 > 4 2.1 < 4 - + 2.1 < 4 2.1 > 4 2.1 == 4 2.1 > 4 2.1 < 4 - + 2.1 < 4 2.1 > 4 2.1 == 4 2.1 > 4 2.1 < 4 - + 2.1 < 4 2.1 > 4 2.1 == 4 2.1 > 4 2.1 < 4 - + 2.1 < 4 2.1 > 4 2.1 == 4 @@ -467,55 +467,55 @@ 2.1 == 4 2.1 > 4 2.1 < 4 - + 2.1 < 4 2.1 > 4 2.1 == 4 2.1 > 4 2.1 < 4 - + 2.1 < 4 2.1 > 4 2.1 == 4 2.1 > 4 2.1 < 4 - + 2.1 < 4 2.1 > 4 2.1 == 4 2.1 > 4 2.1 < 4 - + 2.1 < 4 2.1 > 4 2.1 == 4 2.1 > 4 2.1 < 4 - + 2.1 < 4 2.1 > 4 2.1 == 4 2.1 > 4 2.1 < 4 - + 2.1 < 4 2.1 > 4 2.1 == 4 2.1 > 4 2.1 < 4 - + 2.1 < 4 2.1 > 4 2.1 == 4 2.1 > 4 2.1 < 4 - + 2.1 < 4 2.1 > 4 2.1 == 4 2.1 > 4 2.1 < 4 - + 2.1 < 4 2.1 > 4 2.1 == 4 @@ -527,55 +527,55 @@ 2.1 == 4 2.1 > 4 2.1 < 4 - + 2.1 < 4 2.1 > 4 2.1 == 4 2.1 > 4 2.1 < 4 - + 2.1 < 4 2.1 > 4 2.1 == 4 2.1 > 4 2.1 < 4 - + 2.1 < 4 2.1 > 4 2.1 == 4 2.1 > 4 2.1 < 4 - + 2.1 < 4 2.1 > 4 2.1 == 4 2.1 > 4 2.1 < 4 - + 2.1 < 4 2.1 > 4 2.1 == 4 2.1 > 4 2.1 < 4 - + 2.1 < 4 2.1 > 4 2.1 == 4 2.1 > 4 2.1 < 4 - + 2.1 < 4 2.1 > 4 2.1 == 4 2.1 > 4 2.1 < 4 - + 2.1 < 4 2.1 > 4 2.1 == 4 2.1 > 4 2.1 < 4 - + 2.1 < 4 2.1 > 4 2.1 == 4 @@ -603,55 +603,55 @@ 1234567890L == 3456789012345L 1234567890L > 3456789012345L 1234567890L < 3456789012345L - + 1234567890L < 3456789012345L 1234567890L > 3456789012345L 1234567890L == 3456789012345L 1234567890L > 3456789012345L 1234567890L < 3456789012345L - + 1234567890L < 3456789012345L 1234567890L > 3456789012345L 1234567890L == 3456789012345L 1234567890L > 3456789012345L 1234567890L < 3456789012345L - + 1234567890L < 3456789012345L 1234567890L > 3456789012345L 1234567890L == 3456789012345L 1234567890L > 3456789012345L 1234567890L < 3456789012345L - + 1234567890L < 3456789012345L 1234567890L > 3456789012345L 1234567890L == 3456789012345L 1234567890L > 3456789012345L 1234567890L < 3456789012345L - + 1234567890L < 3456789012345L 1234567890L > 3456789012345L 1234567890L == 3456789012345L 1234567890L > 3456789012345L 1234567890L < 3456789012345L - + 1234567890L < 3456789012345L 1234567890L > 3456789012345L 1234567890L == 3456789012345L 1234567890L > 3456789012345L 1234567890L < 3456789012345L - + 1234567890L < 3456789012345L 1234567890L > 3456789012345L 1234567890L == 3456789012345L 1234567890L > 3456789012345L 1234567890L < 3456789012345L - + 1234567890L < 3456789012345L 1234567890L > 3456789012345L 1234567890L == 3456789012345L 1234567890L > 3456789012345L 1234567890L < 3456789012345L - + 1234567890L < 3456789012345L 1234567890L > 3456789012345L 1234567890L == 3456789012345L @@ -663,55 +663,55 @@ 1234567890L == 3456789012345L 1234567890L > 3456789012345L 1234567890L < 3456789012345L - + 1234567890L < 3456789012345L 1234567890L > 3456789012345L 1234567890L == 3456789012345L 1234567890L > 3456789012345L 1234567890L < 3456789012345L - + 1234567890L < 3456789012345L 1234567890L > 3456789012345L 1234567890L == 3456789012345L 1234567890L > 3456789012345L 1234567890L < 3456789012345L - + 1234567890L < 3456789012345L 1234567890L > 3456789012345L 1234567890L == 3456789012345L 1234567890L > 3456789012345L 1234567890L < 3456789012345L - + 1234567890L < 3456789012345L 1234567890L > 3456789012345L 1234567890L == 3456789012345L 1234567890L > 3456789012345L 1234567890L < 3456789012345L - + 1234567890L < 3456789012345L 1234567890L > 3456789012345L 1234567890L == 3456789012345L 1234567890L > 3456789012345L 1234567890L < 3456789012345L - + 1234567890L < 3456789012345L 1234567890L > 3456789012345L 1234567890L == 3456789012345L 1234567890L > 3456789012345L 1234567890L < 3456789012345L - + 1234567890L < 3456789012345L 1234567890L > 3456789012345L 1234567890L == 3456789012345L 1234567890L > 3456789012345L 1234567890L < 3456789012345L - + 1234567890L < 3456789012345L 1234567890L > 3456789012345L 1234567890L == 3456789012345L 1234567890L > 3456789012345L 1234567890L < 3456789012345L - + 1234567890L < 3456789012345L 1234567890L > 3456789012345L 1234567890L == 3456789012345L @@ -723,55 +723,55 @@ 1234567890L == 3456789012345L 1234567890L > 3456789012345L 1234567890L < 3456789012345L - + 1234567890L < 3456789012345L 1234567890L > 3456789012345L 1234567890L == 3456789012345L 1234567890L > 3456789012345L 1234567890L < 3456789012345L - + 1234567890L < 3456789012345L 1234567890L > 3456789012345L 1234567890L == 3456789012345L 1234567890L > 3456789012345L 1234567890L < 3456789012345L - + 1234567890L < 3456789012345L 1234567890L > 3456789012345L 1234567890L == 3456789012345L 1234567890L > 3456789012345L 1234567890L < 3456789012345L - + 1234567890L < 3456789012345L 1234567890L > 3456789012345L 1234567890L == 3456789012345L 1234567890L > 3456789012345L 1234567890L < 3456789012345L - + 1234567890L < 3456789012345L 1234567890L > 3456789012345L 1234567890L == 3456789012345L 1234567890L > 3456789012345L 1234567890L < 3456789012345L - + 1234567890L < 3456789012345L 1234567890L > 3456789012345L 1234567890L == 3456789012345L 1234567890L > 3456789012345L 1234567890L < 3456789012345L - + 1234567890L < 3456789012345L 1234567890L > 3456789012345L 1234567890L == 3456789012345L 1234567890L > 3456789012345L 1234567890L < 3456789012345L - + 1234567890L < 3456789012345L 1234567890L > 3456789012345L 1234567890L == 3456789012345L 1234567890L > 3456789012345L 1234567890L < 3456789012345L - + 1234567890L < 3456789012345L 1234567890L > 3456789012345L 1234567890L == 3456789012345L Modified: python/trunk/Tools/pybench/Strings.py ============================================================================== --- python/trunk/Tools/pybench/Strings.py (original) +++ python/trunk/Tools/pybench/Strings.py Fri Apr 21 18:34:54 2006 @@ -81,7 +81,7 @@ for i in xrange(self.rounds): pass - + class CompareStrings(Test): @@ -163,7 +163,7 @@ for i in xrange(self.rounds): pass - + class CompareInternedStrings(Test): @@ -245,7 +245,7 @@ for i in xrange(self.rounds): pass - + class CreateStringsWithConcat(Test): @@ -320,7 +320,7 @@ for i in xrange(self.rounds): pass - + class StringSlicing(Test): @@ -334,45 +334,45 @@ for i in xrange(self.rounds): - s[50:] - s[:25] - s[50:55] - s[-1:] - s[:1] - s[2:] - s[11:-11] - - s[50:] - s[:25] - s[50:55] - s[-1:] - s[:1] - s[2:] - s[11:-11] - - s[50:] - s[:25] - s[50:55] - s[-1:] - s[:1] - s[2:] - s[11:-11] - - s[50:] - s[:25] - s[50:55] - s[-1:] - s[:1] - s[2:] - s[11:-11] - - s[50:] - s[:25] - s[50:55] - s[-1:] - s[:1] - s[2:] - s[11:-11] + s[50:] + s[:25] + s[50:55] + s[-1:] + s[:1] + s[2:] + s[11:-11] + + s[50:] + s[:25] + s[50:55] + s[-1:] + s[:1] + s[2:] + s[11:-11] + + s[50:] + s[:25] + s[50:55] + s[-1:] + s[:1] + s[2:] + s[11:-11] + + s[50:] + s[:25] + s[50:55] + s[-1:] + s[:1] + s[2:] + s[11:-11] + + s[50:] + s[:25] + s[50:55] + s[-1:] + s[:1] + s[2:] + s[11:-11] def calibrate(self): @@ -560,5 +560,3 @@ for i in xrange(self.rounds): s = data[i % len_data] - - Modified: python/trunk/Tools/pybench/Tuples.py ============================================================================== --- python/trunk/Tools/pybench/Tuples.py (original) +++ python/trunk/Tools/pybench/Tuples.py Fri Apr 21 18:34:54 2006 @@ -265,7 +265,7 @@ t = tuple(range(100)) for j in r: - + pass class SmallTuples(Test): @@ -362,4 +362,3 @@ for i in xrange(self.rounds): pass - Modified: python/trunk/Tools/pybench/Unicode.py ============================================================================== --- python/trunk/Tools/pybench/Unicode.py (original) +++ python/trunk/Tools/pybench/Unicode.py Fri Apr 21 18:34:54 2006 @@ -86,7 +86,7 @@ for i in xrange(self.rounds): pass - + class CompareUnicode(Test): @@ -168,7 +168,7 @@ for i in xrange(self.rounds): pass - + class CreateUnicodeWithConcat(Test): @@ -243,7 +243,7 @@ for i in xrange(self.rounds): pass - + class UnicodeSlicing(Test): @@ -303,7 +303,7 @@ for i in xrange(self.rounds): pass - + ### String methods class UnicodeMappings(Test): @@ -318,7 +318,7 @@ t = join(map(unichr,range(100)),'') u = join(map(unichr,range(500)),'') v = join(map(unichr,range(1000)),'') - + for i in xrange(self.rounds): s.lower() @@ -375,7 +375,7 @@ t = join(map(unichr,range(100)),'') u = join(map(unichr,range(500)),'') v = join(map(unichr,range(1000)),'') - + for i in xrange(self.rounds): pass @@ -389,7 +389,7 @@ data = (u'abc', u'123', u' ', u'\u1234\u2345\u3456', u'\uFFFF'*10) len_data = len(data) - + for i in xrange(self.rounds): s = data[i % len_data] @@ -447,7 +447,7 @@ data = (u'abc', u'123', u' ', u'\u1234\u2345\u3456', u'\uFFFF'*10) len_data = len(data) - + for i in xrange(self.rounds): s = data[i % len_data] Modified: python/trunk/Tools/pybench/pybench.py ============================================================================== --- python/trunk/Tools/pybench/pybench.py (original) +++ python/trunk/Tools/pybench/pybench.py Fri Apr 21 18:34:54 2006 @@ -38,7 +38,7 @@ __version__ = '1.3' # -# NOTE: Use xrange for all test loops unless you want to face +# NOTE: Use xrange for all test loops unless you want to face # a 20MB process ! # # All tests should have rounds set to values so that a run() @@ -85,7 +85,7 @@ # for comparisons of benchmark runs - tests with unequal version # number will not get compared. version = 1.0 - + # The number of abstract operations done in each round of the # test. An operation is the basic unit of what you want to # measure. The benchmark will output the amount of run-time per @@ -129,7 +129,7 @@ """ Run the test in two phases: first calibrate, then do the actual test. Be careful to keep the calibration timing low w/r to the test timing. - + """ test = self.test calibrate = self.calibrate @@ -144,7 +144,7 @@ offset = offset + t offset = offset / cruns # now the real thing - t = clock() + t = clock() test() t = clock() - t self.last_timing = (t-offset,t,offset) @@ -152,32 +152,32 @@ def calibrate(self): - """ Calibrate the test. + """ Calibrate the test. - This method should execute everything that is needed to - setup and run the test - except for the actual operations - that you intend to measure. pybench uses this method to - measure the test implementation overhead. + This method should execute everything that is needed to + setup and run the test - except for the actual operations + that you intend to measure. pybench uses this method to + measure the test implementation overhead. """ return def test(self): - """ Run the test. + """ Run the test. - The test needs to run self.rounds executing - self.operations number of operations each. + The test needs to run self.rounds executing + self.operations number of operations each. """ # do some tests return - + def stat(self): """ Returns two value: average time per run and average per operation. - + """ runs = len(self.times) if runs == 0: @@ -210,7 +210,7 @@ roundtime = 0 # Average round time version = None # Benchmark version number (see __init__) # as float x.yy - starttime = None # Benchmark start time + starttime = None # Benchmark start time def __init__(self): @@ -254,7 +254,7 @@ print self.roundtime = (clock() - roundtime) / self.rounds print - + def print_stat(self, compare_to=None, hidenoise=0): if not compare_to: @@ -380,7 +380,7 @@ hidenoise = self.values['-d'] warp = self.values['-w'] nogc = self.values['--no-gc'] - + # Switch off GC if nogc: try: @@ -407,7 +407,7 @@ compare_to = bench except IOError: print '* Error opening/reading file',compare_to - compare_to = None + compare_to = None if show_bench: try: From buildbot at python.org Fri Apr 21 18:41:45 2006 From: buildbot at python.org (buildbot at python.org) Date: Fri, 21 Apr 2006 16:41:45 +0000 Subject: [Python-checkins] buildbot warnings in sparc Ubuntu dapper trunk Message-ID: <20060421164145.2C43E1E4010@bag.python.org> The Buildbot has detected a new failure of sparc Ubuntu dapper trunk. Full details are available at: http://www.python.org/dev/buildbot/all/sparc%2520Ubuntu%2520dapper%2520trunk/builds/130 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: andrew.kuchling,thomas.wouters Build Had Warnings: warnings test sincerely, -The Buildbot From python-checkins at python.org Fri Apr 21 18:44:07 2006 From: python-checkins at python.org (thomas.wouters) Date: Fri, 21 Apr 2006 18:44:07 +0200 (CEST) Subject: [Python-checkins] r45623 - in python/branches/p3yk: Doc/howto/unicode.rst Doc/lib/libcodecs.tex Doc/whatsnew/whatsnew25.tex Lib/SimpleXMLRPCServer.py Objects/stringobject.c PCbuild/_bsddb.vcproj PCbuild/_ctypes.vcproj PCbuild/_ctypes_test.vcproj PCbuild/_elementtree.vcproj PCbuild/_msi.vcproj PCbuild/_socket.vcproj PCbuild/_sqlite3.vcproj PCbuild/_testcapi.vcproj PCbuild/_tkinter.vcproj PCbuild/bz2.vcproj PCbuild/pyexpat.vcproj PCbuild/python.vcproj PCbuild/pythoncore.vcproj PCbuild/pythonw.vcproj PCbuild/readme.txt PCbuild/select.vcproj PCbuild/unicodedata.vcproj PCbuild/winsound.vcproj Message-ID: <20060421164407.6CD221E4016@bag.python.org> Author: thomas.wouters Date: Fri Apr 21 18:44:05 2006 New Revision: 45623 Modified: python/branches/p3yk/ (props changed) python/branches/p3yk/Doc/howto/unicode.rst python/branches/p3yk/Doc/lib/libcodecs.tex python/branches/p3yk/Doc/whatsnew/whatsnew25.tex python/branches/p3yk/Lib/SimpleXMLRPCServer.py python/branches/p3yk/Objects/stringobject.c python/branches/p3yk/PCbuild/_bsddb.vcproj python/branches/p3yk/PCbuild/_ctypes.vcproj python/branches/p3yk/PCbuild/_ctypes_test.vcproj python/branches/p3yk/PCbuild/_elementtree.vcproj python/branches/p3yk/PCbuild/_msi.vcproj python/branches/p3yk/PCbuild/_socket.vcproj python/branches/p3yk/PCbuild/_sqlite3.vcproj python/branches/p3yk/PCbuild/_testcapi.vcproj python/branches/p3yk/PCbuild/_tkinter.vcproj python/branches/p3yk/PCbuild/bz2.vcproj python/branches/p3yk/PCbuild/pyexpat.vcproj python/branches/p3yk/PCbuild/python.vcproj python/branches/p3yk/PCbuild/pythoncore.vcproj python/branches/p3yk/PCbuild/pythonw.vcproj python/branches/p3yk/PCbuild/readme.txt python/branches/p3yk/PCbuild/select.vcproj python/branches/p3yk/PCbuild/unicodedata.vcproj python/branches/p3yk/PCbuild/winsound.vcproj Log: Merge with trunk up to revision 45620. Modified: python/branches/p3yk/Doc/howto/unicode.rst ============================================================================== --- python/branches/p3yk/Doc/howto/unicode.rst (original) +++ python/branches/p3yk/Doc/howto/unicode.rst Fri Apr 21 18:44:05 2006 @@ -733,6 +733,7 @@ .. comment Additional topic: building Python w/ UCS2 or UCS4 support .. comment Describe obscure -U switch somewhere? +.. comment Describe use of codecs.StreamRecoder and StreamReaderWriter .. comment Original outline: Modified: python/branches/p3yk/Doc/lib/libcodecs.tex ============================================================================== --- python/branches/p3yk/Doc/lib/libcodecs.tex (original) +++ python/branches/p3yk/Doc/lib/libcodecs.tex Fri Apr 21 18:44:05 2006 @@ -93,21 +93,21 @@ lookup: \begin{funcdesc}{getencoder}{encoding} -Lookup up the codec for the given encoding and return its encoder +Look up the codec for the given encoding and return its encoder function. Raises a \exception{LookupError} in case the encoding cannot be found. \end{funcdesc} \begin{funcdesc}{getdecoder}{encoding} -Lookup up the codec for the given encoding and return its decoder +Look up the codec for the given encoding and return its decoder function. Raises a \exception{LookupError} in case the encoding cannot be found. \end{funcdesc} \begin{funcdesc}{getincrementalencoder}{encoding} -Lookup up the codec for the given encoding and return its incremental encoder +Look up the codec for the given encoding and return its incremental encoder class or factory function. Raises a \exception{LookupError} in case the encoding cannot be found or the @@ -116,7 +116,7 @@ \end{funcdesc} \begin{funcdesc}{getincrementaldecoder}{encoding} -Lookup up the codec for the given encoding and return its incremental decoder +Look up the codec for the given encoding and return its incremental decoder class or factory function. Raises a \exception{LookupError} in case the encoding cannot be found or the @@ -125,14 +125,14 @@ \end{funcdesc} \begin{funcdesc}{getreader}{encoding} -Lookup up the codec for the given encoding and return its StreamReader +Look up the codec for the given encoding and return its StreamReader class or factory function. Raises a \exception{LookupError} in case the encoding cannot be found. \end{funcdesc} \begin{funcdesc}{getwriter}{encoding} -Lookup up the codec for the given encoding and return its StreamWriter +Look up the codec for the given encoding and return its StreamWriter class or factory function. Raises a \exception{LookupError} in case the encoding cannot be found. @@ -353,7 +353,7 @@ the encoding/decoding process during method calls. The joined output of calls to the \method{encode}/\method{decode} method is the -same as if the all single inputs where joined into one, and this input was +same as if all the single inputs were joined into one, and this input was encoded/decoded with the stateless encoder/decoder. @@ -363,7 +363,7 @@ The \class{IncrementalEncoder} class is used for encoding an input in multiple steps. It defines the following methods which every incremental encoder must -define in order to be compatible to the Python codec registry. +define in order to be compatible with the Python codec registry. \begin{classdesc}{IncrementalEncoder}{\optional{errors}} Constructor for a \class{IncrementalEncoder} instance. @@ -410,7 +410,7 @@ The \class{IncrementalDecoder} class is used for decoding an input in multiple steps. It defines the following methods which every incremental decoder must -define in order to be compatible to the Python codec registry. +define in order to be compatible with the Python codec registry. \begin{classdesc}{IncrementalDecoder}{\optional{errors}} Constructor for a \class{IncrementalDecoder} instance. @@ -456,15 +456,15 @@ The \class{StreamWriter} and \class{StreamReader} classes provide generic working interfaces which can be used to implement new -encodings submodules very easily. See \module{encodings.utf_8} for an -example on how this is done. +encoding submodules very easily. See \module{encodings.utf_8} for an +example of how this is done. \subsubsection{StreamWriter Objects \label{stream-writer-objects}} The \class{StreamWriter} class is a subclass of \class{Codec} and defines the following methods which every stream writer must define in -order to be compatible to the Python codec registry. +order to be compatible with the Python codec registry. \begin{classdesc}{StreamWriter}{stream\optional{, errors}} Constructor for a \class{StreamWriter} instance. @@ -473,7 +473,7 @@ free to add additional keyword arguments, but only the ones defined here are used by the Python codec registry. - \var{stream} must be a file-like object open for writing (binary) + \var{stream} must be a file-like object open for writing binary data. The \class{StreamWriter} may implement different error handling @@ -512,19 +512,19 @@ Flushes and resets the codec buffers used for keeping state. Calling this method should ensure that the data on the output is put - into a clean state, that allows appending of new fresh data without + into a clean state that allows appending of new fresh data without having to rescan the whole stream to recover state. \end{methoddesc} In addition to the above methods, the \class{StreamWriter} must also -inherit all other methods and attribute from the underlying stream. +inherit all other methods and attributes from the underlying stream. \subsubsection{StreamReader Objects \label{stream-reader-objects}} The \class{StreamReader} class is a subclass of \class{Codec} and defines the following methods which every stream reader must define in -order to be compatible to the Python codec registry. +order to be compatible with the Python codec registry. \begin{classdesc}{StreamReader}{stream\optional{, errors}} Constructor for a \class{StreamReader} instance. @@ -589,20 +589,20 @@ \var{size}, if given, is passed as size argument to the stream's \method{readline()} method. - If \var{keepends} is false lineends will be stripped from the + If \var{keepends} is false line-endings will be stripped from the lines returned. \versionchanged[\var{keepends} argument added]{2.4} \end{methoddesc} \begin{methoddesc}{readlines}{\optional{sizehint\optional{, keepends}}} - Read all lines available on the input stream and return them as list + Read all lines available on the input stream and return them as a list of lines. - Line breaks are implemented using the codec's decoder method and are + Line-endings are implemented using the codec's decoder method and are included in the list entries if \var{keepends} is true. - \var{sizehint}, if given, is passed as \var{size} argument to the + \var{sizehint}, if given, is passed as the \var{size} argument to the stream's \method{read()} method. \end{methoddesc} @@ -614,7 +614,7 @@ \end{methoddesc} In addition to the above methods, the \class{StreamReader} must also -inherit all other methods and attribute from the underlying stream. +inherit all other methods and attributes from the underlying stream. The next two base classes are included for convenience. They are not needed by the codec registry, but may provide useful in practice. @@ -640,7 +640,7 @@ \class{StreamReaderWriter} instances define the combined interfaces of \class{StreamReader} and \class{StreamWriter} classes. They inherit -all other methods and attribute from the underlying stream. +all other methods and attributes from the underlying stream. \subsubsection{StreamRecoder Objects \label{stream-recoder-objects}} @@ -666,14 +666,14 @@ \var{stream} must be a file-like object. \var{encode}, \var{decode} must adhere to the \class{Codec} - interface, \var{Reader}, \var{Writer} must be factory functions or + interface. \var{Reader}, \var{Writer} must be factory functions or classes providing objects of the \class{StreamReader} and \class{StreamWriter} interface respectively. \var{encode} and \var{decode} are needed for the frontend translation, \var{Reader} and \var{Writer} for the backend translation. The intermediate format used is determined by the two - sets of codecs, e.g. the Unicode codecs will use Unicode as + sets of codecs, e.g. the Unicode codecs will use Unicode as the intermediate encoding. Error handling is done in the same way as defined for the @@ -682,7 +682,7 @@ \class{StreamRecoder} instances define the combined interfaces of \class{StreamReader} and \class{StreamWriter} classes. They inherit -all other methods and attribute from the underlying stream. +all other methods and attributes from the underlying stream. \subsection{Encodings and Unicode\label{encodings-overview}} @@ -695,7 +695,7 @@ memory, CPU endianness and how these arrays are stored as bytes become an issue. Transforming a unicode object into a sequence of bytes is called encoding and recreating the unicode object from the sequence of -bytes is known as decoding. There are many different methods how this +bytes is known as decoding. There are many different methods for how this transformation can be done (these methods are also called encodings). The simplest method is to map the codepoints 0-255 to the bytes \code{0x0}-\code{0xff}. This means that a unicode object that contains @@ -742,7 +742,7 @@ it's a normal character that will be decoded like any other. There's another encoding that is able to encoding the full range of -Unicode characters: UTF-8. UTF-8 is an 8bit encoding, which means +Unicode characters: UTF-8. UTF-8 is an 8-bit encoding, which means there are no issues with byte order in UTF-8. Each byte in a UTF-8 byte sequence consists of two parts: Marker bits (the most significant bits) and payload bits. The marker bits are a sequence of zero to six @@ -762,7 +762,7 @@ The least significant bit of the Unicode character is the rightmost x bit. -As UTF-8 is an 8bit encoding no BOM is required and any \code{U+FEFF} +As UTF-8 is an 8-bit encoding no BOM is required and any \code{U+FEFF} character in the decoded Unicode string (even if it's the first character) is treated as a \samp{ZERO WIDTH NO-BREAK SPACE}. @@ -775,7 +775,7 @@ variant of UTF-8 (that Python 2.5 calls \code{"utf-8-sig"}) for its Notepad program: Before any of the Unicode characters is written to the file, a UTF-8 encoded BOM (which looks like this as a byte sequence: \code{0xef}, -\code{0xbb}, \code{0xbf}) is written. As it's rather improbably that any +\code{0xbb}, \code{0xbf}) is written. As it's rather improbable that any charmap encoded file starts with these byte values (which would e.g. map to LATIN SMALL LETTER I WITH DIAERESIS \\ @@ -794,8 +794,8 @@ \subsection{Standard Encodings\label{standard-encodings}} -Python comes with a number of codecs builtin, either implemented as C -functions, or with dictionaries as mapping tables. The following table +Python comes with a number of codecs built-in, either implemented as C +functions or with dictionaries as mapping tables. The following table lists the codecs by name, together with a few common aliases, and the languages for which the encoding is likely used. Neither the list of aliases nor the list of languages is meant to be exhaustive. Notice @@ -1337,7 +1337,7 @@ UTF-8 codec with BOM signature} \declaremodule{standard}{encodings.utf-8-sig} % XXX utf_8_sig gives TeX errors \modulesynopsis{UTF-8 codec with BOM signature} -\moduleauthor{Walter D\"orwald} +\moduleauthor{Walter D\"orwald}{} \versionadded{2.5} Modified: python/branches/p3yk/Doc/whatsnew/whatsnew25.tex ============================================================================== --- python/branches/p3yk/Doc/whatsnew/whatsnew25.tex (original) +++ python/branches/p3yk/Doc/whatsnew/whatsnew25.tex Fri Apr 21 18:44:05 2006 @@ -4,7 +4,6 @@ % The easy_install stuff % Describe the pkgutil module -% Stateful codec changes % Fix XXX comments % Count up the patches and bugs @@ -35,7 +34,7 @@ %====================================================================== -\section{PEP 243: Uploading Modules to PyPI} +\section{PEP 243: Uploading Modules to PyPI\label{pep-243}} PEP 243 describes an HTTP-based protocol for submitting software packages to a central archive. The Python package index at @@ -61,7 +60,7 @@ %====================================================================== -\section{PEP 308: Conditional Expressions} +\section{PEP 308: Conditional Expressions\label{pep-308}} For a long time, people have been requesting a way to write conditional expressions, expressions that return value A or value B @@ -152,7 +151,7 @@ %====================================================================== -\section{PEP 309: Partial Function Application} +\section{PEP 309: Partial Function Application\label{pep-309}} The \module{functional} module is intended to contain tools for functional-style programming. Currently it only contains a @@ -214,7 +213,7 @@ %====================================================================== -\section{PEP 314: Metadata for Python Software Packages v1.1} +\section{PEP 314: Metadata for Python Software Packages v1.1\label{pep-314}} Some simple dependency support was added to Distutils. The \function{setup()} function now has \code{requires}, \code{provides}, @@ -248,7 +247,7 @@ %====================================================================== -\section{PEP 328: Absolute and Relative Imports} +\section{PEP 328: Absolute and Relative Imports\label{pep-328}} The simpler part of PEP 328 was implemented in Python 2.4: parentheses could now be used to enclose the names imported from a module using @@ -342,7 +341,7 @@ %====================================================================== -\section{PEP 338: Executing Modules as Scripts} +\section{PEP 338: Executing Modules as Scripts\label{pep-338}} The \programopt{-m} switch added in Python 2.4 to execute a module as a script gained a few more abilities. Instead of being implemented in @@ -366,7 +365,7 @@ %====================================================================== -\section{PEP 341: Unified try/except/finally} +\section{PEP 341: Unified try/except/finally\label{pep-341}} Until Python 2.5, the \keyword{try} statement came in two flavours. You could use a \keyword{finally} block to ensure that code @@ -412,7 +411,7 @@ %====================================================================== -\section{PEP 342: New Generator Features\label{section-generators}} +\section{PEP 342: New Generator Features\label{pep-342}} Python 2.5 adds a simple way to pass values \emph{into} a generator. As introduced in Python 2.3, generators only produce output; once a @@ -578,9 +577,9 @@ %====================================================================== -\section{PEP 343: The 'with' statement} +\section{PEP 343: The 'with' statement\label{pep-343}} -The \keyword{with} statement allows a clearer version of code that +The '\keyword{with}' statement allows a clearer version of code that uses \code{try...finally} blocks to ensure that clean-up code is executed. @@ -589,7 +588,7 @@ and show how to write objects called ``context managers'' and ``contexts'' for use with this statement. -The \keyword{with} statement is a new control-flow structure whose +The '\keyword{with}' statement is a new control-flow structure whose basic structure is: \begin{verbatim} @@ -625,11 +624,11 @@ \end{verbatim} After this statement has executed, the file object in \var{f} will -have been automatically closed at this point, even if the 'for' loop +have been automatically closed, even if the 'for' loop raised an exception part-way through the block. The \module{threading} module's locks and condition variables -also support the \keyword{with} statement: +also support the '\keyword{with}' statement: \begin{verbatim} lock = threading.Lock() @@ -658,10 +657,10 @@ print v1.sqrt() \end{verbatim} -\subsection{Writing Context Managers} +\subsection{Writing Context Managers\label{context-managers}} -Under the hood, the \keyword{with} statement is fairly complicated. -Most people will only use \keyword{with} in company with +Under the hood, the '\keyword{with}' statement is fairly complicated. +Most people will only use '\keyword{with}' in company with existing objects that are documented to work as context managers, and don't need to know these details, so you can skip the following section if you like. Authors of new context managers will need to understand the @@ -678,7 +677,7 @@ return a context object. \item The context's \method{__enter__()} method is called. -The value returned is assigned to \var{VAR}. If no \code{as \var{VAR}} +The value returned is assigned to \var{VAR}. If no \code{'as \var{VAR}'} clause is present, the value is simply discarded. \item The code in \var{BLOCK} is executed. @@ -690,7 +689,7 @@ controls whether the exception is re-raised: any false value re-raises the exception, and \code{True} will result in suppressing it. You'll only rarely want to suppress the exception; the -author of the code containing the \keyword{with} statement will +author of the code containing the '\keyword{with}' statement will never realize anything went wrong. \item If \var{BLOCK} didn't raise an exception, @@ -761,7 +760,7 @@ to start a new transaction. In this example, the resulting cursor object would be a useful result, so the method will return it. The user can -then add \code{as cursor} to their \keyword{with} statement +then add \code{as cursor} to their '\keyword{with}' statement to bind the cursor to a variable name. \begin{verbatim} @@ -806,7 +805,7 @@ exactly one value. The code up to the \keyword{yield} will be executed as the \method{__enter__()} method, and the value yielded will be the method's return value that will get bound to the variable -in the \keyword{with} statement's \keyword{as} clause, if any. The +in the '\keyword{with}' statement's \keyword{as} clause, if any. The code after the \keyword{yield} will be executed in the \method{__exit__()} method. Any exception raised in the block will be raised by the \keyword{yield} statement. @@ -854,7 +853,7 @@ There's a \function{nested(\var{mgr1}, \var{mgr2}, ...)} manager that combines a number of context managers so you don't need to write -nested \keyword{with} statements. This example statement does two +nested '\keyword{with}' statements. This example statement does two things, starting a database transaction and acquiring a thread lock: \begin{verbatim} @@ -880,7 +879,7 @@ \seepep{343}{The ``with'' statement}{PEP written by Guido van~Rossum and Nick Coghlan; implemented by Mike Bland, Guido van~Rossum, and -Neal Norwitz. The PEP shows the code generated for a \keyword{with} +Neal Norwitz. The PEP shows the code generated for a '\keyword{with}' statement, which can be helpful in learning how context managers work.} @@ -891,7 +890,7 @@ %====================================================================== -\section{PEP 352: Exceptions as New-Style Classes} +\section{PEP 352: Exceptions as New-Style Classes\label{pep-352}} Exception classes can now be new-style classes, not just classic classes, and the built-in \exception{Exception} class and all the @@ -957,7 +956,7 @@ %====================================================================== -\section{PEP 353: Using ssize_t as the index type\label{section-353}} +\section{PEP 353: Using ssize_t as the index type\label{pep-353}} A wide-ranging change to Python's C API, using a new \ctype{Py_ssize_t} type definition instead of \ctype{int}, @@ -1019,7 +1018,7 @@ %====================================================================== -\section{PEP 357: The '__index__' method} +\section{PEP 357: The '__index__' method\label{pep-357}} The NumPy developers had a problem that could only be solved by adding a new special method, \method{__index__}. When using slice notation, @@ -1092,8 +1091,8 @@ \end{verbatim} \item The \function{min()} and \function{max()} built-in functions -gained a \code{key} keyword argument analogous to the \code{key} -argument for \method{sort()}. This argument supplies a function that +gained a \code{key} keyword parameter analogous to the \code{key} +argument for \method{sort()}. This parameter supplies a function that takes a single argument and is called for every value in the list; \function{min()}/\function{max()} will return the element with the smallest/largest return value from this function. @@ -1186,7 +1185,7 @@ %====================================================================== -\section{New, Improved, and Deprecated Modules} +\section{New, Improved, and Removed Modules} The standard library received many enhancements and bug fixes in Python 2.5. Here's a partial list of the most notable changes, sorted @@ -1196,13 +1195,23 @@ \begin{itemize} -% the cPickle module no longer accepts the deprecated None option in the -% args tuple returned by __reduce__(). - \item The \module{audioop} module now supports the a-LAW encoding, and the code for u-LAW encoding has been improved. (Contributed by Lars Immisch.) +\item The \module{codecs} module gained support for incremental +codecs. The \function{codec.lookup()} function now +returns a \class{CodecInfo} instance instead of a tuple. +\class{CodecInfo} instances behave like a 4-tuple to preserve backward +compatibility but also have the attributes \member{encode}, +\member{decode}, \member{incrementalencoder}, \member{incrementaldecoder}, +\member{streamwriter}, and \member{streamreader}. Incremental codecs +can receive input and produce output in multiple chunks; the output is +the same as if the entire input was fed to the non-incremental codec. +See the \module{codecs} module documentation for details. +(Designed and implemented by Walter D\"orwald.) +% Patch 1436130 + \item The \module{collections} module gained a new type, \class{defaultdict}, that subclasses the standard \class{dict} type. The new type mostly behaves like a dictionary but constructs a @@ -1244,7 +1253,7 @@ raising \exception{ValueError} if the value isn't found. \item New module: The \module{contextlib} module contains helper functions for use -with the new \keyword{with} statement. See +with the new '\keyword{with}' statement. See section~\ref{module-contextlib} for more about this module. (Contributed by Phillip J. Eby.) @@ -1257,6 +1266,11 @@ module's interface, will continue to be maintained in future versions of Python. (Contributed by Armin Rigo.) +Also, the \module{pstats} module used to analyze the data measured by +the profiler now supports directing the output to any file stream +by supplying a \var{stream} argument to the \class{Stats} constructor. +(Contributed by Skip Montanaro.) + \item The \module{csv} module, which parses files in comma-separated value format, received several enhancements and a number of bugfixes. You can now set the maximum size in bytes of a @@ -1302,7 +1316,7 @@ \item The \function{nsmallest()} and \function{nlargest()} functions in the \module{heapq} module -now support a \code{key} keyword argument similar to the one +now support a \code{key} keyword parameter similar to the one provided by the \function{min()}/\function{max()} functions and the \method{sort()} methods. For example: Example: @@ -1375,14 +1389,20 @@ (Contributed by Antti Louko and Diego Petten\`o.) % (Patch 1180695, 1212117) +\item The \module{pickle} and \module{cPickle} modules no +longer accept a return value of \code{None} from the +\method{__reduce__()} method; the method must return a tuple of +arguments instead. The ability to return \code{None} was deprecated +in Python 2.4, so this completes the removal of the feature. + \item The old \module{regex} and \module{regsub} modules, which have been deprecated ever since Python 2.0, have finally been deleted. Other deleted modules: \module{statcache}, \module{tzparse}, \module{whrandom}. -\item The \file{lib-old} directory, +\item Also deleted: the \file{lib-old} directory, which includes ancient modules such as \module{dircmp} and -\module{ni}, was also deleted. \file{lib-old} wasn't on the default +\module{ni}, was removed. \file{lib-old} wasn't on the default \code{sys.path}, so unless your programs explicitly added the directory to \code{sys.path}, this removal shouldn't affect your code. @@ -1824,7 +1844,7 @@ \item The largest change to the C API came from \pep{353}, which modifies the interpreter to use a \ctype{Py_ssize_t} type definition instead of \ctype{int}. See the earlier -section~\ref{section-353} for a discussion of this change. +section~\ref{pep-353} for a discussion of this change. \item The design of the bytecode compiler has changed a great deal, to no longer generate bytecode by traversing the parse tree. Instead @@ -1969,19 +1989,23 @@ characters but doesn't have an encoding declaration. In Python 2.4 this triggered a warning, not a syntax error. -\item The \module{pickle} module no longer uses the deprecated \var{bin} parameter. - \item Previously, the \member{gi_frame} attribute of a generator was always a frame object. Because of the \pep{342} changes -described in section~\ref{section-generators}, it's now possible +described in section~\ref{pep-342}, it's now possible for \member{gi_frame} to be \code{None}. + +\item Library: The \module{pickle} and \module{cPickle} modules no +longer accept a return value of \code{None} from the +\method{__reduce__()} method; the method must return a tuple of +arguments instead. The modules also no longer accept the deprecated +\var{bin} keyword parameter. + \item C API: Many functions now use \ctype{Py_ssize_t} -instead of \ctype{int} to allow processing more data -on 64-bit machines. Extension code may need to make -the same change to avoid warnings and to support 64-bit machines. -See the earlier -section~\ref{section-353} for a discussion of this change. +instead of \ctype{int} to allow processing more data on 64-bit +machines. Extension code may need to make the same change to avoid +warnings and to support 64-bit machines. See the earlier +section~\ref{pep-353} for a discussion of this change. \item C API: The obmalloc changes mean that Modified: python/branches/p3yk/Lib/SimpleXMLRPCServer.py ============================================================================== --- python/branches/p3yk/Lib/SimpleXMLRPCServer.py (original) +++ python/branches/p3yk/Lib/SimpleXMLRPCServer.py Fri Apr 21 18:44:05 2006 @@ -560,6 +560,7 @@ self.handle_xmlrpc(request_text) if __name__ == '__main__': + print 'Running XML-RPC server on port 8000' server = SimpleXMLRPCServer(("localhost", 8000)) server.register_function(pow) server.register_function(lambda x,y: x+y, 'add') Modified: python/branches/p3yk/Objects/stringobject.c ============================================================================== --- python/branches/p3yk/Objects/stringobject.c (original) +++ python/branches/p3yk/Objects/stringobject.c Fri Apr 21 18:44:05 2006 @@ -865,7 +865,7 @@ *p++ = quote; *p = '\0'; _PyString_Resize( - &v, (int) (p - PyString_AS_STRING(v))); + &v, (p - PyString_AS_STRING(v))); return v; } } Modified: python/branches/p3yk/PCbuild/_bsddb.vcproj ============================================================================== --- python/branches/p3yk/PCbuild/_bsddb.vcproj (original) +++ python/branches/p3yk/PCbuild/_bsddb.vcproj Fri Apr 21 18:44:05 2006 @@ -133,7 +133,7 @@ AdditionalOptions=" /USECL:MS_ITANIUM" Optimization="2" InlineFunctionExpansion="1" - AdditionalIncludeDirectories="{MSSDKPATH}\include\Win64\atl;{MSSDKPATH}\include\Win64\crt;{MSSDKPATH}\include\Win64\crt\sys;{MSSDKPATH}\include\Win64\mfc;..\Include;..\PC;"..\..\db-4.4.20\build_win32"" + AdditionalIncludeDirectories="..\Include;..\PC;"..\..\db-4.4.20\build_win32"" PreprocessorDefinitions="NDEBUG;WIN32;_WINDOWS" StringPooling="TRUE" BasicRuntimeChecks="0" @@ -195,7 +195,7 @@ AdditionalOptions=" /USECL:MS_OPTERON" Optimization="2" InlineFunctionExpansion="1" - AdditionalIncludeDirectories="{MSSDKPATH}\include\Win64\atl\amd64;{MSSDKPATH}\include\Win64\crt\amd64;{MSSDKPATH}\include\Win64\crt\amd64\sys;{MSSDKPATH}\include\Win64\mfc\amd64;..\Include;..\PC;"..\..\db-4.4.20\build_win32"" + AdditionalIncludeDirectories="..\Include;..\PC;"..\..\db-4.4.20\build_win32"" PreprocessorDefinitions="NDEBUG;WIN32;_WINDOWS" StringPooling="TRUE" BasicRuntimeChecks="0" Modified: python/branches/p3yk/PCbuild/_ctypes.vcproj ============================================================================== --- python/branches/p3yk/PCbuild/_ctypes.vcproj (original) +++ python/branches/p3yk/PCbuild/_ctypes.vcproj Fri Apr 21 18:44:05 2006 @@ -130,7 +130,7 @@ Name="VCCLCompilerTool" AdditionalOptions=" /USECL:MS_OPTERON" InlineFunctionExpansion="1" - AdditionalIncludeDirectories="{MSSDKPATH}\include\Win64\atl\amd64;{MSSDKPATH}\include\Win64\crt\amd64;{MSSDKPATH}\include\Win64\crt\amd64\sys;{MSSDKPATH}\include\Win64\mfc\amd64;..\Include,..\PC,..\Modules\_ctypes\libffi_msvc" + AdditionalIncludeDirectories="..\Include,..\PC,..\Modules\_ctypes\libffi_msvc" PreprocessorDefinitions="WIN32;NDEBUG;_WINDOWS" StringPooling="TRUE" BasicRuntimeChecks="0" @@ -189,7 +189,7 @@ Name="VCCLCompilerTool" AdditionalOptions=" /USECL:MS_ITANIUM" InlineFunctionExpansion="1" - AdditionalIncludeDirectories="{MSSDKPATH}\include\Win64\atl;{MSSDKPATH}\include\Win64\crt;{MSSDKPATH}\include\Win64\crt\sys;{MSSDKPATH}\include\Win64\mfc;..\Include,..\PC,..\Modules\_ctypes\libffi_msvc" + AdditionalIncludeDirectories="..\Include,..\PC,..\Modules\_ctypes\libffi_msvc" PreprocessorDefinitions="WIN32;NDEBUG;_WINDOWS" StringPooling="TRUE" BasicRuntimeChecks="0" Modified: python/branches/p3yk/PCbuild/_ctypes_test.vcproj ============================================================================== --- python/branches/p3yk/PCbuild/_ctypes_test.vcproj (original) +++ python/branches/p3yk/PCbuild/_ctypes_test.vcproj Fri Apr 21 18:44:05 2006 @@ -126,7 +126,7 @@ Name="VCCLCompilerTool" AdditionalOptions=" /USECL:MS_ITANIUM" Optimization="0" - AdditionalIncludeDirectories="{MSSDKPATH}\include\Win64\atl;{MSSDKPATH}\include\Win64\crt;{MSSDKPATH}\include\Win64\crt\sys;{MSSDKPATH}\include\Win64\mfc;..\Include,..\PC" + AdditionalIncludeDirectories="..\Include,..\PC" PreprocessorDefinitions="WIN32;_DEBUG;_WINDOWS" MinimalRebuild="FALSE" BasicRuntimeChecks="0" @@ -181,7 +181,7 @@ Name="VCCLCompilerTool" AdditionalOptions=" /USECL:MS_OPTERON" InlineFunctionExpansion="1" - AdditionalIncludeDirectories="{MSSDKPATH}\include\Win64\atl\amd64;{MSSDKPATH}\include\Win64\crt\amd64;{MSSDKPATH}\include\Win64\crt\amd64\sys;{MSSDKPATH}\include\Win64\mfc\amd64;..\Include,..\PC" + AdditionalIncludeDirectories="..\Include,..\PC" PreprocessorDefinitions="WIN32;NDEBUG;_WINDOWS" StringPooling="TRUE" BasicRuntimeChecks="0" Modified: python/branches/p3yk/PCbuild/_elementtree.vcproj ============================================================================== --- python/branches/p3yk/PCbuild/_elementtree.vcproj (original) +++ python/branches/p3yk/PCbuild/_elementtree.vcproj Fri Apr 21 18:44:05 2006 @@ -132,7 +132,7 @@ AdditionalOptions=" /USECL:MS_ITANIUM" Optimization="2" InlineFunctionExpansion="1" - AdditionalIncludeDirectories="{MSSDKPATH}\include\Win64\atl;{MSSDKPATH}\include\Win64\crt;{MSSDKPATH}\include\Win64\crt\sys;{MSSDKPATH}\include\Win64\mfc;..\Include,..\PC,..\Modules\expat" + AdditionalIncludeDirectories="..\Include,..\PC,..\Modules\expat" PreprocessorDefinitions="NDEBUG;WIN32;_WINDOWS;XML_NS;XML_DTD;BYTEORDER=1234;XML_CONTEXT_BYTES=1024;USE_PYEXPAT_CAPI;XML_STATIC;HAVE_MEMMOVE" StringPooling="TRUE" BasicRuntimeChecks="0" @@ -193,7 +193,7 @@ AdditionalOptions=" /USECL:MS_OPTERON" Optimization="2" InlineFunctionExpansion="1" - AdditionalIncludeDirectories="{MSSDKPATH}\include\Win64\atl\amd64;{MSSDKPATH}\include\Win64\crt\amd64;{MSSDKPATH}\include\Win64\crt\amd64\sys;{MSSDKPATH}\include\Win64\mfc\amd64;..\Include,..\PC,..\Modules\expat" + AdditionalIncludeDirectories="..\Include,..\PC,..\Modules\expat" PreprocessorDefinitions="NDEBUG;WIN32;_WINDOWS;XML_NS;XML_DTD;BYTEORDER=1234;XML_CONTEXT_BYTES=1024;USE_PYEXPAT_CAPI;XML_STATIC;HAVE_MEMMOVE" StringPooling="TRUE" BasicRuntimeChecks="0" Modified: python/branches/p3yk/PCbuild/_msi.vcproj ============================================================================== --- python/branches/p3yk/PCbuild/_msi.vcproj (original) +++ python/branches/p3yk/PCbuild/_msi.vcproj Fri Apr 21 18:44:05 2006 @@ -132,7 +132,7 @@ AdditionalOptions=" /USECL:MS_ITANIUM" Optimization="2" InlineFunctionExpansion="1" - AdditionalIncludeDirectories="{MSSDKPATH}\include\Win64\atl;{MSSDKPATH}\include\Win64\crt;{MSSDKPATH}\include\Win64\crt\sys;{MSSDKPATH}\include\Win64\mfc;..\Include,..\PC" + AdditionalIncludeDirectories="..\Include,..\PC" PreprocessorDefinitions="WIN32;NDEBUG;_WINDOWS;_USRDLL" StringPooling="TRUE" BasicRuntimeChecks="0" @@ -192,7 +192,7 @@ AdditionalOptions=" /USECL:MS_OPTERON" Optimization="2" InlineFunctionExpansion="1" - AdditionalIncludeDirectories="{MSSDKPATH}\include\Win64\atl\amd64;{MSSDKPATH}\include\Win64\crt\amd64;{MSSDKPATH}\include\Win64\crt\amd64\sys;{MSSDKPATH}\include\Win64\mfc\amd64;..\Include,..\PC" + AdditionalIncludeDirectories="..\Include,..\PC" PreprocessorDefinitions="WIN32;NDEBUG;_WINDOWS;_USRDLL" StringPooling="TRUE" BasicRuntimeChecks="0" Modified: python/branches/p3yk/PCbuild/_socket.vcproj ============================================================================== --- python/branches/p3yk/PCbuild/_socket.vcproj (original) +++ python/branches/p3yk/PCbuild/_socket.vcproj Fri Apr 21 18:44:05 2006 @@ -131,7 +131,7 @@ AdditionalOptions=" /USECL:MS_ITANIUM" Optimization="2" InlineFunctionExpansion="1" - AdditionalIncludeDirectories="{MSSDKPATH}\include\Win64\atl;{MSSDKPATH}\include\Win64\crt;{MSSDKPATH}\include\Win64\crt\sys;{MSSDKPATH}\include\Win64\mfc;..\Include,..\PC" + AdditionalIncludeDirectories="..\Include,..\PC" PreprocessorDefinitions="NDEBUG;WIN32;_WINDOWS" StringPooling="TRUE" BasicRuntimeChecks="0" @@ -192,7 +192,7 @@ AdditionalOptions=" /USECL:MS_OPTERON" Optimization="2" InlineFunctionExpansion="1" - AdditionalIncludeDirectories="{MSSDKPATH}\include\Win64\atl\amd64;{MSSDKPATH}\include\Win64\crt\amd64;{MSSDKPATH}\include\Win64\crt\amd64\sys;{MSSDKPATH}\include\Win64\mfc\amd64;..\Include,..\PC" + AdditionalIncludeDirectories="..\Include,..\PC" PreprocessorDefinitions="NDEBUG;WIN32;_WINDOWS" StringPooling="TRUE" BasicRuntimeChecks="0" Modified: python/branches/p3yk/PCbuild/_sqlite3.vcproj ============================================================================== --- python/branches/p3yk/PCbuild/_sqlite3.vcproj (original) +++ python/branches/p3yk/PCbuild/_sqlite3.vcproj Fri Apr 21 18:44:05 2006 @@ -134,7 +134,7 @@ AdditionalOptions=" /USECL:MS_ITANIUM" Optimization="2" InlineFunctionExpansion="1" - AdditionalIncludeDirectories="{MSSDKPATH}\include\Win64\atl;{MSSDKPATH}\include\Win64\crt;{MSSDKPATH}\include\Win64\crt\sys;{MSSDKPATH}\include\Win64\mfc;..\Include;..\PC;..\..\sqlite-source-3.3.4" + AdditionalIncludeDirectories="..\Include;..\PC;..\..\sqlite-source-3.3.4" PreprocessorDefinitions="NDEBUG;WIN32;_WINDOWS;MODULE_NAME=\"sqlite3\"" StringPooling="TRUE" BasicRuntimeChecks="0" @@ -196,7 +196,7 @@ AdditionalOptions=" /USECL:MS_OPTERON" Optimization="2" InlineFunctionExpansion="1" - AdditionalIncludeDirectories="{MSSDKPATH}\include\Win64\atl\amd64;{MSSDKPATH}\include\Win64\crt\amd64;{MSSDKPATH}\include\Win64\crt\amd64\sys;{MSSDKPATH}\include\Win64\mfc\amd64;..\Include;..\PC;..\..\sqlite-source-3.3.4" + AdditionalIncludeDirectories="..\Include;..\PC;..\..\sqlite-source-3.3.4" PreprocessorDefinitions="NDEBUG;WIN32;_WINDOWS;MODULE_NAME=\"sqlite3\"" StringPooling="TRUE" BasicRuntimeChecks="0" Modified: python/branches/p3yk/PCbuild/_testcapi.vcproj ============================================================================== --- python/branches/p3yk/PCbuild/_testcapi.vcproj (original) +++ python/branches/p3yk/PCbuild/_testcapi.vcproj Fri Apr 21 18:44:05 2006 @@ -129,7 +129,7 @@ AdditionalOptions=" /USECL:MS_ITANIUM" Optimization="2" InlineFunctionExpansion="1" - AdditionalIncludeDirectories="{MSSDKPATH}\include\Win64\atl;{MSSDKPATH}\include\Win64\crt;{MSSDKPATH}\include\Win64\crt\sys;{MSSDKPATH}\include\Win64\mfc;..\Include,..\PC" + AdditionalIncludeDirectories="..\Include,..\PC" PreprocessorDefinitions="WIN32;NDEBUG;_WINDOWS;_USRDLL;MMAP_EXPORTS" StringPooling="TRUE" BasicRuntimeChecks="0" @@ -188,7 +188,7 @@ AdditionalOptions=" /USECL:MS_OPTERON" Optimization="2" InlineFunctionExpansion="1" - AdditionalIncludeDirectories="{MSSDKPATH}\include\Win64\atl\amd64;{MSSDKPATH}\include\Win64\crt\amd64;{MSSDKPATH}\include\Win64\crt\amd64\sys;{MSSDKPATH}\include\Win64\mfc\amd64;..\Include,..\PC" + AdditionalIncludeDirectories="..\Include,..\PC" PreprocessorDefinitions="WIN32;NDEBUG;_WINDOWS;_USRDLL;MMAP_EXPORTS" StringPooling="TRUE" BasicRuntimeChecks="0" Modified: python/branches/p3yk/PCbuild/_tkinter.vcproj ============================================================================== --- python/branches/p3yk/PCbuild/_tkinter.vcproj (original) +++ python/branches/p3yk/PCbuild/_tkinter.vcproj Fri Apr 21 18:44:05 2006 @@ -133,7 +133,7 @@ AdditionalOptions=" /USECL:MS_ITANIUM" Optimization="2" InlineFunctionExpansion="1" - AdditionalIncludeDirectories="{MSSDKPATH}\include\Win64\atl;{MSSDKPATH}\include\Win64\crt;{MSSDKPATH}\include\Win64\crt\sys;{MSSDKPATH}\include\Win64\mfc;..\..\tcltk\include,..\Include,..\PC" + AdditionalIncludeDirectories="..\..\tcltk\include,..\Include,..\PC" PreprocessorDefinitions="NDEBUG;WIN32;_WINDOWS;WITH_APPINIT" StringPooling="TRUE" BasicRuntimeChecks="0" @@ -195,7 +195,7 @@ AdditionalOptions=" /USECL:MS_OPTERON" Optimization="2" InlineFunctionExpansion="1" - AdditionalIncludeDirectories="{MSSDKPATH}\include\Win64\atl\amd64;{MSSDKPATH}\include\Win64\crt\amd64;{MSSDKPATH}\include\Win64\crt\amd64\sys;{MSSDKPATH}\include\Win64\mfc\amd64;..\..\tcltk\include,..\Include,..\PC" + AdditionalIncludeDirectories="..\..\tcltk\include,..\Include,..\PC" PreprocessorDefinitions="NDEBUG;WIN32;_WINDOWS;WITH_APPINIT" StringPooling="TRUE" BasicRuntimeChecks="0" Modified: python/branches/p3yk/PCbuild/bz2.vcproj ============================================================================== --- python/branches/p3yk/PCbuild/bz2.vcproj (original) +++ python/branches/p3yk/PCbuild/bz2.vcproj Fri Apr 21 18:44:05 2006 @@ -140,7 +140,7 @@ AdditionalOptions=" /USECL:MS_ITANIUM" Optimization="2" InlineFunctionExpansion="1" - AdditionalIncludeDirectories="{MSSDKPATH}\include\Win64\atl;{MSSDKPATH}\include\Win64\crt;{MSSDKPATH}\include\Win64\crt\sys;{MSSDKPATH}\include\Win64\mfc;..\Include,..\PC,..\..\bzip2-1.0.3" + AdditionalIncludeDirectories="..\Include,..\PC,..\..\bzip2-1.0.3" PreprocessorDefinitions="NDEBUG;WIN32;_WINDOWS" StringPooling="TRUE" BasicRuntimeChecks="0" @@ -205,7 +205,7 @@ AdditionalOptions=" /USECL:MS_OPTERON" Optimization="2" InlineFunctionExpansion="1" - AdditionalIncludeDirectories="{MSSDKPATH}\include\Win64\atl\amd64;{MSSDKPATH}\include\Win64\crt\amd64;{MSSDKPATH}\include\Win64\crt\amd64\sys;{MSSDKPATH}\include\Win64\mfc\amd64;..\Include,..\PC,..\..\bzip2-1.0.3" + AdditionalIncludeDirectories="..\Include,..\PC,..\..\bzip2-1.0.3" PreprocessorDefinitions="NDEBUG;WIN32;_WINDOWS" StringPooling="TRUE" BasicRuntimeChecks="0" Modified: python/branches/p3yk/PCbuild/pyexpat.vcproj ============================================================================== --- python/branches/p3yk/PCbuild/pyexpat.vcproj (original) +++ python/branches/p3yk/PCbuild/pyexpat.vcproj Fri Apr 21 18:44:05 2006 @@ -131,7 +131,7 @@ AdditionalOptions=" /USECL:MS_ITANIUM" Optimization="2" InlineFunctionExpansion="1" - AdditionalIncludeDirectories="{MSSDKPATH}\include\Win64\atl;{MSSDKPATH}\include\Win64\crt;{MSSDKPATH}\include\Win64\crt\sys;{MSSDKPATH}\include\Win64\mfc;..\Include,..\PC,..\Modules\expat" + AdditionalIncludeDirectories="..\Include,..\PC,..\Modules\expat" PreprocessorDefinitions="NDEBUG;WIN32;_WINDOWS;XML_NS;XML_DTD;BYTEORDER=1234;XML_CONTEXT_BYTES=1024;XML_STATIC;HAVE_MEMMOVE" StringPooling="TRUE" BasicRuntimeChecks="0" @@ -192,7 +192,7 @@ AdditionalOptions=" /USECL:MS_OPTERON" Optimization="2" InlineFunctionExpansion="1" - AdditionalIncludeDirectories="{MSSDKPATH}\include\Win64\atl\amd64;{MSSDKPATH}\include\Win64\crt\amd64;{MSSDKPATH}\include\Win64\crt\amd64\sys;{MSSDKPATH}\include\Win64\mfc\amd64;..\Include,..\PC,..\Modules\expat" + AdditionalIncludeDirectories="..\Include,..\PC,..\Modules\expat" PreprocessorDefinitions="NDEBUG;WIN32;_WINDOWS;XML_NS;XML_DTD;BYTEORDER=1234;XML_CONTEXT_BYTES=1024;XML_STATIC;HAVE_MEMMOVE" StringPooling="TRUE" BasicRuntimeChecks="0" Modified: python/branches/p3yk/PCbuild/python.vcproj ============================================================================== --- python/branches/p3yk/PCbuild/python.vcproj (original) +++ python/branches/p3yk/PCbuild/python.vcproj Fri Apr 21 18:44:05 2006 @@ -140,7 +140,7 @@ AdditionalOptions=" /USECL:MS_ITANIUM /VSEXTCOMP_VERBOSE" Optimization="2" InlineFunctionExpansion="1" - AdditionalIncludeDirectories="{MSSDKPATH}\include\Win64\atl;{MSSDKPATH}\include\Win64\crt;{MSSDKPATH}\include\Win64\crt\sys;{MSSDKPATH}\include\Win64\mfc;..\Include,..\PC" + AdditionalIncludeDirectories="..\Include,..\PC" PreprocessorDefinitions="WIN32;NDEBUG;_CONSOLE" StringPooling="TRUE" BasicRuntimeChecks="0" @@ -204,7 +204,7 @@ AdditionalOptions=" /USECL:MS_OPTERON" Optimization="2" InlineFunctionExpansion="1" - AdditionalIncludeDirectories="{MSSDKPATH}\include\Win64\atl\amd64;{MSSDKPATH}\include\Win64\crt\amd64;{MSSDKPATH}\include\Win64\crt\amd64\sys;{MSSDKPATH}\include\Win64\mfc\amd64;..\Include,..\PC" + AdditionalIncludeDirectories="..\Include,..\PC" PreprocessorDefinitions="WIN32;NDEBUG;_CONSOLE" StringPooling="TRUE" BasicRuntimeChecks="0" Modified: python/branches/p3yk/PCbuild/pythoncore.vcproj ============================================================================== --- python/branches/p3yk/PCbuild/pythoncore.vcproj (original) +++ python/branches/p3yk/PCbuild/pythoncore.vcproj Fri Apr 21 18:44:05 2006 @@ -147,7 +147,7 @@ AdditionalOptions=" /USECL:MS_ITANIUM" Optimization="2" InlineFunctionExpansion="1" - AdditionalIncludeDirectories="{MSSDKPATH}\include\Win64\atl;{MSSDKPATH}\include\Win64\crt;{MSSDKPATH}\include\Win64\crt\sys;{MSSDKPATH}\include\Win64\mfc;..\Include,..\PC" + AdditionalIncludeDirectories="..\Include,..\PC" PreprocessorDefinitions="NDEBUG;WIN32;_WINDOWS;USE_DL_EXPORT" StringPooling="TRUE" BasicRuntimeChecks="0" @@ -214,7 +214,7 @@ AdditionalOptions="/Zm200 /USECL:MS_OPTERON" Optimization="2" InlineFunctionExpansion="1" - AdditionalIncludeDirectories="{MSSDKPATH}\include\Win64\atl\amd64;{MSSDKPATH}\include\Win64\crt\amd64;{MSSDKPATH}\include\Win64\crt\amd64\sys;{MSSDKPATH}\include\Win64\mfc\amd64;..\Include,..\PC" + AdditionalIncludeDirectories="..\Include,..\PC" PreprocessorDefinitions="NDEBUG;WIN32;_WINDOWS;USE_DL_EXPORT" StringPooling="TRUE" BasicRuntimeChecks="0" Modified: python/branches/p3yk/PCbuild/pythonw.vcproj ============================================================================== --- python/branches/p3yk/PCbuild/pythonw.vcproj (original) +++ python/branches/p3yk/PCbuild/pythonw.vcproj Fri Apr 21 18:44:05 2006 @@ -133,7 +133,7 @@ AdditionalOptions=" /USECL:MS_ITANIUM" Optimization="2" InlineFunctionExpansion="1" - AdditionalIncludeDirectories="{MSSDKPATH}\include\Win64\atl;{MSSDKPATH}\include\Win64\crt;{MSSDKPATH}\include\Win64\crt\sys;{MSSDKPATH}\include\Win64\mfc;..\Include,..\PC" + AdditionalIncludeDirectories="..\Include,..\PC" PreprocessorDefinitions="WIN32;NDEBUG;_WINDOWS" StringPooling="TRUE" BasicRuntimeChecks="0" @@ -195,7 +195,7 @@ AdditionalOptions=" /USECL:MS_OPTERON" Optimization="2" InlineFunctionExpansion="1" - AdditionalIncludeDirectories="{MSSDKPATH}\include\Win64\atl\amd64;{MSSDKPATH}\include\Win64\crt\amd64;{MSSDKPATH}\include\Win64\crt\amd64\sys;{MSSDKPATH}\include\Win64\mfc\amd64;..\Include,..\PC" + AdditionalIncludeDirectories="..\Include,..\PC" PreprocessorDefinitions="WIN32;NDEBUG;_WINDOWS" StringPooling="TRUE" BasicRuntimeChecks="0" Modified: python/branches/p3yk/PCbuild/readme.txt ============================================================================== --- python/branches/p3yk/PCbuild/readme.txt (original) +++ python/branches/p3yk/PCbuild/readme.txt Fri Apr 21 18:44:05 2006 @@ -264,7 +264,7 @@ In addition, you need the Visual Studio plugin for external C compilers, from http://sf.net/projects/vsextcomp. The plugin will wrap cl.exe, to locate the proper target compiler, and convert compiler options -accordingly. +accordingly. The project files require atleast version 0.8. Building for AMD64 ------------------ Modified: python/branches/p3yk/PCbuild/select.vcproj ============================================================================== --- python/branches/p3yk/PCbuild/select.vcproj (original) +++ python/branches/p3yk/PCbuild/select.vcproj Fri Apr 21 18:44:05 2006 @@ -21,7 +21,7 @@ Name="VCCLCompilerTool" Optimization="2" InlineFunctionExpansion="1" - AdditionalIncludeDirectories="..\Include,..\PC,..\..\select113" + AdditionalIncludeDirectories="..\Include,..\PC" PreprocessorDefinitions="NDEBUG;WIN32;_WINDOWS" StringPooling="TRUE" RuntimeLibrary="2" @@ -77,7 +77,7 @@ Author: thomas.heller Date: Fri Apr 21 18:48:56 2006 New Revision: 45624 Added: python/trunk/Modules/_ctypes/libffi_msvc/mingwin32.S Modified: python/trunk/Modules/_ctypes/_ctypes.c python/trunk/Modules/_ctypes/libffi/fficonfig.py.in python/trunk/Modules/_ctypes/libffi_msvc/ffi.c python/trunk/Modules/_ctypes/libffi_msvc/ffi.h python/trunk/Modules/_ctypes/libffi_msvc/ffi_common.h python/trunk/Modules/_ctypes/libffi_msvc/ffitarget.h python/trunk/Modules/_ctypes/libffi_msvc/prep_cif.c python/trunk/Modules/_ctypes/libffi_msvc/win32.S Log: Merge in changes from ctypes 0.9.9.6 upstream version. Modified: python/trunk/Modules/_ctypes/_ctypes.c ============================================================================== --- python/trunk/Modules/_ctypes/_ctypes.c (original) +++ python/trunk/Modules/_ctypes/_ctypes.c Fri Apr 21 18:48:56 2006 @@ -1223,6 +1223,19 @@ return value; } } +/* function pointer */ + if (CFuncPtrObject_Check(value)) { + PyCArgObject *parg; + CFuncPtrObject *func; + func = (CFuncPtrObject *)value; + parg = new_CArgObject(); + parg->pffi_type = &ffi_type_pointer; + parg->tag = 'P'; + Py_INCREF(value); + parg->value.p = *(void **)func->b_ptr; + parg->obj = value; + return (PyObject *)parg; + } /* c_char_p, c_wchar_p */ stgd = PyObject_stgdict(value); if (stgd && CDataObject_Check(value) && stgd->proto && PyString_Check(stgd->proto)) { @@ -4407,6 +4420,8 @@ if (PointerTypeObject_Check(arg)) return 1; + if (CFuncPtrTypeObject_Check(arg)) + return 1; dict = PyType_stgdict(arg); if (dict) { if (PyString_Check(dict->proto) @@ -4566,7 +4581,7 @@ #endif PyModule_AddObject(m, "FUNCFLAG_CDECL", PyInt_FromLong(FUNCFLAG_CDECL)); PyModule_AddObject(m, "FUNCFLAG_PYTHONAPI", PyInt_FromLong(FUNCFLAG_PYTHONAPI)); - PyModule_AddStringConstant(m, "__version__", "0.9.9.4"); + PyModule_AddStringConstant(m, "__version__", "0.9.9.6"); PyModule_AddObject(m, "_memmove_addr", PyLong_FromVoidPtr(memmove)); PyModule_AddObject(m, "_memset_addr", PyLong_FromVoidPtr(memset)); Modified: python/trunk/Modules/_ctypes/libffi/fficonfig.py.in ============================================================================== --- python/trunk/Modules/_ctypes/libffi/fficonfig.py.in (original) +++ python/trunk/Modules/_ctypes/libffi/fficonfig.py.in Fri Apr 21 18:48:56 2006 @@ -31,5 +31,6 @@ ffi_sources = [os.path.join('@srcdir@', f) for f in ffi_sources] ffi_cflags = '@CFLAGS@' +# I think this may no longer be needed: if sys.platform == "openbsd3": ffi_cflags += " -fno-stack-protector" Modified: python/trunk/Modules/_ctypes/libffi_msvc/ffi.c ============================================================================== --- python/trunk/Modules/_ctypes/libffi_msvc/ffi.c (original) +++ python/trunk/Modules/_ctypes/libffi_msvc/ffi.c Fri Apr 21 18:48:56 2006 @@ -26,8 +26,6 @@ OTHER DEALINGS IN THE SOFTWARE. ----------------------------------------------------------------------- */ -#ifndef __x86_64__ - #include #include @@ -143,11 +141,7 @@ /*@-declundef@*/ /*@-exportheader@*/ -#ifdef _MSC_VER extern int -#else -extern void -#endif ffi_call_SYSV(void (*)(char *, extended_cif *), /*@out@*/ extended_cif *, unsigned, unsigned, @@ -156,14 +150,9 @@ /*@=declundef@*/ /*@=exportheader@*/ -#if defined(X86_WIN32) || defined(_MSC_VER) /*@-declundef@*/ /*@-exportheader@*/ -#ifdef _MSC_VER extern int -#else -extern void -#endif ffi_call_STDCALL(void (*)(char *, extended_cif *), /*@out@*/ extended_cif *, unsigned, unsigned, @@ -171,13 +160,8 @@ void (*fn)()); /*@=declundef@*/ /*@=exportheader@*/ -#endif /* X86_WIN32 || _MSC_VER*/ -#ifdef _MSC_VER int -#else -void -#endif ffi_call(/*@dependent@*/ ffi_cif *cif, void (*fn)(), /*@out@*/ void *rvalue, @@ -206,24 +190,18 @@ { case FFI_SYSV: /*@-usedef@*/ -#ifdef _MSC_VER - return -#endif - ffi_call_SYSV(ffi_prep_args, &ecif, cif->bytes, - cif->flags, ecif.rvalue, fn); + return ffi_call_SYSV(ffi_prep_args, &ecif, cif->bytes, + cif->flags, ecif.rvalue, fn); /*@=usedef@*/ break; -#if defined(X86_WIN32) || defined(_MSC_VER) + case FFI_STDCALL: /*@-usedef@*/ -#ifdef _MSC_VER - return -#endif - ffi_call_STDCALL(ffi_prep_args, &ecif, cif->bytes, - cif->flags, ecif.rvalue, fn); + return ffi_call_STDCALL(ffi_prep_args, &ecif, cif->bytes, + cif->flags, ecif.rvalue, fn); /*@=usedef@*/ break; -#endif /* X86_WIN32 */ + default: FFI_ASSERT(0); break; @@ -236,23 +214,10 @@ static void ffi_prep_incoming_args_SYSV (char *stack, void **ret, void** args, ffi_cif* cif); -#ifndef _MSC_VER -static void ffi_closure_SYSV (ffi_closure *) - __attribute__ ((regparm(1))); -static void ffi_closure_raw_SYSV (ffi_raw_closure *) - __attribute__ ((regparm(1))); -#endif - /* This function is jumped to by the trampoline */ -#ifdef _MSC_VER static void __fastcall ffi_closure_SYSV (ffi_closure *closure, int *argp) -#else -static void -ffi_closure_SYSV (closure) - ffi_closure *closure; -#endif { // this is our return value storage long double res; @@ -262,11 +227,11 @@ void **arg_area; unsigned short rtype; void *resp = (void*)&res; -#ifdef _MSC_VER +//#ifdef _MSC_VER void *args = &argp[1]; -#else - void *args = __builtin_dwarf_cfa (); -#endif +//#else +// void *args = __builtin_dwarf_cfa (); +//#endif cif = closure->cif; arg_area = (void**) alloca (cif->nargs * sizeof (void*)); @@ -390,7 +355,7 @@ /* MOV EDX, ESP is 0x8b 0xd4 */ -#ifdef _MSC_VER +//#ifdef _MSC_VER #define FFI_INIT_TRAMPOLINE(TRAMP,FUN,CTX,BYTES) \ { unsigned char *__tramp = (unsigned char*)(TRAMP); \ @@ -407,18 +372,18 @@ *(unsigned short*) &__tramp[13] = BYTES; \ } -#else -#define FFI_INIT_TRAMPOLINE(TRAMP,FUN,CTX,BYTES) \ -({ unsigned char *__tramp = (unsigned char*)(TRAMP); \ - unsigned int __fun = (unsigned int)(FUN); \ - unsigned int __ctx = (unsigned int)(CTX); \ - unsigned int __dis = __fun - ((unsigned int) __tramp + FFI_TRAMPOLINE_SIZE); \ - *(unsigned char*) &__tramp[0] = 0xb8; \ - *(unsigned int*) &__tramp[1] = __ctx; /* movl __ctx, %eax */ \ - *(unsigned char *) &__tramp[5] = 0xe9; \ - *(unsigned int*) &__tramp[6] = __dis; /* jmp __fun */ \ - }) -#endif +//#else +//#define FFI_INIT_TRAMPOLINE(TRAMP,FUN,CTX,BYTES) \ +//({ unsigned char *__tramp = (unsigned char*)(TRAMP); \ +// unsigned int __fun = (unsigned int)(FUN); \ +// unsigned int __ctx = (unsigned int)(CTX); \ +// unsigned int __dis = __fun - ((unsigned int) __tramp + FFI_TRAMPOLINE_SIZE); \ +// *(unsigned char*) &__tramp[0] = 0xb8; \ +// *(unsigned int*) &__tramp[1] = __ctx; /* movl __ctx, %eax */ \ +// *(unsigned char *) &__tramp[5] = 0xe9; \ +// *(unsigned int*) &__tramp[6] = __dis; /* jmp __fun */ \ +// }) +//#endif /* the cif must already be prep'ed */ @@ -433,10 +398,8 @@ if (cif->abi == FFI_SYSV) bytes = 0; -#ifdef _MSC_VER else if (cif->abi == FFI_STDCALL) bytes = cif->bytes; -#endif else return FFI_BAD_ABI; @@ -450,5 +413,3 @@ return FFI_OK; } - -#endif /* __x86_64__ */ Modified: python/trunk/Modules/_ctypes/libffi_msvc/ffi.h ============================================================================== --- python/trunk/Modules/_ctypes/libffi_msvc/ffi.h (original) +++ python/trunk/Modules/_ctypes/libffi_msvc/ffi.h Fri Apr 21 18:48:56 2006 @@ -272,11 +272,7 @@ /*@dependent@*/ /*@out@*/ /*@partial@*/ ffi_type *rtype, /*@dependent@*/ ffi_type **atypes); -#ifdef _MSC_VER int -#else -void -#endif ffi_call(/*@dependent@*/ ffi_cif *cif, void (*fn)(), /*@out@*/ void *rvalue, Modified: python/trunk/Modules/_ctypes/libffi_msvc/ffi_common.h ============================================================================== --- python/trunk/Modules/_ctypes/libffi_msvc/ffi_common.h (original) +++ python/trunk/Modules/_ctypes/libffi_msvc/ffi_common.h Fri Apr 21 18:48:56 2006 @@ -13,24 +13,7 @@ #endif #include - -/* Do not move this. Some versions of AIX are very picky about where - this is positioned. */ -#ifdef __GNUC__ -# define alloca __builtin_alloca -#else -# if HAVE_ALLOCA_H -# include -# else -# ifdef _AIX - #pragma alloca -# else -# ifndef alloca /* predefined by HP cc +Olibcalls */ -char *alloca (); -# endif -# endif -# endif -#endif +#include /* Check for the existence of memcpy. */ #if STDC_HEADERS Modified: python/trunk/Modules/_ctypes/libffi_msvc/ffitarget.h ============================================================================== --- python/trunk/Modules/_ctypes/libffi_msvc/ffitarget.h (original) +++ python/trunk/Modules/_ctypes/libffi_msvc/ffitarget.h Fri Apr 21 18:48:56 2006 @@ -43,23 +43,21 @@ FFI_FIRST_ABI = 0, /* ---- Intel x86 Win32 ---------- */ -#if defined(X86_WIN32) || defined(_MSC_VER) FFI_SYSV, FFI_STDCALL, /* TODO: Add fastcall support for the sake of completeness */ FFI_DEFAULT_ABI = FFI_SYSV, -#endif /* ---- Intel x86 and AMD x86-64 - */ -#if !defined(X86_WIN32) && (defined(__i386__) || defined(__x86_64__)) - FFI_SYSV, - FFI_UNIX64, /* Unix variants all use the same ABI for x86-64 */ -#ifdef __i386__ - FFI_DEFAULT_ABI = FFI_SYSV, -#else - FFI_DEFAULT_ABI = FFI_UNIX64, -#endif -#endif +/* #if !defined(X86_WIN32) && (defined(__i386__) || defined(__x86_64__)) */ +/* FFI_SYSV, */ +/* FFI_UNIX64,*/ /* Unix variants all use the same ABI for x86-64 */ +/* #ifdef __i386__ */ +/* FFI_DEFAULT_ABI = FFI_SYSV, */ +/* #else */ +/* FFI_DEFAULT_ABI = FFI_UNIX64, */ +/* #endif */ +/* #endif */ FFI_LAST_ABI = FFI_DEFAULT_ABI + 1 } ffi_abi; Added: python/trunk/Modules/_ctypes/libffi_msvc/mingwin32.S ============================================================================== --- (empty file) +++ python/trunk/Modules/_ctypes/libffi_msvc/mingwin32.S Fri Apr 21 18:48:56 2006 @@ -0,0 +1,228 @@ +/* ----------------------------------------------------------------------- + win32.S - Copyright (c) 1996, 1998, 2001, 2002 Red Hat, Inc. + Copyright (c) 2001 John Beniton + Copyright (c) 2002 Ranjit Mathew + + + X86 Foreign Function Interface + + Permission is hereby granted, free of charge, to any person obtaining + a copy of this software and associated documentation files (the + ``Software''), to deal in the Software without restriction, including + without limitation the rights to use, copy, modify, merge, publish, + distribute, sublicense, and/or sell copies of the Software, and to + permit persons to whom the Software is furnished to do so, subject to + the following conditions: + + The above copyright notice and this permission notice shall be included + in all copies or substantial portions of the Software. + + THE SOFTWARE IS PROVIDED ``AS IS'', WITHOUT WARRANTY OF ANY KIND, EXPRESS + OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF + MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. + IN NO EVENT SHALL CYGNUS SOLUTIONS BE LIABLE FOR ANY CLAIM, DAMAGES OR + OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, + ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + OTHER DEALINGS IN THE SOFTWARE. + ----------------------------------------------------------------------- */ + +#define LIBFFI_ASM +#include +#include + +.text + +.globl ffi_prep_args + + # This assumes we are using gas. + .balign 16 +.globl _ffi_call_SYSV + +_ffi_call_SYSV: + pushl %ebp + movl %esp,%ebp + + # Make room for all of the new args. + movl 16(%ebp),%ecx + subl %ecx,%esp + + movl %esp,%eax + + # Place all of the ffi_prep_args in position + pushl 12(%ebp) + pushl %eax + call *8(%ebp) + + # Return stack to previous state and call the function + addl $8,%esp + + # FIXME: Align the stack to a 128-bit boundary to avoid + # potential performance hits. + + call *28(%ebp) + + # Remove the space we pushed for the args + movl 16(%ebp),%ecx + addl %ecx,%esp + + # Load %ecx with the return type code + movl 20(%ebp),%ecx + + # If the return value pointer is NULL, assume no return value. + cmpl $0,24(%ebp) + jne retint + + # Even if there is no space for the return value, we are + # obliged to handle floating-point values. + cmpl $2,%ecx # Float_type + jne noretval + fstp %st(0) + + jmp epilogue + +retint: + cmpl $1,%ecx # Int_type + jne retfloat + # Load %ecx with the pointer to storage for the return value + movl 24(%ebp),%ecx + movl %eax,0(%ecx) + jmp epilogue + +retfloat: + cmpl $2,%ecx # Float_type + jne retdouble + # Load %ecx with the pointer to storage for the return value + movl 24(%ebp),%ecx + fstps (%ecx) + jmp epilogue + +retdouble: + cmpl $3,%ecx # Double_type + jne retlongdouble + # Load %ecx with the pointer to storage for the return value + movl 24(%ebp),%ecx + fstpl (%ecx) + jmp epilogue + +retlongdouble: + cmpl $4,%ecx # Longdouble_type + jne retint64 + # Load %ecx with the pointer to storage for the return value + movl 24(%ebp),%ecx + fstpt (%ecx) + jmp epilogue + +retint64: + cmpl $12,%ecx # SINT64_type + jne retstruct + # Load %ecx with the pointer to storage for the return value + movl 24(%ebp),%ecx + movl %eax,0(%ecx) + movl %edx,4(%ecx) + +retstruct: + # Nothing to do! + +noretval: +epilogue: + movl %ebp,%esp + popl %ebp + ret + +.ffi_call_SYSV_end: + + # This assumes we are using gas. + .balign 16 +.globl _ffi_call_STDCALL + +_ffi_call_STDCALL: + pushl %ebp + movl %esp,%ebp + + # Make room for all of the new args. + movl 16(%ebp),%ecx + subl %ecx,%esp + + movl %esp,%eax + + # Place all of the ffi_prep_args in position + pushl 12(%ebp) + pushl %eax + call *8(%ebp) + + # Return stack to previous state and call the function + addl $8,%esp + + # FIXME: Align the stack to a 128-bit boundary to avoid + # potential performance hits. + + call *28(%ebp) + + # stdcall functions pop arguments off the stack themselves + + # Load %ecx with the return type code + movl 20(%ebp),%ecx + + # If the return value pointer is NULL, assume no return value. + cmpl $0,24(%ebp) + jne sc_retint + + # Even if there is no space for the return value, we are + # obliged to handle floating-point values. + cmpl $2,%ecx # Float_type + jne sc_noretval + fstp %st(0) + + jmp sc_epilogue + +sc_retint: + cmpl $1,%ecx # Int_type + jne sc_retfloat + # Load %ecx with the pointer to storage for the return value + movl 24(%ebp),%ecx + movl %eax,0(%ecx) + jmp sc_epilogue + +sc_retfloat: + cmpl $2,%ecx # Float_type + jne sc_retdouble + # Load %ecx with the pointer to storage for the return value + movl 24(%ebp),%ecx + fstps (%ecx) + jmp sc_epilogue + +sc_retdouble: + cmpl $2,%ecx # Double_type + jne sc_retlongdouble + # Load %ecx with the pointer to storage for the return value + movl 24(%ebp),%ecx + fstpl (%ecx) + jmp sc_epilogue + +sc_retlongdouble: + cmpl $4,%ecx # Longdouble_type + jne sc_retint64 + # Load %ecx with the pointer to storage for the return value + movl 24(%ebp),%ecx + fstpt (%ecx) + jmp sc_epilogue + +sc_retint64: + cmpl $12,%ecx # SINT64_Type + jne sc_retstruct + # Load %ecx with the pointer to storage for the return value + movl 24(%ebp),%ecx + movl %eax,0(%ecx) + movl %edx,4(%ecx) + +sc_retstruct: + # Nothing to do! + +sc_noretval: +sc_epilogue: + movl %ebp,%esp + popl %ebp + ret + +.ffi_call_STDCALL_end: + Modified: python/trunk/Modules/_ctypes/libffi_msvc/prep_cif.c ============================================================================== --- python/trunk/Modules/_ctypes/libffi_msvc/prep_cif.c (original) +++ python/trunk/Modules/_ctypes/libffi_msvc/prep_cif.c Fri Apr 21 18:48:56 2006 @@ -147,7 +147,7 @@ else #endif { -#ifndef _MSC_VER +#if !defined(_MSC_VER) && !defined(__MINGW32__) /* Don't know if this is a libffi bug or not. At least on Windows with MSVC, function call parameters are *not* aligned in the same way as structure fields are, they are Modified: python/trunk/Modules/_ctypes/libffi_msvc/win32.S ============================================================================== --- python/trunk/Modules/_ctypes/libffi_msvc/win32.S (original) +++ python/trunk/Modules/_ctypes/libffi_msvc/win32.S Fri Apr 21 18:48:56 2006 @@ -41,7 +41,11 @@ _ffi_call_SYSV: pushl %ebp movl %esp,%ebp - + + #THe: save previous %esi, and store the current stack pointer in %esi + pushl %esi + movl %esp,%esi + # Make room for all of the new args. movl 16(%ebp),%ecx subl %ecx,%esp @@ -64,7 +68,9 @@ # Remove the space we pushed for the args movl 16(%ebp),%ecx addl %ecx,%esp - + + sub %esp,%esi # calculate stack pointer difference + # Load %ecx with the return type code movl 20(%ebp),%ecx @@ -125,6 +131,8 @@ noretval: epilogue: + movl %esi,%eax # return the stack pointer detlta in %eax + popl %esi # restore previous %esi movl %ebp,%esp popl %ebp ret @@ -139,6 +147,10 @@ pushl %ebp movl %esp,%ebp + #THe: save previous %esi, and store the current stack pointer in %esi + pushl %esi + movl %esp,%esi + # Make room for all of the new args. movl 16(%ebp),%ecx subl %ecx,%esp @@ -158,6 +170,8 @@ call *28(%ebp) + sub %esp,%esi # difference in stack + # stdcall functions pop arguments off the stack themselves # Load %ecx with the return type code @@ -220,6 +234,8 @@ sc_noretval: sc_epilogue: + movl %esi,%eax # return the stack difference + popl %esi # restore previous %esi value movl %ebp,%esp popl %ebp ret From python-checkins at python.org Fri Apr 21 18:51:05 2006 From: python-checkins at python.org (thomas.heller) Date: Fri, 21 Apr 2006 18:51:05 +0200 (CEST) Subject: [Python-checkins] r45625 - in python/trunk/Lib/ctypes: __init__.py _loader.py test/test_bitfields.py test/test_byteswap.py test/test_callbacks.py test/test_cast.py test/test_cfuncs.py test/test_checkretval.py test/test_find.py test/test_funcptr.py test/test_functions.py test/test_libc.py test/test_loading.py test/test_pointers.py test/test_posix.py test/test_prototypes.py test/test_refcounts.py test/test_returnfuncptrs.py test/test_slicing.py test/test_stringptr.py test/test_unicode.py test/test_values.py test/test_win32.py util.py Message-ID: <20060421165105.8A5201E4006@bag.python.org> Author: thomas.heller Date: Fri Apr 21 18:51:04 2006 New Revision: 45625 Added: python/trunk/Lib/ctypes/test/test_find.py python/trunk/Lib/ctypes/util.py Removed: python/trunk/Lib/ctypes/_loader.py python/trunk/Lib/ctypes/test/test_posix.py Modified: python/trunk/Lib/ctypes/__init__.py python/trunk/Lib/ctypes/test/test_bitfields.py python/trunk/Lib/ctypes/test/test_byteswap.py python/trunk/Lib/ctypes/test/test_callbacks.py python/trunk/Lib/ctypes/test/test_cast.py python/trunk/Lib/ctypes/test/test_cfuncs.py python/trunk/Lib/ctypes/test/test_checkretval.py python/trunk/Lib/ctypes/test/test_funcptr.py python/trunk/Lib/ctypes/test/test_functions.py python/trunk/Lib/ctypes/test/test_libc.py python/trunk/Lib/ctypes/test/test_loading.py python/trunk/Lib/ctypes/test/test_pointers.py python/trunk/Lib/ctypes/test/test_prototypes.py python/trunk/Lib/ctypes/test/test_refcounts.py python/trunk/Lib/ctypes/test/test_returnfuncptrs.py python/trunk/Lib/ctypes/test/test_slicing.py python/trunk/Lib/ctypes/test/test_stringptr.py python/trunk/Lib/ctypes/test/test_unicode.py python/trunk/Lib/ctypes/test/test_values.py python/trunk/Lib/ctypes/test/test_win32.py Log: Merge in changes from ctypes 0.9.9.6 upstream version. Modified: python/trunk/Lib/ctypes/__init__.py ============================================================================== --- python/trunk/Lib/ctypes/__init__.py (original) +++ python/trunk/Lib/ctypes/__init__.py Fri Apr 21 18:51:04 2006 @@ -3,7 +3,7 @@ import os as _os, sys as _sys from itertools import chain as _chain -__version__ = "0.9.9.4" +__version__ = "0.9.9.6" from _ctypes import Union, Structure, Array from _ctypes import _Pointer @@ -23,8 +23,6 @@ from _ctypes import FUNCFLAG_CDECL as _FUNCFLAG_CDECL, \ FUNCFLAG_PYTHONAPI as _FUNCFLAG_PYTHONAPI -from ctypes._loader import LibraryLoader - """ WINOLEAPI -> HRESULT WINOLEAPI_(type) @@ -72,9 +70,11 @@ The function prototype can be called in three ways to create a callable object: - prototype(funct) - returns a C callable function calling funct - prototype(vtbl_index, method_name[, paramflags]) - a Python callable that calls a COM method - prototype(funct_name, dll[, paramflags]) - a Python callable that calls an exported function in a dll + prototype(integer address) -> foreign function + prototype(callable) -> create and return a C callable function from callable + prototype(integer index, method name[, paramflags]) -> foreign function calling a COM method + prototype((ordinal number, dll object)[, paramflags]) -> foreign function exported by ordinal + prototype((function name, dll object)[, paramflags]) -> foreign function exported by name """ try: return _c_functype_cache[(restype, argtypes)] @@ -352,6 +352,23 @@ _flags_ = _FUNCFLAG_STDCALL _restype_ = HRESULT +class LibraryLoader(object): + def __init__(self, dlltype): + self._dlltype = dlltype + + def __getattr__(self, name): + if name[0] == '_': + raise AttributeError(name) + dll = self._dlltype(name) + setattr(self, name, dll) + return dll + + def __getitem__(self, name): + return getattr(self, name) + + def LoadLibrary(self, name): + return self._dlltype(name) + cdll = LibraryLoader(CDLL) pydll = LibraryLoader(PyDLL) @@ -402,7 +419,12 @@ _restype_ = restype _flags_ = _FUNCFLAG_CDECL | _FUNCFLAG_PYTHONAPI return CFunctionType -cast = PYFUNCTYPE(py_object, c_void_p, py_object)(_cast_addr) +_cast = PYFUNCTYPE(py_object, c_void_p, py_object)(_cast_addr) + +def cast(obj, typ): + result = _cast(obj, typ) + result.__keepref = obj + return result _string_at = CFUNCTYPE(py_object, c_void_p, c_int)(_string_at_addr) def string_at(ptr, size=0): Deleted: /python/trunk/Lib/ctypes/_loader.py ============================================================================== --- /python/trunk/Lib/ctypes/_loader.py Fri Apr 21 18:51:04 2006 +++ (empty file) @@ -1,262 +0,0 @@ -import sys, os -import ctypes - -if os.name in ("nt", "ce"): - from _ctypes import LoadLibrary as dlopen -else: - from _ctypes import dlopen -from _ctypes import RTLD_LOCAL, RTLD_GLOBAL - -# _findLib(name) returns an iterable of possible names for a library. -if os.name in ("nt", "ce"): - def _findLib(name): - return [name] - -if os.name == "posix" and sys.platform == "darwin": - from ctypes.macholib.dyld import dyld_find as _dyld_find - def _findLib(name): - possible = ['lib%s.dylib' % name, - '%s.dylib' % name, - '%s.framework/%s' % (name, name)] - for name in possible: - try: - return [_dyld_find(name)] - except ValueError: - continue - return [] - -elif os.name == "posix": - # Andreas Degert's find functions, using gcc, /sbin/ldconfig, objdump - import re, tempfile - - def _findLib_gcc(name): - expr = '[^\(\)\s]*lib%s\.[^\(\)\s]*' % name - cmd = 'if type gcc &>/dev/null; then CC=gcc; else CC=cc; fi;' \ - '$CC -Wl,-t -o /dev/null 2>&1 -l' + name - try: - fdout, outfile = tempfile.mkstemp() - fd = os.popen(cmd) - trace = fd.read() - err = fd.close() - finally: - try: - os.unlink(outfile) - except OSError, e: - if e.errno != errno.ENOENT: - raise - res = re.search(expr, trace) - if not res: - return None - return res.group(0) - - def _findLib_ld(name): - expr = '/[^\(\)\s]*lib%s\.[^\(\)\s]*' % name - res = re.search(expr, os.popen('/sbin/ldconfig -p 2>/dev/null').read()) - if not res: - cmd = 'ldd %s 2>/dev/null' % sys.executable - res = re.search(expr, os.popen(cmd).read()) - if not res: - return None - return res.group(0) - - def _get_soname(f): - cmd = "objdump -p -j .dynamic 2>/dev/null " + f - res = re.search(r'\sSONAME\s+([^\s]+)', os.popen(cmd).read()) - if not res: - return f - return res.group(1) - - def _findLib(name): - lib = _findLib_ld(name) - if not lib: - lib = _findLib_gcc(name) - if not lib: - return [name] - return [_get_soname(lib)] - -class LibraryLoader(object): - """Loader for shared libraries. - - Shared libraries are accessed when compiling/linking a program, - and when the program is run. The purpose of the 'find' method is - to locate a library similar to what the compiler does (on machines - with several versions of a shared library the most recent should - be loaded), while 'load' acts like when the program is run, and - uses the runtime loader directly. 'load_version' works like - 'load' but tries to be platform independend (for cases where this - makes sense). Loading via attribute access is a shorthand - notation especially useful for interactive use.""" - - - def __init__(self, dlltype, mode=RTLD_LOCAL): - """Create a library loader instance which loads libraries by - creating an instance of 'dlltype'. 'mode' can be RTLD_LOCAL - or RTLD_GLOBAL, it is ignored on Windows. - """ - self._dlltype = dlltype - self._mode = mode - - def load(self, libname, mode=None): - """Load and return the library with the given libname. On - most systems 'libname' is the filename of the shared library; - when it's not a pathname it will be searched in a system - dependend list of locations (on many systems additional search - paths can be specified by an environment variable). Sometimes - the extension (like '.dll' on Windows) can be omitted. - - 'mode' allows to override the default flags specified in the - constructor, it is ignored on Windows. - """ - if mode is None: - mode = self._mode - return self._load(libname, mode) - - def load_library(self, libname, mode=None): - """Load and return the library with the given libname. This - method passes the specified 'libname' directly to the - platform's library loading function (dlopen, or LoadLibrary). - - 'mode' allows to override the default flags specified in the - constructor, it is ignored on Windows. - """ - if mode is None: - mode = self._mode - return self._dlltype(libname, mode) - - # alias name for backwards compatiblity - LoadLibrary = load_library - - # Helpers for load and load_version - assembles a filename from name and filename - if os.name in ("nt", "ce"): - # Windows (XXX what about cygwin?) - def _plat_load_version(self, name, version, mode): - # not sure if this makes sense - if version is not None: - return self.load(name + version, mode) - return self.load(name, mode) - - _load = load_library - - elif os.name == "posix" and sys.platform == "darwin": - # Mac OS X - def _plat_load_version(self, name, version, mode): - if version: - return self.load("lib%s.%s.dylib" % (name, version), mode) - return self.load("lib%s.dylib" % name, mode) - - def _load(self, libname, mode): - # _dyld_find raises ValueError, convert this into OSError - try: - pathname = _dyld_find(libname) - except ValueError: - raise OSError("Library %s could not be found" % libname) - return self.load_library(pathname, mode) - - elif os.name == "posix": - # Posix - def _plat_load_version(self, name, version, mode): - if version: - return self.load("lib%s.so.%s" % (name, version), mode) - return self.load("lib%s.so" % name, mode) - - _load = load_library - - else: - # Others, TBD - def _plat_load_version(self, name, version, mode=None): - return self.load(name, mode) - - _load = load_library - - def load_version(self, name, version=None, mode=None): - """Build a (system dependend) filename from 'name' and - 'version', then load and return it. 'name' is the library - name without any prefix like 'lib' and suffix like '.so' or - '.dylib'. This method should be used if a library is - available on different platforms, using the particular naming - convention of each platform. - - 'mode' allows to override the default flags specified in the - constructor, it is ignored on Windows. - """ - return self._plat_load_version(name, version, mode) - - def find(self, name, mode=None): - """Try to find a library, load and return it. 'name' is the - library name without any prefix like 'lib', suffix like '.so', - '.dylib' or version number (this is the form used for the - posix linker option '-l'). - - 'mode' allows to override the default flags specified in the - constructor, it is ignored on Windows. - - On windows, this method does the same as the 'load' method. - - On other platforms, this function might call other programs - like the compiler to find the library. When using ctypes to - write a shared library wrapping, consider using .load() or - .load_version() instead. - """ - for libname in _findLib(name): - try: - return self.load(libname, mode) - except OSError: - continue - raise OSError("Library %r not found" % name) - - def __getattr__(self, name): - """Load a library via attribute access. Calls - .load_version(). The result is cached.""" - if name.startswith("_"): - raise AttributeError(name) - dll = self.load_version(name) - setattr(self, name, dll) - return dll - -################################################################ -# test code - -class CDLL(object): - def __init__(self, name, mode): - self._handle = dlopen(name, mode) - self._name = name - - def __repr__(self): - return "<%s '%s', handle %x at %x>" % \ - (self.__class__.__name__, self._name, - (self._handle & (sys.maxint*2 + 1)), - id(self)) - -cdll = LibraryLoader(CDLL) - -def test(): - if os.name == "nt": - print cdll.msvcrt - print cdll.load("msvcrt") - # load_version looks more like an artefact: - print cdll.load_version("msvcr", "t") - print cdll.find("msvcrt") - - if os.name == "posix": - # find and load_version - print cdll.find("m") - print cdll.find("c") - print cdll.load_version("crypto", "0.9.7") - - # getattr - print cdll.m - print cdll.bz2 - - # load - if sys.platform == "darwin": - print cdll.load("libm.dylib") - print cdll.load("libcrypto.dylib") - print cdll.load("libSystem.dylib") - print cdll.load("System.framework/System") - else: - print cdll.load("libm.so") - print cdll.load("libcrypt.so") - print cdll.find("crypt") - -if __name__ == "__main__": - test() Modified: python/trunk/Lib/ctypes/test/test_bitfields.py ============================================================================== --- python/trunk/Lib/ctypes/test/test_bitfields.py (original) +++ python/trunk/Lib/ctypes/test/test_bitfields.py Fri Apr 21 18:51:04 2006 @@ -24,7 +24,7 @@ ("R", c_short, 6), ("S", c_short, 7)] -func = cdll.load(_ctypes_test.__file__).unpack_bitfields +func = CDLL(_ctypes_test.__file__).unpack_bitfields func.argtypes = POINTER(BITS), c_char ##for n in "ABCDEFGHIMNOPQRS": Modified: python/trunk/Lib/ctypes/test/test_byteswap.py ============================================================================== --- python/trunk/Lib/ctypes/test/test_byteswap.py (original) +++ python/trunk/Lib/ctypes/test/test_byteswap.py Fri Apr 21 18:51:04 2006 @@ -15,7 +15,7 @@ class Test(unittest.TestCase): def X_test(self): - print sys.byteorder + print >> sys.stderr, sys.byteorder for i in range(32): bits = BITS() setattr(bits, "i%s" % i, 1) Modified: python/trunk/Lib/ctypes/test/test_callbacks.py ============================================================================== --- python/trunk/Lib/ctypes/test/test_callbacks.py (original) +++ python/trunk/Lib/ctypes/test/test_callbacks.py Fri Apr 21 18:51:04 2006 @@ -115,7 +115,7 @@ def test_integrate(self): # Derived from some then non-working code, posted by David Foster - dll = cdll.load(_ctypes_test.__file__) + dll = CDLL(_ctypes_test.__file__) # The function prototype called by 'integrate': double func(double); CALLBACK = CFUNCTYPE(c_double, c_double) Modified: python/trunk/Lib/ctypes/test/test_cast.py ============================================================================== --- python/trunk/Lib/ctypes/test/test_cast.py (original) +++ python/trunk/Lib/ctypes/test/test_cast.py Fri Apr 21 18:51:04 2006 @@ -23,33 +23,24 @@ def test_address2pointer(self): array = (c_int * 3)(42, 17, 2) - # on AMD64, sizeof(int) == 4 and sizeof(void *) == 8. - # By default, cast would convert a Python int (or long) into - # a C int, which would be too short to represent a pointer - # on this platform. - - # So we have to wrap the address into a c_void_p for this to work. - # - # XXX Better would be to hide the differences in the cast function. address = addressof(array) ptr = cast(c_void_p(address), POINTER(c_int)) self.failUnlessEqual([ptr[i] for i in range(3)], [42, 17, 2]) + ptr = cast(address, POINTER(c_int)) + self.failUnlessEqual([ptr[i] for i in range(3)], [42, 17, 2]) + def test_ptr2array(self): array = (c_int * 3)(42, 17, 2) -## # Hm, already tested above. -## ptr = cast(array, POINTER(c_int)) -## self.failUnlessEqual([ptr[i] for i in range(3)], [42, 17, 2]) - -# print cast(addressof(array), c_int * 3)[:] -## ptr = cast(addressof(ptr) - -## print ptr[0], ptr[1], ptr[2] -## ptr = POINTER(c_int).from_address(addressof(array)) -## # XXX this crashes: -## print ptr[0], ptr[1], ptr[2] + from sys import getrefcount + + before = getrefcount(array) + ptr = cast(array, POINTER(c_int)) + self.failUnlessEqual(getrefcount(array), before + 1) + del ptr + self.failUnlessEqual(getrefcount(array), before) if __name__ == "__main__": unittest.main() Modified: python/trunk/Lib/ctypes/test/test_cfuncs.py ============================================================================== --- python/trunk/Lib/ctypes/test/test_cfuncs.py (original) +++ python/trunk/Lib/ctypes/test/test_cfuncs.py Fri Apr 21 18:51:04 2006 @@ -7,7 +7,7 @@ import _ctypes_test class CFunctions(unittest.TestCase): - _dll = cdll.load(_ctypes_test.__file__) + _dll = CDLL(_ctypes_test.__file__) def S(self): return c_longlong.in_dll(self._dll, "last_tf_arg_s").value Modified: python/trunk/Lib/ctypes/test/test_checkretval.py ============================================================================== --- python/trunk/Lib/ctypes/test/test_checkretval.py (original) +++ python/trunk/Lib/ctypes/test/test_checkretval.py Fri Apr 21 18:51:04 2006 @@ -14,7 +14,7 @@ def test_checkretval(self): import _ctypes_test - dll = cdll.load(_ctypes_test.__file__) + dll = CDLL(_ctypes_test.__file__) self.failUnlessEqual(42, dll._testfunc_p_p(42)) dll._testfunc_p_p.restype = CHECKED Added: python/trunk/Lib/ctypes/test/test_find.py ============================================================================== --- (empty file) +++ python/trunk/Lib/ctypes/test/test_find.py Fri Apr 21 18:51:04 2006 @@ -0,0 +1,90 @@ +import unittest +import os, sys +from ctypes import * +from ctypes.util import find_library +from ctypes.test import is_resource_enabled + +if sys.platform == "win32": + lib_gl = find_library("OpenGL32") + lib_glu = find_library("Glu32") + lib_glut = find_library("glut32") + lib_gle = None +elif sys.platform == "darwin": + lib_gl = lib_glu = find_library("OpenGL") + lib_glut = find_library("GLUT") + lib_gle = None +else: + lib_gl = find_library("GL") + lib_glu = find_library("GLU") + lib_glut = find_library("glut") + lib_gle = find_library("gle") + +## print, for debugging +if is_resource_enabled("printing"): + if lib_gl or lib_glu or lib_glut or lib_gle: + print "OpenGL libraries:" + for item in (("GL", lib_gl), + ("GLU", lib_glu), + ("glut", lib_glut), + ("gle", lib_gle)): + print "\t", item + + +# On some systems, loading the OpenGL libraries needs the RTLD_GLOBAL mode. +class Test_OpenGL_libs(unittest.TestCase): + def setUp(self): + self.gl = self.glu = self.gle = self.glut = None + if lib_gl: + self.gl = CDLL(lib_gl, mode=RTLD_GLOBAL) + if lib_glu: + self.glu = CDLL(lib_glu, RTLD_GLOBAL) + if lib_glut: + self.glut = CDLL(lib_glut) + if lib_gle: + self.gle = CDLL(lib_gle) + + if lib_gl: + def test_gl(self): + if self.gl: + self.gl.glClearIndex + + if lib_glu: + def test_glu(self): + if self.glu: + self.glu.gluBeginCurve + + if lib_glut: + def test_glut(self): + if self.glut: + self.glut.glutWireTetrahedron + + if lib_gle: + def test_gle(self): + if self.gle: + self.gle.gleGetJoinStyle + +##if os.name == "posix" and sys.platform != "darwin": + +## # On platforms where the default shared library suffix is '.so', +## # at least some libraries can be loaded as attributes of the cdll +## # object, since ctypes now tries loading the lib again +## # with '.so' appended of the first try fails. +## # +## # Won't work for libc, unfortunately. OTOH, it isn't +## # needed for libc since this is already mapped into the current +## # process (?) +## # +## # On MAC OSX, it won't work either, because dlopen() needs a full path, +## # and the default suffix is either none or '.dylib'. + +## class LoadLibs(unittest.TestCase): +## def test_libm(self): +## import math +## libm = cdll.libm +## sqrt = libm.sqrt +## sqrt.argtypes = (c_double,) +## sqrt.restype = c_double +## self.failUnlessEqual(sqrt(2), math.sqrt(2)) + +if __name__ == "__main__": + unittest.main() Modified: python/trunk/Lib/ctypes/test/test_funcptr.py ============================================================================== --- python/trunk/Lib/ctypes/test/test_funcptr.py (original) +++ python/trunk/Lib/ctypes/test/test_funcptr.py Fri Apr 21 18:51:04 2006 @@ -8,7 +8,7 @@ WINFUNCTYPE = CFUNCTYPE import _ctypes_test -lib = cdll.load(_ctypes_test.__file__) +lib = CDLL(_ctypes_test.__file__) class CFuncPtrTestCase(unittest.TestCase): def test_basic(self): Modified: python/trunk/Lib/ctypes/test/test_functions.py ============================================================================== --- python/trunk/Lib/ctypes/test/test_functions.py (original) +++ python/trunk/Lib/ctypes/test/test_functions.py Fri Apr 21 18:51:04 2006 @@ -15,9 +15,9 @@ WINFUNCTYPE = CFUNCTYPE import _ctypes_test -dll = cdll.load(_ctypes_test.__file__) +dll = CDLL(_ctypes_test.__file__) if sys.platform == "win32": - windll = windll.load(_ctypes_test.__file__) + windll = WinDLL(_ctypes_test.__file__) class POINT(Structure): _fields_ = [("x", c_int), ("y", c_int)] Modified: python/trunk/Lib/ctypes/test/test_libc.py ============================================================================== --- python/trunk/Lib/ctypes/test/test_libc.py (original) +++ python/trunk/Lib/ctypes/test/test_libc.py Fri Apr 21 18:51:04 2006 @@ -4,7 +4,7 @@ from ctypes import * import _ctypes_test -lib = cdll.load(_ctypes_test.__file__) +lib = CDLL(_ctypes_test.__file__) class LibTest(unittest.TestCase): def test_sqrt(self): Modified: python/trunk/Lib/ctypes/test/test_loading.py ============================================================================== --- python/trunk/Lib/ctypes/test/test_loading.py (original) +++ python/trunk/Lib/ctypes/test/test_loading.py Fri Apr 21 18:51:04 2006 @@ -1,6 +1,8 @@ from ctypes import * import sys, unittest import os, StringIO +from ctypes.util import find_library +from ctypes.test import is_resource_enabled libc_name = None if os.name == "nt": @@ -18,39 +20,49 @@ libc_name = line.split()[4] else: libc_name = line.split()[2] -## print "libc_name is", libc_name break +if is_resource_enabled("printing"): + print "libc_name is", libc_name + class LoaderTest(unittest.TestCase): unknowndll = "xxrandomnamexx" if libc_name is not None: def test_load(self): - cdll.load(libc_name) - cdll.load(os.path.basename(libc_name)) - self.assertRaises(OSError, cdll.load, self.unknowndll) + CDLL(libc_name) + CDLL(os.path.basename(libc_name)) + self.assertRaises(OSError, CDLL, self.unknowndll) if libc_name is not None and os.path.basename(libc_name) == "libc.so.6": def test_load_version(self): - cdll.load_version("c", "6") + cdll.LoadLibrary("libc.so.6") # linux uses version, libc 9 should not exist - self.assertRaises(OSError, cdll.load_version, "c", "9") - self.assertRaises(OSError, cdll.load_version, self.unknowndll, "") + self.assertRaises(OSError, cdll.LoadLibrary, "libc.so.9") + self.assertRaises(OSError, cdll.LoadLibrary, self.unknowndll) - def test_find(self): - name = "c" - cdll.find(name) - self.assertRaises(OSError, cdll.find, self.unknowndll) + def test_find(self): + for name in ("c", "m"): + lib = find_library(name) + if lib: + cdll.LoadLibrary(lib) + CDLL(lib) if os.name in ("nt", "ce"): def test_load_library(self): + if is_resource_enabled("printing"): + print find_library("kernel32") + print find_library("user32") + if os.name == "nt": - windll.load_library("kernel32").GetModuleHandleW + windll.kernel32.GetModuleHandleW + windll["kernel32"].GetModuleHandleW windll.LoadLibrary("kernel32").GetModuleHandleW WinDLL("kernel32").GetModuleHandleW elif os.name == "ce": - windll.load_library("coredll").GetModuleHandleW + windll.coredll.GetModuleHandleW + windll["coredll"].GetModuleHandleW windll.LoadLibrary("coredll").GetModuleHandleW WinDLL("coredll").GetModuleHandleW Modified: python/trunk/Lib/ctypes/test/test_pointers.py ============================================================================== --- python/trunk/Lib/ctypes/test/test_pointers.py (original) +++ python/trunk/Lib/ctypes/test/test_pointers.py Fri Apr 21 18:51:04 2006 @@ -20,7 +20,7 @@ self.failUnlessRaises(TypeError, A, c_ulong(33)) def test_pass_pointers(self): - dll = cdll.load(_ctypes_test.__file__) + dll = CDLL(_ctypes_test.__file__) func = dll._testfunc_p_p func.restype = c_long @@ -35,7 +35,7 @@ self.failUnlessEqual(res[0], 12345678) def test_change_pointers(self): - dll = cdll.load(_ctypes_test.__file__) + dll = CDLL(_ctypes_test.__file__) func = dll._testfunc_p_p i = c_int(87654) @@ -70,7 +70,7 @@ return 0 callback = PROTOTYPE(func) - dll = cdll.load(_ctypes_test.__file__) + dll = CDLL(_ctypes_test.__file__) # This function expects a function pointer, # and calls this with an integer pointer as parameter. # The int pointer points to a table containing the numbers 1..10 @@ -156,7 +156,7 @@ def test_charpp( self ): """Test that a character pointer-to-pointer is correctly passed""" - dll = cdll.load(_ctypes_test.__file__) + dll = CDLL(_ctypes_test.__file__) func = dll._testfunc_c_p_p func.restype = c_char_p argv = (c_char_p * 2)() Deleted: /python/trunk/Lib/ctypes/test/test_posix.py ============================================================================== --- /python/trunk/Lib/ctypes/test/test_posix.py Fri Apr 21 18:51:04 2006 +++ (empty file) @@ -1,40 +0,0 @@ -import unittest, os, sys -from ctypes import * - -if os.name == "posix" and sys.platform == "linux2": - # I don't really know on which platforms this works, - # later it should use the find_library stuff to avoid - # hardcoding the names. - - class TestRTLD_GLOBAL(unittest.TestCase): - def test_GL(self): - if os.path.exists('/usr/lib/libGL.so'): - cdll.load('libGL.so', mode=RTLD_GLOBAL) - if os.path.exists('/usr/lib/libGLU.so'): - cdll.load('libGLU.so') - -##if os.name == "posix" and sys.platform != "darwin": - -## # On platforms where the default shared library suffix is '.so', -## # at least some libraries can be loaded as attributes of the cdll -## # object, since ctypes now tries loading the lib again -## # with '.so' appended of the first try fails. -## # -## # Won't work for libc, unfortunately. OTOH, it isn't -## # needed for libc since this is already mapped into the current -## # process (?) -## # -## # On MAC OSX, it won't work either, because dlopen() needs a full path, -## # and the default suffix is either none or '.dylib'. - -## class LoadLibs(unittest.TestCase): -## def test_libm(self): -## import math -## libm = cdll.libm -## sqrt = libm.sqrt -## sqrt.argtypes = (c_double,) -## sqrt.restype = c_double -## self.failUnlessEqual(sqrt(2), math.sqrt(2)) - -if __name__ == "__main__": - unittest.main() Modified: python/trunk/Lib/ctypes/test/test_prototypes.py ============================================================================== --- python/trunk/Lib/ctypes/test/test_prototypes.py (original) +++ python/trunk/Lib/ctypes/test/test_prototypes.py Fri Apr 21 18:51:04 2006 @@ -22,7 +22,7 @@ # In this case, there would have to be an additional reference to the argument... import _ctypes_test -testdll = cdll.load(_ctypes_test.__file__) +testdll = CDLL(_ctypes_test.__file__) # Return machine address `a` as a (possibly long) non-negative integer. # Starting with Python 2.5, id(anything) is always non-negative, and Modified: python/trunk/Lib/ctypes/test/test_refcounts.py ============================================================================== --- python/trunk/Lib/ctypes/test/test_refcounts.py (original) +++ python/trunk/Lib/ctypes/test/test_refcounts.py Fri Apr 21 18:51:04 2006 @@ -6,7 +6,7 @@ OtherCallback = ctypes.CFUNCTYPE(ctypes.c_int, ctypes.c_int, ctypes.c_ulonglong) import _ctypes_test -dll = ctypes.cdll.load(_ctypes_test.__file__) +dll = ctypes.CDLL(_ctypes_test.__file__) class RefcountTestCase(unittest.TestCase): Modified: python/trunk/Lib/ctypes/test/test_returnfuncptrs.py ============================================================================== --- python/trunk/Lib/ctypes/test/test_returnfuncptrs.py (original) +++ python/trunk/Lib/ctypes/test/test_returnfuncptrs.py Fri Apr 21 18:51:04 2006 @@ -8,7 +8,7 @@ def test_with_prototype(self): # The _ctypes_test shared lib/dll exports quite some functions for testing. # The get_strchr function returns a *pointer* to the C strchr function. - dll = cdll.load(_ctypes_test.__file__) + dll = CDLL(_ctypes_test.__file__) get_strchr = dll.get_strchr get_strchr.restype = CFUNCTYPE(c_char_p, c_char_p, c_char) strchr = get_strchr() @@ -18,7 +18,7 @@ self.assertRaises(TypeError, strchr, "abcdef") def test_without_prototype(self): - dll = cdll.load(_ctypes_test.__file__) + dll = CDLL(_ctypes_test.__file__) get_strchr = dll.get_strchr # the default 'c_int' would not work on systems where sizeof(int) != sizeof(void *) get_strchr.restype = c_void_p Modified: python/trunk/Lib/ctypes/test/test_slicing.py ============================================================================== --- python/trunk/Lib/ctypes/test/test_slicing.py (original) +++ python/trunk/Lib/ctypes/test/test_slicing.py Fri Apr 21 18:51:04 2006 @@ -37,7 +37,7 @@ def test_char_ptr(self): s = "abcdefghijklmnopqrstuvwxyz\0" - dll = cdll.load(_ctypes_test.__file__) + dll = CDLL(_ctypes_test.__file__) dll.my_strdup.restype = POINTER(c_char) res = dll.my_strdup(s) self.failUnlessEqual(res[:len(s)], s) @@ -65,7 +65,7 @@ def test_wchar_ptr(self): s = u"abcdefghijklmnopqrstuvwxyz\0" - dll = cdll.load(_ctypes_test.__file__) + dll = CDLL(_ctypes_test.__file__) dll.my_wcsdup.restype = POINTER(c_wchar) dll.my_wcsdup.argtypes = POINTER(c_wchar), res = dll.my_wcsdup(s) Modified: python/trunk/Lib/ctypes/test/test_stringptr.py ============================================================================== --- python/trunk/Lib/ctypes/test/test_stringptr.py (original) +++ python/trunk/Lib/ctypes/test/test_stringptr.py Fri Apr 21 18:51:04 2006 @@ -3,7 +3,7 @@ import _ctypes_test -lib = cdll.load(_ctypes_test.__file__) +lib = CDLL(_ctypes_test.__file__) class StringPtrTestCase(unittest.TestCase): Modified: python/trunk/Lib/ctypes/test/test_unicode.py ============================================================================== --- python/trunk/Lib/ctypes/test/test_unicode.py (original) +++ python/trunk/Lib/ctypes/test/test_unicode.py Fri Apr 21 18:51:04 2006 @@ -8,7 +8,7 @@ pass else: import _ctypes_test - dll = ctypes.cdll.load(_ctypes_test.__file__) + dll = ctypes.CDLL(_ctypes_test.__file__) wcslen = dll.my_wcslen wcslen.argtypes = [ctypes.c_wchar_p] @@ -66,7 +66,7 @@ self.failUnlessEqual(buf[:], u"ab\0\0\0\0") import _ctypes_test - func = ctypes.cdll.load(_ctypes_test.__file__)._testfunc_p_p + func = ctypes.CDLL(_ctypes_test.__file__)._testfunc_p_p class StringTestCase(UnicodeTestCase): def setUp(self): Modified: python/trunk/Lib/ctypes/test/test_values.py ============================================================================== --- python/trunk/Lib/ctypes/test/test_values.py (original) +++ python/trunk/Lib/ctypes/test/test_values.py Fri Apr 21 18:51:04 2006 @@ -10,7 +10,7 @@ class ValuesTestCase(unittest.TestCase): def test_an_integer(self): - ctdll = cdll.load(_ctypes_test.__file__) + ctdll = CDLL(_ctypes_test.__file__) an_integer = c_int.in_dll(ctdll, "an_integer") x = an_integer.value self.failUnlessEqual(x, ctdll.get_an_integer()) @@ -18,7 +18,7 @@ self.failUnlessEqual(x*2, ctdll.get_an_integer()) def test_undefined(self): - ctdll = cdll.load(_ctypes_test.__file__) + ctdll = CDLL(_ctypes_test.__file__) self.assertRaises(ValueError, c_int.in_dll, ctdll, "Undefined_Symbol") class Win_ValuesTestCase(unittest.TestCase): Modified: python/trunk/Lib/ctypes/test/test_win32.py ============================================================================== --- python/trunk/Lib/ctypes/test/test_win32.py (original) +++ python/trunk/Lib/ctypes/test/test_win32.py Fri Apr 21 18:51:04 2006 @@ -54,7 +54,7 @@ ("right", c_long), ("bottom", c_long)] - dll = cdll.load(_ctypes_test.__file__) + dll = CDLL(_ctypes_test.__file__) pt = POINT(10, 10) rect = RECT(0, 0, 20, 20) Added: python/trunk/Lib/ctypes/util.py ============================================================================== --- (empty file) +++ python/trunk/Lib/ctypes/util.py Fri Apr 21 18:51:04 2006 @@ -0,0 +1,122 @@ +import sys, os +import ctypes + +# find_library(name) returns the pathname of a library, or None. +if os.name == "nt": + def find_library(name): + # See MSDN for the REAL search order. + for directory in os.environ['PATH'].split(os.pathsep): + fname = os.path.join(directory, name) + if os.path.exists(fname): + return fname + if fname.lower().endswith(".dll"): + continue + fname = fname + ".dll" + if os.path.exists(fname): + return fname + return None + +if os.name == "ce": + # search path according to MSDN: + # - absolute path specified by filename + # - The .exe launch directory + # - the Windows directory + # - ROM dll files (where are they?) + # - OEM specified search path: HKLM\Loader\SystemPath + def find_library(name): + return name + +if os.name == "posix" and sys.platform == "darwin": + from ctypes.macholib.dyld import dyld_find as _dyld_find + def find_library(name): + possible = ['lib%s.dylib' % name, + '%s.dylib' % name, + '%s.framework/%s' % (name, name)] + for name in possible: + try: + return _dyld_find(name) + except ValueError: + continue + return None + +elif os.name == "posix": + # Andreas Degert's find functions, using gcc, /sbin/ldconfig, objdump + import re, tempfile + + def _findLib_gcc(name): + expr = '[^\(\)\s]*lib%s\.[^\(\)\s]*' % name + cmd = 'if type gcc &>/dev/null; then CC=gcc; else CC=cc; fi;' \ + '$CC -Wl,-t -o /dev/null 2>&1 -l' + name + try: + fdout, outfile = tempfile.mkstemp() + fd = os.popen(cmd) + trace = fd.read() + err = fd.close() + finally: + try: + os.unlink(outfile) + except OSError, e: + if e.errno != errno.ENOENT: + raise + res = re.search(expr, trace) + if not res: + return None + return res.group(0) + + def _findLib_ld(name): + expr = '/[^\(\)\s]*lib%s\.[^\(\)\s]*' % name + res = re.search(expr, os.popen('/sbin/ldconfig -p 2>/dev/null').read()) + if not res: + # Hm, this works only for libs needed by the python executable. + cmd = 'ldd %s 2>/dev/null' % sys.executable + res = re.search(expr, os.popen(cmd).read()) + if not res: + return None + return res.group(0) + + def _get_soname(f): + cmd = "objdump -p -j .dynamic 2>/dev/null " + f + res = re.search(r'\sSONAME\s+([^\s]+)', os.popen(cmd).read()) + if not res: + return None + return res.group(1) + + def find_library(name): + lib = _findLib_ld(name) or _findLib_gcc(name) + if not lib: + return None + return _get_soname(lib) + +################################################################ +# test code + +def test(): + from ctypes import cdll + if os.name == "nt": + print cdll.msvcrt + print cdll.load("msvcrt") + print find_library("msvcrt") + + if os.name == "posix": + # find and load_version + print find_library("m") + print find_library("c") + print find_library("bz2") + + # getattr +## print cdll.m +## print cdll.bz2 + + # load + if sys.platform == "darwin": + print cdll.LoadLibrary("libm.dylib") + print cdll.LoadLibrary("libcrypto.dylib") + print cdll.LoadLibrary("libSystem.dylib") + print cdll.LoadLibrary("System.framework/System") + else: + print cdll.LoadLibrary("libm.so") + print cdll.LoadLibrary("libcrypt.so") + print find_library("crypt") + +if __name__ == "__main__": + test() From python-checkins at python.org Fri Apr 21 19:06:43 2006 From: python-checkins at python.org (phillip.eby) Date: Fri, 21 Apr 2006 19:06:43 +0200 (CEST) Subject: [Python-checkins] r45626 - sandbox/trunk/setuptools/setuptools/package_index.py Message-ID: <20060421170643.DE2321E4015@bag.python.org> Author: phillip.eby Date: Fri Apr 21 19:06:43 2006 New Revision: 45626 Modified: sandbox/trunk/setuptools/setuptools/package_index.py Log: Ignore bdist_dumb distributions when looking for downloads. Modified: sandbox/trunk/setuptools/setuptools/package_index.py ============================================================================== --- sandbox/trunk/setuptools/setuptools/package_index.py (original) +++ sandbox/trunk/setuptools/setuptools/package_index.py Fri Apr 21 19:06:43 2006 @@ -96,7 +96,6 @@ ``pkg_resources.normalize_path()`` on it before passing it to this routine! """ - # Generate alternative interpretations of a source distro name # Because some packages are ambiguous as to name/versions split # e.g. "adns-python-1.1.0", "egenix-mx-commercial", etc. @@ -110,6 +109,11 @@ # versions in distribution archive names (sdist and bdist). parts = basename.split('-') + if not py_version: + for i,p in enumerate(parts[2:]): + if len(p)==5 and p.startswith('py2.'): + return # It's a bdist_dumb, not an sdist -- bail out + for p in range(1,len(parts)+1): yield Distribution( location, metadata, '-'.join(parts[:p]), '-'.join(parts[p:]), @@ -117,10 +121,6 @@ platform = platform ) - - - - class PackageIndex(Environment): """A distribution index that scans web pages for download URLs""" @@ -534,6 +534,8 @@ def retry_sf_download(self, url, filename): try: return self._download_to(url, filename) + except (KeyboardInterrupt,SystemExit): + raise except: scheme, server, path, param, query, frag = urlparse.urlparse(url) if server!='dl.sourceforge.net': @@ -570,8 +572,6 @@ - - def open_url(self, url): try: return urllib2.urlopen(url) @@ -672,7 +672,7 @@ # DNS-bl0ck1n9 f1r3w4llz sUx0rs! _sf_mirrors[:] = ['dl.sourceforge.net'] return random.choice(_sf_mirrors) - + From python-checkins at python.org Fri Apr 21 19:08:24 2006 From: python-checkins at python.org (phillip.eby) Date: Fri, 21 Apr 2006 19:08:24 +0200 (CEST) Subject: [Python-checkins] r45627 - sandbox/branches/setuptools-0.6/setuptools/package_index.py Message-ID: <20060421170824.E80A11E400B@bag.python.org> Author: phillip.eby Date: Fri Apr 21 19:08:23 2006 New Revision: 45627 Modified: sandbox/branches/setuptools-0.6/setuptools/package_index.py Log: Ignore bdist_dumb distributions when looking at download URLs Modified: sandbox/branches/setuptools-0.6/setuptools/package_index.py ============================================================================== --- sandbox/branches/setuptools-0.6/setuptools/package_index.py (original) +++ sandbox/branches/setuptools-0.6/setuptools/package_index.py Fri Apr 21 19:08:23 2006 @@ -96,7 +96,6 @@ ``pkg_resources.normalize_path()`` on it before passing it to this routine! """ - # Generate alternative interpretations of a source distro name # Because some packages are ambiguous as to name/versions split # e.g. "adns-python-1.1.0", "egenix-mx-commercial", etc. @@ -110,6 +109,11 @@ # versions in distribution archive names (sdist and bdist). parts = basename.split('-') + if not py_version: + for i,p in enumerate(parts[2:]): + if len(p)==5 and p.startswith('py2.'): + return # It's a bdist_dumb, not an sdist -- bail out + for p in range(1,len(parts)+1): yield Distribution( location, metadata, '-'.join(parts[:p]), '-'.join(parts[p:]), @@ -117,10 +121,6 @@ platform = platform ) - - - - class PackageIndex(Environment): """A distribution index that scans web pages for download URLs""" @@ -534,6 +534,8 @@ def retry_sf_download(self, url, filename): try: return self._download_to(url, filename) + except (KeyboardInterrupt,SystemExit): + raise except: scheme, server, path, param, query, frag = urlparse.urlparse(url) if server!='dl.sourceforge.net': @@ -570,8 +572,6 @@ - - def open_url(self, url): try: return urllib2.urlopen(url) From buildbot at python.org Fri Apr 21 19:20:20 2006 From: buildbot at python.org (buildbot at python.org) Date: Fri, 21 Apr 2006 17:20:20 +0000 Subject: [Python-checkins] buildbot warnings in alpha Tru64 5.1 trunk Message-ID: <20060421172020.868B31E400B@bag.python.org> The Buildbot has detected a new failure of alpha Tru64 5.1 trunk. Full details are available at: http://www.python.org/dev/buildbot/all/alpha%2520Tru64%25205.1%2520trunk/builds/307 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: george.yoshida,hyeshik.chang Build Had Warnings: warnings test sincerely, -The Buildbot From python-checkins at python.org Fri Apr 21 19:38:55 2006 From: python-checkins at python.org (phillip.eby) Date: Fri, 21 Apr 2006 19:38:55 +0200 (CEST) Subject: [Python-checkins] r45628 - sandbox/trunk/setuptools/setuptools/sandbox.py Message-ID: <20060421173855.997571E4006@bag.python.org> Author: phillip.eby Date: Fri Apr 21 19:38:55 2006 New Revision: 45628 Modified: sandbox/trunk/setuptools/setuptools/sandbox.py Log: Recognize 'U' as a valid read-only mode for open() Modified: sandbox/trunk/setuptools/setuptools/sandbox.py ============================================================================== --- sandbox/trunk/setuptools/setuptools/sandbox.py (original) +++ sandbox/trunk/setuptools/setuptools/sandbox.py Fri Apr 21 19:38:55 2006 @@ -158,7 +158,7 @@ raise SandboxViolation(operation, args, kw) def _open(self, path, mode='r', *args, **kw): - if mode not in ('r', 'rt', 'rb', 'rU') and not self._ok(path): + if mode not in ('r', 'rt', 'rb', 'rU', 'U') and not self._ok(path): self._violation("open", path, mode, *args, **kw) return _open(path,mode,*args,**kw) From python-checkins at python.org Fri Apr 21 19:40:38 2006 From: python-checkins at python.org (phillip.eby) Date: Fri, 21 Apr 2006 19:40:38 +0200 (CEST) Subject: [Python-checkins] r45629 - sandbox/branches/setuptools-0.6/setuptools/sandbox.py Message-ID: <20060421174038.B52E41E4018@bag.python.org> Author: phillip.eby Date: Fri Apr 21 19:40:38 2006 New Revision: 45629 Modified: sandbox/branches/setuptools-0.6/setuptools/sandbox.py Log: Recognize 'U' as a valid read-only mode for open() Modified: sandbox/branches/setuptools-0.6/setuptools/sandbox.py ============================================================================== --- sandbox/branches/setuptools-0.6/setuptools/sandbox.py (original) +++ sandbox/branches/setuptools-0.6/setuptools/sandbox.py Fri Apr 21 19:40:38 2006 @@ -158,7 +158,7 @@ raise SandboxViolation(operation, args, kw) def _open(self, path, mode='r', *args, **kw): - if mode not in ('r', 'rt', 'rb', 'rU') and not self._ok(path): + if mode not in ('r', 'rt', 'rb', 'rU', 'U') and not self._ok(path): self._violation("open", path, mode, *args, **kw) return _open(path,mode,*args,**kw) From buildbot at python.org Fri Apr 21 21:37:57 2006 From: buildbot at python.org (buildbot at python.org) Date: Fri, 21 Apr 2006 19:37:57 +0000 Subject: [Python-checkins] buildbot warnings in x86 Ubuntu dapper (icc) trunk Message-ID: <20060421193757.6D6991E4005@bag.python.org> The Buildbot has detected a new failure of x86 Ubuntu dapper (icc) trunk. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520Ubuntu%2520dapper%2520%2528icc%2529%2520trunk/builds/213 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: thomas.heller Build Had Warnings: warnings test sincerely, -The Buildbot From python-checkins at python.org Fri Apr 21 23:18:12 2006 From: python-checkins at python.org (tim.peters) Date: Fri, 21 Apr 2006 23:18:12 +0200 (CEST) Subject: [Python-checkins] r45631 - in python/trunk: Lib/ntpath.py Lib/test/test_threaded_import.py Lib/test/threaded_import_hangers.py Misc/NEWS Message-ID: <20060421211812.2E45D1E4012@bag.python.org> Author: tim.peters Date: Fri Apr 21 23:18:10 2006 New Revision: 45631 Added: python/trunk/Lib/test/threaded_import_hangers.py (contents, props changed) Modified: python/trunk/Lib/ntpath.py python/trunk/Lib/test/test_threaded_import.py python/trunk/Misc/NEWS Log: SF bug #1473760 TempFile can hang on Windows. Python 2.4 changed ntpath.abspath to do an import inside the function. As a result, due to Python's import lock, anything calling abspath on Windows (directly, or indirectly like tempfile.TemporaryFile) hung when it was called from a thread spawned as a side effect of importing a module. This is a depressingly frequent problem, and deserves a more general fix. I'm settling for a micro-fix here because this specific one accounts for a report of Zope Corp's ZEO hanging on Windows, and it was an odd way to change abspath to begin with (ntpath needs a different implementation depending on whether we're actually running on Windows, and the _obvious_ way to arrange for that is not to bury a possibly-failing import _inside_ the function). Note that if/when other micro-fixes of this kind get made, the new Lib/test/threaded_import_hangers.py is a convenient place to add tests for them. Modified: python/trunk/Lib/ntpath.py ============================================================================== --- python/trunk/Lib/ntpath.py (original) +++ python/trunk/Lib/ntpath.py Fri Apr 21 23:18:10 2006 @@ -481,27 +481,28 @@ # Return an absolute path. -def abspath(path): - """Return the absolute version of a path""" - try: - from nt import _getfullpathname - except ImportError: # Not running on Windows - mock up something sensible. - global abspath - def _abspath(path): - if not isabs(path): - path = join(os.getcwd(), path) - return normpath(path) - abspath = _abspath - return _abspath(path) - - if path: # Empty path must return current working directory. - try: - path = _getfullpathname(path) - except WindowsError: - pass # Bad path - return unchanged. - else: - path = os.getcwd() - return normpath(path) +try: + from nt import _getfullpathname + +except ImportError: # not running on Windows - mock up something sensible + def abspath(path): + """Return the absolute version of a path.""" + if not isabs(path): + path = join(os.getcwd(), path) + return normpath(path) + +else: # use native Windows method on Windows + def abspath(path): + """Return the absolute version of a path.""" + + if path: # Empty path must return current working directory. + try: + path = _getfullpathname(path) + except WindowsError: + pass # Bad path - return unchanged. + else: + path = os.getcwd() + return normpath(path) # realpath is a no-op on systems without islink support realpath = abspath Modified: python/trunk/Lib/test/test_threaded_import.py ============================================================================== --- python/trunk/Lib/test/test_threaded_import.py (original) +++ python/trunk/Lib/test/test_threaded_import.py Fri Apr 21 23:18:10 2006 @@ -6,7 +6,7 @@ # randrange, and then Python hangs. import thread -from test.test_support import verbose, TestSkipped +from test.test_support import verbose, TestSkipped, TestFailed critical_section = thread.allocate_lock() done = thread.allocate_lock() @@ -25,6 +25,23 @@ if finished: done.release() +def test_import_hangers(): + import sys + if verbose: + print "testing import hangers ...", + + from test import threaded_import_hangers + + try: + if threaded_import_hangers.errors: + raise TestFailed(threaded_import_hangers.errors) + elif verbose: + print "OK." + finally: + # In case this test is run again, make sure the helper module + # gets loaded from scratch again. + del sys.modules['test.threaded_import_hangers'] + # Tricky: When regrtest imports this module, the thread running regrtest # grabs the import lock and won't let go of it until this module returns. # All other threads attempting an import hang for the duration. Since @@ -53,5 +70,7 @@ print "OK." done.release() + test_import_hangers() + if __name__ == "__main__": test_main() Added: python/trunk/Lib/test/threaded_import_hangers.py ============================================================================== --- (empty file) +++ python/trunk/Lib/test/threaded_import_hangers.py Fri Apr 21 23:18:10 2006 @@ -0,0 +1,42 @@ +# This is a helper module for test_threaded_import. The test imports this +# module, and this module tries to run various Python library functions in +# their own thread, as a side effect of being imported. If the spawned +# thread doesn't complete in TIMEOUT seconds, an "appeared to hang" message +# is appended to the module-global `errors` list. That list remains empty +# if (and only if) all functions tested complete. + +TIMEOUT = 10 + +import threading + +import tempfile +import os.path + +errors = [] + +# This class merely runs a function in its own thread T. The thread importing +# this module holds the import lock, so if the function called by T tries +# to do its own imports it will block waiting for this module's import +# to complete. +class Worker(threading.Thread): + def __init__(self, function, args): + threading.Thread.__init__(self) + self.function = function + self.args = args + + def run(self): + self.function(*self.args) + +for name, func, args in [ + # Bug 147376: TemporaryFile hung on Windows, starting in Python 2.4. + ("tempfile.TemporaryFile", tempfile.TemporaryFile, ()), + + # The real cause for bug 147376: ntpath.abspath() caused the hang. + ("os.path.abspath", os.path.abspath, ('.',)), + ]: + + t = Worker(func, args) + t.start() + t.join(TIMEOUT) + if t.isAlive(): + errors.append("%s appeared to hang" % name) Modified: python/trunk/Misc/NEWS ============================================================================== --- python/trunk/Misc/NEWS (original) +++ python/trunk/Misc/NEWS Fri Apr 21 23:18:10 2006 @@ -86,6 +86,9 @@ Library ------- +- Bug #1473760: ``tempfile.TemporaryFile()`` could hang on Windows, when + called from a thread spawned as a side effect of importing a module. + - New modules: setuptools, easy_install, and pkg_resources, to support building, installing, and using Python eggs, respectively. From python-checkins at python.org Fri Apr 21 23:22:01 2006 From: python-checkins at python.org (tim.peters) Date: Fri, 21 Apr 2006 23:22:01 +0200 (CEST) Subject: [Python-checkins] r45632 - in python/branches/release24-maint: Lib/ntpath.py Lib/test/test_threaded_import.py Lib/test/threaded_import_hangers.py Misc/NEWS Message-ID: <20060421212201.9F3FF1E4006@bag.python.org> Author: tim.peters Date: Fri Apr 21 23:22:00 2006 New Revision: 45632 Added: python/branches/release24-maint/Lib/test/threaded_import_hangers.py - copied unchanged from r45631, python/trunk/Lib/test/threaded_import_hangers.py Modified: python/branches/release24-maint/Lib/ntpath.py python/branches/release24-maint/Lib/test/test_threaded_import.py python/branches/release24-maint/Misc/NEWS Log: Merge rev 45631 from trunk. SF bug #1473760 TempFile can hang on Windows. Python 2.4 changed ntpath.abspath to do an import inside the function. As a result, due to Python's import lock, anything calling abspath on Windows (directly, or indirectly like tempfile.TemporaryFile) hung when it was called from a thread spawned as a side effect of importing a module. This is a depressingly frequent problem, and deserves a more general fix. I'm settling for a micro-fix here because this specific one accounts for a report of Zope Corp's ZEO hanging on Windows, and it was an odd way to change abspath to begin with (ntpath needs a different implementation depending on whether we're actually running on Windows, and the _obvious_ way to arrange for that is not to bury a possibly-failing import _inside_ the function). Note that if/when other micro-fixes of this kind get made, the new Lib/test/threaded_import_hangers.py is a convenient place to add tests for them. Modified: python/branches/release24-maint/Lib/ntpath.py ============================================================================== --- python/branches/release24-maint/Lib/ntpath.py (original) +++ python/branches/release24-maint/Lib/ntpath.py Fri Apr 21 23:22:00 2006 @@ -482,27 +482,28 @@ # Return an absolute path. -def abspath(path): - """Return the absolute version of a path""" - try: - from nt import _getfullpathname - except ImportError: # Not running on Windows - mock up something sensible. - global abspath - def _abspath(path): - if not isabs(path): - path = join(os.getcwd(), path) - return normpath(path) - abspath = _abspath - return _abspath(path) - - if path: # Empty path must return current working directory. - try: - path = _getfullpathname(path) - except WindowsError: - pass # Bad path - return unchanged. - else: - path = os.getcwd() - return normpath(path) +try: + from nt import _getfullpathname + +except ImportError: # not running on Windows - mock up something sensible + def abspath(path): + """Return the absolute version of a path.""" + if not isabs(path): + path = join(os.getcwd(), path) + return normpath(path) + +else: # use native Windows method on Windows + def abspath(path): + """Return the absolute version of a path.""" + + if path: # Empty path must return current working directory. + try: + path = _getfullpathname(path) + except WindowsError: + pass # Bad path - return unchanged. + else: + path = os.getcwd() + return normpath(path) # realpath is a no-op on systems without islink support realpath = abspath Modified: python/branches/release24-maint/Lib/test/test_threaded_import.py ============================================================================== --- python/branches/release24-maint/Lib/test/test_threaded_import.py (original) +++ python/branches/release24-maint/Lib/test/test_threaded_import.py Fri Apr 21 23:22:00 2006 @@ -6,7 +6,7 @@ # randrange, and then Python hangs. import thread -from test.test_support import verbose, TestSkipped +from test.test_support import verbose, TestSkipped, TestFailed critical_section = thread.allocate_lock() done = thread.allocate_lock() @@ -25,6 +25,23 @@ if finished: done.release() +def test_import_hangers(): + import sys + if verbose: + print "testing import hangers ...", + + from test import threaded_import_hangers + + try: + if threaded_import_hangers.errors: + raise TestFailed(threaded_import_hangers.errors) + elif verbose: + print "OK." + finally: + # In case this test is run again, make sure the helper module + # gets loaded from scratch again. + del sys.modules['test.threaded_import_hangers'] + # Tricky: When regrtest imports this module, the thread running regrtest # grabs the import lock and won't let go of it until this module returns. # All other threads attempting an import hang for the duration. Since @@ -53,5 +70,7 @@ print "OK." done.release() + test_import_hangers() + if __name__ == "__main__": test_main() Modified: python/branches/release24-maint/Misc/NEWS ============================================================================== --- python/branches/release24-maint/Misc/NEWS (original) +++ python/branches/release24-maint/Misc/NEWS Fri Apr 21 23:22:00 2006 @@ -29,6 +29,9 @@ Library ------- +- Bug #1473760: ``tempfile.TemporaryFile()`` could hang on Windows, when + called from a thread spawned as a side effect of importing a module. + - The ``__del__`` method of class ``local`` in module ``_threading_local`` returned before accomplishing any of its intended cleanup. @@ -43,6 +46,9 @@ raise an exception some day. But dicts have been allowed, and "mostly worked", so support for them won't go away without warning. +- New modules: setuptools, easy_install, and pkg_resources, to support + building, installing, and using Python eggs, respectively. + Tools/Demos ----------- From python-checkins at python.org Fri Apr 21 23:35:21 2006 From: python-checkins at python.org (tim.peters) Date: Fri, 21 Apr 2006 23:35:21 +0200 (CEST) Subject: [Python-checkins] r45633 - python/branches/release24-maint/Misc/NEWS Message-ID: <20060421213521.82E601E401A@bag.python.org> Author: tim.peters Date: Fri Apr 21 23:35:20 2006 New Revision: 45633 Modified: python/branches/release24-maint/Misc/NEWS Log: Oops! 2.5 news about setuptools snuck into 2.4 when I resolved merge conflicts from a recent backport. Modified: python/branches/release24-maint/Misc/NEWS ============================================================================== --- python/branches/release24-maint/Misc/NEWS (original) +++ python/branches/release24-maint/Misc/NEWS Fri Apr 21 23:35:20 2006 @@ -46,9 +46,6 @@ raise an exception some day. But dicts have been allowed, and "mostly worked", so support for them won't go away without warning. -- New modules: setuptools, easy_install, and pkg_resources, to support - building, installing, and using Python eggs, respectively. - Tools/Demos ----------- From python-checkins at python.org Fri Apr 21 23:53:38 2006 From: python-checkins at python.org (phillip.eby) Date: Fri, 21 Apr 2006 23:53:38 +0200 (CEST) Subject: [Python-checkins] r45634 - python/trunk/Doc/whatsnew/whatsnew25.tex Message-ID: <20060421215338.45CFD1E4006@bag.python.org> Author: phillip.eby Date: Fri Apr 21 23:53:37 2006 New Revision: 45634 Modified: python/trunk/Doc/whatsnew/whatsnew25.tex Log: Guido wrote contextlib, not me, but thanks anyway. ;) Modified: python/trunk/Doc/whatsnew/whatsnew25.tex ============================================================================== --- python/trunk/Doc/whatsnew/whatsnew25.tex (original) +++ python/trunk/Doc/whatsnew/whatsnew25.tex Fri Apr 21 23:53:37 2006 @@ -1255,7 +1255,6 @@ \item New module: The \module{contextlib} module contains helper functions for use with the new '\keyword{with}' statement. See section~\ref{module-contextlib} for more about this module. -(Contributed by Phillip J. Eby.) \item New module: The \module{cProfile} module is a C implementation of the existing \module{profile} module that has much lower overhead. From buildbot at python.org Sat Apr 22 00:01:21 2006 From: buildbot at python.org (buildbot at python.org) Date: Fri, 21 Apr 2006 22:01:21 +0000 Subject: [Python-checkins] buildbot warnings in amd64 gentoo trunk Message-ID: <20060421220121.9E3DD1E4017@bag.python.org> The Buildbot has detected a new failure of amd64 gentoo trunk. Full details are available at: http://www.python.org/dev/buildbot/all/amd64%2520gentoo%2520trunk/builds/594 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: tim.peters Build Had Warnings: warnings failed slave lost sincerely, -The Buildbot From buildbot at python.org Sat Apr 22 00:13:24 2006 From: buildbot at python.org (buildbot at python.org) Date: Fri, 21 Apr 2006 22:13:24 +0000 Subject: [Python-checkins] buildbot warnings in g4 osx.4 trunk Message-ID: <20060421221324.D8E6F1E4006@bag.python.org> The Buildbot has detected a new failure of g4 osx.4 trunk. Full details are available at: http://www.python.org/dev/buildbot/all/g4%2520osx.4%2520trunk/builds/503 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: tim.peters Build Had Warnings: warnings test sincerely, -The Buildbot From buildbot at python.org Sat Apr 22 00:13:30 2006 From: buildbot at python.org (buildbot at python.org) Date: Fri, 21 Apr 2006 22:13:30 +0000 Subject: [Python-checkins] buildbot warnings in x86 XP-2 trunk Message-ID: <20060421221331.25FED1E4006@bag.python.org> The Buildbot has detected a new failure of x86 XP-2 trunk. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520XP-2%2520trunk/builds/343 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: tim.peters Build Had Warnings: warnings test sincerely, -The Buildbot From buildbot at python.org Sat Apr 22 00:13:46 2006 From: buildbot at python.org (buildbot at python.org) Date: Fri, 21 Apr 2006 22:13:46 +0000 Subject: [Python-checkins] buildbot failure in g4 osx.4 2.4 Message-ID: <20060421221346.D09B61E4006@bag.python.org> The Buildbot has detected a new failure of g4 osx.4 2.4. Full details are available at: http://www.python.org/dev/buildbot/all/g4%2520osx.4%25202.4/builds/95 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch branches/release24-maint] HEAD Blamelist: tim.peters BUILD FAILED: failed svn sincerely, -The Buildbot From buildbot at python.org Sat Apr 22 00:13:51 2006 From: buildbot at python.org (buildbot at python.org) Date: Fri, 21 Apr 2006 22:13:51 +0000 Subject: [Python-checkins] buildbot warnings in ppc Debian unstable trunk Message-ID: <20060421221351.632021E4006@bag.python.org> The Buildbot has detected a new failure of ppc Debian unstable trunk. Full details are available at: http://www.python.org/dev/buildbot/all/ppc%2520Debian%2520unstable%2520trunk/builds/239 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: tim.peters Build Had Warnings: warnings test sincerely, -The Buildbot From buildbot at python.org Sat Apr 22 00:13:55 2006 From: buildbot at python.org (buildbot at python.org) Date: Fri, 21 Apr 2006 22:13:55 +0000 Subject: [Python-checkins] buildbot failure in ppc Debian unstable 2.4 Message-ID: <20060421221355.DC5731E4006@bag.python.org> The Buildbot has detected a new failure of ppc Debian unstable 2.4. Full details are available at: http://www.python.org/dev/buildbot/all/ppc%2520Debian%2520unstable%25202.4/builds/31 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch branches/release24-maint] HEAD Blamelist: tim.peters BUILD FAILED: failed svn sincerely, -The Buildbot From buildbot at python.org Sat Apr 22 00:13:59 2006 From: buildbot at python.org (buildbot at python.org) Date: Fri, 21 Apr 2006 22:13:59 +0000 Subject: [Python-checkins] buildbot failure in ppc Debian unstable trunk Message-ID: <20060421221359.AFBDC1E400B@bag.python.org> The Buildbot has detected a new failure of ppc Debian unstable trunk. Full details are available at: http://www.python.org/dev/buildbot/all/ppc%2520Debian%2520unstable%2520trunk/builds/240 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: phillip.eby BUILD FAILED: failed svn sincerely, -The Buildbot From buildbot at python.org Sat Apr 22 00:14:19 2006 From: buildbot at python.org (buildbot at python.org) Date: Fri, 21 Apr 2006 22:14:19 +0000 Subject: [Python-checkins] buildbot failure in x86 XP-2 2.4 Message-ID: <20060421221419.8E15B1E400B@bag.python.org> The Buildbot has detected a new failure of x86 XP-2 2.4. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520XP-2%25202.4/builds/80 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch branches/release24-maint] HEAD Blamelist: tim.peters BUILD FAILED: failed svn sincerely, -The Buildbot From buildbot at python.org Sat Apr 22 00:14:22 2006 From: buildbot at python.org (buildbot at python.org) Date: Fri, 21 Apr 2006 22:14:22 +0000 Subject: [Python-checkins] buildbot failure in g4 osx.4 trunk Message-ID: <20060421221422.4BCEA1E400B@bag.python.org> The Buildbot has detected a new failure of g4 osx.4 trunk. Full details are available at: http://www.python.org/dev/buildbot/all/g4%2520osx.4%2520trunk/builds/504 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: phillip.eby BUILD FAILED: failed svn sincerely, -The Buildbot From buildbot at python.org Sat Apr 22 00:14:56 2006 From: buildbot at python.org (buildbot at python.org) Date: Fri, 21 Apr 2006 22:14:56 +0000 Subject: [Python-checkins] buildbot failure in x86 XP-2 trunk Message-ID: <20060421221456.D8A1E1E4006@bag.python.org> The Buildbot has detected a new failure of x86 XP-2 trunk. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520XP-2%2520trunk/builds/344 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: phillip.eby BUILD FAILED: failed svn sincerely, -The Buildbot From buildbot at python.org Sat Apr 22 00:15:12 2006 From: buildbot at python.org (buildbot at python.org) Date: Fri, 21 Apr 2006 22:15:12 +0000 Subject: [Python-checkins] buildbot warnings in x86 XP trunk Message-ID: <20060421221512.36B341E4010@bag.python.org> The Buildbot has detected a new failure of x86 XP trunk. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520XP%2520trunk/builds/525 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: tim.peters Build Had Warnings: warnings test sincerely, -The Buildbot From buildbot at python.org Sat Apr 22 00:15:30 2006 From: buildbot at python.org (buildbot at python.org) Date: Fri, 21 Apr 2006 22:15:30 +0000 Subject: [Python-checkins] buildbot warnings in x86 OpenBSD 2.4 Message-ID: <20060421221530.EA0881E400C@bag.python.org> The Buildbot has detected a new failure of x86 OpenBSD 2.4. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520OpenBSD%25202.4/builds/67 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch branches/release24-maint] HEAD Blamelist: tim.peters Build Had Warnings: warnings test sincerely, -The Buildbot From buildbot at python.org Sat Apr 22 00:15:30 2006 From: buildbot at python.org (buildbot at python.org) Date: Fri, 21 Apr 2006 22:15:30 +0000 Subject: [Python-checkins] buildbot warnings in sparc solaris10 gcc trunk Message-ID: <20060421221531.224C91E400B@bag.python.org> The Buildbot has detected a new failure of sparc solaris10 gcc trunk. Full details are available at: http://www.python.org/dev/buildbot/all/sparc%2520solaris10%2520gcc%2520trunk/builds/510 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: tim.peters Build Had Warnings: warnings test sincerely, -The Buildbot From buildbot at python.org Sat Apr 22 00:15:44 2006 From: buildbot at python.org (buildbot at python.org) Date: Fri, 21 Apr 2006 22:15:44 +0000 Subject: [Python-checkins] buildbot failure in sparc solaris10 gcc 2.4 Message-ID: <20060421221544.F0B9D1E400D@bag.python.org> The Buildbot has detected a new failure of sparc solaris10 gcc 2.4. Full details are available at: http://www.python.org/dev/buildbot/all/sparc%2520solaris10%2520gcc%25202.4/builds/97 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch branches/release24-maint] HEAD Blamelist: tim.peters BUILD FAILED: failed svn sincerely, -The Buildbot From buildbot at python.org Sat Apr 22 00:15:56 2006 From: buildbot at python.org (buildbot at python.org) Date: Fri, 21 Apr 2006 22:15:56 +0000 Subject: [Python-checkins] buildbot failure in x86 OpenBSD trunk Message-ID: <20060421221556.725791E400C@bag.python.org> The Buildbot has detected a new failure of x86 OpenBSD trunk. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520OpenBSD%2520trunk/builds/409 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: phillip.eby BUILD FAILED: failed svn sincerely, -The Buildbot From buildbot at python.org Sat Apr 22 00:15:59 2006 From: buildbot at python.org (buildbot at python.org) Date: Fri, 21 Apr 2006 22:15:59 +0000 Subject: [Python-checkins] buildbot failure in sparc solaris10 gcc trunk Message-ID: <20060421221559.9560B1E400F@bag.python.org> The Buildbot has detected a new failure of sparc solaris10 gcc trunk. Full details are available at: http://www.python.org/dev/buildbot/all/sparc%2520solaris10%2520gcc%2520trunk/builds/511 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: phillip.eby BUILD FAILED: failed svn sincerely, -The Buildbot From buildbot at python.org Sat Apr 22 00:16:13 2006 From: buildbot at python.org (buildbot at python.org) Date: Fri, 21 Apr 2006 22:16:13 +0000 Subject: [Python-checkins] buildbot failure in x86 OpenBSD 2.4 Message-ID: <20060421221613.A37341E4006@bag.python.org> The Buildbot has detected a new failure of x86 OpenBSD 2.4. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520OpenBSD%25202.4/builds/68 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch branches/release24-maint] HEAD Blamelist: tim.peters BUILD FAILED: failed svn sincerely, -The Buildbot From buildbot at python.org Sat Apr 22 00:17:34 2006 From: buildbot at python.org (buildbot at python.org) Date: Fri, 21 Apr 2006 22:17:34 +0000 Subject: [Python-checkins] buildbot failure in x86 XP 2.4 Message-ID: <20060421221734.654311E4006@bag.python.org> The Buildbot has detected a new failure of x86 XP 2.4. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520XP%25202.4/builds/93 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch branches/release24-maint] HEAD Blamelist: tim.peters BUILD FAILED: failed svn sincerely, -The Buildbot From buildbot at python.org Sat Apr 22 00:19:27 2006 From: buildbot at python.org (buildbot at python.org) Date: Fri, 21 Apr 2006 22:19:27 +0000 Subject: [Python-checkins] buildbot failure in x86 XP trunk Message-ID: <20060421221927.DE1E91E4006@bag.python.org> The Buildbot has detected a new failure of x86 XP trunk. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520XP%2520trunk/builds/526 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: phillip.eby BUILD FAILED: failed svn sincerely, -The Buildbot From buildbot at python.org Sat Apr 22 00:25:55 2006 From: buildbot at python.org (buildbot at python.org) Date: Fri, 21 Apr 2006 22:25:55 +0000 Subject: [Python-checkins] buildbot warnings in x86 gentoo trunk Message-ID: <20060421222555.EA1C91E4006@bag.python.org> The Buildbot has detected a new failure of x86 gentoo trunk. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520gentoo%2520trunk/builds/574 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: tim.peters Build Had Warnings: warnings test sincerely, -The Buildbot From buildbot at python.org Sat Apr 22 00:25:59 2006 From: buildbot at python.org (buildbot at python.org) Date: Fri, 21 Apr 2006 22:25:59 +0000 Subject: [Python-checkins] buildbot failure in x86 gentoo 2.4 Message-ID: <20060421222600.053851E4016@bag.python.org> The Buildbot has detected a new failure of x86 gentoo 2.4. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520gentoo%25202.4/builds/99 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch branches/release24-maint] HEAD Blamelist: tim.peters BUILD FAILED: failed svn sincerely, -The Buildbot From buildbot at python.org Sat Apr 22 00:26:05 2006 From: buildbot at python.org (buildbot at python.org) Date: Fri, 21 Apr 2006 22:26:05 +0000 Subject: [Python-checkins] buildbot failure in x86 gentoo trunk Message-ID: <20060421222605.3F7EC1E4011@bag.python.org> The Buildbot has detected a new failure of x86 gentoo trunk. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520gentoo%2520trunk/builds/575 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: phillip.eby BUILD FAILED: failed svn sincerely, -The Buildbot From buildbot at python.org Sat Apr 22 00:26:38 2006 From: buildbot at python.org (buildbot at python.org) Date: Fri, 21 Apr 2006 22:26:38 +0000 Subject: [Python-checkins] buildbot failure in amd64 gentoo 2.4 Message-ID: <20060421222638.6A33A1E4006@bag.python.org> The Buildbot has detected a new failure of amd64 gentoo 2.4. Full details are available at: http://www.python.org/dev/buildbot/all/amd64%2520gentoo%25202.4/builds/103 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch branches/release24-maint] HEAD Blamelist: tim.peters BUILD FAILED: failed svn sincerely, -The Buildbot From buildbot at python.org Sat Apr 22 00:28:47 2006 From: buildbot at python.org (buildbot at python.org) Date: Fri, 21 Apr 2006 22:28:47 +0000 Subject: [Python-checkins] buildbot warnings in alpha Tru64 5.1 trunk Message-ID: <20060421222847.7FF501E4006@bag.python.org> The Buildbot has detected a new failure of alpha Tru64 5.1 trunk. Full details are available at: http://www.python.org/dev/buildbot/all/alpha%2520Tru64%25205.1%2520trunk/builds/310 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: tim.peters Build Had Warnings: warnings test sincerely, -The Buildbot From buildbot at python.org Sat Apr 22 00:30:04 2006 From: buildbot at python.org (buildbot at python.org) Date: Fri, 21 Apr 2006 22:30:04 +0000 Subject: [Python-checkins] buildbot failure in alpha Tru64 5.1 2.4 Message-ID: <20060421223004.43DB81E4015@bag.python.org> The Buildbot has detected a new failure of alpha Tru64 5.1 2.4. Full details are available at: http://www.python.org/dev/buildbot/all/alpha%2520Tru64%25205.1%25202.4/builds/55 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch branches/release24-maint] HEAD Blamelist: tim.peters BUILD FAILED: failed svn sincerely, -The Buildbot From buildbot at python.org Sat Apr 22 00:30:37 2006 From: buildbot at python.org (buildbot at python.org) Date: Fri, 21 Apr 2006 22:30:37 +0000 Subject: [Python-checkins] buildbot failure in x86 cygwin trunk Message-ID: <20060421223038.0352A1E4015@bag.python.org> The Buildbot has detected a new failure of x86 cygwin trunk. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520cygwin%2520trunk/builds/244 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: phillip.eby BUILD FAILED: failed svn sincerely, -The Buildbot From buildbot at python.org Sat Apr 22 00:31:41 2006 From: buildbot at python.org (buildbot at python.org) Date: Fri, 21 Apr 2006 22:31:41 +0000 Subject: [Python-checkins] buildbot failure in alpha Tru64 5.1 trunk Message-ID: <20060421223141.303C81E400B@bag.python.org> The Buildbot has detected a new failure of alpha Tru64 5.1 trunk. Full details are available at: http://www.python.org/dev/buildbot/all/alpha%2520Tru64%25205.1%2520trunk/builds/311 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: phillip.eby BUILD FAILED: failed svn sincerely, -The Buildbot From buildbot at python.org Sat Apr 22 00:31:56 2006 From: buildbot at python.org (buildbot at python.org) Date: Fri, 21 Apr 2006 22:31:56 +0000 Subject: [Python-checkins] buildbot failure in hppa Ubuntu dapper trunk Message-ID: <20060421223158.6506C1E400B@bag.python.org> The Buildbot has detected a new failure of hppa Ubuntu dapper trunk. Full details are available at: http://www.python.org/dev/buildbot/all/hppa%2520Ubuntu%2520dapper%2520trunk/builds/253 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: phillip.eby BUILD FAILED: failed svn sincerely, -The Buildbot From buildbot at python.org Sat Apr 22 00:32:21 2006 From: buildbot at python.org (buildbot at python.org) Date: Fri, 21 Apr 2006 22:32:21 +0000 Subject: [Python-checkins] buildbot failure in x86 cygwin 2.4 Message-ID: <20060421223221.902ED1E400B@bag.python.org> The Buildbot has detected a new failure of x86 cygwin 2.4. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520cygwin%25202.4/builds/40 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch branches/release24-maint] HEAD Blamelist: tim.peters BUILD FAILED: failed svn sincerely, -The Buildbot From buildbot at python.org Sat Apr 22 00:37:45 2006 From: buildbot at python.org (buildbot at python.org) Date: Fri, 21 Apr 2006 22:37:45 +0000 Subject: [Python-checkins] buildbot failure in x86 Ubuntu dapper (icc) 2.4 Message-ID: <20060421223745.B89711E4006@bag.python.org> The Buildbot has detected a new failure of x86 Ubuntu dapper (icc) 2.4. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520Ubuntu%2520dapper%2520%2528icc%2529%25202.4/builds/35 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch branches/release24-maint] HEAD Blamelist: tim.peters BUILD FAILED: failed svn sincerely, -The Buildbot From buildbot at python.org Sat Apr 22 00:38:00 2006 From: buildbot at python.org (buildbot at python.org) Date: Fri, 21 Apr 2006 22:38:00 +0000 Subject: [Python-checkins] buildbot failure in x86 Ubuntu dapper (icc) trunk Message-ID: <20060421223801.19BA31E4006@bag.python.org> The Buildbot has detected a new failure of x86 Ubuntu dapper (icc) trunk. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520Ubuntu%2520dapper%2520%2528icc%2529%2520trunk/builds/215 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: phillip.eby BUILD FAILED: failed svn sincerely, -The Buildbot From buildbot at python.org Sat Apr 22 00:38:22 2006 From: buildbot at python.org (buildbot at python.org) Date: Fri, 21 Apr 2006 22:38:22 +0000 Subject: [Python-checkins] buildbot failure in hppa Ubuntu dapper 2.4 Message-ID: <20060421223822.BA28E1E4006@bag.python.org> The Buildbot has detected a new failure of hppa Ubuntu dapper 2.4. Full details are available at: http://www.python.org/dev/buildbot/all/hppa%2520Ubuntu%2520dapper%25202.4/builds/32 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch branches/release24-maint] HEAD Blamelist: tim.peters BUILD FAILED: failed svn sincerely, -The Buildbot From thomas at python.org Sat Apr 22 00:44:49 2006 From: thomas at python.org (Thomas Wouters) Date: Sat, 22 Apr 2006 00:44:49 +0200 Subject: [Python-checkins] buildbot failure in hppa Ubuntu dapper 2.4 In-Reply-To: <20060421223822.BA28E1E4006@bag.python.org> References: <20060421223822.BA28E1E4006@bag.python.org> Message-ID: <9e804ac0604211544y1d6de068yf21ec2dee374562c@mail.gmail.com> On 4/22/06, buildbot at python.org wrote: > BUILD FAILED: failed svn FWIW, the svn apache went down shortly after the regular apache on www.python.org went down. It took some fiddling to get it back up, but it should be working fine now. -- Thomas Wouters Hi! I'm a .signature virus! copy me into your .signature file to help me spread! -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-checkins/attachments/20060422/faa6e2d8/attachment.html From buildbot at python.org Sat Apr 22 01:20:30 2006 From: buildbot at python.org (buildbot at python.org) Date: Fri, 21 Apr 2006 23:20:30 +0000 Subject: [Python-checkins] buildbot warnings in sparc Ubuntu dapper 2.4 Message-ID: <20060421232030.935881E4006@bag.python.org> The Buildbot has detected a new failure of sparc Ubuntu dapper 2.4. Full details are available at: http://www.python.org/dev/buildbot/all/sparc%2520Ubuntu%2520dapper%25202.4/builds/29 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch branches/release24-maint] HEAD Blamelist: tim.peters Build Had Warnings: warnings test sincerely, -The Buildbot From python-checkins at python.org Sat Apr 22 03:50:35 2006 From: python-checkins at python.org (andrew.kuchling) Date: Sat, 22 Apr 2006 03:50:35 +0200 (CEST) Subject: [Python-checkins] r45635 - sandbox/trunk/mailbox/libmailbox.tex Message-ID: <20060422015035.2518F1E4017@bag.python.org> Author: andrew.kuchling Date: Sat Apr 22 03:50:34 2006 New Revision: 45635 Modified: sandbox/trunk/mailbox/libmailbox.tex Log: Typo fix; formatting tweak Modified: sandbox/trunk/mailbox/libmailbox.tex ============================================================================== --- sandbox/trunk/mailbox/libmailbox.tex (original) +++ sandbox/trunk/mailbox/libmailbox.tex Sat Apr 22 03:50:34 2006 @@ -303,7 +303,7 @@ \begin{methoddesc}{clean}{} Delete temporary files from the mailbox that have not been accessed in the last 36 hours. The Maildir specification says that mail-reading programs -should do this occassionally. +should do this occasionally. \end{methoddesc} Some \class{Mailbox} methods implemented by \class{Maildir} deserve special @@ -326,7 +326,8 @@ \begin{methoddesc}{lock}{} \methodline{unlock}{} Maildir mailboxes do not support (or require) locking, so these methods do -nothing. \end{methoddesc} +nothing. +\end{methoddesc} \begin{methoddesc}{close}{} \class{Maildir} instances do not keep any open files and the underlying From python-checkins at python.org Sat Apr 22 03:51:41 2006 From: python-checkins at python.org (andrew.kuchling) Date: Sat, 22 Apr 2006 03:51:41 +0200 (CEST) Subject: [Python-checkins] r45636 - python/trunk/Doc/mac/scripting.tex Message-ID: <20060422015141.4AFBB1E4006@bag.python.org> Author: andrew.kuchling Date: Sat Apr 22 03:51:41 2006 New Revision: 45636 Modified: python/trunk/Doc/mac/scripting.tex Log: Typo fixes Modified: python/trunk/Doc/mac/scripting.tex ============================================================================== --- python/trunk/Doc/mac/scripting.tex (original) +++ python/trunk/Doc/mac/scripting.tex Sat Apr 22 03:51:41 2006 @@ -12,7 +12,7 @@ the conceptual model and the terminology, and documents the standard suite. The "Open Scripting Architecture" document explains how to use OSA from an application programmers point of view. In the Apple Help -Viewer these book sare located in the Developer Documentation, Core +Viewer these books are located in the Developer Documentation, Core Technologies section. @@ -49,7 +49,7 @@ The generated output is a package with a number of modules, one for every suite used in the program plus an \module{__init__} module to glue it all together. The Python inheritance graph follows the AppleScript -inheritance graph, so if a programs dictionary specifies that it +inheritance graph, so if a program's dictionary specifies that it includes support for the Standard Suite, but extends one or two verbs with extra arguments then the output suite will contain a module \module{Standard_Suite} that imports and re-exports everything from From python-checkins at python.org Sat Apr 22 03:52:18 2006 From: python-checkins at python.org (andrew.kuchling) Date: Sat, 22 Apr 2006 03:52:18 +0200 (CEST) Subject: [Python-checkins] r45637 - python/branches/release24-maint/Doc/mac/scripting.tex Message-ID: <20060422015218.CBC611E4006@bag.python.org> Author: andrew.kuchling Date: Sat Apr 22 03:52:18 2006 New Revision: 45637 Modified: python/branches/release24-maint/Doc/mac/scripting.tex Log: Typo fixes Modified: python/branches/release24-maint/Doc/mac/scripting.tex ============================================================================== --- python/branches/release24-maint/Doc/mac/scripting.tex (original) +++ python/branches/release24-maint/Doc/mac/scripting.tex Sat Apr 22 03:52:18 2006 @@ -12,7 +12,7 @@ the conceptual model and the terminology, and documents the standard suite. The "Open Scripting Architecture" document explains how to use OSA from an application programmers point of view. In the Apple Help -Viewer these book sare located in the Developer Documentation, Core +Viewer these books are located in the Developer Documentation, Core Technologies section. @@ -49,7 +49,7 @@ The generated output is a package with a number of modules, one for every suite used in the program plus an \module{__init__} module to glue it all together. The Python inheritance graph follows the AppleScript -inheritance graph, so if a programs dictionary specifies that it +inheritance graph, so if a program's dictionary specifies that it includes support for the Standard Suite, but extends one or two verbs with extra arguments then the output suite will contain a module \module{Standard_Suite} that imports and re-exports everything from From python-checkins at python.org Sat Apr 22 04:00:05 2006 From: python-checkins at python.org (andrew.kuchling) Date: Sat, 22 Apr 2006 04:00:05 +0200 (CEST) Subject: [Python-checkins] r45638 - python/trunk/Modules/rotatingtree.h Message-ID: <20060422020005.5D5F11E4006@bag.python.org> Author: andrew.kuchling Date: Sat Apr 22 03:58:40 2006 New Revision: 45638 Modified: python/trunk/Modules/rotatingtree.h Log: Fix comment typo Modified: python/trunk/Modules/rotatingtree.h ============================================================================== --- python/trunk/Modules/rotatingtree.h (original) +++ python/trunk/Modules/rotatingtree.h Sat Apr 22 03:58:40 2006 @@ -4,7 +4,7 @@ * * It's a dict-like data structure that works best when accesses are not * random, but follow a strong pattern. The one implemented here is for - * accesses patterns where the same small set of keys is looked up over + * access patterns where the same small set of keys is looked up over * and over again, and this set of keys evolves slowly over time. */ From python-checkins at python.org Sat Apr 22 04:06:03 2006 From: python-checkins at python.org (andrew.kuchling) Date: Sat, 22 Apr 2006 04:06:03 +0200 (CEST) Subject: [Python-checkins] r45639 - python/trunk/Lib/test/test_old_mailbox.py Message-ID: <20060422020603.F1BAC1E4006@bag.python.org> Author: andrew.kuchling Date: Sat Apr 22 04:06:03 2006 New Revision: 45639 Added: python/trunk/Lib/test/test_old_mailbox.py - copied, changed from r45597, python/trunk/Lib/test/test_mailbox.py Log: Make copy of test_mailbox.py. We'll still want to check the backward compatibility classes in the new mailbox.py that I'll be committing in a few minutes. One change has been made: the tests use len(mbox) instead of len(mbox.boxes). The 'boxes' attribute was never documented and contains some internal state that seems unlikely to have been useful. Copied: python/trunk/Lib/test/test_old_mailbox.py (from r45597, python/trunk/Lib/test/test_mailbox.py) ============================================================================== --- python/trunk/Lib/test/test_mailbox.py (original) +++ python/trunk/Lib/test/test_old_mailbox.py Sat Apr 22 04:06:03 2006 @@ -1,3 +1,6 @@ +# This set of tests exercises the backward-compatibility class +# in mailbox.py (the ones without write support). + import mailbox import os import time @@ -63,17 +66,15 @@ def test_empty_maildir(self): """Test an empty maildir mailbox""" # Test for regression on bug #117490: - # Make sure the boxes attribute actually gets set. self.mbox = mailbox.Maildir(test_support.TESTFN) - self.assert_(hasattr(self.mbox, "boxes")) - self.assert_(len(self.mbox.boxes) == 0) + self.assert_(len(self.mbox) == 0) self.assert_(self.mbox.next() is None) self.assert_(self.mbox.next() is None) def test_nonempty_maildir_cur(self): self.createMessage("cur") self.mbox = mailbox.Maildir(test_support.TESTFN) - self.assert_(len(self.mbox.boxes) == 1) + self.assert_(len(self.mbox) == 1) self.assert_(self.mbox.next() is not None) self.assert_(self.mbox.next() is None) self.assert_(self.mbox.next() is None) @@ -81,7 +82,7 @@ def test_nonempty_maildir_new(self): self.createMessage("new") self.mbox = mailbox.Maildir(test_support.TESTFN) - self.assert_(len(self.mbox.boxes) == 1) + self.assert_(len(self.mbox) == 1) self.assert_(self.mbox.next() is not None) self.assert_(self.mbox.next() is None) self.assert_(self.mbox.next() is None) @@ -90,7 +91,7 @@ self.createMessage("cur") self.createMessage("new") self.mbox = mailbox.Maildir(test_support.TESTFN) - self.assert_(len(self.mbox.boxes) == 2) + self.assert_(len(self.mbox) == 2) self.assert_(self.mbox.next() is not None) self.assert_(self.mbox.next() is not None) self.assert_(self.mbox.next() is None) From buildbot at python.org Sat Apr 22 04:15:15 2006 From: buildbot at python.org (buildbot at python.org) Date: Sat, 22 Apr 2006 02:15:15 +0000 Subject: [Python-checkins] buildbot warnings in x86 OpenBSD trunk Message-ID: <20060422021515.BCBB91E4006@bag.python.org> The Buildbot has detected a new failure of x86 OpenBSD trunk. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520OpenBSD%2520trunk/builds/410 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: andrew.kuchling Build Had Warnings: warnings test sincerely, -The Buildbot From buildbot at python.org Sat Apr 22 04:39:19 2006 From: buildbot at python.org (buildbot at python.org) Date: Sat, 22 Apr 2006 02:39:19 +0000 Subject: [Python-checkins] buildbot warnings in x86 OpenBSD 2.4 Message-ID: <20060422023919.C0ECE1E4006@bag.python.org> The Buildbot has detected a new failure of x86 OpenBSD 2.4. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520OpenBSD%25202.4/builds/69 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch branches/release24-maint] HEAD Blamelist: andrew.kuchling Build Had Warnings: warnings test sincerely, -The Buildbot From buildbot at python.org Sat Apr 22 04:40:28 2006 From: buildbot at python.org (buildbot at python.org) Date: Sat, 22 Apr 2006 02:40:28 +0000 Subject: [Python-checkins] buildbot warnings in x86 XP-2 trunk Message-ID: <20060422024029.173331E4006@bag.python.org> The Buildbot has detected a new failure of x86 XP-2 trunk. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520XP-2%2520trunk/builds/348 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: andrew.kuchling Build Had Warnings: warnings test sincerely, -The Buildbot From buildbot at python.org Sat Apr 22 04:41:09 2006 From: buildbot at python.org (buildbot at python.org) Date: Sat, 22 Apr 2006 02:41:09 +0000 Subject: [Python-checkins] buildbot warnings in x86 cygwin 2.4 Message-ID: <20060422024110.386731E4006@bag.python.org> The Buildbot has detected a new failure of x86 cygwin 2.4. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520cygwin%25202.4/builds/41 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch branches/release24-maint] HEAD Blamelist: andrew.kuchling Build Had Warnings: warnings test sincerely, -The Buildbot From buildbot at python.org Sat Apr 22 04:44:39 2006 From: buildbot at python.org (buildbot at python.org) Date: Sat, 22 Apr 2006 02:44:39 +0000 Subject: [Python-checkins] buildbot warnings in amd64 gentoo trunk Message-ID: <20060422024439.339981E4006@bag.python.org> The Buildbot has detected a new failure of amd64 gentoo trunk. Full details are available at: http://www.python.org/dev/buildbot/all/amd64%2520gentoo%2520trunk/builds/597 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: andrew.kuchling Build Had Warnings: warnings test sincerely, -The Buildbot From buildbot at python.org Sat Apr 22 04:50:14 2006 From: buildbot at python.org (buildbot at python.org) Date: Sat, 22 Apr 2006 02:50:14 +0000 Subject: [Python-checkins] buildbot warnings in x86 gentoo trunk Message-ID: <20060422025014.3240B1E400F@bag.python.org> The Buildbot has detected a new failure of x86 gentoo trunk. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520gentoo%2520trunk/builds/577 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: andrew.kuchling Build Had Warnings: warnings test sincerely, -The Buildbot From buildbot at python.org Sat Apr 22 04:56:35 2006 From: buildbot at python.org (buildbot at python.org) Date: Sat, 22 Apr 2006 02:56:35 +0000 Subject: [Python-checkins] buildbot warnings in x86 cygwin trunk Message-ID: <20060422025636.07CD21E4006@bag.python.org> The Buildbot has detected a new failure of x86 cygwin trunk. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520cygwin%2520trunk/builds/246 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: andrew.kuchling Build Had Warnings: warnings test sincerely, -The Buildbot From buildbot at python.org Sat Apr 22 04:58:28 2006 From: buildbot at python.org (buildbot at python.org) Date: Sat, 22 Apr 2006 02:58:28 +0000 Subject: [Python-checkins] buildbot warnings in hppa Ubuntu dapper trunk Message-ID: <20060422025828.AA7CD1E4006@bag.python.org> The Buildbot has detected a new failure of hppa Ubuntu dapper trunk. Full details are available at: http://www.python.org/dev/buildbot/all/hppa%2520Ubuntu%2520dapper%2520trunk/builds/254 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: andrew.kuchling Build Had Warnings: warnings test sincerely, -The Buildbot From buildbot at python.org Sat Apr 22 05:09:51 2006 From: buildbot at python.org (buildbot at python.org) Date: Sat, 22 Apr 2006 03:09:51 +0000 Subject: [Python-checkins] buildbot warnings in hppa Ubuntu dapper 2.4 Message-ID: <20060422030951.489721E4006@bag.python.org> The Buildbot has detected a new failure of hppa Ubuntu dapper 2.4. Full details are available at: http://www.python.org/dev/buildbot/all/hppa%2520Ubuntu%2520dapper%25202.4/builds/33 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch branches/release24-maint] HEAD Blamelist: andrew.kuchling Build Had Warnings: warnings test sincerely, -The Buildbot From buildbot at python.org Sat Apr 22 05:10:21 2006 From: buildbot at python.org (buildbot at python.org) Date: Sat, 22 Apr 2006 03:10:21 +0000 Subject: [Python-checkins] buildbot warnings in sparc solaris10 gcc 2.4 Message-ID: <20060422031022.143FB1E4006@bag.python.org> The Buildbot has detected a new failure of sparc solaris10 gcc 2.4. Full details are available at: http://www.python.org/dev/buildbot/all/sparc%2520solaris10%2520gcc%25202.4/builds/99 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch branches/release24-maint] HEAD Blamelist: andrew.kuchling Build Had Warnings: warnings test sincerely, -The Buildbot From buildbot at python.org Sat Apr 22 05:35:13 2006 From: buildbot at python.org (buildbot at python.org) Date: Sat, 22 Apr 2006 03:35:13 +0000 Subject: [Python-checkins] buildbot warnings in x86 XP trunk Message-ID: <20060422033513.76A411E4006@bag.python.org> The Buildbot has detected a new failure of x86 XP trunk. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520XP%2520trunk/builds/528 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: andrew.kuchling Build Had Warnings: warnings test sincerely, -The Buildbot From buildbot at python.org Sat Apr 22 05:40:56 2006 From: buildbot at python.org (buildbot at python.org) Date: Sat, 22 Apr 2006 03:40:56 +0000 Subject: [Python-checkins] buildbot warnings in g4 osx.4 trunk Message-ID: <20060422034056.EE8D01E4006@bag.python.org> The Buildbot has detected a new failure of g4 osx.4 trunk. Full details are available at: http://www.python.org/dev/buildbot/all/g4%2520osx.4%2520trunk/builds/506 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: andrew.kuchling Build Had Warnings: warnings test sincerely, -The Buildbot From buildbot at python.org Sat Apr 22 05:44:38 2006 From: buildbot at python.org (buildbot at python.org) Date: Sat, 22 Apr 2006 03:44:38 +0000 Subject: [Python-checkins] buildbot warnings in ppc Debian unstable trunk Message-ID: <20060422034438.1BA9C1E4006@bag.python.org> The Buildbot has detected a new failure of ppc Debian unstable trunk. Full details are available at: http://www.python.org/dev/buildbot/all/ppc%2520Debian%2520unstable%2520trunk/builds/242 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: andrew.kuchling Build Had Warnings: warnings test sincerely, -The Buildbot From buildbot at python.org Sat Apr 22 05:47:45 2006 From: buildbot at python.org (buildbot at python.org) Date: Sat, 22 Apr 2006 03:47:45 +0000 Subject: [Python-checkins] buildbot warnings in ia64 Debian unstable trunk Message-ID: <20060422034746.033451E4006@bag.python.org> The Buildbot has detected a new failure of ia64 Debian unstable trunk. Full details are available at: http://www.python.org/dev/buildbot/all/ia64%2520Debian%2520unstable%2520trunk/builds/234 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: andrew.kuchling Build Had Warnings: warnings test sincerely, -The Buildbot From python-checkins at python.org Sat Apr 22 07:53:00 2006 From: python-checkins at python.org (tim.peters) Date: Sat, 22 Apr 2006 07:53:00 +0200 (CEST) Subject: [Python-checkins] r45641 - in python/trunk/Lib: mailbox.py test/test_mailbox.py Message-ID: <20060422055300.5D0B61E4006@bag.python.org> Author: tim.peters Date: Sat Apr 22 07:52:59 2006 New Revision: 45641 Modified: python/trunk/Lib/mailbox.py python/trunk/Lib/test/test_mailbox.py Log: Whitespace normalization. Modified: python/trunk/Lib/mailbox.py ============================================================================== --- python/trunk/Lib/mailbox.py (original) +++ python/trunk/Lib/mailbox.py Sat Apr 22 07:52:59 2006 @@ -266,7 +266,7 @@ if e.errno == errno.ENOENT: pass else: - raise + raise def __setitem__(self, key, message): """Replace the keyed message; raise KeyError if it doesn't exist.""" @@ -1641,7 +1641,7 @@ self._labels.remove(label) except ValueError: pass - + def get_visible(self): """Return a Message representation of visible headers.""" return Message(self._visible) @@ -1840,7 +1840,7 @@ except OSError, e: if e.errno == errno.EEXIST: os.remove(pre_lock.name) - raise ExternalClashError('dot lock unavailable: %s' % + raise ExternalClashError('dot lock unavailable: %s' % f.name) else: raise Modified: python/trunk/Lib/test/test_mailbox.py ============================================================================== --- python/trunk/Lib/test/test_mailbox.py (original) +++ python/trunk/Lib/test/test_mailbox.py Sat Apr 22 07:52:59 2006 @@ -1380,7 +1380,7 @@ msg.update_visible() for label in ('unseen', 'deleted', 'filed', 'answered', 'forwarded', 'edited', 'resent'): - msg.add_label(label) + msg.add_label(label) msg2 = mailbox.BabylMessage(msg) self.assert_(msg2.get_labels() == ['unseen', 'deleted', 'filed', 'answered', 'forwarded', 'edited', @@ -1673,17 +1673,17 @@ X-Original-To: gkj+person at localhost Delivered-To: gkj+person at localhost Received: from localhost (localhost [127.0.0.1]) - by andy.gregorykjohnson.com (Postfix) with ESMTP id 356ED9DD17 - for ; Wed, 13 Jul 2005 17:23:16 -0400 (EDT) + by andy.gregorykjohnson.com (Postfix) with ESMTP id 356ED9DD17 + for ; Wed, 13 Jul 2005 17:23:16 -0400 (EDT) Delivered-To: gkj at sundance.gregorykjohnson.com Received: from localhost [127.0.0.1] - by localhost with POP3 (fetchmail-6.2.5) - for gkj+person at localhost (single-drop); Wed, 13 Jul 2005 17:23:16 -0400 (EDT) + by localhost with POP3 (fetchmail-6.2.5) + for gkj+person at localhost (single-drop); Wed, 13 Jul 2005 17:23:16 -0400 (EDT) Received: from andy.gregorykjohnson.com (andy.gregorykjohnson.com [64.32.235.228]) - by sundance.gregorykjohnson.com (Postfix) with ESMTP id 5B056316746 - for ; Wed, 13 Jul 2005 17:23:11 -0400 (EDT) + by sundance.gregorykjohnson.com (Postfix) with ESMTP id 5B056316746 + for ; Wed, 13 Jul 2005 17:23:11 -0400 (EDT) Received: by andy.gregorykjohnson.com (Postfix, from userid 1000) - id 490CD9DD17; Wed, 13 Jul 2005 17:23:11 -0400 (EDT) + id 490CD9DD17; Wed, 13 Jul 2005 17:23:11 -0400 (EDT) Date: Wed, 13 Jul 2005 17:23:11 -0400 From: "Gregory K. Johnson" To: gkj at gregorykjohnson.com @@ -1701,7 +1701,7 @@ This is a sample message. --- +-- Gregory K. Johnson --NMuMz9nt05w80d4+ @@ -1720,17 +1720,17 @@ "X-Original-To":"gkj+person at localhost", "Delivered-To":"gkj+person at localhost", "Received":"""from localhost (localhost [127.0.0.1]) - by andy.gregorykjohnson.com (Postfix) with ESMTP id 356ED9DD17 - for ; Wed, 13 Jul 2005 17:23:16 -0400 (EDT)""", + by andy.gregorykjohnson.com (Postfix) with ESMTP id 356ED9DD17 + for ; Wed, 13 Jul 2005 17:23:16 -0400 (EDT)""", "Delivered-To":"gkj at sundance.gregorykjohnson.com", "Received":"""from localhost [127.0.0.1] - by localhost with POP3 (fetchmail-6.2.5) - for gkj+person at localhost (single-drop); Wed, 13 Jul 2005 17:23:16 -0400 (EDT)""", + by localhost with POP3 (fetchmail-6.2.5) + for gkj+person at localhost (single-drop); Wed, 13 Jul 2005 17:23:16 -0400 (EDT)""", "Received":"""from andy.gregorykjohnson.com (andy.gregorykjohnson.com [64.32.235.228]) - by sundance.gregorykjohnson.com (Postfix) with ESMTP id 5B056316746 - for ; Wed, 13 Jul 2005 17:23:11 -0400 (EDT)""", + by sundance.gregorykjohnson.com (Postfix) with ESMTP id 5B056316746 + for ; Wed, 13 Jul 2005 17:23:11 -0400 (EDT)""", "Received":"""by andy.gregorykjohnson.com (Postfix, from userid 1000) - id 490CD9DD17; Wed, 13 Jul 2005 17:23:11 -0400 (EDT)""", + id 490CD9DD17; Wed, 13 Jul 2005 17:23:11 -0400 (EDT)""", "Date":"Wed, 13 Jul 2005 17:23:11 -0400", "From":""""Gregory K. Johnson" """, "To":"gkj at gregorykjohnson.com", @@ -1742,7 +1742,7 @@ _sample_payloads = ("""This is a sample message. --- +-- Gregory K. Johnson """, """H4sICM2D1UIAA3RleHQAC8nILFYAokSFktSKEoW0zJxUPa7wzJIMhZLyfIWczLzUYj0uAHTs From python-checkins at python.org Sat Apr 22 08:07:47 2006 From: python-checkins at python.org (neal.norwitz) Date: Sat, 22 Apr 2006 08:07:47 +0200 (CEST) Subject: [Python-checkins] r45642 - python/trunk/Doc/Makefile.deps Message-ID: <20060422060747.8882A1E4006@bag.python.org> Author: neal.norwitz Date: Sat Apr 22 08:07:46 2006 New Revision: 45642 Modified: python/trunk/Doc/Makefile.deps Log: Add libctypes as a dep Modified: python/trunk/Doc/Makefile.deps ============================================================================== --- python/trunk/Doc/Makefile.deps (original) +++ python/trunk/Doc/Makefile.deps Sat Apr 22 08:07:46 2006 @@ -155,6 +155,7 @@ lib/required_2.py \ lib/libtempfile.tex \ lib/liberrno.tex \ + lib/libctypes.tex \ lib/libsomeos.tex \ lib/libsignal.tex \ lib/libsocket.tex \ From buildbot at python.org Sat Apr 22 08:22:04 2006 From: buildbot at python.org (buildbot at python.org) Date: Sat, 22 Apr 2006 06:22:04 +0000 Subject: [Python-checkins] buildbot warnings in x86 XP-2 trunk Message-ID: <20060422062204.1EEE21E4026@bag.python.org> The Buildbot has detected a new failure of x86 XP-2 trunk. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520XP-2%2520trunk/builds/351 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: neal.norwitz Build Had Warnings: warnings failed slave lost sincerely, -The Buildbot From martin at v.loewis.de Sat Apr 22 09:48:32 2006 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat, 22 Apr 2006 09:48:32 +0200 Subject: [Python-checkins] r45585 - in external/ctypes: .cvsignore ACKS ANNOUNCE ChangeLog LICENSE.txt MANIFEST.in README.CVS README.txt ctypes-dev.el ctypes/.CTYPES_DEVEL ctypes/.cvsignore ctypes/__init__.py ctypes/_endian.py ctypes/_loader.py ctypes/mac In-Reply-To: References: Message-ID: <4449DFD0.7020207@v.loewis.de> Neal Norwitz wrote: >> New Revision: 45585 > > I tried to diff this to see what changed, but I don't get anything > returned. Does anyone know how to view the full diff of this checkin? Try svn log -r45585 -v http://svn.python.org/projects svn diff -r45584:45585 http://svn.python.org/projects Regards, Martin From python-checkins at python.org Sat Apr 22 13:15:43 2006 From: python-checkins at python.org (martin.v.loewis) Date: Sat, 22 Apr 2006 13:15:43 +0200 (CEST) Subject: [Python-checkins] r45643 - in python/trunk/Modules: cjkcodecs/multibytecodec.c operator.c posixmodule.c Message-ID: <20060422111543.0E12C1E4006@bag.python.org> Author: martin.v.loewis Date: Sat Apr 22 13:15:41 2006 New Revision: 45643 Modified: python/trunk/Modules/cjkcodecs/multibytecodec.c python/trunk/Modules/operator.c python/trunk/Modules/posixmodule.c Log: Fix more ssize_t problems. Modified: python/trunk/Modules/cjkcodecs/multibytecodec.c ============================================================================== --- python/trunk/Modules/cjkcodecs/multibytecodec.c (original) +++ python/trunk/Modules/cjkcodecs/multibytecodec.c Sat Apr 22 13:15:41 2006 @@ -831,7 +831,7 @@ { while (buf->inbuf < buf->inbuf_end) { Py_ssize_t inleft, outleft; - int r; + Py_ssize_t r; inleft = (Py_ssize_t)(buf->inbuf_end - buf->inbuf); outleft = (Py_ssize_t)(buf->outbuf_end - buf->outbuf); Modified: python/trunk/Modules/operator.c ============================================================================== --- python/trunk/Modules/operator.c (original) +++ python/trunk/Modules/operator.c Sat Apr 22 13:15:41 2006 @@ -48,6 +48,12 @@ if(-1 == (r=AOP(a1,a2))) return NULL; \ return PyInt_FromLong(r); } +#define spamn2(OP,AOP) static PyObject *OP(PyObject *s, PyObject *a) { \ + PyObject *a1, *a2; Py_ssize_t r; \ + if(! PyArg_UnpackTuple(a,#OP,2,2,&a1,&a2)) return NULL; \ + if(-1 == (r=AOP(a1,a2))) return NULL; \ + return PyInt_FromSsize_t(r); } + #define spami2b(OP,AOP) static PyObject *OP(PyObject *s, PyObject *a) { \ PyObject *a1, *a2; long r; \ if(! PyArg_UnpackTuple(a,#OP,2,2,&a1,&a2)) return NULL; \ @@ -99,8 +105,8 @@ spamoi(op_irepeat , PySequence_InPlaceRepeat) spami2b(op_contains , PySequence_Contains) spami2b(sequenceIncludes, PySequence_Contains) -spami2(indexOf , PySequence_Index) -spami2(countOf , PySequence_Count) +spamn2(indexOf , PySequence_Index) +spamn2(countOf , PySequence_Count) spami(isMappingType , PyMapping_Check) spam2(op_getitem , PyObject_GetItem) spam2n(op_delitem , PyObject_DelItem) Modified: python/trunk/Modules/posixmodule.c ============================================================================== --- python/trunk/Modules/posixmodule.c (original) +++ python/trunk/Modules/posixmodule.c Sat Apr 22 13:15:41 2006 @@ -2440,7 +2440,7 @@ PyObject *key, *val, *keys=NULL, *vals=NULL; Py_ssize_t i, pos, argc, envc; PyObject *(*getitem)(PyObject *, Py_ssize_t); - int lastarg = 0; + Py_ssize_t lastarg = 0; /* execve has three arguments: (path, argv, env), where argv is a list or tuple of strings and env is a dictionary @@ -2581,7 +2581,8 @@ char *path; PyObject *argv; char **argvlist; - int mode, i, argc; + int mode, i; + Py_ssize_t argc; Py_intptr_t spawnval; PyObject *(*getitem)(PyObject *, Py_ssize_t); @@ -2670,10 +2671,11 @@ char **argvlist; char **envlist; PyObject *key, *val, *keys=NULL, *vals=NULL, *res=NULL; - int mode, i, pos, argc, envc; + int mode, pos, envc; + Py_ssize_t argc, i; Py_intptr_t spawnval; PyObject *(*getitem)(PyObject *, Py_ssize_t); - int lastarg = 0; + Py_ssize_t lastarg = 0; /* spawnve has four arguments: (mode, path, argv, env), where argv is a list or tuple of strings and env is a dictionary @@ -4374,7 +4376,7 @@ char modulepath[_MAX_PATH]; struct stat statinfo; GetModuleFileName(NULL, modulepath, sizeof(modulepath)); - for (i = x = 0; modulepath[i]; i++) + for (x = i = 0; modulepath[i]; i++) if (modulepath[i] == SEP) x = i+1; modulepath[x] = '\0'; From python-checkins at python.org Sat Apr 22 13:40:05 2006 From: python-checkins at python.org (martin.v.loewis) Date: Sat, 22 Apr 2006 13:40:05 +0200 (CEST) Subject: [Python-checkins] r45644 - in python/trunk: Modules/_codecsmodule.c Modules/_hotshot.c Objects/classobject.c Python/ast.c Python/ceval.c Python/codecs.c Message-ID: <20060422114005.E557E1E400F@bag.python.org> Author: martin.v.loewis Date: Sat Apr 22 13:40:03 2006 New Revision: 45644 Modified: python/trunk/Modules/_codecsmodule.c python/trunk/Modules/_hotshot.c python/trunk/Objects/classobject.c python/trunk/Python/ast.c python/trunk/Python/ceval.c python/trunk/Python/codecs.c Log: Fix more ssize_t issues. Modified: python/trunk/Modules/_codecsmodule.c ============================================================================== --- python/trunk/Modules/_codecsmodule.c (original) +++ python/trunk/Modules/_codecsmodule.c Sat Apr 22 13:40:03 2006 @@ -169,7 +169,7 @@ static PyObject *codec_tuple(PyObject *unicode, - int len) + Py_ssize_t len) { PyObject *v,*w; @@ -181,7 +181,7 @@ return NULL; } PyTuple_SET_ITEM(v,0,unicode); - w = PyInt_FromLong(len); + w = PyInt_FromSsize_t(len); if (w == NULL) { Py_DECREF(v); return NULL; @@ -213,7 +213,7 @@ PyObject *str; const char *errors = NULL; char *buf; - int len; + Py_ssize_t len; if (!PyArg_ParseTuple(args, "O!|z:escape_encode", &PyString_Type, &str, &errors)) Modified: python/trunk/Modules/_hotshot.c ============================================================================== --- python/trunk/Modules/_hotshot.c (original) +++ python/trunk/Modules/_hotshot.c Sat Apr 22 13:40:03 2006 @@ -1420,7 +1420,7 @@ char *buffer; char cwdbuffer[PATH_MAX]; PyObject *temp; - int i, len; + Py_ssize_t i, len; buffer = get_version_string(); if (buffer == NULL) { Modified: python/trunk/Objects/classobject.c ============================================================================== --- python/trunk/Objects/classobject.c (original) +++ python/trunk/Objects/classobject.c Sat Apr 22 13:40:03 2006 @@ -320,7 +320,7 @@ } sname = PyString_AsString(name); if (sname[0] == '_' && sname[1] == '_') { - int n = PyString_Size(name); + Py_ssize_t n = PyString_Size(name); if (sname[n-1] == '_' && sname[n-2] == '_') { char *err = NULL; if (strcmp(sname, "__dict__") == 0) @@ -380,7 +380,7 @@ PyObject *mod = PyDict_GetItemString(op->cl_dict, "__module__"); PyObject *name = op->cl_name; PyObject *res; - int m, n; + Py_ssize_t m, n; if (name == NULL || !PyString_Check(name)) return class_repr(op); @@ -638,7 +638,7 @@ PyObject_GC_Del(inst); } else { - int refcnt = inst->ob_refcnt; + Py_ssize_t refcnt = inst->ob_refcnt; /* __del__ resurrected it! Make it look like the original * Py_DECREF never happened. */ @@ -778,7 +778,7 @@ PyObject *func, *args, *res, *tmp; char *sname = PyString_AsString(name); if (sname[0] == '_' && sname[1] == '_') { - int n = PyString_Size(name); + Py_ssize_t n = PyString_Size(name); if (sname[n-1] == '_' && sname[n-2] == '_') { if (strcmp(sname, "__dict__") == 0) { if (PyEval_GetRestricted()) { @@ -1263,7 +1263,7 @@ */ PyErr_Clear(); return _PySequence_IterSearch((PyObject *)inst, member, - PY_ITERSEARCH_CONTAINS); + PY_ITERSEARCH_CONTAINS) > 0; } else return -1; Modified: python/trunk/Python/ast.c ============================================================================== --- python/trunk/Python/ast.c (original) +++ python/trunk/Python/ast.c Sat Apr 22 13:40:03 2006 @@ -3034,7 +3034,7 @@ if (*s & 0x80) { /* XXX inefficient */ PyObject *w; char *r; - int rn, i; + Py_ssize_t rn, i; w = decode_utf8(&s, end, "utf-16-be"); if (w == NULL) { Py_DECREF(u); Modified: python/trunk/Python/ceval.c ============================================================================== --- python/trunk/Python/ceval.c (original) +++ python/trunk/Python/ceval.c Sat Apr 22 13:40:03 2006 @@ -1560,7 +1560,7 @@ /* XXX move into writeobject() ? */ if (PyString_Check(v)) { char *s = PyString_AS_STRING(v); - int len = PyString_GET_SIZE(v); + Py_ssize_t len = PyString_GET_SIZE(v); if (len == 0 || !isspace(Py_CHARMASK(s[len-1])) || s[len-1] == ' ') @@ -1569,7 +1569,7 @@ #ifdef Py_USING_UNICODE else if (PyUnicode_Check(v)) { Py_UNICODE *s = PyUnicode_AS_UNICODE(v); - int len = PyUnicode_GET_SIZE(v); + Py_ssize_t len = PyUnicode_GET_SIZE(v); if (len == 0 || !Py_UNICODE_ISSPACE(s[len-1]) || s[len-1] == ' ') Modified: python/trunk/Python/codecs.c ============================================================================== --- python/trunk/Python/codecs.c (original) +++ python/trunk/Python/codecs.c Sat Apr 22 13:40:03 2006 @@ -95,7 +95,7 @@ { PyInterpreterState *interp; PyObject *result, *args = NULL, *v; - int i, len; + Py_ssize_t i, len; if (encoding == NULL) { PyErr_BadArgument(); From buildbot at python.org Sat Apr 22 13:41:31 2006 From: buildbot at python.org (buildbot at python.org) Date: Sat, 22 Apr 2006 11:41:31 +0000 Subject: [Python-checkins] buildbot failure in x86 OpenBSD trunk Message-ID: <20060422114131.88F951E400C@bag.python.org> The Buildbot has detected a new failure of x86 OpenBSD trunk. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520OpenBSD%2520trunk/builds/416 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: The web-page 'force build' button was pressed by 'Martin von Loewis': force clean Build Source Stamp: [branch foo] HEAD Blamelist: BUILD FAILED: failed svn sincerely, -The Buildbot From buildbot at python.org Sat Apr 22 13:59:06 2006 From: buildbot at python.org (buildbot at python.org) Date: Sat, 22 Apr 2006 11:59:06 +0000 Subject: [Python-checkins] buildbot warnings in x86 OpenBSD trunk Message-ID: <20060422115907.051951E4006@bag.python.org> The Buildbot has detected a new failure of x86 OpenBSD trunk. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520OpenBSD%2520trunk/builds/417 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: martin.v.loewis Build Had Warnings: warnings test sincerely, -The Buildbot From buildbot at python.org Sat Apr 22 14:41:13 2006 From: buildbot at python.org (buildbot at python.org) Date: Sat, 22 Apr 2006 12:41:13 +0000 Subject: [Python-checkins] buildbot warnings in ia64 Debian unstable trunk Message-ID: <20060422124113.F07FB1E4006@bag.python.org> The Buildbot has detected a new failure of ia64 Debian unstable trunk. Full details are available at: http://www.python.org/dev/buildbot/all/ia64%2520Debian%2520unstable%2520trunk/builds/239 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: martin.v.loewis Build Had Warnings: warnings test sincerely, -The Buildbot From buildbot at python.org Sat Apr 22 15:19:09 2006 From: buildbot at python.org (buildbot at python.org) Date: Sat, 22 Apr 2006 13:19:09 +0000 Subject: [Python-checkins] buildbot warnings in sparc Ubuntu dapper trunk Message-ID: <20060422131909.69E471E4006@bag.python.org> The Buildbot has detected a new failure of sparc Ubuntu dapper trunk. Full details are available at: http://www.python.org/dev/buildbot/all/sparc%2520Ubuntu%2520dapper%2520trunk/builds/136 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: martin.v.loewis Build Had Warnings: warnings test sincerely, -The Buildbot From python-checkins at python.org Sat Apr 22 17:10:50 2006 From: python-checkins at python.org (george.yoshida) Date: Sat, 22 Apr 2006 17:10:50 +0200 (CEST) Subject: [Python-checkins] r45645 - python/trunk/Doc/lib/libcollections.tex Message-ID: <20060422151050.87E7B1E4020@bag.python.org> Author: george.yoshida Date: Sat Apr 22 17:10:49 2006 New Revision: 45645 Modified: python/trunk/Doc/lib/libcollections.tex Log: Typo fixes Modified: python/trunk/Doc/lib/libcollections.tex ============================================================================== --- python/trunk/Doc/lib/libcollections.tex (original) +++ python/trunk/Doc/lib/libcollections.tex Sat Apr 22 17:10:49 2006 @@ -59,12 +59,12 @@ \begin{methoddesc}{pop}{} Remove and return an element from the right side of the deque. - If no elements are present, raises a \exception{IndexError}. + If no elements are present, raises an \exception{IndexError}. \end{methoddesc} \begin{methoddesc}{popleft}{} Remove and return an element from the left side of the deque. - If no elements are present, raises a \exception{IndexError}. + If no elements are present, raises an \exception{IndexError}. \end{methoddesc} \begin{methoddesc}{remove}{value} From python-checkins at python.org Sat Apr 22 17:12:37 2006 From: python-checkins at python.org (george.yoshida) Date: Sat, 22 Apr 2006 17:12:37 +0200 (CEST) Subject: [Python-checkins] r45646 - python/branches/release24-maint/Doc/lib/libcollections.tex Message-ID: <20060422151237.524931E400C@bag.python.org> Author: george.yoshida Date: Sat Apr 22 17:12:37 2006 New Revision: 45646 Modified: python/branches/release24-maint/Doc/lib/libcollections.tex Log: Typo fixes Modified: python/branches/release24-maint/Doc/lib/libcollections.tex ============================================================================== --- python/branches/release24-maint/Doc/lib/libcollections.tex (original) +++ python/branches/release24-maint/Doc/lib/libcollections.tex Sat Apr 22 17:12:37 2006 @@ -56,12 +56,12 @@ \begin{methoddesc}{pop}{} Remove and return an element from the right side of the deque. - If no elements are present, raises a \exception{IndexError}. + If no elements are present, raises an \exception{IndexError}. \end{methoddesc} \begin{methoddesc}{popleft}{} Remove and return an element from the left side of the deque. - If no elements are present, raises a \exception{IndexError}. + If no elements are present, raises an \exception{IndexError}. \end{methoddesc} \begin{methoddesc}{rotate}{n} From python-checkins at python.org Sat Apr 22 17:19:55 2006 From: python-checkins at python.org (martin.v.loewis) Date: Sat, 22 Apr 2006 17:19:55 +0200 (CEST) Subject: [Python-checkins] r45647 - python/trunk/PC/example_nt/example.c python/trunk/PC/example_nt/example.def python/trunk/PC/example_nt/example.vcproj Message-ID: <20060422151955.131001E4008@bag.python.org> Author: martin.v.loewis Date: Sat Apr 22 17:19:54 2006 New Revision: 45647 Removed: python/trunk/PC/example_nt/example.def Modified: python/trunk/PC/example_nt/example.c python/trunk/PC/example_nt/example.vcproj Log: Port to Python 2.5. Drop .DEF file. Change output file names to .pyd. Modified: python/trunk/PC/example_nt/example.c ============================================================================== --- python/trunk/PC/example_nt/example.c (original) +++ python/trunk/PC/example_nt/example.c Sat Apr 22 17:19:54 2006 @@ -13,7 +13,7 @@ {NULL, NULL} }; -void +PyMODINIT_FUNC initexample(void) { Py_InitModule("example", example_methods); Deleted: /python/trunk/PC/example_nt/example.def ============================================================================== --- /python/trunk/PC/example_nt/example.def Sat Apr 22 17:19:54 2006 +++ (empty file) @@ -1,2 +0,0 @@ -EXPORTS - initexample Modified: python/trunk/PC/example_nt/example.vcproj ============================================================================== --- python/trunk/PC/example_nt/example.vcproj (original) +++ python/trunk/PC/example_nt/example.vcproj Sat Apr 22 17:19:54 2006 @@ -39,12 +39,12 @@ - - Author: george.yoshida Date: Sat Apr 22 17:27:14 2006 New Revision: 45648 Modified: python/trunk/Doc/lib/libprofile.tex Log: - add versionadded tag - make arbitrary arguments come last Modified: python/trunk/Doc/lib/libprofile.tex ============================================================================== --- python/trunk/Doc/lib/libprofile.tex (original) +++ python/trunk/Doc/lib/libprofile.tex Sat Apr 22 17:27:14 2006 @@ -391,7 +391,7 @@ % (This \stmodindex use may be hard to change ;-( ) \stmodindex{pstats} -\begin{classdesc}{Stats}{filename\optional{, \moreargs\optional{, stream=sys.stdout}}} +\begin{classdesc}{Stats}{filename\optional{, stream=sys.stdout\optional{, \moreargs}}} This class constructor creates an instance of a ``statistics object'' from a \var{filename} (or set of filenames). \class{Stats} objects are manipulated by methods, in order to print useful reports. You may specify @@ -409,6 +409,8 @@ processes can be considered in a single report. If additional files need to be combined with data in an existing \class{Stats} object, the \method{add()} method can be used. + +\versionchanged[The \var{stream} parameter was added]{2.5} \end{classdesc} From python-checkins at python.org Sat Apr 22 17:48:16 2006 From: python-checkins at python.org (hyeshik.chang) Date: Sat, 22 Apr 2006 17:48:16 +0200 (CEST) Subject: [Python-checkins] r45649 - python/trunk/Lib/test/test_codecencodings_cn.py python/trunk/Lib/test/test_codecencodings_hk.py python/trunk/Lib/test/test_codecencodings_jp.py python/trunk/Lib/test/test_codecencodings_kr.py python/trunk/Lib/test/test_codecencodings_tw.py python/trunk/Lib/test/test_codecmaps_cn.py python/trunk/Lib/test/test_codecmaps_hk.py python/trunk/Lib/test/test_codecmaps_jp.py python/trunk/Lib/test/test_codecmaps_kr.py python/trunk/Lib/test/test_codecmaps_tw.py python/trunk/Lib/test/test_multibytecodec.py Message-ID: <20060422154816.84AB21E4008@bag.python.org> Author: hyeshik.chang Date: Sat Apr 22 17:48:15 2006 New Revision: 45649 Modified: python/trunk/Lib/test/test_codecencodings_cn.py python/trunk/Lib/test/test_codecencodings_hk.py python/trunk/Lib/test/test_codecencodings_jp.py python/trunk/Lib/test/test_codecencodings_kr.py python/trunk/Lib/test/test_codecencodings_tw.py python/trunk/Lib/test/test_codecmaps_cn.py python/trunk/Lib/test/test_codecmaps_hk.py python/trunk/Lib/test/test_codecmaps_jp.py python/trunk/Lib/test/test_codecmaps_kr.py python/trunk/Lib/test/test_codecmaps_tw.py python/trunk/Lib/test/test_multibytecodec.py Log: Remove $CJKCodecs$ RCS tags. The CJKCodecs isn't maintained outside anymore. Modified: python/trunk/Lib/test/test_codecencodings_cn.py ============================================================================== --- python/trunk/Lib/test/test_codecencodings_cn.py (original) +++ python/trunk/Lib/test/test_codecencodings_cn.py Sat Apr 22 17:48:15 2006 @@ -3,7 +3,6 @@ # test_codecencodings_cn.py # Codec encoding tests for PRC encodings. # -# $CJKCodecs: test_codecencodings_cn.py,v 1.2 2004/06/19 06:09:55 perky Exp $ from test import test_support from test import test_multibytecodec_support Modified: python/trunk/Lib/test/test_codecencodings_hk.py ============================================================================== --- python/trunk/Lib/test/test_codecencodings_hk.py (original) +++ python/trunk/Lib/test/test_codecencodings_hk.py Sat Apr 22 17:48:15 2006 @@ -3,7 +3,6 @@ # test_codecencodings_hk.py # Codec encoding tests for HongKong encodings. # -# $CJKCodecs: test_codecencodings_hk.py,v 1.1 2004/07/10 17:35:20 perky Exp $ from test import test_support from test import test_multibytecodec_support Modified: python/trunk/Lib/test/test_codecencodings_jp.py ============================================================================== --- python/trunk/Lib/test/test_codecencodings_jp.py (original) +++ python/trunk/Lib/test/test_codecencodings_jp.py Sat Apr 22 17:48:15 2006 @@ -3,7 +3,6 @@ # test_codecencodings_jp.py # Codec encoding tests for Japanese encodings. # -# $CJKCodecs: test_codecencodings_jp.py,v 1.3 2004/06/19 06:09:55 perky Exp $ from test import test_support from test import test_multibytecodec_support Modified: python/trunk/Lib/test/test_codecencodings_kr.py ============================================================================== --- python/trunk/Lib/test/test_codecencodings_kr.py (original) +++ python/trunk/Lib/test/test_codecencodings_kr.py Sat Apr 22 17:48:15 2006 @@ -3,7 +3,6 @@ # test_codecencodings_kr.py # Codec encoding tests for ROK encodings. # -# $CJKCodecs: test_codecencodings_kr.py,v 1.2 2004/06/19 06:09:55 perky Exp $ from test import test_support from test import test_multibytecodec_support Modified: python/trunk/Lib/test/test_codecencodings_tw.py ============================================================================== --- python/trunk/Lib/test/test_codecencodings_tw.py (original) +++ python/trunk/Lib/test/test_codecencodings_tw.py Sat Apr 22 17:48:15 2006 @@ -3,7 +3,6 @@ # test_codecencodings_tw.py # Codec encoding tests for ROC encodings. # -# $CJKCodecs: test_codecencodings_tw.py,v 1.2 2004/06/19 06:09:55 perky Exp $ from test import test_support from test import test_multibytecodec_support Modified: python/trunk/Lib/test/test_codecmaps_cn.py ============================================================================== --- python/trunk/Lib/test/test_codecmaps_cn.py (original) +++ python/trunk/Lib/test/test_codecmaps_cn.py Sat Apr 22 17:48:15 2006 @@ -3,7 +3,6 @@ # test_codecmaps_cn.py # Codec mapping tests for PRC encodings # -# $CJKCodecs: test_codecmaps_cn.py,v 1.3 2004/06/19 06:09:55 perky Exp $ from test import test_support from test import test_multibytecodec_support Modified: python/trunk/Lib/test/test_codecmaps_hk.py ============================================================================== --- python/trunk/Lib/test/test_codecmaps_hk.py (original) +++ python/trunk/Lib/test/test_codecmaps_hk.py Sat Apr 22 17:48:15 2006 @@ -3,7 +3,6 @@ # test_codecmaps_hk.py # Codec mapping tests for HongKong encodings # -# $CJKCodecs: test_codecmaps_hk.py,v 1.1 2004/07/10 17:35:20 perky Exp $ from test import test_support from test import test_multibytecodec_support Modified: python/trunk/Lib/test/test_codecmaps_jp.py ============================================================================== --- python/trunk/Lib/test/test_codecmaps_jp.py (original) +++ python/trunk/Lib/test/test_codecmaps_jp.py Sat Apr 22 17:48:15 2006 @@ -3,7 +3,6 @@ # test_codecmaps_jp.py # Codec mapping tests for Japanese encodings # -# $CJKCodecs: test_codecmaps_jp.py,v 1.3 2004/06/19 06:09:55 perky Exp $ from test import test_support from test import test_multibytecodec_support Modified: python/trunk/Lib/test/test_codecmaps_kr.py ============================================================================== --- python/trunk/Lib/test/test_codecmaps_kr.py (original) +++ python/trunk/Lib/test/test_codecmaps_kr.py Sat Apr 22 17:48:15 2006 @@ -3,7 +3,6 @@ # test_codecmaps_kr.py # Codec mapping tests for ROK encodings # -# $CJKCodecs: test_codecmaps_kr.py,v 1.3 2004/06/19 06:09:55 perky Exp $ from test import test_support from test import test_multibytecodec_support Modified: python/trunk/Lib/test/test_codecmaps_tw.py ============================================================================== --- python/trunk/Lib/test/test_codecmaps_tw.py (original) +++ python/trunk/Lib/test/test_codecmaps_tw.py Sat Apr 22 17:48:15 2006 @@ -3,7 +3,6 @@ # test_codecmaps_tw.py # Codec mapping tests for ROC encodings # -# $CJKCodecs: test_codecmaps_tw.py,v 1.3 2004/06/19 06:09:55 perky Exp $ from test import test_support from test import test_multibytecodec_support Modified: python/trunk/Lib/test/test_multibytecodec.py ============================================================================== --- python/trunk/Lib/test/test_multibytecodec.py (original) +++ python/trunk/Lib/test/test_multibytecodec.py Sat Apr 22 17:48:15 2006 @@ -3,7 +3,6 @@ # test_multibytecodec.py # Unit test for multibytecodec itself # -# $CJKCodecs: test_multibytecodec.py,v 1.8 2004/06/19 06:09:55 perky Exp $ from test import test_support from test import test_multibytecodec_support From buildbot at python.org Sat Apr 22 21:32:54 2006 From: buildbot at python.org (buildbot at python.org) Date: Sat, 22 Apr 2006 19:32:54 +0000 Subject: [Python-checkins] buildbot failure in x86 XP-2 trunk Message-ID: <20060422193254.64F5C1E4010@bag.python.org> The Buildbot has detected a new failure of x86 XP-2 trunk. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520XP-2%2520trunk/builds/352 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: george.yoshida,hyeshik.chang,martin.v.loewis BUILD FAILED: failed failed slave lost sincerely, -The Buildbot From python-checkins at python.org Sat Apr 22 23:21:18 2006 From: python-checkins at python.org (brett.cannon) Date: Sat, 22 Apr 2006 23:21:18 +0200 (CEST) Subject: [Python-checkins] r45651 - peps/trunk/pep-3100.txt Message-ID: <20060422212118.B59211E400B@bag.python.org> Author: brett.cannon Date: Sat Apr 22 23:21:18 2006 New Revision: 45651 Modified: peps/trunk/pep-3100.txt Log: Mention how func_whatever is to be renamed __whatever__ Modified: peps/trunk/pep-3100.txt ============================================================================== --- peps/trunk/pep-3100.txt (original) +++ peps/trunk/pep-3100.txt Sat Apr 22 23:21:18 2006 @@ -92,6 +92,8 @@ * __builtins__ should get a different name *or* completely unified with __builtin__. Keeping both with confusingly similar spellings and semantics is evil. +* Attributes on functions of the form ``func_whatever`` will be renamed + ``__whatever__`` [25]_ To be removed: @@ -288,6 +290,9 @@ .. [24] python-3000 email http://mail.python.org/pipermail/python-3000/2006-April/000996.html +.. [25] python-3000 email ("Pronouncement on parameter lists") + http://mail.python.org/pipermail/python-3000/2006-April/001175.html + .. [#pep238] PEP 238 (Changing the Division Operator) http://www.python.org/dev/peps/pep-0238 From python-checkins at python.org Sun Apr 23 05:48:00 2006 From: python-checkins at python.org (greg.ward) Date: Sun, 23 Apr 2006 05:48:00 +0200 (CEST) Subject: [Python-checkins] r45654 - in python/trunk: Doc/lib/liboptparse.tex Lib/optparse.py Lib/test/test_optparse.py Misc/NEWS Message-ID: <20060423034800.849871E4010@bag.python.org> Author: greg.ward Date: Sun Apr 23 05:47:58 2006 New Revision: 45654 Modified: python/trunk/Doc/lib/liboptparse.tex python/trunk/Lib/optparse.py python/trunk/Lib/test/test_optparse.py python/trunk/Misc/NEWS Log: Update optparse to Optik 1.5.1. Modified: python/trunk/Doc/lib/liboptparse.tex ============================================================================== --- python/trunk/Doc/lib/liboptparse.tex (original) +++ python/trunk/Doc/lib/liboptparse.tex Sun Apr 23 05:47:58 2006 @@ -35,9 +35,9 @@ \end{verbatim} As it parses the command line, \code{optparse} sets attributes of the -\var{options} object returned by \method{parse{\_}args()} based on user-supplied +\code{options} object returned by \method{parse{\_}args()} based on user-supplied command-line values. When \method{parse{\_}args()} returns from parsing this -command line, \var{options.filename} will be \code{"outfile"} and +command line, \code{options.filename} will be \code{"outfile"} and \code{options.verbose} will be \code{False}. \code{optparse} supports both long and short options, allows short options to be merged together, and allows options to be associated with their arguments in a variety of @@ -100,8 +100,8 @@ single letter, e.g. \code{"-x"} or \code{"-F"}. Also, traditional \UNIX{} syntax allows multiple options to be merged into a single argument, e.g. \code{"-x -F"} is equivalent to \code{"-xF"}. The GNU project -introduced \code{"{--}"} followed by a series of hyphen-separated words, -e.g. \code{"{--}file"} or \code{"{--}dry-run"}. These are the only two option +introduced \code{"-{}-"} followed by a series of hyphen-separated words, +e.g. \code{"-{}-file"} or \code{"-{}-dry-run"}. These are the only two option syntaxes provided by \module{optparse}. Some other option syntaxes that the world has seen include: @@ -170,7 +170,7 @@ prog -v --report /tmp/report.txt foo bar \end{verbatim} -\code{"-v"} and \code{"{--}report"} are both options. Assuming that +\code{"-v"} and \code{"-{}-report"} are both options. Assuming that \longprogramopt{report} takes one argument, \code{"/tmp/report.txt"} is an option argument. \code{"foo"} and \code{"bar"} are positional arguments. @@ -287,12 +287,12 @@ \method{parse{\_}args()} returns two values: \begin{itemize} \item {} -\var{options}, an object containing values for all of your options{---}e.g. if \code{"-{}-file"} takes a single string argument, then -\var{options.file} will be the filename supplied by the user, or +\code{options}, an object containing values for all of your options{---}e.g. if \code{"-{}-file"} takes a single string argument, then +\code{options.file} will be the filename supplied by the user, or \code{None} if the user did not supply that option \item {} -\var{args}, the list of positional arguments leftover after parsing +\code{args}, the list of positional arguments leftover after parsing options \end{itemize} @@ -309,7 +309,7 @@ adding new actions is an advanced topic covered in section~\ref{optparse-extending}, Extending \module{optparse}. Most actions tell \module{optparse} to store a value in some variable{---}for example, take a string from the command line and store it in an -attribute of \var{options}. +attribute of \code{options}. If you don't specify an option action, \module{optparse} defaults to \code{store}. @@ -333,8 +333,8 @@ \end{verbatim} When \module{optparse} sees the option string \code{"-f"}, it consumes the next -argument, \code{"foo.txt"}, and stores it in \var{options.filename}. So, -after this call to \method{parse{\_}args()}, \var{options.filename} is +argument, \code{"foo.txt"}, and stores it in \code{options.filename}. So, +after this call to \method{parse{\_}args()}, \code{options.filename} is \code{"foo.txt"}. Some other option types supported by \module{optparse} are \code{int} and \code{float}. @@ -379,7 +379,7 @@ Flag options{---}set a variable to true or false when a particular option is seen{---}are quite common. \module{optparse} supports them with two separate actions, \code{store{\_}true} and \code{store{\_}false}. For example, you might have a -\var{verbose} flag that is turned on with \code{"-v"} and off with \code{"-q"}: +\code{verbose} flag that is turned on with \code{"-v"} and off with \code{"-q"}: \begin{verbatim} parser.add_option("-v", action="store_true", dest="verbose") parser.add_option("-q", action="store_false", dest="verbose") @@ -421,7 +421,7 @@ destination, which is assigned before the command line is parsed. First, consider the verbose/quiet example. If we want \module{optparse} to set -\var{verbose} to \code{True} unless \code{"-q"} is seen, then we can do this: +\code{verbose} to \code{True} unless \code{"-q"} is seen, then we can do this: \begin{verbatim} parser.add_option("-v", action="store_true", dest="verbose", default=True) parser.add_option("-q", action="store_false", dest="verbose") @@ -441,7 +441,7 @@ parser.add_option("-q", action="store_false", dest="verbose", default=True) \end{verbatim} -Again, the default value for \var{verbose} will be \code{True}: the last +Again, the default value for \code{verbose} will be \code{True}: the last default value supplied for any particular destination is the one that counts. @@ -566,7 +566,7 @@ parser = OptionParser(usage="%prog [-f] [-q]", version="%prog 1.0") \end{verbatim} -Note that \code{"{\%}prog"} is expanded just like it is in \var{usage}. Apart +Note that \code{"{\%}prog"} is expanded just like it is in \code{usage}. Apart from that, \code{version} can contain anything you like. When you supply it, \module{optparse} automatically adds a \code{"-{}-version"} option to your parser. If it encounters this option on the command line, it expands your @@ -580,14 +580,14 @@ \end{verbatim} -\subsubsection{How \module{optparse} handles errors\label{optparse-how-optik-handles-errors}} +\subsubsection{How \module{optparse} handles errors\label{optparse-how-optparse-handles-errors}} There are two broad classes of errors that \module{optparse} has to worry about: programmer errors and user errors. Programmer errors are usually -erroneous calls to \code{parse.add{\_}option()}, e.g. invalid option strings, +erroneous calls to \code{parser.add{\_}option()}, e.g. invalid option strings, unknown option attributes, missing option attributes, etc. These are dealt with in the usual way: raise an exception (either -\exception{optparse.OptionError} or \exception{TypeError}) and let the program crash. +\code{optparse.OptionError} or \code{TypeError}) and let the program crash. Handling user errors is much more important, since they are guaranteed to happen no matter how stable your code is. \module{optparse} can automatically @@ -659,12 +659,66 @@ if __name__ == "__main__": main() \end{verbatim} -% $Id: tutorial.txt 415 2004-09-30 02:26:17Z greg $ +% $Id: tutorial.txt 505 2005-07-22 01:52:40Z gward $ \subsection{Reference Guide\label{optparse-reference-guide}} +\subsubsection{Creating the parser\label{optparse-creating-parser}} + +The first step in using \module{optparse} is to create an OptionParser instance: +\begin{verbatim} +parser = OptionParser(...) +\end{verbatim} + +The OptionParser constructor has no required arguments, but a number of +optional keyword arguments. You should always pass them as keyword +arguments, i.e. do not rely on the order in which the arguments are +declared. +\begin{quote} +\begin{description} +\item[\code{usage} (default: \code{"{\%}prog {[}options]"})] +The usage summary to print when your program is run incorrectly or +with a help option. When \module{optparse} prints the usage string, it expands +\code{{\%}prog} to \code{os.path.basename(sys.argv{[}0])} (or to \code{prog} if +you passed that keyword argument). To suppress a usage message, +pass the special value \code{optparse.SUPPRESS{\_}USAGE}. +\item[\code{option{\_}list} (default: \code{{[}]})] +A list of Option objects to populate the parser with. The options +in \code{option{\_}list} are added after any options in +\code{standard{\_}option{\_}list} (a class attribute that may be set by +OptionParser subclasses), but before any version or help options. +Deprecated; use \method{add{\_}option()} after creating the parser instead. +\item[\code{option{\_}class} (default: optparse.Option)] +Class to use when adding options to the parser in \method{add{\_}option()}. +\item[\code{version} (default: \code{None})] +A version string to print when the user supplies a version option. +If you supply a true value for \code{version}, \module{optparse} automatically adds +a version option with the single option string \code{"-{}-version"}. The +substring \code{"{\%}prog"} is expanded the same as for \code{usage}. +\item[\code{conflict{\_}handler} (default: \code{"error"})] +Specifies what to do when options with conflicting option strings +are added to the parser; see section~\ref{optparse-conflicts-between-options}, Conflicts between options. +\item[\code{description} (default: \code{None})] +A paragraph of text giving a brief overview of your program. \module{optparse} +reformats this paragraph to fit the current terminal width and +prints it when the user requests help (after \code{usage}, but before +the list of options). +\item[\code{formatter} (default: a new IndentedHelpFormatter)] +An instance of optparse.HelpFormatter that will be used for +printing help text. \module{optparse} provides two concrete classes for this +purpose: IndentedHelpFormatter and TitledHelpFormatter. +\item[\code{add{\_}help{\_}option} (default: \code{True})] +If true, \module{optparse} will add a help option (with option strings \code{"-h"} +and \code{"-{}-help"}) to the parser. +\item[\code{prog}] +The string to use when expanding \code{"{\%}prog"} in \code{usage} and +\code{version} instead of \code{os.path.basename(sys.argv{[}0])}. +\end{description} +\end{quote} + + \subsubsection{Populating the parser\label{optparse-populating-parser}} There are several ways to populate the parser with options. The @@ -708,38 +762,34 @@ specify any number of short or long option strings, but you must specify at least one overall option string. -The canonical way to create an Option instance is by calling -\function{make{\_}option()}, so that is what will be shown here. However, the -most common and convenient way is to use \code{parser.add{\_}option()}. Note -that \function{make{\_}option()} and \code{parser.add{\_}option()} have identical call -signatures: +The canonical way to create an Option instance is with the +\method{add{\_}option()} method of \class{OptionParser}: \begin{verbatim} -make_option(opt_str, ..., attr=value, ...) -parser.add_option(opt_str, ..., attr=value, ...) +parser.add_option(opt_str[, ...], attr=value, ...) \end{verbatim} To define an option with only a short option string: \begin{verbatim} -make_option("-f", attr=value, ...) +parser.add_option("-f", attr=value, ...) \end{verbatim} And to define an option with only a long option string: \begin{verbatim} -make_option("--foo", attr=value, ...) +parser.add_option("--foo", attr=value, ...) \end{verbatim} -The \code{attr=value} keyword arguments define option attributes, -i.e. attributes of the Option object. The most important option -attribute is \member{action}, and it largely determines what other attributes -are relevant or required. If you pass irrelevant option attributes, or -fail to pass required ones, \module{optparse} raises an OptionError exception -explaining your mistake. - -An options's \emph{action} determines what \module{optparse} does when it encounters -this option on the command-line. The actions hard-coded into \module{optparse} are: +The keyword arguments define attributes of the new Option object. The +most important option attribute is \member{action}, and it largely determines +which other attributes are relevant or required. If you pass irrelevant +option attributes, or fail to pass required ones, \module{optparse} raises an +OptionError exception explaining your mistake. + +An options's \emph{action} determines what \module{optparse} does when it encounters this +option on the command-line. The standard option actions hard-coded into +\module{optparse} are: \begin{description} \item[\code{store}] -store this option's argument {[}default] +store this option's argument (default) \item[\code{store{\_}const}] store a constant value \item[\code{store{\_}true}] @@ -748,6 +798,8 @@ store a false value \item[\code{append}] append this option's argument to a list +\item[\code{append{\_}const}] +append a constant value to a list \item[\code{count}] increment a counter by one \item[\code{callback}] @@ -762,24 +814,25 @@ below.) As you can see, most actions involve storing or updating a value -somewhere. \module{optparse} always creates an instance of \code{optparse.Values} -specifically for this purpose; we refer to this instance as \var{options}. -Option arguments (and various other values) are stored as attributes of -this object, according to the \member{dest} (destination) option attribute. +somewhere. \module{optparse} always creates a special object for this, +conventionally called \code{options} (it happens to be an instance of +\code{optparse.Values}). Option arguments (and various other values) are +stored as attributes of this object, according to the \member{dest} +(destination) option attribute. For example, when you call \begin{verbatim} parser.parse_args() \end{verbatim} -one of the first things \module{optparse} does is create the \var{options} object: +one of the first things \module{optparse} does is create the \code{options} object: \begin{verbatim} options = Values() \end{verbatim} If one of the options in this parser is defined with \begin{verbatim} -make_option("-f", "--file", action="store", type="string", dest="filename") +parser.add_option("-f", "--file", action="store", type="string", dest="filename") \end{verbatim} and the command-line being parsed includes any of the following: @@ -790,8 +843,7 @@ --file foo \end{verbatim} -then \module{optparse}, on seeing the \programopt{-f} or \longprogramopt{file} option, will do the -equivalent of +then \module{optparse}, on seeing this option, will do the equivalent of \begin{verbatim} options.filename = "foo" \end{verbatim} @@ -912,6 +964,13 @@ \end{verbatim} \item {} +\code{append{\_}const} {[}required: \code{const}; relevant: \member{dest}] + +Like \code{store{\_}const}, but the value \code{const} is appended to \member{dest}; +as with \code{append}, \member{dest} defaults to \code{None}, and an an empty list is +automatically created the first time the option is encountered. + +\item {} \code{count} {[}relevant: \member{dest}] Increment the integer stored at \member{dest}. If no default value is @@ -939,14 +998,9 @@ \code{callback} {[}required: \code{callback}; relevant: \member{type}, \code{nargs}, \code{callback{\_}args}, \code{callback{\_}kwargs}] -Call the function specified by \code{callback}. The signature of -this function should be +Call the function specified by \code{callback}, which is called as \begin{verbatim} -func(option : Option, - opt : string, - value : any, - parser : OptionParser, - *args, **kwargs) +func(option, opt_str, value, parser, *args, **kwargs) \end{verbatim} See section~\ref{optparse-option-callbacks}, Option Callbacks for more detail. @@ -956,7 +1010,7 @@ Prints a complete help message for all the options in the current option parser. The help message is constructed from -the \var{usage} string passed to OptionParser's constructor and +the \code{usage} string passed to OptionParser's constructor and the \member{help} string passed to every option. If no \member{help} string is supplied for an option, it will still be @@ -1007,6 +1061,87 @@ \end{itemize} +\subsubsection{Option attributes\label{optparse-option-attributes}} + +The following option attributes may be passed as keyword arguments +to \code{parser.add{\_}option()}. If you pass an option attribute +that is not relevant to a particular option, or fail to pass a required +option attribute, \module{optparse} raises OptionError. +\begin{itemize} +\item {} +\member{action} (default: \code{"store"}) + +Determines \module{optparse}'s behaviour when this option is seen on the command +line; the available options are documented above. + +\item {} +\member{type} (default: \code{"string"}) + +The argument type expected by this option (e.g., \code{"string"} or +\code{"int"}); the available option types are documented below. + +\item {} +\member{dest} (default: derived from option strings) + +If the option's action implies writing or modifying a value somewhere, +this tells \module{optparse} where to write it: \member{dest} names an attribute of the +\code{options} object that \module{optparse} builds as it parses the command line. + +\item {} +\code{default} (deprecated) + +The value to use for this option's destination if the option is not +seen on the command line. Deprecated; use \code{parser.set{\_}defaults()} +instead. + +\item {} +\code{nargs} (default: 1) + +How many arguments of type \member{type} should be consumed when this +option is seen. If {\textgreater} 1, \module{optparse} will store a tuple of values to +\member{dest}. + +\item {} +\code{const} + +For actions that store a constant value, the constant value to store. + +\item {} +\code{choices} + +For options of type \code{"choice"}, the list of strings the user +may choose from. + +\item {} +\code{callback} + +For options with action \code{"callback"}, the callable to call when this +option is seen. See section~\ref{optparse-option-callbacks}, Option Callbacks for detail on the arguments +passed to \code{callable}. + +\item {} +\code{callback{\_}args}, \code{callback{\_}kwargs} + +Additional positional and keyword arguments to pass to \code{callback} +after the four standard callback arguments. + +\item {} +\member{help} + +Help text to print for this option when listing all available options +after the user supplies a \member{help} option (such as \code{"-{}-help"}). +If no help text is supplied, the option will be listed without help +text. To hide this option, use the special value \code{SUPPRESS{\_}HELP}. + +\item {} +\code{metavar} (default: derived from option strings) + +Stand-in for the option argument(s) to use when printing help text. +See section~\ref{optparse-tutorial}, the tutorial for an example. + +\end{itemize} + + \subsubsection{Standard option types\label{optparse-standard-option-types}} \module{optparse} has six built-in option types: \code{string}, \code{int}, \code{long}, @@ -1017,22 +1152,74 @@ text on the command line is stored in the destination (or passed to the callback) as-is. -Integer arguments are passed to \code{int()} to convert them to Python -integers. If \code{int()} fails, so will \module{optparse}, although with a more -useful error message. (Internally, \module{optparse} raises -\exception{OptionValueError}; OptionParser catches this exception higher -up and terminates your program with a useful error message.) - -Likewise, \code{float} arguments are passed to \code{float()} for conversion, -\code{long} arguments to \code{long()}, and \code{complex} arguments to -\code{complex()}. Apart from that, they are handled identically to integer -arguments. +Integer arguments (type \code{int} or \code{long}) are parsed as follows: +\begin{quote} +\begin{itemize} +\item {} +if the number starts with \code{0x}, it is parsed as a hexadecimal number + +\item {} +if the number starts with \code{0}, it is parsed as an octal number + +\item {} +if the number starts with \code{0b}, is is parsed as a binary number + +\item {} +otherwise, the number is parsed as a decimal number + +\end{itemize} +\end{quote} + +The conversion is done by calling either \code{int()} or \code{long()} with +the appropriate base (2, 8, 10, or 16). If this fails, so will \module{optparse}, +although with a more useful error message. + +\code{float} and \code{complex} option arguments are converted directly with +\code{float()} and \code{complex()}, with similar error-handling. \code{choice} options are a subtype of \code{string} options. The \code{choices} option attribute (a sequence of strings) defines the set of allowed -option arguments. \code{optparse.option.check{\_}choice()} compares +option arguments. \code{optparse.check{\_}choice()} compares user-supplied option arguments against this master list and raises -\exception{OptionValueError} if an invalid string is given. +OptionValueError if an invalid string is given. + + +\subsubsection{Parsing arguments\label{optparse-parsing-arguments}} + +The whole point of creating and populating an OptionParser is to call +its \method{parse{\_}args()} method: +\begin{verbatim} +(options, args) = parser.parse_args(args=None, options=None) +\end{verbatim} + +where the input parameters are +\begin{description} +\item[\code{args}] +the list of arguments to process (\code{sys.argv{[}1:]} by default) +\item[\code{options}] +object to store option arguments in (a new instance of +optparse.Values by default) +\end{description} + +and the return values are +\begin{description} +\item[\code{options}] +the same object as was passed in as \code{options}, or the new +optparse.Values instance created by \module{optparse} +\item[\code{args}] +the leftover positional arguments after all options have been +processed +\end{description} + +The most common usage is to supply neither keyword argument. If you +supply a \code{values} object, it will be repeatedly modified with a +\code{setattr()} call for every option argument written to an option +destination, and finally returned by \method{parse{\_}args()}. + +If \method{parse{\_}args()} encounters any errors in the argument list, it calls +the OptionParser's \method{error()} method with an appropriate end-user error +message. This ultimately terminates your process with an exit status of +2 (the traditional \UNIX{} exit status for command-line errors). \subsubsection{Querying and manipulating your option parser\label{optparse-querying-manipulating-option-parser}} @@ -1050,9 +1237,8 @@ If the OptionParser has an option corresponding to \code{opt{\_}str}, that option is removed. If that option provided any other option strings, all of those option strings become invalid. - If \code{opt{\_}str} does not occur in any option belonging to this -OptionParser, raises \exception{ValueError}. +OptionParser, raises ValueError. \end{description} @@ -1074,20 +1260,20 @@ mechanism. You can set the conflict-handling mechanism either in the constructor: \begin{verbatim} -parser = OptionParser(..., conflict_handler="...") +parser = OptionParser(..., conflict_handler=handler) \end{verbatim} or with a separate call: \begin{verbatim} -parser.set_conflict_handler("...") +parser.set_conflict_handler(handler) \end{verbatim} -The available conflict-handling mechanisms are: +The available conflict handlers are: \begin{quote} \begin{description} \item[\code{error} (default)] assume option conflicts are a programming error and raise -\exception{OptionConflictError} +OptionConflictError \item[\code{resolve}] resolve option conflicts intelligently (see below) \end{description} @@ -1131,7 +1317,78 @@ -n, --noisy be noisy --dry-run new dry-run option \end{verbatim} -% $Id: reference.txt 415 2004-09-30 02:26:17Z greg $ + + +\subsubsection{Cleanup\label{optparse-cleanup}} + +OptionParser instances have several cyclic references. This should not +be a problem for Python's garbage collector, but you may wish to break +the cyclic references explicitly by calling \code{destroy()} on your +OptionParser once you are done with it. This is particularly useful in +long-running applications where large object graphs are reachable from +your OptionParser. + + +\subsubsection{Other methods\label{optparse-other-methods}} + +OptionParser supports several other public methods: +\begin{itemize} +\item {} +\code{set{\_}usage(usage)} + +Set the usage string according to the rules described above for the +\code{usage} constructor keyword argument. Passing \code{None} sets the +default usage string; use \code{SUPPRESS{\_}USAGE} to suppress a usage +message. + +\item {} +\code{enable{\_}interspersed{\_}args()}, \code{disable{\_}interspersed{\_}args()} + +Enable/disable positional arguments interspersed with options, similar +to GNU getopt (enabled by default). For example, if \code{"-a"} and +\code{"-b"} are both simple options that take no arguments, \module{optparse} +normally accepts this syntax: +\begin{verbatim} +prog -a arg1 -b arg2 +\end{verbatim} + +and treats it as equivalent to +\begin{verbatim} +prog -a -b arg1 arg2 +\end{verbatim} + +To disable this feature, call \code{disable{\_}interspersed{\_}args()}. This +restores traditional \UNIX{} syntax, where option parsing stops with the +first non-option argument. + +\item {} +\code{set{\_}defaults(dest=value, ...)} + +Set default values for several option destinations at once. Using +\method{set{\_}defaults()} is the preferred way to set default values for +options, since multiple options can share the same destination. For +example, if several ``mode'' options all set the same destination, any +one of them can set the default, and the last one wins: +\begin{verbatim} +parser.add_option("--advanced", action="store_const", + dest="mode", const="advanced", + default="novice") # overridden below +parser.add_option("--novice", action="store_const", + dest="mode", const="novice", + default="advanced") # overrides above setting +\end{verbatim} + +To avoid this confusion, use \method{set{\_}defaults()}: +\begin{verbatim} +parser.set_defaults(mode="advanced") +parser.add_option("--advanced", action="store_const", + dest="mode", const="advanced") +parser.add_option("--novice", action="store_const", + dest="mode", const="novice") +\end{verbatim} + +\end{itemize} +% $Id: reference.txt 505 2005-07-22 01:52:40Z gward $ \subsection{Option Callbacks\label{optparse-option-callbacks}} @@ -1234,7 +1491,7 @@ the current list of leftover arguments, ie. arguments that have been consumed but are neither options nor option arguments. Feel free to modify \code{parser.largs}, e.g. by adding more -arguments to it. (This list will become \var{args}, the second +arguments to it. (This list will become \code{args}, the second return value of \method{parse{\_}args()}.) \item[\code{parser.rargs}] the current list of remaining arguments, ie. with \code{opt{\_}str} and @@ -1260,7 +1517,7 @@ \subsubsection{Raising errors in a callback\label{optparse-raising-errors-in-callback}} -The callback function should raise \exception{OptionValueError} if there are any +The callback function should raise OptionValueError if there are any problems with the option or its argument(s). \module{optparse} catches this and terminates the program, printing the error message you supply to stderr. Your message should be clear, concise, accurate, and mention Modified: python/trunk/Lib/optparse.py ============================================================================== --- python/trunk/Lib/optparse.py (original) +++ python/trunk/Lib/optparse.py Sun Apr 23 05:47:58 2006 @@ -16,7 +16,7 @@ # Python developers: please do not make changes to this file, since # it is automatically generated from the Optik source code. -__version__ = "1.5a2" +__version__ = "1.5.1" __all__ = ['Option', 'SUPPRESS_HELP', @@ -35,8 +35,8 @@ 'BadOptionError'] __copyright__ = """ -Copyright (c) 2001-2004 Gregory P. Ward. All rights reserved. -Copyright (c) 2002-2004 Python Software Foundation. All rights reserved. +Copyright (c) 2001-2006 Gregory P. Ward. All rights reserved. +Copyright (c) 2002-2006 Python Software Foundation. All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are @@ -67,21 +67,26 @@ """ import sys, os +import types import textwrap -try: - from gettext import gettext as _ -except ImportError: - _ = lambda arg: arg def _repr(self): return "<%s at 0x%x: %s>" % (self.__class__.__name__, id(self), self) # This file was generated from: -# Id: option_parser.py 421 2004-10-26 00:45:16Z greg -# Id: option.py 422 2004-10-26 00:53:47Z greg -# Id: help.py 367 2004-07-24 23:21:21Z gward -# Id: errors.py 367 2004-07-24 23:21:21Z gward +# Id: option_parser.py 509 2006-04-20 00:58:24Z gward +# Id: option.py 509 2006-04-20 00:58:24Z gward +# Id: help.py 509 2006-04-20 00:58:24Z gward +# Id: errors.py 509 2006-04-20 00:58:24Z gward + +try: + from gettext import gettext +except ImportError: + def gettext(message): + return message +_ = gettext + class OptParseError (Exception): def __init__(self, msg): @@ -120,8 +125,25 @@ class BadOptionError (OptParseError): """ - Raised if an invalid or ambiguous option is seen on the command-line. + Raised if an invalid option is seen on the command line. """ + def __init__(self, opt_str): + self.opt_str = opt_str + + def __str__(self): + return _("no such option: %s") % self.opt_str + +class AmbiguousOptionError (BadOptionError): + """ + Raised if an ambiguous option is seen on the command line. + """ + def __init__(self, opt_str, possibilities): + BadOptionError.__init__(self, opt_str) + self.possibilities = possibilities + + def __str__(self): + return (_("ambiguous option: %s (%s?)") + % (self.opt_str, ", ".join(self.possibilities))) class HelpFormatter: @@ -223,15 +245,30 @@ def format_heading(self, heading): raise NotImplementedError, "subclasses must implement" - def format_description(self, description): - if not description: - return "" - desc_width = self.width - self.current_indent + def _format_text(self, text): + """ + Format a paragraph of free-form text for inclusion in the + help output at the current indentation level. + """ + text_width = self.width - self.current_indent indent = " "*self.current_indent - return textwrap.fill(description, - desc_width, + return textwrap.fill(text, + text_width, initial_indent=indent, - subsequent_indent=indent) + "\n" + subsequent_indent=indent) + + def format_description(self, description): + if description: + return self._format_text(description) + "\n" + else: + return "" + + def format_epilog(self, epilog): + if epilog: + return "\n" + self._format_text(epilog) + "\n" + else: + return "" + def expand_default(self, option): if self.parser is None or not self.default_tag: @@ -328,7 +365,7 @@ self, indent_increment, max_help_position, width, short_first) def format_usage(self, usage): - return _("usage: %s\n") % usage + return _("Usage: %s\n") % usage def format_heading(self, heading): return "%*s%s:\n" % (self.current_indent, "", heading) @@ -353,8 +390,27 @@ return "%s\n%s\n" % (heading, "=-"[self.level] * len(heading)) -_builtin_cvt = { "int" : (int, _("integer")), - "long" : (long, _("long integer")), +def _parse_num(val, type): + if val[:2].lower() == "0x": # hexadecimal + radix = 16 + elif val[:2].lower() == "0b": # binary + radix = 2 + val = val[2:] or "0" # have to remove "0b" prefix + elif val[:1] == "0": # octal + radix = 8 + else: # decimal + radix = 10 + + return type(val, radix) + +def _parse_int(val): + return _parse_num(val, int) + +def _parse_long(val): + return _parse_num(val, long) + +_builtin_cvt = { "int" : (_parse_int, _("integer")), + "long" : (_parse_long, _("long integer")), "float" : (float, _("floating-point")), "complex" : (complex, _("complex")) } @@ -422,6 +478,7 @@ "store_true", "store_false", "append", + "append_const", "count", "callback", "help", @@ -435,6 +492,7 @@ "store_true", "store_false", "append", + "append_const", "count") # The set of actions for which it makes sense to supply a value @@ -448,6 +506,10 @@ ALWAYS_TYPED_ACTIONS = ("store", "append") + # The set of actions which take a 'const' attribute. + CONST_ACTIONS = ("store_const", + "append_const") + # The set of known types for option parsers. Again, listed here for # constructor argument validation. TYPES = ("string", "int", "long", "float", "complex", "choice") @@ -572,9 +634,17 @@ # No type given? "string" is the most sensible default. self.type = "string" else: - # Allow type objects as an alternative to their names. - if type(self.type) is type: + # Allow type objects or builtin type conversion functions + # (int, str, etc.) as an alternative to their names. (The + # complicated check of __builtin__ is only necessary for + # Python 2.1 and earlier, and is short-circuited by the + # first check on modern Pythons.) + import __builtin__ + if ( type(self.type) is types.TypeType or + (hasattr(self.type, "__name__") and + getattr(__builtin__, self.type.__name__, None) is self.type) ): self.type = self.type.__name__ + if self.type == "str": self.type = "string" @@ -589,7 +659,7 @@ if self.choices is None: raise OptionError( "must supply a list of choices for type 'choice'", self) - elif type(self.choices) not in (tuple, list): + elif type(self.choices) not in (types.TupleType, types.ListType): raise OptionError( "choices must be a list of strings ('%s' supplied)" % str(type(self.choices)).split("'")[1], self) @@ -613,7 +683,7 @@ self.dest = self._short_opts[0][1] def _check_const(self): - if self.action != "store_const" and self.const is not None: + if self.action not in self.CONST_ACTIONS and self.const is not None: raise OptionError( "'const' must not be supplied for action %r" % self.action, self) @@ -633,12 +703,12 @@ raise OptionError( "callback not callable: %r" % self.callback, self) if (self.callback_args is not None and - type(self.callback_args) is not tuple): + type(self.callback_args) is not types.TupleType): raise OptionError( "callback_args, if supplied, must be a tuple: not %r" % self.callback_args, self) if (self.callback_kwargs is not None and - type(self.callback_kwargs) is not dict): + type(self.callback_kwargs) is not types.DictType): raise OptionError( "callback_kwargs, if supplied, must be a dict: not %r" % self.callback_kwargs, self) @@ -720,6 +790,8 @@ setattr(values, dest, False) elif action == "append": values.ensure_value(dest, []).append(value) + elif action == "append_const": + values.ensure_value(dest, []).append(self.const) elif action == "count": setattr(values, dest, values.ensure_value(dest, 0) + 1) elif action == "callback": @@ -748,11 +820,9 @@ True, False except NameError: (True, False) = (1, 0) -try: - basestring -except NameError: - basestring = (str, unicode) +def isbasestring(x): + return isinstance(x, types.StringType) or isinstance(x, types.UnicodeType) class Values: @@ -766,16 +836,13 @@ __repr__ = _repr - def __eq__(self, other): + def __cmp__(self, other): if isinstance(other, Values): - return self.__dict__ == other.__dict__ - elif isinstance(other, dict): - return self.__dict__ == other + return cmp(self.__dict__, other.__dict__) + elif isinstance(other, types.DictType): + return cmp(self.__dict__, other) else: - return False - - def __ne__(self, other): - return not (self == other) + return -1 def _update_careful(self, dict): """ @@ -893,6 +960,13 @@ return self.description + def destroy(self): + """see OptionParser.destroy().""" + del self._short_opt + del self._long_opt + del self.defaults + + # -- Option-adding methods ----------------------------------------- def _check_conflict(self, option): @@ -926,7 +1000,7 @@ """add_option(Option) add_option(opt_str, ..., kwarg=val, ...) """ - if type(args[0]) is str: + if type(args[0]) is types.StringType: option = self.option_class(*args, **kwargs) elif len(args) == 1 and not kwargs: option = args[0] @@ -1018,6 +1092,11 @@ def set_title(self, title): self.title = title + def destroy(self): + """see OptionParser.destroy().""" + OptionContainer.destroy(self) + del self.option_list + # -- Help-formatting methods --------------------------------------- def format_help(self, formatter): @@ -1044,6 +1123,8 @@ prog : string the name of the current program (to override os.path.basename(sys.argv[0])). + epilog : string + paragraph of help text to print after option help option_groups : [OptionGroup] list of option groups in this parser (option groups are @@ -1102,7 +1183,8 @@ description=None, formatter=None, add_help_option=True, - prog=None): + prog=None, + epilog=None): OptionContainer.__init__( self, option_class, conflict_handler, description) self.set_usage(usage) @@ -1114,6 +1196,7 @@ formatter = IndentedHelpFormatter() self.formatter = formatter self.formatter.set_parser(self) + self.epilog = epilog # Populate the option list; initial sources are the # standard_option_list class attribute, the 'option_list' @@ -1124,6 +1207,22 @@ self._init_parsing_state() + + def destroy(self): + """ + Declare that you are done with this OptionParser. This cleans up + reference cycles so the OptionParser (and all objects referenced by + it) can be garbage-collected promptly. After calling destroy(), the + OptionParser is unusable. + """ + OptionContainer.destroy(self) + for group in self.option_groups: + group.destroy() + del self.option_list + del self.option_groups + del self.formatter + + # -- Private methods ----------------------------------------------- # (used by our or OptionContainer's constructor) @@ -1167,7 +1266,7 @@ elif usage is SUPPRESS_USAGE: self.usage = None # For backwards compatibility with Optik 1.3 and earlier. - elif usage.startswith("usage:" + " "): + elif usage.lower().startswith("usage: "): self.usage = usage[7:] else: self.usage = usage @@ -1201,7 +1300,7 @@ defaults = self.defaults.copy() for option in self._get_all_options(): default = defaults.get(option.dest) - if isinstance(default, basestring): + if isbasestring(default): opt_str = option.get_opt_string() defaults[option.dest] = option.check_value(opt_str, default) @@ -1212,7 +1311,7 @@ def add_option_group(self, *args, **kwargs): # XXX lots of overlap with OptionContainer.add_option() - if type(args[0]) is str: + if type(args[0]) is types.StringType: group = OptionGroup(self, *args, **kwargs) elif len(args) == 1 and not kwargs: group = args[0] @@ -1276,7 +1375,7 @@ try: stop = self._process_args(largs, rargs, values) except (BadOptionError, OptionValueError), err: - self.error(err.msg) + self.error(str(err)) args = largs + rargs return self.check_values(values, args) @@ -1401,7 +1500,7 @@ i += 1 # we have consumed a character if not option: - self.error(_("no such option: %s") % opt) + raise BadOptionError(opt) if option.takes_value(): # Any characters left in arg? Pretend they're the # next arg, and stop consuming characters of arg. @@ -1501,7 +1600,7 @@ formatter = self.formatter formatter.store_option_strings(self) result = [] - result.append(formatter.format_heading(_("options"))) + result.append(formatter.format_heading(_("Options"))) formatter.indent() if self.option_list: result.append(OptionContainer.format_option_help(self, formatter)) @@ -1513,6 +1612,9 @@ # Drop the last "\n", or the header if no options or option groups: return "".join(result[:-1]) + def format_epilog(self, formatter): + return formatter.format_epilog(self.epilog) + def format_help(self, formatter=None): if formatter is None: formatter = self.formatter @@ -1522,6 +1624,7 @@ if self.description: result.append(self.format_description(formatter) + "\n") result.append(self.format_option_help(formatter)) + result.append(self.format_epilog(formatter)) return "".join(result) def print_help(self, file=None): @@ -1555,11 +1658,10 @@ if len(possibilities) == 1: return possibilities[0] elif not possibilities: - raise BadOptionError(_("no such option: %s") % s) + raise BadOptionError(s) else: # More than one possible completion: ambiguous prefix. - raise BadOptionError(_("ambiguous option: %s (%s?)") - % (s, ", ".join(possibilities))) + raise AmbiguousOptionError(s, possibilities) # Some day, there might be many Option classes. As of Optik 1.3, the Modified: python/trunk/Lib/test/test_optparse.py ============================================================================== --- python/trunk/Lib/test/test_optparse.py (original) +++ python/trunk/Lib/test/test_optparse.py Sun Apr 23 05:47:58 2006 @@ -10,17 +10,22 @@ import sys import os +import re import copy +import types import unittest from cStringIO import StringIO from pprint import pprint from test import test_support + from optparse import make_option, Option, IndentedHelpFormatter, \ TitledHelpFormatter, OptionParser, OptionContainer, OptionGroup, \ SUPPRESS_HELP, SUPPRESS_USAGE, OptionError, OptionConflictError, \ - BadOptionError, OptionValueError, Values, _match_abbrev + BadOptionError, OptionValueError, Values +from optparse import _match_abbrev +from optparse import _parse_num # Do the right thing with boolean values for all known Python versions. try: @@ -28,6 +33,7 @@ except NameError: (True, False) = (1, 0) +retype = type(re.compile('')) class InterceptedError(Exception): def __init__(self, @@ -96,7 +102,8 @@ args -- positional arguments to `func` kwargs -- keyword arguments to `func` expected_exception -- exception that should be raised - expected_output -- output we expect to see + expected_message -- expected exception message (or pattern + if a compiled regex object) Returns the exception raised for further testing. """ @@ -109,14 +116,23 @@ func(*args, **kwargs) except expected_exception, err: actual_message = str(err) - self.assertEqual(actual_message, - expected_message, + if isinstance(expected_message, retype): + self.assert_(expected_message.search(actual_message), """\ +expected exception message pattern: +/%s/ +actual exception message: +'''%s''' +""" % (expected_message.pattern, actual_message)) + else: + self.assertEqual(actual_message, + expected_message, + """\ expected exception message: -'''%(expected_message)s''' +'''%s''' actual exception message: -'''%(actual_message)s''' -""" % locals()) +'''%s''' +""" % (expected_message, actual_message)) return err else: @@ -157,7 +173,9 @@ sys.stdout = save_stdout except InterceptedError, err: - self.assertEqual(output, expected_output) + if output != expected_output: + self.fail("expected: \n'''\n" + expected_output + + "'''\nbut got \n'''\n" + output + "'''") self.assertEqual(err.exit_status, expected_status) self.assertEqual(err.exit_message, expected_error) else: @@ -366,6 +384,23 @@ self.assertRaises(self.parser.remove_option, ('foo',), None, ValueError, "no such option 'foo'") + def test_refleak(self): + # If an OptionParser is carrying around a reference to a large + # object, various cycles can prevent it from being GC'd in + # a timely fashion. destroy() breaks the cycles to ensure stuff + # can be cleaned up. + big_thing = [42] + refcount = sys.getrefcount(big_thing) + parser = OptionParser() + parser.add_option("-a", "--aaarggh") + parser.big_thing = big_thing + + parser.destroy() + #self.assertEqual(refcount, sys.getrefcount(big_thing)) + del parser + self.assertEqual(refcount, sys.getrefcount(big_thing)) + + class TestOptionValues(BaseTest): def setUp(self): pass @@ -391,13 +426,21 @@ def setUp(self): self.parser = OptionParser() - def test_type_aliases(self): - self.parser.add_option("-x", type=int) + def test_str_aliases_string(self): + self.parser.add_option("-s", type="str") + self.assertEquals(self.parser.get_option("-s").type, "string") + + def test_new_type_object(self): self.parser.add_option("-s", type=str) - self.parser.add_option("-t", type="str") + self.assertEquals(self.parser.get_option("-s").type, "string") + self.parser.add_option("-x", type=int) self.assertEquals(self.parser.get_option("-x").type, "int") + + def test_old_type_object(self): + self.parser.add_option("-s", type=types.StringType) self.assertEquals(self.parser.get_option("-s").type, "string") - self.assertEquals(self.parser.get_option("-t").type, "string") + self.parser.add_option("-x", type=types.IntType) + self.assertEquals(self.parser.get_option("-x").type, "int") # Custom type for testing processing of default values. @@ -487,13 +530,13 @@ save_argv = sys.argv[:] try: sys.argv[0] = os.path.join("foo", "bar", "baz.py") - parser = OptionParser("usage: %prog ...", version="%prog 1.2") - expected_usage = "usage: baz.py ...\n" + parser = OptionParser("%prog ...", version="%prog 1.2") + expected_usage = "Usage: baz.py ...\n" self.assertUsage(parser, expected_usage) self.assertVersion(parser, "baz.py 1.2") self.assertHelp(parser, expected_usage + "\n" + - "options:\n" + "Options:\n" " --version show program's version number and exit\n" " -h, --help show this help message and exit\n") finally: @@ -505,7 +548,7 @@ usage="%prog arg arg") parser.remove_option("-h") parser.remove_option("--version") - expected_usage = "usage: thingy arg arg\n" + expected_usage = "Usage: thingy arg arg\n" self.assertUsage(parser, expected_usage) self.assertVersion(parser, "thingy 0.1") self.assertHelp(parser, expected_usage + "\n") @@ -515,9 +558,9 @@ def setUp(self): self.parser = OptionParser(prog="test") self.help_prefix = """\ -usage: test [options] +Usage: test [options] -options: +Options: -h, --help show this help message and exit """ self.file_help = "read from FILE [default: %default]" @@ -699,13 +742,16 @@ self.assertParseOK(["-a", "--", "foo", "bar"], {'a': "--", 'boo': None, 'foo': None}, ["foo", "bar"]), + self.assertParseOK(["-a", "--", "--foo", "bar"], + {'a': "--", 'boo': None, 'foo': ["bar"]}, + []), def test_short_option_joined_and_separator(self): self.assertParseOK(["-ab", "--", "--foo", "bar"], {'a': "b", 'boo': None, 'foo': None}, ["--foo", "bar"]), - def test_invalid_option_becomes_positional_arg(self): + def test_hyphen_becomes_positional_arg(self): self.assertParseOK(["-ab", "-", "--foo", "bar"], {'a': "b", 'boo': None, 'foo': ["bar"]}, ["-"]) @@ -870,6 +916,8 @@ type="float", dest="point") self.parser.add_option("-f", "--foo", action="append", nargs=2, type="int", dest="foo") + self.parser.add_option("-z", "--zero", action="append_const", + dest="foo", const=(0, 0)) def test_nargs_append(self): self.assertParseOK(["-f", "4", "-3", "blah", "--foo", "1", "666"], @@ -885,6 +933,11 @@ {'point': None, 'foo':[(3, 4)]}, []) + def test_nargs_append_const(self): + self.assertParseOK(["--zero", "--foo", "3", "4", "-z"], + {'point': None, 'foo':[(0, 0), (3, 4), (0, 0)]}, + []) + class TestVersion(BaseTest): def test_version(self): self.parser = InterceptingOptionParser(usage=SUPPRESS_USAGE, @@ -960,8 +1013,14 @@ self.parser.add_option("-a", None, type="string", dest="a") self.parser.add_option("-f", "--file", type="file", dest="file") + def tearDown(self): + if os.path.isdir(test_support.TESTFN): + os.rmdir(test_support.TESTFN) + elif os.path.isfile(test_support.TESTFN): + os.unlink(test_support.TESTFN) + class MyOption (Option): - def check_file (option, opt, value): + def check_file(option, opt, value): if not os.path.exists(value): raise OptionValueError("%s: file does not exist" % value) elif not os.path.isfile(value): @@ -972,25 +1031,23 @@ TYPE_CHECKER = copy.copy(Option.TYPE_CHECKER) TYPE_CHECKER["file"] = check_file - def test_extend_file(self): + def test_filetype_ok(self): open(test_support.TESTFN, "w").close() self.assertParseOK(["--file", test_support.TESTFN, "-afoo"], {'file': test_support.TESTFN, 'a': 'foo'}, []) - os.unlink(test_support.TESTFN) - - def test_extend_file_nonexistent(self): + def test_filetype_noexist(self): self.assertParseFail(["--file", test_support.TESTFN, "-afoo"], "%s: file does not exist" % test_support.TESTFN) - def test_file_irregular(self): + def test_filetype_notfile(self): os.mkdir(test_support.TESTFN) self.assertParseFail(["--file", test_support.TESTFN, "-afoo"], "%s: not a regular file" % test_support.TESTFN) - os.rmdir(test_support.TESTFN) + class TestExtendAddActions(BaseTest): def setUp(self): @@ -1003,7 +1060,7 @@ STORE_ACTIONS = Option.STORE_ACTIONS + ("extend",) TYPED_ACTIONS = Option.TYPED_ACTIONS + ("extend",) - def take_action (self, action, dest, opt, value, values, parser): + def take_action(self, action, dest, opt, value, values, parser): if action == "extend": lvalue = value.split(",") values.ensure_value(dest, []).extend(lvalue) @@ -1072,7 +1129,7 @@ callback=lambda: None, type="string", help="foo") - expected_help = ("options:\n" + expected_help = ("Options:\n" " -t TEST, --test=TEST foo\n") self.assertHelp(parser, expected_help) @@ -1085,7 +1142,7 @@ dest="points", default=[])] self.parser = OptionParser(option_list=options) - def process_tuple (self, option, opt, value, parser_, len, type): + def process_tuple(self, option, opt, value, parser_, len, type): self.assertEqual(len, 3) self.assert_(type is int) @@ -1110,7 +1167,7 @@ self.parser = OptionParser(option_list=options) # Callback that meddles in rargs, largs - def process_n (self, option, opt, value, parser_): + def process_n(self, option, opt, value, parser_): # option is -3, -5, etc. nargs = int(opt[1:]) rargs = parser_.rargs @@ -1139,7 +1196,7 @@ callback=self.process_many, type="int")] self.parser = OptionParser(option_list=options) - def process_many (self, option, opt, value, parser_): + def process_many(self, option, opt, value, parser_): if opt == "-a": self.assertEqual(value, ("foo", "bar")) elif opt == "--apple": @@ -1162,7 +1219,7 @@ self.parser.add_option("--foo-bar", action="callback", callback=self.check_abbrev) - def check_abbrev (self, option, opt, value, parser): + def check_abbrev(self, option, opt, value, parser): self.assertEqual(opt, "--foo-bar") def test_abbrev_callback_expansion(self): @@ -1177,7 +1234,7 @@ self.parser = InterceptingOptionParser(usage=SUPPRESS_USAGE, option_list=options) - def variable_args (self, option, opt, value, parser): + def variable_args(self, option, opt, value, parser): self.assert_(value is None) done = 0 value = [] @@ -1229,7 +1286,7 @@ self.parser = InterceptingOptionParser(usage=SUPPRESS_USAGE, option_list=options) - def show_version (self, option, opt, value, parser): + def show_version(self, option, opt, value, parser): parser.values.show_version = 1 class TestConflict(ConflictBase): @@ -1280,7 +1337,7 @@ def test_conflict_resolve_help(self): self.assertOutput(["-h"], """\ -options: +Options: --verbose increment verbosity -h, --help show this help message and exit -v, --version show version @@ -1319,7 +1376,7 @@ def test_conflict_override_help(self): self.assertOutput(["-h"], """\ -options: +Options: -h, --help show this help message and exit -n, --dry-run dry run mode """) @@ -1332,9 +1389,9 @@ # -- Other testing. ---------------------------------------------------- _expected_help_basic = """\ -usage: bar.py [options] +Usage: bar.py [options] -options: +Options: -a APPLE throw APPLEs at basket -b NUM, --boo=NUM shout "boo!" NUM times (in order to frighten away all the evil spirits that cause trouble and mayhem) @@ -1343,9 +1400,9 @@ """ _expected_help_long_opts_first = """\ -usage: bar.py [options] +Usage: bar.py [options] -options: +Options: -a APPLE throw APPLEs at basket --boo=NUM, -b NUM shout "boo!" NUM times (in order to frighten away all the evil spirits that cause trouble and mayhem) @@ -1358,7 +1415,7 @@ ===== bar.py [options] -options +Options ======= -a APPLE throw APPLEs at basket --boo=NUM, -b NUM shout "boo!" NUM times (in order to frighten away all the @@ -1368,9 +1425,9 @@ """ _expected_help_short_lines = """\ -usage: bar.py [options] +Usage: bar.py [options] -options: +Options: -a APPLE throw APPLEs at basket -b NUM, --boo=NUM shout "boo!" NUM times (in order to frighten away all the evil spirits @@ -1382,15 +1439,8 @@ class TestHelp(BaseTest): def setUp(self): - self.orig_columns = os.environ.get('COLUMNS') self.parser = self.make_parser(80) - def tearDown(self): - if self.orig_columns is None: - del os.environ['COLUMNS'] - else: - os.environ['COLUMNS'] = self.orig_columns - def make_parser(self, columns): options = [ make_option("-a", type="string", dest='a', @@ -1419,7 +1469,7 @@ self.assertHelpEquals(_expected_help_basic) def test_help_old_usage(self): - self.parser.set_usage("usage: %prog [options]") + self.parser.set_usage("Usage: %prog [options]") self.assertHelpEquals(_expected_help_basic) def test_help_long_opts_first(self): @@ -1449,13 +1499,13 @@ group.add_option("-g", action="store_true", help="Group option.") self.parser.add_option_group(group) - self.assertHelpEquals("""\ -usage: bar.py [options] + expect = """\ +Usage: bar.py [options] This is the program description for bar.py. bar.py has an option group as well as single options. -options: +Options: -a APPLE throw APPLEs at basket -b NUM, --boo=NUM shout "boo!" NUM times (in order to frighten away all the evil spirits that cause trouble and mayhem) @@ -1467,9 +1517,12 @@ that some of them bite. -g Group option. -""") +""" + self.assertHelpEquals(expect) + self.parser.epilog = "Please report bugs to /dev/null." + self.assertHelpEquals(expect + "\nPlease report bugs to /dev/null.\n") class TestMatchAbbrev(BaseTest): @@ -1490,6 +1543,43 @@ BadOptionError, "ambiguous option: --f (%s?)" % possibilities) +class TestParseNumber(BaseTest): + def setUp(self): + self.parser = InterceptingOptionParser() + self.parser.add_option("-n", type=int) + self.parser.add_option("-l", type=long) + + def test_parse_num_fail(self): + self.assertRaises( + _parse_num, ("", int), {}, + ValueError, + re.compile(r"invalid literal for int().*: '?'?")) + self.assertRaises( + _parse_num, ("0xOoops", long), {}, + ValueError, + re.compile(r"invalid literal for long().*: '?0xOoops'?")) + + def test_parse_num_ok(self): + self.assertEqual(_parse_num("0", int), 0) + self.assertEqual(_parse_num("0x10", int), 16) + self.assertEqual(_parse_num("0XA", long), 10L) + self.assertEqual(_parse_num("010", long), 8L) + self.assertEqual(_parse_num("0b11", int), 3) + self.assertEqual(_parse_num("0b", long), 0L) + + def test_numeric_options(self): + self.assertParseOK(["-n", "42", "-l", "0x20"], + { "n": 42, "l": 0x20 }, []) + self.assertParseOK(["-n", "0b0101", "-l010"], + { "n": 5, "l": 8 }, []) + self.assertParseFail(["-n008"], + "option -n: invalid integer value: '008'") + self.assertParseFail(["-l0b0123"], + "option -l: invalid long integer value: '0b0123'") + self.assertParseFail(["-l", "0x12x"], + "option -l: invalid long integer value: '0x12x'") + + def _testclasses(): mod = sys.modules[__name__] return [getattr(mod, name) for name in dir(mod) if name.startswith('Test')] Modified: python/trunk/Misc/NEWS ============================================================================== --- python/trunk/Misc/NEWS (original) +++ python/trunk/Misc/NEWS Sun Apr 23 05:47:58 2006 @@ -86,6 +86,11 @@ Library ------- +- Updated optparse module to Optik 1.5.1 (allow numeric constants in + hex, octal, or binary; add ``append_const`` action; keep going if + gettext cannot be imported; added ``OptionParser.destroy()`` method; + added ``epilog`` for better help generation). + - Bug #1473760: ``tempfile.TemporaryFile()`` could hang on Windows, when called from a thread spawned as a side effect of importing a module. From buildbot at python.org Sun Apr 23 05:56:21 2006 From: buildbot at python.org (buildbot at python.org) Date: Sun, 23 Apr 2006 03:56:21 +0000 Subject: [Python-checkins] buildbot failure in x86 cygwin trunk Message-ID: <20060423035621.2C9141E4007@bag.python.org> The Buildbot has detected a new failure of x86 cygwin trunk. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520cygwin%2520trunk/builds/255 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: greg.ward BUILD FAILED: failed svn sincerely, -The Buildbot From buildbot at python.org Sun Apr 23 06:56:20 2006 From: buildbot at python.org (buildbot at python.org) Date: Sun, 23 Apr 2006 04:56:20 +0000 Subject: [Python-checkins] buildbot warnings in x86 Ubuntu dapper (icc) trunk Message-ID: <20060423045621.269251E4008@bag.python.org> The Buildbot has detected a new failure of x86 Ubuntu dapper (icc) trunk. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520Ubuntu%2520dapper%2520%2528icc%2529%2520trunk/builds/224 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: greg.ward Build Had Warnings: warnings test sincerely, -The Buildbot From python-checkins at python.org Sun Apr 23 07:14:11 2006 From: python-checkins at python.org (nick.coghlan) Date: Sun, 23 Apr 2006 07:14:11 +0200 (CEST) Subject: [Python-checkins] r45655 - peps/trunk/pep-0343.txt Message-ID: <20060423051411.5DAF41E4007@bag.python.org> Author: nick.coghlan Date: Sun Apr 23 07:14:10 2006 New Revision: 45655 Modified: peps/trunk/pep-0343.txt Log: Clarify the original meanings I intended for various terms, and record the fact that gaining consensus on the terminology is still an open issue Modified: peps/trunk/pep-0343.txt ============================================================================== --- peps/trunk/pep-0343.txt (original) +++ peps/trunk/pep-0343.txt Sun Apr 23 07:14:10 2006 @@ -7,7 +7,7 @@ Type: Standards Track Content-Type: text/plain Created: 13-May-2005 -Post-History: 2-Jun-2005, 16-Oct-2005, 29-Oct-2005 +Post-History: 2-Jun-2005, 16-Oct-2005, 29-Oct-2005, 23-Apr-2006 Abstract @@ -19,6 +19,7 @@ section on Resolved Issues). It's still at Draft status until Guido gives a final blessing to the updated PEP. + Author's Note This PEP was originally written in first person by Guido, and @@ -26,6 +27,13 @@ on python-dev. Any first person references are from Guido's original. + Python's alpha release cycle revealed terminology problems in this + PEP and in the associated documentation and implementation [14]. + The current version of the PEP reflects the implementation and + documentation as at Python 2.5a2. The PEP will be updated to + reflect any changes made to the terminology prior to the final + Python 2.5 release. + Introduction After a lot of discussion about PEP 340 and alternatives, I @@ -225,9 +233,9 @@ The translation of the above statement is: - ctx = (EXPR).__context__() - exit = ctx.__exit__ # Not calling it yet - value = ctx.__enter__() + mgr = (EXPR).__context__() + exit = mgr.__exit__ # Not calling it yet + value = mgr.__enter__() exc = True try: try: @@ -244,7 +252,7 @@ if exc: exit(None, None, None) - Here, the lowercase variables (ctx, exit, value, exc) are internal + Here, the lowercase variables (mgr, exit, value, exc) are internal variables and not accessible to the user; they will most likely be implemented as special registers or stack positions. @@ -260,37 +268,37 @@ as exceptions by __exit__().) The call to the __context__() method serves a similar purpose to - that of the __iter__() method of iterator and iterables. An - object with with simple state requirements (such as + that of the __iter__() method of iterator and iterables. A context + object with simple state requirements (such as threading.RLock) may provide its own __enter__() and __exit__() methods, and simply return 'self' from its __context__ method. On - the other hand, an object with more complex state requirements - (such as decimal.Context) may return a distinct context object + the other hand, a context object with more complex state requirements + (such as decimal.Context) may return a distinct context manager each time its __context__ method is invoked. If the "as VAR" part of the syntax is omitted, the "VAR =" part of - the translation is omitted (but ctx.__enter__() is still called). + the translation is omitted (but mgr.__enter__() is still called). - The calling convention for ctx.__exit__() is as follows. If the + The calling convention for mgr.__exit__() is as follows. If the finally-suite was reached through normal completion of BLOCK or through a non-local goto (a break, continue or return statement in - BLOCK), ctx.__exit__() is called with three None arguments. If + BLOCK), mgr.__exit__() is called with three None arguments. If the finally-suite was reached through an exception raised in - BLOCK, ctx.__exit__() is called with three arguments representing + BLOCK, mgr.__exit__() is called with three arguments representing the exception type, value, and traceback. - IMPORTANT: if ctx.__exit__() returns a "true" value, the exception + IMPORTANT: if mgr.__exit__() returns a "true" value, the exception is "swallowed". That is, if it returns "true", execution continues at the next statement after the with-statement, even if an exception happened inside the with-statement. However, if the with-statement was left via a non-local goto (break, continue or - return), this non-local return is resumed when ctx.__exit__() + return), this non-local return is resumed when mgr.__exit__() returns regardless of the return value. The motivation for this - detail is to make it possible for ctx.__exit__() to swallow + detail is to make it possible for mgr.__exit__() to swallow exceptions, without making it too easy (since the default return value, None, is false and this causes the exception to be re-raised). The main use case for swallowing exceptions is to - make it possible to write the @contextmanager decorator so thatn + make it possible to write the @contextmanager decorator so that a try/except block in a decorated generator behaves exactly as if the body of the generator were expanded in-line at the place of the with-statement. @@ -330,6 +338,17 @@ methods should avoid raising errors unless they have actually failed. (And allowing the original error to proceed isn't a failure.) + + Objects returned by __context__() methods should also provide a + __context__() method that returns self. This allows a program to + retrieve the context manager directly without breaking anything. + For example, the following should work just as well as the normal + case where the extra variable isn't used: + + mgr = (EXPR).__context__() + with mgr as VAR: + BLOCK + Transition Plan @@ -406,13 +425,13 @@ finally: f.close() # Ditto for errors here (however unlikely) - A robust builtin implementation of this decorator will be made + A robust implementation of this decorator will be made part of the standard library. Just as generator-iterator functions are very useful for writing - __iter__() methods for iterables, generator-context functions will - be very useful for writing __context__() methods for contexts. - These methods will still need to be decorated using the + __iter__() methods for iterables, generator context functions will + be very useful for writing __context__() methods for context + objects. These methods will still need to be decorated using the contextmanager decorator. To ensure an obvious error message if the decorator is left out, generator-iterator objects will NOT be given a native context - if you want to ensure a generator is closed @@ -446,7 +465,7 @@ is entered). OTOH such mistakes are easily diagnosed; for example, the - generator-context decorator above raises RuntimeError when a + generator context decorator above raises RuntimeError when a second with-statement calls f.__enter__() again. A similar error can be raised if __enter__ is invoked on a closed file object. @@ -470,19 +489,70 @@ returns an iterator (this means that all iterators are iterables, but not all iterables are iterators). - This PEP proposes that the protocol used by the with statement be - known as the "context management protocol", and that objects that - implement that protocol be known as "context managers". The term - "context manager" then encompasses all objects with a __context__() - method that returns a context object. (This means that all contexts - are context managers, but not all context managers are contexts). - - The term "context" is based on the concept that the context object - defines a context of execution for the code that forms the body - of the with statement. + This PEP proposes that the protocol consisting of the __enter__() + and __exit__() methods, and a __context__() method that returns + self be known as the "context management protocol", and that + objects that implement that protocol be known as "context + managers". + + The term "context object" then encompasses all objects with a + __context__() method that returns a context manager. The protocol + these objects implement is called the "context protocol". This + means that all context managers are context objects, but not all + context objects are context managers, just as all iterators are + iterables, but not all iterables are iterators. + + These terms are based on the concept that the context object + defines a context of execution for the code that forms the body of + the with statement. The role of the context manager is to + translate the context object's stored state into an active + manipulation of the runtime environment to setup and tear down the + desired runtime context for the duration of the with statement. + For example, a synchronisation lock's context manager acquires the + lock when entering the with statement, and releases the lock when + leaving it. The runtime context established within the body of the + with statement is that the synchronisation lock is currently held. + + The general term "context" is unfortunately ambiguous. If necessary, + it can be made more explicit by using the terms "context objext" for + objects providing a __context__() method and "runtime context" for + the runtime environment modifications made by the context manager. + When solely discussing use of the with statement, the distinction + between the two shouldn't matter as the context object fully + defines the changes made to the runtime context. The distinction is + more important when discussing the process of implementing context + objects and context managers. + +Open Issues + + 1. As noted earlier, the standard terminology section has not yet + met with consensus on python-dev. It will be refined throughout + the Python 2.5 release cycle based on user feedback on the + usability of the documentation. + + 2. The original resolution was for the decorator to make a context + manager from a generator to be a builtin called "contextmanager". + The shorter term "context" was considered too ambiguous and + potentially confusing [9]. + The different flavours of generators could then be described as: + - A "generator function" is an undecorated function containing + the 'yield' keyword, and the objects produced by + such functions are "generator-iterators". The term + "generator" may refer to either a generator function or a + generator-iterator depending on the situation. + - A "generator context function" is a generator function to + which the "contextmanager" decorator is applied and the + objects produced by such functions are "generator-context- + managers". The term "generator context" may refer to either + a generator context function or a generator-context-manager + depending on the situation. + + In the Python 2.5 implementation, the decorator is actually part + of the standard library module contextlib. The ongoing + terminology review may lead to it being renamed + "contextlib.context" (with the existence of the underlying context + manager being an implementation detail). - In cases where the general term "context" would be ambiguous, it - can be made explicit by expanding it to "manageable context". Resolved Issues @@ -570,21 +640,7 @@ works without having to first understand the mechanics of how generator context managers are implemented. - 6. The decorator to make a context manager from a generator will be - a builtin called "contextmanager". The shorter term "context" was - considered too ambiguous and potentially confusing [9]. - The different flavours of generators can then be described as: - - A "generator function" is an undecorated function containing - the 'yield' keyword, and the objects produced by - such functions are "generator-iterators". The term - "generator" may refer to either a generator function or a - generator-iterator depending on the situation. - - A "generator context function" is a generator function to - which the "contextmanager" decorator is applied and the - objects produced by such functions are "generator-context- - managers". The term "generator context" may refer to either a - generator context function or a generator-context-manager - depending on the situation. + 6. See point 2 in open issues :) 7. A generator function used to implement a __context__ method will need to be decorated with the contextmanager decorator in order @@ -609,7 +665,7 @@ appropriate objects, such as threading.RLock, will be able to be used directly in with statements. - The tense used in the names of the example context managers is not + The tense used in the names of the example contexts is not arbitrary. Past tense ("-ed") is used when the name refers to an action which is done in the __enter__ method and undone in the __exit__ method. Progressive tense ("-ing") is used when the name @@ -634,7 +690,7 @@ # if via return or by an uncaught exception). PEP 319 gives a use case for also having an unlocked() - template; this can be written very similarly (just swap the + context; this can be written very similarly (just swap the acquire() and release() calls). 2. A template for opening a file that ensures the file is closed @@ -903,8 +959,10 @@ Reference Implementation - There is no implementation at this time. This PEP was accepted - by Guido at his EuroPython keynote, 27 June 2005. + This PEP was first accepted by Guido at his EuroPython + keynote, 27 June 2005. + It was accepted again later, with the __context__ method added. + The PEP was implemented for Python 2.5a1 References @@ -945,6 +1003,9 @@ [13] http://mail.python.org/pipermail/python-dev/2006-February/061903.html + [14] + http://mail.python.org/pipermail/python-dev/2006-April/063859.html + Copyright This document has been placed in the public domain. From python-checkins at python.org Sun Apr 23 10:46:03 2006 From: python-checkins at python.org (george.yoshida) Date: Sun, 23 Apr 2006 10:46:03 +0200 (CEST) Subject: [Python-checkins] r45657 - python/branches/release24-maint/Doc/whatsnew/whatsnew24.tex Message-ID: <20060423084603.C08B31E4007@bag.python.org> Author: george.yoshida Date: Sun Apr 23 10:46:03 2006 New Revision: 45657 Modified: python/branches/release24-maint/Doc/whatsnew/whatsnew24.tex Log: fix tex markup Modified: python/branches/release24-maint/Doc/whatsnew/whatsnew24.tex ============================================================================== --- python/branches/release24-maint/Doc/whatsnew/whatsnew24.tex (original) +++ python/branches/release24-maint/Doc/whatsnew/whatsnew24.tex Sun Apr 23 10:46:03 2006 @@ -1656,11 +1656,11 @@ \item Python can now be built with additional profiling for the interpreter itself, intended as an aid to people developing the - Python core. Providing \longprogramopt{--enable-profiling} to the + Python core. Providing \longprogramopt{enable-profiling} to the \program{configure} script will let you profile the interpreter with - \program{gprof}, and providing the \longprogramopt{--with-tsc} + \program{gprof}, and providing the \longprogramopt{with-tsc} switch enables profiling using the Pentium's Time-Stamp-Counter - register. Note that the \longprogramopt{--with-tsc} switch is slightly + register. Note that the \longprogramopt{with-tsc} switch is slightly misnamed, because the profiling feature also works on the PowerPC platform, though that processor architecture doesn't call that register ``the TSC register''. (Contributed by Jeremy Hylton.) From python-checkins at python.org Sun Apr 23 11:27:14 2006 From: python-checkins at python.org (george.yoshida) Date: Sun, 23 Apr 2006 11:27:14 +0200 (CEST) Subject: [Python-checkins] r45658 - python/trunk/Doc/ref/ref4.tex Message-ID: <20060423092714.583C41E4007@bag.python.org> Author: george.yoshida Date: Sun Apr 23 11:27:10 2006 New Revision: 45658 Modified: python/trunk/Doc/ref/ref4.tex Log: wrap SyntaxError with \exception{} Modified: python/trunk/Doc/ref/ref4.tex ============================================================================== --- python/trunk/Doc/ref/ref4.tex (original) +++ python/trunk/Doc/ref/ref4.tex Sun Apr 23 11:27:10 2006 @@ -127,7 +127,7 @@ If the wild card form of import --- \samp{import *} --- is used in a function and the function contains or is a nested block with free -variables, the compiler will raise a SyntaxError. +variables, the compiler will raise a \exception{SyntaxError}. If \keyword{exec} is used in a function and the function contains or is a nested block with free variables, the compiler will raise a From python-checkins at python.org Sun Apr 23 11:28:10 2006 From: python-checkins at python.org (george.yoshida) Date: Sun, 23 Apr 2006 11:28:10 +0200 (CEST) Subject: [Python-checkins] r45659 - python/branches/release24-maint/Doc/ref/ref4.tex Message-ID: <20060423092810.84FC51E4007@bag.python.org> Author: george.yoshida Date: Sun Apr 23 11:28:10 2006 New Revision: 45659 Modified: python/branches/release24-maint/Doc/ref/ref4.tex Log: wrap SyntaxError with \exception{} Modified: python/branches/release24-maint/Doc/ref/ref4.tex ============================================================================== --- python/branches/release24-maint/Doc/ref/ref4.tex (original) +++ python/branches/release24-maint/Doc/ref/ref4.tex Sun Apr 23 11:28:10 2006 @@ -127,7 +127,7 @@ If the wild card form of import --- \samp{import *} --- is used in a function and the function contains or is a nested block with free -variables, the compiler will raise a SyntaxError. +variables, the compiler will raise a \exception{SyntaxError}. If \keyword{exec} is used in a function and the function contains or is a nested block with free variables, the compiler will raise a From buildbot at python.org Sun Apr 23 11:57:51 2006 From: buildbot at python.org (buildbot at python.org) Date: Sun, 23 Apr 2006 09:57:51 +0000 Subject: [Python-checkins] buildbot warnings in hppa Ubuntu dapper trunk Message-ID: <20060423095751.883181E4003@bag.python.org> The Buildbot has detected a new failure of hppa Ubuntu dapper trunk. Full details are available at: http://www.python.org/dev/buildbot/all/hppa%2520Ubuntu%2520dapper%2520trunk/builds/263 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: george.yoshida Build Had Warnings: warnings test sincerely, -The Buildbot From buildbot at python.org Sun Apr 23 12:23:06 2006 From: buildbot at python.org (buildbot at python.org) Date: Sun, 23 Apr 2006 10:23:06 +0000 Subject: [Python-checkins] buildbot warnings in hppa Ubuntu dapper 2.4 Message-ID: <20060423102306.C01841E4003@bag.python.org> The Buildbot has detected a new failure of hppa Ubuntu dapper 2.4. Full details are available at: http://www.python.org/dev/buildbot/all/hppa%2520Ubuntu%2520dapper%25202.4/builds/36 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch branches/release24-maint] HEAD Blamelist: george.yoshida Build Had Warnings: warnings test sincerely, -The Buildbot From buildbot at python.org Sun Apr 23 12:27:05 2006 From: buildbot at python.org (buildbot at python.org) Date: Sun, 23 Apr 2006 10:27:05 +0000 Subject: [Python-checkins] buildbot warnings in sparc Ubuntu dapper 2.4 Message-ID: <20060423102705.D30861E4003@bag.python.org> The Buildbot has detected a new failure of sparc Ubuntu dapper 2.4. Full details are available at: http://www.python.org/dev/buildbot/all/sparc%2520Ubuntu%2520dapper%25202.4/builds/33 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch branches/release24-maint] HEAD Blamelist: george.yoshida Build Had Warnings: warnings test sincerely, -The Buildbot From buildbot at python.org Sun Apr 23 12:56:37 2006 From: buildbot at python.org (buildbot at python.org) Date: Sun, 23 Apr 2006 10:56:37 +0000 Subject: [Python-checkins] buildbot warnings in sparc solaris10 gcc 2.4 Message-ID: <20060423105637.BD8131E4003@bag.python.org> The Buildbot has detected a new failure of sparc solaris10 gcc 2.4. Full details are available at: http://www.python.org/dev/buildbot/all/sparc%2520solaris10%2520gcc%25202.4/builds/102 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch branches/release24-maint] HEAD Blamelist: george.yoshida Build Had Warnings: warnings test sincerely, -The Buildbot From buildbot at python.org Sun Apr 23 13:06:21 2006 From: buildbot at python.org (buildbot at python.org) Date: Sun, 23 Apr 2006 11:06:21 +0000 Subject: [Python-checkins] buildbot warnings in ia64 Debian unstable 2.4 Message-ID: <20060423110621.5932B1E4003@bag.python.org> The Buildbot has detected a new failure of ia64 Debian unstable 2.4. Full details are available at: http://www.python.org/dev/buildbot/all/ia64%2520Debian%2520unstable%25202.4/builds/36 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch branches/release24-maint] HEAD Blamelist: george.yoshida Build Had Warnings: warnings test sincerely, -The Buildbot From buildbot at python.org Sun Apr 23 13:25:52 2006 From: buildbot at python.org (buildbot at python.org) Date: Sun, 23 Apr 2006 11:25:52 +0000 Subject: [Python-checkins] buildbot warnings in x86 Ubuntu dapper (icc) 2.4 Message-ID: <20060423112552.BD4641E4016@bag.python.org> The Buildbot has detected a new failure of x86 Ubuntu dapper (icc) 2.4. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520Ubuntu%2520dapper%2520%2528icc%2529%25202.4/builds/40 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch branches/release24-maint] HEAD Blamelist: george.yoshida Build Had Warnings: warnings test sincerely, -The Buildbot From python-checkins at python.org Sun Apr 23 13:59:26 2006 From: python-checkins at python.org (ronald.oussoren) Date: Sun, 23 Apr 2006 13:59:26 +0200 (CEST) Subject: [Python-checkins] r45660 - python/trunk/Modules/posixmodule.c python/trunk/Modules/socketmodule.c python/trunk/Modules/timemodule.c Message-ID: <20060423115926.AD7EA1E4003@bag.python.org> Author: ronald.oussoren Date: Sun Apr 23 13:59:25 2006 New Revision: 45660 Modified: python/trunk/Modules/posixmodule.c python/trunk/Modules/socketmodule.c python/trunk/Modules/timemodule.c Log: Patch 1471925 - Weak linking support for OSX This patch causes several symbols in the socket and posix module to be weakly linked on OSX and disables usage of ftime on OSX. These changes make it possible to use a binary build on OSX 10.4 on a 10.3 system. Modified: python/trunk/Modules/posixmodule.c ============================================================================== --- python/trunk/Modules/posixmodule.c (original) +++ python/trunk/Modules/posixmodule.c Sun Apr 23 13:59:25 2006 @@ -13,6 +13,18 @@ /* See also ../Dos/dosmodule.c */ +#ifdef __APPLE__ + /* + * Step 1 of support for weak-linking a number of symbols existing on + * OSX 10.4 and later, see the comment in the #ifdef __APPLE__ block + * at the end of this file for more information. + */ +# pragma weak lchown +# pragma weak statvfs +# pragma weak fstatvfs + +#endif /* __APPLE__ */ + #define PY_SSIZE_T_CLEAN #include "Python.h" @@ -8266,6 +8278,45 @@ PyModule_AddObject(m, "statvfs_result", (PyObject*) &StatVFSResultType); initialized = 1; + +#ifdef __APPLE__ + /* + * Step 2 of weak-linking support on Mac OS X. + * + * The code below removes functions that are not available on the + * currently active platform. + * + * This block allow one to use a python binary that was build on + * OSX 10.4 on OSX 10.3, without loosing access to new APIs on + * OSX 10.4. + */ +#ifdef HAVE_FSTATVFS + if (fstatvfs == NULL) { + if (PyObject_DelAttrString(m, "fstatvfs") == -1) { + return; + } + } +#endif /* HAVE_FSTATVFS */ + +#ifdef HAVE_STATVFS + if (statvfs == NULL) { + if (PyObject_DelAttrString(m, "statvfs") == -1) { + return; + } + } +#endif /* HAVE_STATVFS */ + +# ifdef HAVE_LCHOWN + if (lchown == NULL) { + if (PyObject_DelAttrString(m, "lchown") == -1) { + return; + } + } +#endif /* HAVE_LCHOWN */ + + +#endif /* __APPLE__ */ + } #ifdef __cplusplus Modified: python/trunk/Modules/socketmodule.c ============================================================================== --- python/trunk/Modules/socketmodule.c (original) +++ python/trunk/Modules/socketmodule.c Sun Apr 23 13:59:25 2006 @@ -61,6 +61,15 @@ */ +#ifdef __APPLE__ + /* + * inet_aton is not available on OSX 10.3, yet we want to use a binary + * that was build on 10.4 or later to work on that release, weak linking + * comes to the rescue. + */ +# pragma weak inet_aton +#endif + #include "Python.h" #include "structmember.h" @@ -306,6 +315,11 @@ older releases don't have */ #undef HAVE_GETADDRINFO #endif + +#ifdef HAVE_INET_ATON +#define USE_INET_ATON_WEAKLINK +#endif + #endif /* I know this is a bad practice, but it is the easiest... */ @@ -3333,7 +3347,9 @@ #endif #ifdef HAVE_INET_ATON struct in_addr buf; -#else +#endif + +#if !defined(HAVE_INET_ATON) || defined(USE_INET_ATON_WEAKLINK) /* Have to use inet_addr() instead */ unsigned long packed_addr; #endif @@ -3344,6 +3360,10 @@ #ifdef HAVE_INET_ATON + +#ifdef USE_INET_ATON_WEAKLINK + if (inet_aton != NULL) { +#endif if (inet_aton(ip_addr, &buf)) return PyString_FromStringAndSize((char *)(&buf), sizeof(buf)); @@ -3352,7 +3372,14 @@ "illegal IP address string passed to inet_aton"); return NULL; -#else /* ! HAVE_INET_ATON */ +#ifdef USE_INET_ATON_WEAKLINK + } else { +#endif + +#endif + +#if !defined(HAVE_INET_ATON) || defined(USE_INET_ATON_WEAKLINK) + /* special-case this address as inet_addr might return INADDR_NONE * for this */ if (strcmp(ip_addr, "255.255.255.255") == 0) { @@ -3369,6 +3396,11 @@ } return PyString_FromStringAndSize((char *) &packed_addr, sizeof(packed_addr)); + +#ifdef USE_INET_ATON_WEAKLINK + } +#endif + #endif } Modified: python/trunk/Modules/timemodule.c ============================================================================== --- python/trunk/Modules/timemodule.c (original) +++ python/trunk/Modules/timemodule.c Sun Apr 23 13:59:25 2006 @@ -5,6 +5,18 @@ #include "structseq.h" #include "timefuncs.h" +#ifdef __APPLE__ +#if defined(HAVE_GETTIMEOFDAY) && defined(HAVE_FTIME) + /* + * floattime falls back to ftime when getttimeofday fails because the latter + * might fail on some platforms. This fallback is unwanted on MacOSX because + * that makes it impossible to use a binary build on OSX 10.4 on earlier + * releases of the OS. Therefore claim we don't support ftime. + */ +# undef HAVE_FTIME +#endif +#endif + #include #include @@ -842,6 +854,7 @@ return (double)t.tv_sec + t.tv_usec*0.000001; #endif /* !GETTIMEOFDAY_NO_TZ */ } + #endif /* !HAVE_GETTIMEOFDAY */ { #if defined(HAVE_FTIME) From python-checkins at python.org Sun Apr 23 14:36:24 2006 From: python-checkins at python.org (ronald.oussoren) Date: Sun, 23 Apr 2006 14:36:24 +0200 (CEST) Subject: [Python-checkins] r45661 - python/trunk/Modules/selectmodule.c Message-ID: <20060423123624.41CA11E4003@bag.python.org> Author: ronald.oussoren Date: Sun Apr 23 14:36:23 2006 New Revision: 45661 Modified: python/trunk/Modules/selectmodule.c Log: Patch 1471761 - test for broken poll at runtime This patch checks if poll is broken when the select module is loaded instead of doing so at configure-time. This functionality is only active on Mac OS X. Modified: python/trunk/Modules/selectmodule.c ============================================================================== --- python/trunk/Modules/selectmodule.c (original) +++ python/trunk/Modules/selectmodule.c Sun Apr 23 14:36:23 2006 @@ -8,6 +8,13 @@ #include "Python.h" +#ifdef __APPLE__ + /* Perform runtime testing for a broken poll on OSX to make it easier + * to use the same binary on multiple releases of the OS. + */ +#undef HAVE_BROKEN_POLL +#endif + /* Windows #defines FD_SETSIZE to 64 if FD_SETSIZE isn't already defined. 64 is too small (too many people have bumped into that limit). Here we boost it. @@ -618,7 +625,38 @@ return NULL; return (PyObject *)rv; } -#endif /* HAVE_POLL && !HAVE_BROKEN_POLL */ + +#ifdef __APPLE__ +/* + * On some systems poll() sets errno on invalid file descriptors. We test + * for this at runtime because this bug may be fixed or introduced between + * OS releases. + */ +static int select_have_broken_poll(void) +{ + int poll_test; + int filedes[2]; + + struct pollfd poll_struct = { 0, POLLIN|POLLPRI|POLLOUT, 0 }; + + /* Create a file descriptor to make invalid */ + if (pipe(filedes) < 0) { + return 1; + } + poll_struct.fd = filedes[0]; + close(filedes[0]); + close(filedes[1]); + poll_test = poll(&poll_struct, 1, 0); + if (poll_test < 0) { + return 1; + } else if (poll_test == 0 && poll_struct.revents != POLLNVAL) { + return 1; + } + return 0; +} +#endif /* __APPLE__ */ + +#endif /* HAVE_POLL */ PyDoc_STRVAR(select_doc, "select(rlist, wlist, xlist[, timeout]) -> (rlist, wlist, xlist)\n\ @@ -645,9 +683,9 @@ static PyMethodDef select_methods[] = { {"select", select_select, METH_VARARGS, select_doc}, -#if defined(HAVE_POLL) && !defined(HAVE_BROKEN_POLL) +#if defined(HAVE_POLL) {"poll", select_poll, METH_VARARGS, poll_doc}, -#endif /* HAVE_POLL && !HAVE_BROKEN_POLL */ +#endif /* HAVE_POLL */ {0, 0}, /* sentinel */ }; @@ -668,29 +706,40 @@ SelectError = PyErr_NewException("select.error", NULL, NULL); Py_INCREF(SelectError); PyModule_AddObject(m, "error", SelectError); -#if defined(HAVE_POLL) && !defined(HAVE_BROKEN_POLL) - poll_Type.ob_type = &PyType_Type; - PyModule_AddIntConstant(m, "POLLIN", POLLIN); - PyModule_AddIntConstant(m, "POLLPRI", POLLPRI); - PyModule_AddIntConstant(m, "POLLOUT", POLLOUT); - PyModule_AddIntConstant(m, "POLLERR", POLLERR); - PyModule_AddIntConstant(m, "POLLHUP", POLLHUP); - PyModule_AddIntConstant(m, "POLLNVAL", POLLNVAL); +#if defined(HAVE_POLL) + +#ifdef __APPLE__ + if (select_have_broken_poll()) { + if (PyObject_DelAttrString(m, "poll") == -1) { + PyErr_Clear(); + } + } else { +#else + { +#endif + poll_Type.ob_type = &PyType_Type; + PyModule_AddIntConstant(m, "POLLIN", POLLIN); + PyModule_AddIntConstant(m, "POLLPRI", POLLPRI); + PyModule_AddIntConstant(m, "POLLOUT", POLLOUT); + PyModule_AddIntConstant(m, "POLLERR", POLLERR); + PyModule_AddIntConstant(m, "POLLHUP", POLLHUP); + PyModule_AddIntConstant(m, "POLLNVAL", POLLNVAL); #ifdef POLLRDNORM - PyModule_AddIntConstant(m, "POLLRDNORM", POLLRDNORM); + PyModule_AddIntConstant(m, "POLLRDNORM", POLLRDNORM); #endif #ifdef POLLRDBAND - PyModule_AddIntConstant(m, "POLLRDBAND", POLLRDBAND); + PyModule_AddIntConstant(m, "POLLRDBAND", POLLRDBAND); #endif #ifdef POLLWRNORM - PyModule_AddIntConstant(m, "POLLWRNORM", POLLWRNORM); + PyModule_AddIntConstant(m, "POLLWRNORM", POLLWRNORM); #endif #ifdef POLLWRBAND - PyModule_AddIntConstant(m, "POLLWRBAND", POLLWRBAND); + PyModule_AddIntConstant(m, "POLLWRBAND", POLLWRBAND); #endif #ifdef POLLMSG - PyModule_AddIntConstant(m, "POLLMSG", POLLMSG); + PyModule_AddIntConstant(m, "POLLMSG", POLLMSG); #endif -#endif /* HAVE_POLL && !HAVE_BROKEN_POLL */ + } +#endif /* HAVE_POLL */ } From buildbot at python.org Sun Apr 23 14:59:09 2006 From: buildbot at python.org (buildbot at python.org) Date: Sun, 23 Apr 2006 12:59:09 +0000 Subject: [Python-checkins] buildbot warnings in x86 cygwin trunk Message-ID: <20060423125909.007511E4003@bag.python.org> The Buildbot has detected a new failure of x86 cygwin trunk. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520cygwin%2520trunk/builds/257 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: ronald.oussoren Build Had Warnings: warnings test sincerely, -The Buildbot From python-checkins at python.org Sun Apr 23 17:13:33 2006 From: python-checkins at python.org (nick.coghlan) Date: Sun, 23 Apr 2006 17:13:33 +0200 (CEST) Subject: [Python-checkins] r45662 - python/trunk/Doc/lib/libstdtypes.tex Message-ID: <20060423151333.9A25B1E4007@bag.python.org> Author: nick.coghlan Date: Sun Apr 23 17:13:32 2006 New Revision: 45662 Modified: python/trunk/Doc/lib/libstdtypes.tex Log: Add a Context Types section to parallel the Iterator Types section (uses the same terminology as the 2.5a1 implementation) Modified: python/trunk/Doc/lib/libstdtypes.tex ============================================================================== --- python/trunk/Doc/lib/libstdtypes.tex (original) +++ python/trunk/Doc/lib/libstdtypes.tex Sun Apr 23 17:13:32 2006 @@ -1753,6 +1753,102 @@ \end{memberdesc} +\subsection{Context Types \label{typecontext}} + +\versionadded{2.5} +\index{context protocol} +\index{context management protocol} +\index{protocol!context} +\index{protocol!context management} + +Python's \keyword{with} statement supports the concept of a runtime +context defined by a context object. This is implemented using three +distinct methods; these are used to allow user-defined classes to +define a context. + +The \dfn{context protocol} consists of a single method that needs +to be provided for an object to define a runtime context: + +\begin{methoddesc}[context]{__context__}{} + Return a context manager object. The object is required to support + the context management protocol described below. If an object + supports different kinds of runtime context, additional methods can + be provided to specifically request context managers for those + kinds of context. (An example of an object supporting multiple kinds + of context would be a synchronisation object which supported both + a locked context for normal thread synchronisation and an unlocked + context to temporarily release a held lock while performing a + potentially long running operation) +\end{methoddesc} + +The context manager objects themselves are required to support the +following three methods, which together form the +\dfn{context management protocol}: + +\begin{methoddesc}[context manager]{__context__}{} + Return the context manager object itself. This is required to + allow both contexts and context managers to be used with the + \keyword{with} statement. +\end{methoddesc} + +\begin{methoddesc}[context manager]{__enter__}{} + Set up the runtime context and return either the defining context + object or another object related to the runtime context. The value + returned by this method is bound to the identifier in the + \keyword{as} clause of \keyword{with} statements using this context. + (An example of a context with a context manager that returns the + original context object is file objects, which are returned from + __enter__() to allow \function{open()} to be used directly in a with + statement. An example of a context manager that returns a related + object is \code{decimal.Context} which returns a copy of the original + context to allow changes to be made to the current decimal context + without affecting code outside the \keyword{with} statement). +\end{methoddesc} + +\begin{methoddesc}[context manager]{__exit__}{exc_type, exc_val, exc_tb} + Tear down the runtime context and return a Boolean flag indicating if + an expection that occurred should be suppressed. If an exception + occurred while executing the body of the \keyword{with} statement, the + arguments contain the exception type, value and traceback information. + Otherwise, all three arguments are \var{None}. + Returning a true value from this method will cause the \keyword{with} + statement to suppress the exception and continue execution with the + statement immediately following the \keyword{with} statement. Otherwise + the exception continues propagating after this method has finished + executing. Exceptions that occur during execution of this method will + replace any exception that occurred in the body of the \keyword{with} + statement. + The exception passed in should never be reraised explicitly - instead, + this method should return a false value to indicate that the method + completed successfully and does not want to suppress the raised + exception. This allows context management code (such as + \code{contextlib.nested}) to easily detect whether or not an + \method{__exit__()} method has actually failed. +\end{methoddesc} + +Python defines several context objects to support easy thread +synchronisation, prompt closure of files or other objects, and +thread-safe manipulation of the decimal arithmetic context. The +specific types are not important beyond their implementation of +the context protocol. + +Python's generators and the \code{contextlib.contextmanager} +decorator provide a convenient way to implement the context +and context management protocols. If a context object's +\method{__context__()} method is implemented as a generator decorated +with the \code{contextlib.contextmanager} decorator, it will +automatically return a context manager object supplying the +necessary \method{__context__()}, \method{__enter__()} and +\method{__exit__()} methods. + +Note that there is no specific slot for any of these methods in the +type structure for Python objects in the Python/C API. Extension +types wanting to define these methods must provide them as a normal +Python accessible method. Compared to the overhead of setting up the +runtime context, the overhead of a single class dictionary lookup +is negligible. + + \subsection{Other Built-in Types \label{typesother}} The interpreter supports several other kinds of objects. From python-checkins at python.org Sun Apr 23 17:14:38 2006 From: python-checkins at python.org (nick.coghlan) Date: Sun, 23 Apr 2006 17:14:38 +0200 (CEST) Subject: [Python-checkins] r45663 - python/trunk/Doc/lib/libcontextlib.tex Message-ID: <20060423151438.3ED531E4007@bag.python.org> Author: nick.coghlan Date: Sun Apr 23 17:14:37 2006 New Revision: 45663 Modified: python/trunk/Doc/lib/libcontextlib.tex Log: Update contextlib documentation to use the same terminology as the module implementation Modified: python/trunk/Doc/lib/libcontextlib.tex ============================================================================== --- python/trunk/Doc/lib/libcontextlib.tex (original) +++ python/trunk/Doc/lib/libcontextlib.tex Sun Apr 23 17:14:37 2006 @@ -49,8 +49,9 @@ the error (if any), or ensure that some cleanup takes place. Note that you can use \code{@contextmanager} to define a context -manager's \method{__context__} method. This is usually more convenient -than creating another class just to serve as a context. For example: +object's \method{__context__} method. This is usually more convenient +than creating another class just to serve as a context manager. +For example: \begin{verbatim} from __future__ import with_statement @@ -97,10 +98,10 @@ do_something() \end{verbatim} -Note that if one of the nested contexts' \method{__exit__()} method +Note that if the \method{__exit__()} method of one of the nested context managers raises an exception, any previous exception state will be lost; the new -exception will be passed to the outer contexts' \method{__exit__()} -method(s), if any. In general, \method{__exit__()} methods should avoid +exception will be passed to the \method{__exit__()} methods of any remaining +outer context managers. In general, \method{__exit__()} methods should avoid raising exceptions, and in particular they should not re-raise a passed-in exception. \end{funcdesc} @@ -127,9 +128,9 @@ from contextlib import closing import codecs -with closing(codecs.open("foo", encoding="utf8")) as f: +with closing(urllib.urlopen('http://www.python.org')) as f: for line in f: - print line.encode("latin1") + print line \end{verbatim} without needing to explicitly close \code{f}. Even if an error occurs, From buildbot at python.org Sun Apr 23 17:30:26 2006 From: buildbot at python.org (buildbot at python.org) Date: Sun, 23 Apr 2006 15:30:26 +0000 Subject: [Python-checkins] buildbot warnings in x86 XP trunk Message-ID: <20060423153026.4DCFC1E4007@bag.python.org> The Buildbot has detected a new failure of x86 XP trunk. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520XP%2520trunk/builds/541 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: nick.coghlan Build Had Warnings: warnings failed slave lost sincerely, -The Buildbot From buildbot at python.org Sun Apr 23 17:30:49 2006 From: buildbot at python.org (buildbot at python.org) Date: Sun, 23 Apr 2006 15:30:49 +0000 Subject: [Python-checkins] buildbot failure in x86 XP trunk Message-ID: <20060423153049.989131E4007@bag.python.org> The Buildbot has detected a new failure of x86 XP trunk. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520XP%2520trunk/builds/542 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: gerhard.haering BUILD FAILED: failed compile sincerely, -The Buildbot From python-checkins at python.org Sun Apr 23 17:39:17 2006 From: python-checkins at python.org (nick.coghlan) Date: Sun, 23 Apr 2006 17:39:17 +0200 (CEST) Subject: [Python-checkins] r45666 - python/trunk/Doc/ref/ref3.tex python/trunk/Doc/ref/ref7.tex Message-ID: <20060423153917.08AEC1E4007@bag.python.org> Author: nick.coghlan Date: Sun Apr 23 17:39:16 2006 New Revision: 45666 Modified: python/trunk/Doc/ref/ref3.tex python/trunk/Doc/ref/ref7.tex Log: Update with statement documentation to use same terminology as 2.5a1 implementation Modified: python/trunk/Doc/ref/ref3.tex ============================================================================== --- python/trunk/Doc/ref/ref3.tex (original) +++ python/trunk/Doc/ref/ref3.tex Sun Apr 23 17:39:16 2006 @@ -2116,42 +2116,54 @@ \versionadded{2.5} -A \dfn{context manager} is an object that manages the entry to, and exit -from, a \dfn{context} surrounding a block of code. Context managers are -normally invoked using the \keyword{with} statement (described in -section~\ref{with}), but can also be used by directly invoking their -methods. +A \dfn{context object} is an object that defines the runtime context +to be established when executing a \keyword{with} statement. The +context object provides a \dfn{context manager} which manages the +entry into, and the exit from, the desired runtime context for the +execution of the block of code. Context managers are normally +invoked using the \keyword{with} statement (described in +section~\ref{with}), but can also be used by directly invoking +their methods. + \stindex{with} \index{context manager} -\index{context} +\index{context object} + +Typical uses of contexts and context managers include saving and +restoring various kinds of global state, locking and unlocking +resources, closing opened files, etc. -Typical uses of context managers include saving and restoring various -kinds of global state, locking and unlocking resources, closing opened -files, etc. +For more information on contexts and context manager objects, see +``\ulink{Context Types}{../lib/typecontext.html}'' in the +\citetitle[../lib/lib.html]{Python Library Reference}. -\begin{methoddesc}[context manager]{__context__}{self} +\begin{methoddesc}[context]{__context__}{self} Invoked when the object is used as the context expression of a \keyword{with} statement. The return value must implement -\method{__enter__()} and \method{__exit__()} methods. Simple context -managers that wish to directly -implement \method{__enter__()} and \method{__exit__()} should just -return \var{self}. +\method{__enter__()} and \method{__exit__()} methods. Simple contexts +may be able to implement \method{__enter__()} and \method{__exit__()} +directly without requiring a separate context manager object and +should just return \var{self}. -Context managers written in Python can also implement this method using +Context objects written in Python can also implement this method using a generator function decorated with the \function{contextlib.contextmanager} decorator, as this can be simpler than writing individual \method{__enter__()} and \method{__exit__()} -methods when the state to be managed is complex. +methods on a separate object when the state to be managed is complex. + +Context manager objects also need to implement this method; they are +required to return themselves. \end{methoddesc} -\begin{methoddesc}[context]{__enter__}{self} -Enter the context defined by this object. The \keyword{with} statement +\begin{methoddesc}[context manager]{__enter__}{self} +Enter the context managed by this object. The \keyword{with} statement will bind this method's return value to the target(s) specified in the \keyword{as} clause of the statement, if any. \end{methoddesc} -\begin{methoddesc}[context]{__exit__}{exc_type, exc_value, traceback} -Exit the context defined by this object. The parameters describe the +\begin{methoddesc}[context manager]{__exit__} +{self, exc_type, exc_value, traceback} +Exit the context managed by this object. The parameters describe the exception that caused the context to be exited. If the context was exited without an exception, all three arguments will be \constant{None}. Modified: python/trunk/Doc/ref/ref7.tex ============================================================================== --- python/trunk/Doc/ref/ref7.tex (original) +++ python/trunk/Doc/ref/ref7.tex Sun Apr 23 17:39:16 2006 @@ -329,13 +329,12 @@ \begin{enumerate} -\item The expression is evaluated, to obtain a context manager -object. +\item The expression is evaluated, to obtain a context object. -\item The context manager's \method{__context__()} method is invoked to -obtain a context object. +\item The context object's \method{__context__()} method is invoked to +obtain a context manager object. -\item The context object's \method{__enter__()} method is invoked. +\item The context manager's \method{__enter__()} method is invoked. \item If a target list was included in the \keyword{with} statement, the return value from \method{__enter__()} is assigned to it. @@ -348,8 +347,8 @@ \item The suite is executed. -\item The context object's \method{__exit__()} method is invoked. If an -exception caused the suite to be exited, its type, value, and +\item The context manager's \method{__exit__()} method is invoked. If +an exception caused the suite to be exited, its type, value, and traceback are passed as arguments to \method{__exit__()}. Otherwise, three \constant{None} arguments are supplied. From python-checkins at python.org Sun Apr 23 18:05:06 2006 From: python-checkins at python.org (nick.coghlan) Date: Sun, 23 Apr 2006 18:05:06 +0200 (CEST) Subject: [Python-checkins] r45667 - python/trunk/Doc/tut/tut.tex Message-ID: <20060423160506.06D171E4007@bag.python.org> Author: nick.coghlan Date: Sun Apr 23 18:05:04 2006 New Revision: 45667 Modified: python/trunk/Doc/tut/tut.tex Log: Add a (very) brief mention of the with statement to the end of chapter 8 Modified: python/trunk/Doc/tut/tut.tex ============================================================================== --- python/trunk/Doc/tut/tut.tex (original) +++ python/trunk/Doc/tut/tut.tex Sun Apr 23 18:05:04 2006 @@ -941,9 +941,9 @@ u'abc' >>> str(u"abc") 'abc' ->>> u"äöü" +>>> u"�" u'\xe4\xf6\xfc' ->>> str(u"äöü") +>>> str(u"�") Traceback (most recent call last): File "", line 1, in ? UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-2: ordinal not in range(128) @@ -955,7 +955,7 @@ for encodings are preferred. \begin{verbatim} ->>> u"äöü".encode('utf-8') +>>> u"�".encode('utf-8') '\xc3\xa4\xc3\xb6\xc3\xbc' \end{verbatim} @@ -3744,6 +3744,36 @@ for releasing external resources (such as files or network connections), regardless of whether the use of the resource was successful. +\section{Predefined Clean-up Actions \label{cleanup-with}} + +Some objects define standard clean-up actions to be undertaken when +the object is no longer needed, regardless of whether or not the +operation using the object succeeded or failed. +Look at the following example, which tries to open a file and print +its contents to the screen. + +\begin{verbatim} +for line in open("myfile.txt"): + print line +\end{verbatim} + +The problem with this code is that it leaves the file open for an +indeterminate amount of time after the code has finished executing. +This is not an issue in simple scripts, but can be a problem for +larger applications. The \keyword{with} statement allows +objects like files to be used in a way that ensures they are +always cleaned up promptly and correctly. + +\begin{verbatim} +with open("myfile.txt") as f: + for line in f: + print line +\end{verbatim} + +After the statement is executed, the file \var{f} is always closed, +even if a problem was encountered while processing the lines. Other +objects which provide predefined clean-up actions will indicate +this in their documentation. \chapter{Classes \label{classes}} From buildbot at python.org Sun Apr 23 18:21:32 2006 From: buildbot at python.org (buildbot at python.org) Date: Sun, 23 Apr 2006 16:21:32 +0000 Subject: [Python-checkins] buildbot warnings in sparc Ubuntu dapper trunk Message-ID: <20060423162132.6F99A1E4020@bag.python.org> The Buildbot has detected a new failure of sparc Ubuntu dapper trunk. Full details are available at: http://www.python.org/dev/buildbot/all/sparc%2520Ubuntu%2520dapper%2520trunk/builds/141 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: ronald.oussoren Build Had Warnings: warnings test sincerely, -The Buildbot From python-checkins at python.org Sun Apr 23 18:35:20 2006 From: python-checkins at python.org (nick.coghlan) Date: Sun, 23 Apr 2006 18:35:20 +0200 (CEST) Subject: [Python-checkins] r45668 - python/trunk/Doc/tut/tut.tex Message-ID: <20060423163520.6C9721E4016@bag.python.org> Author: nick.coghlan Date: Sun Apr 23 18:35:19 2006 New Revision: 45668 Modified: python/trunk/Doc/tut/tut.tex Log: Take 2 on mentioning the with statement, this time without inadvertently killing the Unicode examples Modified: python/trunk/Doc/tut/tut.tex ============================================================================== --- python/trunk/Doc/tut/tut.tex (original) +++ python/trunk/Doc/tut/tut.tex Sun Apr 23 18:35:19 2006 @@ -941,9 +941,9 @@ u'abc' >>> str(u"abc") 'abc' ->>> u"�" +>>> u"äöü" u'\xe4\xf6\xfc' ->>> str(u"�") +>>> str(u"äöü") Traceback (most recent call last): File "", line 1, in ? UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-2: ordinal not in range(128) @@ -955,7 +955,7 @@ for encodings are preferred. \begin{verbatim} ->>> u"�".encode('utf-8') +>>> u"äöü".encode('utf-8') '\xc3\xa4\xc3\xb6\xc3\xbc' \end{verbatim} @@ -3744,6 +3744,7 @@ for releasing external resources (such as files or network connections), regardless of whether the use of the resource was successful. + \section{Predefined Clean-up Actions \label{cleanup-with}} Some objects define standard clean-up actions to be undertaken when @@ -3775,6 +3776,7 @@ objects which provide predefined clean-up actions will indicate this in their documentation. + \chapter{Classes \label{classes}} Python's class mechanism adds classes to the language with a minimum From python-checkins at python.org Sun Apr 23 19:04:08 2006 From: python-checkins at python.org (nick.coghlan) Date: Sun, 23 Apr 2006 19:04:08 +0200 (CEST) Subject: [Python-checkins] r45669 - python/trunk/Misc/NEWS Message-ID: <20060423170408.570381E4007@bag.python.org> Author: nick.coghlan Date: Sun Apr 23 19:04:07 2006 New Revision: 45669 Modified: python/trunk/Misc/NEWS Log: Backdated NEWS entry to record the implementation of PEP 338 for alpha 1 Modified: python/trunk/Misc/NEWS ============================================================================== --- python/trunk/Misc/NEWS (original) +++ python/trunk/Misc/NEWS Sun Apr 23 19:04:07 2006 @@ -181,6 +181,9 @@ Core and builtins ----------------- +- PEP 338: -m command line switch now delegates to runpy.run_module + allowing it to support modules in packages and zipfiles + - On Windows, .DLL is not an accepted file name extension for extension modules anymore; extensions are only found if they end in .PYD. @@ -670,6 +673,10 @@ Library ------- +- PEP 338: new module runpy defines a run_module function to support + executing modules which provide access to source code or a code object + via the PEP 302 import mechanisms. + - The email module's parsedate_tz function now sets the daylight savings flag to -1 (unknown) since it can't tell from the date whether it should be set. From buildbot at python.org Sun Apr 23 19:25:13 2006 From: buildbot at python.org (buildbot at python.org) Date: Sun, 23 Apr 2006 17:25:13 +0000 Subject: [Python-checkins] buildbot warnings in x86 Ubuntu dapper (icc) trunk Message-ID: <20060423172514.1D45B1E4007@bag.python.org> The Buildbot has detected a new failure of x86 Ubuntu dapper (icc) trunk. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520Ubuntu%2520dapper%2520%2528icc%2529%2520trunk/builds/229 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: gerhard.haering,nick.coghlan Build Had Warnings: warnings test sincerely, -The Buildbot From python-checkins at python.org Sun Apr 23 20:13:45 2006 From: python-checkins at python.org (tim.peters) Date: Sun, 23 Apr 2006 20:13:45 +0200 (CEST) Subject: [Python-checkins] r45670 - python/trunk/Lib/optparse.py Message-ID: <20060423181345.E5D471E4007@bag.python.org> Author: tim.peters Date: Sun Apr 23 20:13:45 2006 New Revision: 45670 Modified: python/trunk/Lib/optparse.py Log: Whitespace normalization. Modified: python/trunk/Lib/optparse.py ============================================================================== --- python/trunk/Lib/optparse.py (original) +++ python/trunk/Lib/optparse.py Sun Apr 23 20:13:45 2006 @@ -256,7 +256,7 @@ text_width, initial_indent=indent, subsequent_indent=indent) - + def format_description(self, description): if description: return self._format_text(description) + "\n" @@ -1212,7 +1212,7 @@ """ Declare that you are done with this OptionParser. This cleans up reference cycles so the OptionParser (and all objects referenced by - it) can be garbage-collected promptly. After calling destroy(), the + it) can be garbage-collected promptly. After calling destroy(), the OptionParser is unusable. """ OptionContainer.destroy(self) From python-checkins at python.org Sun Apr 23 21:14:28 2006 From: python-checkins at python.org (skip.montanaro) Date: Sun, 23 Apr 2006 21:14:28 +0200 (CEST) Subject: [Python-checkins] r45671 - python/trunk/Doc/lib/lib.tex python/trunk/Doc/lib/libtrace.tex Message-ID: <20060423191428.C9D281E4007@bag.python.org> Author: skip.montanaro Date: Sun Apr 23 21:14:27 2006 New Revision: 45671 Added: python/trunk/Doc/lib/libtrace.tex (contents, props changed) Modified: python/trunk/Doc/lib/lib.tex Log: first cut at trace module doc Modified: python/trunk/Doc/lib/lib.tex ============================================================================== --- python/trunk/Doc/lib/lib.tex (original) +++ python/trunk/Doc/lib/lib.tex Sun Apr 23 21:14:27 2006 @@ -360,7 +360,7 @@ \input{libprofile} % The Python Profiler \input{libhotshot} % unmaintained C profiler \input{libtimeit} - +\input{libtrace} % ============= % PYTHON ENGINE Added: python/trunk/Doc/lib/libtrace.tex ============================================================================== --- (empty file) +++ python/trunk/Doc/lib/libtrace.tex Sun Apr 23 21:14:27 2006 @@ -0,0 +1,95 @@ +\section{\module{trace} --- + Trace or track Python statement execution} + +\declaremodule{standard}{trace} +\modulesynopsis{Trace or track Python statement execution.} + +The \module{trace} module allows you to trace program execution, generate +annotated statement coverage listings, print caller/callee relationships and +list functions executed during a program run. It can be used in another +program or from the command line. + +\subsection{Command Line Usage} + +The \module{trace} module can be invoked from the command line. It can be +as simple as + +\begin{verbatim} +python -m trace --count somefile.py ... +\end{verbatim} + +The above will generate annotated listings of all Python modules imported +during the execution of somefile.py. + +\subsection{Command Line Arguments} + +\begin{description} +\item[--trace, -t]{Display lines as they are executed.} +\item[--count, -c]{Produce a set of annotated listing files upon program +completion that shows how many times each statement was executed.} +\item[--report, -r]{Produce an annotated list from an earlier program run that +used the \code{--count} and \code{--file} arguments.} +\item[--no-report, -R]{Do not generate annotated listings. This is useful +if you intend to make several runs with \code{--count} then produce a single +set of annotated listings at the end.} +\item[--listfuncs, -l]{List the functions executed by running the program.} +\item[--trackcalls, -T]{Generate calling relationships exposed by running the +program.} +\item[--file, -f]{Name a file containing (or to contain) counts.} +\item[--coverdir, -C]{Name a directory in which to save annotated listing +files.} +\item[--missing, -m]{When generating annotated listings, mark lines which +were not executed with \code{>>>>>>}.} +\item[--summary -s]{When using \code{--count} or \code{--report}, write a +brief summary to stdout for each file processed.} +\item[--ignore-module]{Ignore the named module and its submodules (if it is +a package). May be given multiple times.} +\item[--ignore-dir]{Ignore all modules and packages in the named directory +and subdirectories. May be given multiple times.} +\end{description} + +\subsection{Program Usage} + +\begin{classdesc}{Trace}{\optional{count=1\optional{,trace=1\optional{,countfuncs=0\optional{,countcallers=0\optional{,ignoremods=()\optional{,ignoredirs=()\optional{,infile=None\optional{,outfile=None}}}}}}}}} + +Create an object to trace execution of a single statement or expression. +All parameters are optional. \var{count} enables counting of line numbers. +\var{trace} enables line execution tracing. \var{countfuncs} enables +listing of the functions called during the run. \var{countcallers} enables +call relationship tracking. \var{ignoremods} is a list of modules or +packages to ignore. \var{ignoredirs} is a list of directories whose modules +or packages should be ignored. \var{infile} is the file from which to read +stored count information. \var{outfile} is a file in which to write updated +count information. + +\end{classdesc} + +\begin{methoddesc}[Trace]{run}{cmd} +Run \code{cmd} under control of the Trace object with the current tracing +parameters. +\end{methoddesc} + +\begin{methoddesc}[Trace]{run}{cmd\optional{,globals=None\optional{,locals=None}}} +Run \code{cmd} under control of the Trace object with the current tracing +parameters in the defined global and local environments. If not defined, +\code{globals} and \code{locals} default to empty dictionaries. +\end{methoddesc} + +\begin{methoddesc}[Trace]{runfunc}{func, *args, **kwds} +Call \code{function} with the given arguments under control of the Trace +object with the current tracing parameters. +\end{methoddesc} + +\subsubsection{Example} + +\begin{verbatim} +# create a Trace object, telling it what to ignore, and whether to +# do tracing or line-counting or both. +trace = trace.Trace(ignoredirs=[sys.prefix, sys.exec_prefix,], trace=0, + count=1) +# run the new command using the given trace +trace.run('main()') +# make a report, telling it where you want output +r = trace.results() +r.write_results(show_missing=True) +\end{verbatim} From python-checkins at python.org Sun Apr 23 21:26:33 2006 From: python-checkins at python.org (skip.montanaro) Date: Sun, 23 Apr 2006 21:26:33 +0200 (CEST) Subject: [Python-checkins] r45672 - python/trunk/Doc/lib/libtrace.tex Message-ID: <20060423192633.DB6301E4008@bag.python.org> Author: skip.montanaro Date: Sun Apr 23 21:26:33 2006 New Revision: 45672 Modified: python/trunk/Doc/lib/libtrace.tex Log: minor tweak Modified: python/trunk/Doc/lib/libtrace.tex ============================================================================== --- python/trunk/Doc/lib/libtrace.tex (original) +++ python/trunk/Doc/lib/libtrace.tex Sun Apr 23 21:26:33 2006 @@ -19,7 +19,7 @@ \end{verbatim} The above will generate annotated listings of all Python modules imported -during the execution of somefile.py. +during the execution of \code{somefile.py}. \subsection{Command Line Arguments} @@ -69,7 +69,7 @@ parameters. \end{methoddesc} -\begin{methoddesc}[Trace]{run}{cmd\optional{,globals=None\optional{,locals=None}}} +\begin{methoddesc}[Trace]{runctx}{cmd\optional{,globals=None\optional{,locals=None}}} Run \code{cmd} under control of the Trace object with the current tracing parameters in the defined global and local environments. If not defined, \code{globals} and \code{locals} default to empty dictionaries. From python-checkins at python.org Sun Apr 23 21:30:50 2006 From: python-checkins at python.org (skip.montanaro) Date: Sun, 23 Apr 2006 21:30:50 +0200 (CEST) Subject: [Python-checkins] r45673 - python/trunk/Doc/lib/libtrace.tex Message-ID: <20060423193050.DD66D1E400B@bag.python.org> Author: skip.montanaro Date: Sun Apr 23 21:30:50 2006 New Revision: 45673 Modified: python/trunk/Doc/lib/libtrace.tex Log: it's always helpful if the example works... Modified: python/trunk/Doc/lib/libtrace.tex ============================================================================== --- python/trunk/Doc/lib/libtrace.tex (original) +++ python/trunk/Doc/lib/libtrace.tex Sun Apr 23 21:30:50 2006 @@ -83,13 +83,15 @@ \subsubsection{Example} \begin{verbatim} +import sys + # create a Trace object, telling it what to ignore, and whether to # do tracing or line-counting or both. -trace = trace.Trace(ignoredirs=[sys.prefix, sys.exec_prefix,], trace=0, +tracer = trace.Trace(ignoredirs=[sys.prefix, sys.exec_prefix,], trace=0, count=1) -# run the new command using the given trace -trace.run('main()') -# make a report, telling it where you want output -r = trace.results() -r.write_results(show_missing=True) +# run the new command using the given tracer +tracer.run('main()') +# make a report, placing output in /tmp +r = tracer.results() +r.write_results(show_missing=True, coverdir="/tmp") \end{verbatim} From python-checkins at python.org Sun Apr 23 21:32:14 2006 From: python-checkins at python.org (skip.montanaro) Date: Sun, 23 Apr 2006 21:32:14 +0200 (CEST) Subject: [Python-checkins] r45674 - python/trunk/Lib/trace.py Message-ID: <20060423193214.E54E71E4008@bag.python.org> Author: skip.montanaro Date: Sun Apr 23 21:32:14 2006 New Revision: 45674 Modified: python/trunk/Lib/trace.py Log: correct example Modified: python/trunk/Lib/trace.py ============================================================================== --- python/trunk/Lib/trace.py (original) +++ python/trunk/Lib/trace.py Sun Apr 23 21:32:14 2006 @@ -35,15 +35,17 @@ trace.py --trackcalls spam.py eggs Sample use, programmatically - # create a Trace object, telling it what to ignore, and whether to - # do tracing or line-counting or both. - trace = trace.Trace(ignoredirs=[sys.prefix, sys.exec_prefix,], trace=0, - count=1) - # run the new command using the given trace - trace.run('main()') - # make a report, telling it where you want output - r = trace.results() - r.write_results(show_missing=True) + import sys + + # create a Trace object, telling it what to ignore, and whether to + # do tracing or line-counting or both. + tracer = trace.Trace(ignoredirs=[sys.prefix, sys.exec_prefix,], trace=0, + count=1) + # run the new command using the given tracer + tracer.run('main()') + # make a report, placing output in /tmp + r = tracer.results() + r.write_results(show_missing=True, coverdir="/tmp") """ import linecache From buildbot at python.org Sun Apr 23 22:44:37 2006 From: buildbot at python.org (buildbot at python.org) Date: Sun, 23 Apr 2006 20:44:37 +0000 Subject: [Python-checkins] buildbot warnings in hppa Ubuntu dapper trunk Message-ID: <20060423204437.4EC2D1E400B@bag.python.org> The Buildbot has detected a new failure of hppa Ubuntu dapper trunk. Full details are available at: http://www.python.org/dev/buildbot/all/hppa%2520Ubuntu%2520dapper%2520trunk/builds/271 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: skip.montanaro Build Had Warnings: warnings test sincerely, -The Buildbot From python-checkins at python.org Sun Apr 23 23:01:06 2006 From: python-checkins at python.org (andrew.kuchling) Date: Sun, 23 Apr 2006 23:01:06 +0200 (CEST) Subject: [Python-checkins] r45675 - python/trunk/Doc/whatsnew/whatsnew25.tex Message-ID: <20060423210106.E73D61E400B@bag.python.org> Author: andrew.kuchling Date: Sun Apr 23 23:01:04 2006 New Revision: 45675 Modified: python/trunk/Doc/whatsnew/whatsnew25.tex Log: Edits to the PEP 343 section Modified: python/trunk/Doc/whatsnew/whatsnew25.tex ============================================================================== --- python/trunk/Doc/whatsnew/whatsnew25.tex (original) +++ python/trunk/Doc/whatsnew/whatsnew25.tex Sun Apr 23 23:01:04 2006 @@ -585,8 +585,7 @@ In this section, I'll discuss the statement as it will commonly be used. In the next section, I'll examine the implementation details -and show how to write objects called ``context managers'' and -``contexts'' for use with this statement. +and show how to write objects for use with this statement. The '\keyword{with}' statement is a new control-flow structure whose basic structure is: @@ -596,13 +595,13 @@ with-block \end{verbatim} -The expression is evaluated, and it should result in a type of object -that's called a context manager. The context manager can return a +The expression is evaluated, and it should result in an object that +supports the context management protocol. This object may return a value that can optionally be bound to the name \var{variable}. (Note -carefully: \var{variable} is \emph{not} assigned the result of -\var{expression}.) One method of the context manager is run before -\var{with-block} is executed, and another method is run after the -block is done, even if the block raised an exception. +carefully that \var{variable} is \emph{not} assigned the result of +\var{expression}.) The object can then run set-up code +before \var{with-block} is executed and some clean-up code +is executed after the block is done, even if the block raised an exception. To enable the statement in Python 2.5, you need to add the following directive to your module: @@ -613,7 +612,8 @@ The statement will always be enabled in Python 2.6. -Some standard Python objects can now behave as context managers. File +Some standard Python objects now support the context management +protocol and can be used with the '\keyword{with}' statement. File objects are one example: \begin{verbatim} @@ -637,12 +637,11 @@ ... \end{verbatim} -The lock is acquired before the block is executed, and always released once +The lock is acquired before the block is executed and always released once the block is complete. The \module{decimal} module's contexts, which encapsulate the desired -precision and rounding characteristics for computations, can also be -used as context managers. +precision and rounding characteristics for computations, also work. \begin{verbatim} import decimal @@ -660,47 +659,45 @@ \subsection{Writing Context Managers\label{context-managers}} Under the hood, the '\keyword{with}' statement is fairly complicated. -Most people will only use '\keyword{with}' in company with -existing objects that are documented to work as context managers, and -don't need to know these details, so you can skip the following section if -you like. Authors of new context managers will need to understand the -details of the underlying implementation. +Most people will only use '\keyword{with}' in company with existing +objects and don't need to know these details, so you can skip the +following section if you like. Authors of new objects will need to +understand the details of the underlying implementation. A high-level explanation of the context management protocol is: \begin{itemize} \item The expression is evaluated and should result in an object -that's a context manager, meaning that it has a -\method{__context__()} method. +with a \method{__context__()} method. \item This object's \method{__context__()} method is called, and must -return a context object. +return another object that has \method{__enter__()} and +\method{__exit__()}. -\item The context's \method{__enter__()} method is called. -The value returned is assigned to \var{VAR}. If no \code{'as \var{VAR}'} -clause is present, the value is simply discarded. +\item This object's \method{__enter__()} method is called. The value +returned is assigned to \var{VAR}. If no \code{'as \var{VAR}'} clause +is present, the value is simply discarded. \item The code in \var{BLOCK} is executed. -\item If \var{BLOCK} raises an exception, the context object's +\item If \var{BLOCK} raises an exception, the \method{__exit__(\var{type}, \var{value}, \var{traceback})} is called with the exception's information, the same values returned by -\function{sys.exc_info()}. The method's return value -controls whether the exception is re-raised: any false value -re-raises the exception, and \code{True} will result in suppressing it. -You'll only rarely want to suppress the exception; the -author of the code containing the '\keyword{with}' statement will -never realize anything went wrong. +\function{sys.exc_info()}. The method's return value controls whether +the exception is re-raised: any false value re-raises the exception, +and \code{True} will result in suppressing it. You'll only rarely +want to suppress the exception; the author of the code containing the +'\keyword{with}' statement will never realize anything went wrong. \item If \var{BLOCK} didn't raise an exception, -the context object's \method{__exit__()} is still called, +the \method{__exit__()} method is still called, but \var{type}, \var{value}, and \var{traceback} are all \code{None}. \end{itemize} Let's think through an example. I won't present detailed code but -will only sketch the necessary code. The example will be writing a -context manager for a database that supports transactions. +will only sketch the methods necessary for a database that supports +transactions. (For people unfamiliar with database terminology: a set of changes to the database are grouped into a transaction. Transactions can be @@ -721,15 +718,15 @@ # ... more operations ... \end{verbatim} -The transaction should either be committed if the code in the block -runs flawlessly, or rolled back if there's an exception. +The transaction should be committed if the code in the block +runs flawlessly or rolled back if there's an exception. First, the \class{DatabaseConnection} needs a \method{__context__()} -method. Sometimes an object can be its own context manager and can -simply return \code{self}; the \module{threading} module's lock objects -can do this. For our database example, though, we need to -create a new object; I'll call this class \class{DatabaseContext}. -Our \method{__context__()} must therefore look like this: +method. Sometimes an object can simply return \code{self}; the +\module{threading} module's lock objects do this, for example. For +our database example, though, we need to create a new object; I'll +call this class \class{DatabaseContext}. Our \method{__context__()} +method must therefore look like this: \begin{verbatim} class DatabaseConnection: @@ -746,9 +743,9 @@ "Rolls back current transaction" \end{verbatim} -The context needs the connection object so that the connection -object's \method{commit()} or \method{rollback()} methods can be -called: +Instance of \class{DatabaseContext} need the connection object so that +the connection object's \method{commit()} or \method{rollback()} +methods can be called: \begin{verbatim} class DatabaseContext: @@ -756,12 +753,11 @@ self.connection = connection \end{verbatim} -The \method {__enter__()} method is pretty easy, having only -to start a new transaction. In this example, -the resulting cursor object would be a useful result, -so the method will return it. The user can -then add \code{as cursor} to their '\keyword{with}' statement -to bind the cursor to a variable name. +The \method {__enter__()} method is pretty easy, having only to start +a new transaction. For this application the resulting cursor object +would be a useful result, so the method will return it. The user can +then add \code{as cursor} to their '\keyword{with}' statement to bind +the cursor to a variable name. \begin{verbatim} class DatabaseContext: @@ -798,17 +794,18 @@ \subsection{The contextlib module\label{module-contextlib}} The new \module{contextlib} module provides some functions and a -decorator that are useful for writing context managers. +decorator that are useful for writing objects for use with the +'\keyword{with}' statement. The decorator is called \function{contextmanager}, and lets you write -a simple context manager as a generator. The generator should yield -exactly one value. The code up to the \keyword{yield} will be -executed as the \method{__enter__()} method, and the value yielded -will be the method's return value that will get bound to the variable -in the '\keyword{with}' statement's \keyword{as} clause, if any. The -code after the \keyword{yield} will be executed in the -\method{__exit__()} method. Any exception raised in the block -will be raised by the \keyword{yield} statement. +a simple context manager as a generator function. The generator +should yield exactly one value. The code up to the \keyword{yield} +will be executed as the \method{__enter__()} method, and the value +yielded will be the method's return value that will get bound to the +variable in the '\keyword{with}' statement's \keyword{as} clause, if +any. The code after the \keyword{yield} will be executed in the +\method{__exit__()} method. Any exception raised in the block will be +raised by the \keyword{yield} statement. Our database example from the previous section could be written using this decorator as: @@ -832,8 +829,9 @@ ... \end{verbatim} -You can also use this decorator to write the \method{__context__()} method -for a class without creating a new class for the context: +You can also use this decorator to write the \method{__context__()} +method for a class without creating a new class to act as the context +manager: \begin{verbatim} class DatabaseConnection: @@ -851,8 +849,8 @@ \end{verbatim} -There's a \function{nested(\var{mgr1}, \var{mgr2}, ...)} manager that -combines a number of context managers so you don't need to write +There's a \function{nested(\var{mgr1}, \var{mgr2}, ...)} function that +combines a number of contexts so you don't need to write nested '\keyword{with}' statements. This example statement does two things, starting a database transaction and acquiring a thread lock: @@ -862,7 +860,7 @@ ... \end{verbatim} -Finally, the \function{closing(\var{object})} context manager +Finally, the \function{closing(\var{object})} function returns \var{object} so that it can be bound to a variable, and calls \code{\var{object}.close()} at the end of the block. @@ -880,8 +878,7 @@ \seepep{343}{The ``with'' statement}{PEP written by Guido van~Rossum and Nick Coghlan; implemented by Mike Bland, Guido van~Rossum, and Neal Norwitz. The PEP shows the code generated for a '\keyword{with}' -statement, which can be helpful in learning how context managers -work.} +statement, which can be helpful in learning how the statement works.} \seeurl{../lib/module-contextlib.html}{The documentation for the \module{contextlib} module.} From python-checkins at python.org Sun Apr 23 23:51:12 2006 From: python-checkins at python.org (andrew.kuchling) Date: Sun, 23 Apr 2006 23:51:12 +0200 (CEST) Subject: [Python-checkins] r45676 - python/trunk/Doc/whatsnew/whatsnew25.tex Message-ID: <20060423215112.345921E4015@bag.python.org> Author: andrew.kuchling Date: Sun Apr 23 23:51:10 2006 New Revision: 45676 Modified: python/trunk/Doc/whatsnew/whatsnew25.tex Log: Add two items Modified: python/trunk/Doc/whatsnew/whatsnew25.tex ============================================================================== --- python/trunk/Doc/whatsnew/whatsnew25.tex (original) +++ python/trunk/Doc/whatsnew/whatsnew25.tex Sun Apr 23 23:51:10 2006 @@ -1340,6 +1340,30 @@ (Contributed by Raymond Hettinger.) +\item The \module{mailbox} module underwent a massive rewrite to add +the capability to modify mailboxes in addition to reading them. A new +set of classes that include \class{mbox}, \class{MH}, and +\class{Maildir} are used to read mailboxes, and have an +\method{add(\var{message})} method to add messages, +\method{remove(\var{key})} to remove messages, and +\method{lock()}/\method{unlock()} to lock/unlock the mailbox. The +following example converts a maildir-format mailbox into an mbox-format one: + +\begin{verbatim} +import mailbox + +# 'factory=None' uses email.Message.Message as the class representing +# individual messages. +src = mailbox.Maildir('maildir', factory=None) +dest = mailbox.mbox('/tmp/mbox') + +for msg in src: + dest.add(msg) +\end{verbatim} + +(Contributed by Gregory K. Johnson. Funding was provided by Google's +2005 Summer of Code.) + \item The \module{nis} module now supports accessing domains other than the system default domain by supplying a \var{domain} argument to the \function{nis.match()} and \function{nis.maps()} functions. @@ -1354,6 +1378,11 @@ lets you easily sort lists using multiple fields. (Contributed by Raymond Hettinger.) +\item The \module{optparse} module was updated to version 1.5.1 of the +Optik library. The \class{OptionParser} class gained an +\member{epilog} attribute, a string that will be printed after the +help message, and a \method{destroy()} method to break reference +cycles created by the object. (Contributed by Greg Ward.) \item The \module{os} module underwent several changes. The \member{stat_float_times} variable now defaults to true, meaning that From python-checkins at python.org Mon Apr 24 04:03:17 2006 From: python-checkins at python.org (tim.peters) Date: Mon, 24 Apr 2006 04:03:17 +0200 (CEST) Subject: [Python-checkins] r45677 - in python/trunk: Doc/lib/libdoctest.tex Misc/NEWS Message-ID: <20060424020317.EBFE91E4002@bag.python.org> Author: tim.peters Date: Mon Apr 24 04:03:16 2006 New Revision: 45677 Modified: python/trunk/Doc/lib/libdoctest.tex python/trunk/Misc/NEWS Log: Bug #1337990: clarified that `doctest` does not support examples requiring both expected output and an exception. I'll backport to 2.4 next. Modified: python/trunk/Doc/lib/libdoctest.tex ============================================================================== --- python/trunk/Doc/lib/libdoctest.tex (original) +++ python/trunk/Doc/lib/libdoctest.tex Mon Apr 24 04:03:16 2006 @@ -407,10 +407,13 @@ \subsubsection{What About Exceptions?\label{doctest-exceptions}} No problem, provided that the traceback is the only output produced by -the example: just paste in the traceback. Since tracebacks contain -details that are likely to change rapidly (for example, exact file paths -and line numbers), this is one case where doctest works hard to be -flexible in what it accepts. +the example: just paste in the traceback.\footnote{Examples containing + both expected output and an exception are not supported. Trying + to guess where one ends and the other begins is too error-prone, + and that also makes for a confusing test.} +Since tracebacks contain details that are likely to change rapidly (for +example, exact file paths and line numbers), this is one case where doctest +works hard to be flexible in what it accepts. Simple example: Modified: python/trunk/Misc/NEWS ============================================================================== --- python/trunk/Misc/NEWS (original) +++ python/trunk/Misc/NEWS Mon Apr 24 04:03:16 2006 @@ -171,6 +171,8 @@ Documentation ------------- +- Bug #1337990: clarified that ``doctest`` does not support examples + requiring both expected output and an exception. What's New in Python 2.5 alpha 1? From python-checkins at python.org Mon Apr 24 04:07:14 2006 From: python-checkins at python.org (tim.peters) Date: Mon, 24 Apr 2006 04:07:14 +0200 (CEST) Subject: [Python-checkins] r45678 - in python/branches/release24-maint: Doc/lib/libdoctest.tex Misc/NEWS Message-ID: <20060424020714.771761E4002@bag.python.org> Author: tim.peters Date: Mon Apr 24 04:07:13 2006 New Revision: 45678 Modified: python/branches/release24-maint/Doc/lib/libdoctest.tex python/branches/release24-maint/Misc/NEWS Log: Merge rev 45677 from the trunk. Bug #1337990: clarified that ``doctest`` does not support examples requiring both expected output and an exception. Modified: python/branches/release24-maint/Doc/lib/libdoctest.tex ============================================================================== --- python/branches/release24-maint/Doc/lib/libdoctest.tex (original) +++ python/branches/release24-maint/Doc/lib/libdoctest.tex Mon Apr 24 04:07:13 2006 @@ -407,10 +407,13 @@ \subsubsection{What About Exceptions?\label{doctest-exceptions}} No problem, provided that the traceback is the only output produced by -the example: just paste in the traceback. Since tracebacks contain -details that are likely to change rapidly (for example, exact file paths -and line numbers), this is one case where doctest works hard to be -flexible in what it accepts. +the example: just paste in the traceback.\footnote{Examples containing + both expected output and an exception are not supported. Trying + to guess where one ends and the other begins is too error-prone, + and that also makes for a confusing test.} +Since tracebacks contain details that are likely to change rapidly (for +example, exact file paths and line numbers), this is one case where doctest +works hard to be flexible in what it accepts. Simple example: Modified: python/branches/release24-maint/Misc/NEWS ============================================================================== --- python/branches/release24-maint/Misc/NEWS (original) +++ python/branches/release24-maint/Misc/NEWS Mon Apr 24 04:07:13 2006 @@ -54,6 +54,12 @@ - Fix test_long failure on Tru64 with gcc by using -mieee gcc option. +Documentation +------------- + +- Bug #1337990: clarified that ``doctest`` does not support examples + requiring both expected output and an exception. + What's New in Python 2.4.3? =========================== From python-checkins at python.org Mon Apr 24 05:04:49 2006 From: python-checkins at python.org (nick.coghlan) Date: Mon, 24 Apr 2006 05:04:49 +0200 (CEST) Subject: [Python-checkins] r45679 - python/trunk/Misc/NEWS Message-ID: <20060424030449.6A6C31E4002@bag.python.org> Author: nick.coghlan Date: Mon Apr 24 05:04:43 2006 New Revision: 45679 Modified: python/trunk/Misc/NEWS Log: Note changes made to PEP 343 related documentation Modified: python/trunk/Misc/NEWS ============================================================================== --- python/trunk/Misc/NEWS (original) +++ python/trunk/Misc/NEWS Mon Apr 24 05:04:43 2006 @@ -171,6 +171,10 @@ Documentation ------------- +- PEP 343: Added Context Types section to the library reference + and attempted to bring other PEP 343 related documentation into + line with the implementation and/or python-dev discussions. + - Bug #1337990: clarified that ``doctest`` does not support examples requiring both expected output and an exception. @@ -415,7 +419,7 @@ subclasses of str always behaved. int/long/float, conversion of an instance to the base class has been moved to the proper nb_* magic slot and out of PyNumber_*(). - Thanks Walter Dörwald. + Thanks Walter D�wald. - Descriptors defined in C with a PyGetSetDef structure, where the setter is NULL, now raise an AttributeError when attempting to set or delete the From python-checkins at python.org Mon Apr 24 05:22:31 2006 From: python-checkins at python.org (nick.coghlan) Date: Mon, 24 Apr 2006 05:22:31 +0200 (CEST) Subject: [Python-checkins] r45680 - peps/trunk/pep-0343.txt Message-ID: <20060424032231.DB3EC1E4002@bag.python.org> Author: nick.coghlan Date: Mon Apr 24 05:22:30 2006 New Revision: 45680 Modified: peps/trunk/pep-0343.txt Log: Change proposed terminology from context object to context specifier Modified: peps/trunk/pep-0343.txt ============================================================================== --- peps/trunk/pep-0343.txt (original) +++ peps/trunk/pep-0343.txt Mon Apr 24 05:22:30 2006 @@ -29,6 +29,9 @@ Python's alpha release cycle revealed terminology problems in this PEP and in the associated documentation and implementation [14]. + So while the PEP is already accepted, this refers to the + implementation rather than the exact terminology. + The current version of the PEP reflects the implementation and documentation as at Python 2.5a2. The PEP will be updated to reflect any changes made to the terminology prior to the final @@ -269,12 +272,12 @@ The call to the __context__() method serves a similar purpose to that of the __iter__() method of iterator and iterables. A context - object with simple state requirements (such as + specifier with simple state requirements (such as threading.RLock) may provide its own __enter__() and __exit__() methods, and simply return 'self' from its __context__ method. On - the other hand, a context object with more complex state requirements - (such as decimal.Context) may return a distinct context manager - each time its __context__ method is invoked. + the other hand, a context specifier with more complex state + requirements (such as decimal.Context) may return a distinct + context manager each time its __context__ method is invoked. If the "as VAR" part of the syntax is omitted, the "VAR =" part of the translation is omitted (but mgr.__enter__() is still called). @@ -321,9 +324,10 @@ of a database transaction roll-back decision. To facilitate chaining of contexts in Python code that directly - manipulates context objects, __exit__() methods should *not* - re-raise the error that is passed in to them, because it is always - the responsibility of the *caller* to do any reraising in that case. + manipulates context specifiers and managers, __exit__() methods + should *not* re-raise the error that is passed in to them, because + it is always the responsibility of the *caller* to do any reraising + in that case. That way, if the caller needs to tell whether the __exit__() invocation *failed* (as opposed to successfully cleaning up before @@ -349,6 +353,9 @@ with mgr as VAR: BLOCK + The with statement implementation and examples like the nested() + function require this behaviour in order to be able to deal + transparently with both context specifiers and context managers. Transition Plan @@ -431,7 +438,7 @@ Just as generator-iterator functions are very useful for writing __iter__() methods for iterables, generator context functions will be very useful for writing __context__() methods for context - objects. These methods will still need to be decorated using the + specifiers. These methods will still need to be decorated using the contextmanager decorator. To ensure an obvious error message if the decorator is left out, generator-iterator objects will NOT be given a native context - if you want to ensure a generator is closed @@ -486,8 +493,7 @@ statement is called the iterator protocol and an iterator is any object that properly implements that protocol. The term "iterable" then encompasses all objects with an __iter__() method that - returns an iterator (this means that all iterators are iterables, - but not all iterables are iterators). + returns an iterator. This PEP proposes that the protocol consisting of the __enter__() and __exit__() methods, and a __context__() method that returns @@ -495,17 +501,18 @@ objects that implement that protocol be known as "context managers". - The term "context object" then encompasses all objects with a + The term "context specifier" then encompasses all objects with a __context__() method that returns a context manager. The protocol - these objects implement is called the "context protocol". This - means that all context managers are context objects, but not all - context objects are context managers, just as all iterators are - iterables, but not all iterables are iterators. + these objects implement is called the "context specification + protocol". This means that all context managers are context + specifiers, but not all context specifiers are context managers, + just as all iterators are iterables, but not all iterables are + iterators. - These terms are based on the concept that the context object + These terms are based on the concept that the context specifier defines a context of execution for the code that forms the body of the with statement. The role of the context manager is to - translate the context object's stored state into an active + translate the context specifier's stored state into an active manipulation of the runtime environment to setup and tear down the desired runtime context for the duration of the with statement. For example, a synchronisation lock's context manager acquires the @@ -514,21 +521,36 @@ with statement is that the synchronisation lock is currently held. The general term "context" is unfortunately ambiguous. If necessary, - it can be made more explicit by using the terms "context objext" for - objects providing a __context__() method and "runtime context" for - the runtime environment modifications made by the context manager. - When solely discussing use of the with statement, the distinction - between the two shouldn't matter as the context object fully - defines the changes made to the runtime context. The distinction is - more important when discussing the process of implementing context - objects and context managers. + it can be made more explicit by using the terms "context specifier" + for objects providing a __context__() method and "runtime context" + for the runtime environment modifications made by the context + manager. When solely discussing use of the with statement, the + distinction between the two shouldn't matter as the context + specifier fully defines the changes made to the runtime context. + The distinction is more important when discussing the process of + implementing context specifiers and context managers. Open Issues - 1. As noted earlier, the standard terminology section has not yet - met with consensus on python-dev. It will be refined throughout - the Python 2.5 release cycle based on user feedback on the - usability of the documentation. + 1. After this PEP was originally approved, a subsequent discussion + on python-dev [4] settled on the term "context manager" for + objects which provide __enter__ and __exit__ methods, and + "context management protocol" for the protocol itself. With the + addition of the __context__ method to the protocol, the natural + adjustment is to call all objects which provide a __context__ + method "context managers", and the objects with __enter__ and + __exit__ methods "contexts" (or "manageable contexts" in + situations where the general term "context" would be ambiguous). + + As noted above, the Python 2.5 release cycle revealed problems + with the previously agreed terminology. The updated standard + terminology section has not yet met with consensus on + python-dev. It will be refined throughout the Python 2.5 release + cycle based on user feedback on the usability of the + documentation. + The first change made as a result of the current discussion is + replacement of the term "context object" with + "context specifier". 2. The original resolution was for the decorator to make a context manager from a generator to be a builtin called "contextmanager". @@ -595,16 +617,8 @@ and for uninitialized objects (and for a variety of miscellaneous conditions)." - 3. After this PEP was originally approved, a subsequent discussion - on python-dev [4] settled on the term "context manager" for - objects which provide __enter__ and __exit__ methods, and - "context management protocol" for the protocol itself. With the - addition of the __context__ method to the protocol, the natural - adjustment is to call all objects which provide a __context__ - method "context managers", and the objects with __enter__ and - __exit__ methods "contexts" (or "manageable contexts" in - situations where the general term "context" would be ambiguous). - This is now documented in the "Standard Terminology" section. + 3. See item 1 in open issues :) + 4. The originally approved version of this PEP did not include a __context__ method - the method was only added to the PEP after @@ -640,7 +654,7 @@ works without having to first understand the mechanics of how generator context managers are implemented. - 6. See point 2 in open issues :) + 6. See item 2 in open issues :) 7. A generator function used to implement a __context__ method will need to be decorated with the contextmanager decorator in order From buildbot at python.org Mon Apr 24 05:29:27 2006 From: buildbot at python.org (buildbot at python.org) Date: Mon, 24 Apr 2006 03:29:27 +0000 Subject: [Python-checkins] buildbot warnings in sparc Ubuntu dapper trunk Message-ID: <20060424032928.5BFB91E4002@bag.python.org> The Buildbot has detected a new failure of sparc Ubuntu dapper trunk. Full details are available at: http://www.python.org/dev/buildbot/all/sparc%2520Ubuntu%2520dapper%2520trunk/builds/146 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: tim.peters Build Had Warnings: warnings test sincerely, -The Buildbot From buildbot at python.org Mon Apr 24 05:43:02 2006 From: buildbot at python.org (buildbot at python.org) Date: Mon, 24 Apr 2006 03:43:02 +0000 Subject: [Python-checkins] buildbot failure in x86 cygwin trunk Message-ID: <20060424034302.530971E4002@bag.python.org> The Buildbot has detected a new failure of x86 cygwin trunk. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520cygwin%2520trunk/builds/269 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: nick.coghlan BUILD FAILED: failed compile sincerely, -The Buildbot From python-checkins at python.org Mon Apr 24 06:17:07 2006 From: python-checkins at python.org (nick.coghlan) Date: Mon, 24 Apr 2006 06:17:07 +0200 (CEST) Subject: [Python-checkins] r45681 - in python/trunk/Doc: lib/libcontextlib.tex lib/libstdtypes.tex ref/ref3.tex ref/ref7.tex Message-ID: <20060424041707.054771E4002@bag.python.org> Author: nick.coghlan Date: Mon Apr 24 06:17:02 2006 New Revision: 45681 Modified: python/trunk/Doc/lib/libcontextlib.tex python/trunk/Doc/lib/libstdtypes.tex python/trunk/Doc/ref/ref3.tex python/trunk/Doc/ref/ref7.tex Log: Change PEP 343 related documentation to use the term context specifier instead of context object Modified: python/trunk/Doc/lib/libcontextlib.tex ============================================================================== --- python/trunk/Doc/lib/libcontextlib.tex (original) +++ python/trunk/Doc/lib/libcontextlib.tex Mon Apr 24 06:17:02 2006 @@ -46,12 +46,17 @@ block, it is reraised inside the generator at the point where the yield occurred. Thus, you can use a \keyword{try}...\keyword{except}...\keyword{finally} statement to trap -the error (if any), or ensure that some cleanup takes place. +the error (if any), or ensure that some cleanup takes place. If an +exception is trapped merely in order to log it or to perform some +action (rather than to suppress it entirely), the generator must +reraise that exception. Otherwise the \keyword{with} statement will +treat the exception as having been handled, and resume execution with +the statement immediately following the \keyword{with} statement. Note that you can use \code{@contextmanager} to define a context -object's \method{__context__} method. This is usually more convenient -than creating another class just to serve as a context manager. -For example: +specifier's \method{__context__} method. This is usually more +convenient than creating another class just to serve as a context +manager. For example: \begin{verbatim} from __future__ import with_statement @@ -78,7 +83,7 @@ \end{funcdesc} \begin{funcdesc}{nested}{ctx1\optional{, ctx2\optional{, ...}}} -Combine multiple context managers into a single nested context manager. +Combine multiple context specifiers into a single nested context manager. Code like this: @@ -98,11 +103,15 @@ do_something() \end{verbatim} -Note that if the \method{__exit__()} method of one of the nested context managers -raises an exception, any previous exception state will be lost; the new -exception will be passed to the \method{__exit__()} methods of any remaining -outer context managers. In general, \method{__exit__()} methods should avoid -raising exceptions, and in particular they should not re-raise a +Note that if the \method{__exit__()} method of one of the nested +context managers indicates an exception should be suppressed, no +exception information will be passed to any remaining outer context +managers. Similarly, if the \method{__exit__()} method of one of the +nested context managers raises an exception, any previous exception +state will be lost; the new exception will be passed to the +\method{__exit__()} methods of any remaining outer context managers. +In general, \method{__exit__()} methods should avoid raising +exceptions, and in particular they should not re-raise a passed-in exception. \end{funcdesc} Modified: python/trunk/Doc/lib/libstdtypes.tex ============================================================================== --- python/trunk/Doc/lib/libstdtypes.tex (original) +++ python/trunk/Doc/lib/libstdtypes.tex Mon Apr 24 06:17:02 2006 @@ -1756,20 +1756,21 @@ \subsection{Context Types \label{typecontext}} \versionadded{2.5} -\index{context protocol} +\index{context specification protocol} \index{context management protocol} -\index{protocol!context} +\index{protocol!context specification} \index{protocol!context management} Python's \keyword{with} statement supports the concept of a runtime -context defined by a context object. This is implemented using three -distinct methods; these are used to allow user-defined classes to -define a context. +context defined by a context specifier. This is implemented using +three distinct methods; these are used to allow user-defined +classes to define a context. + +The \dfn{context specification protocol} consists of a single +method that needs to be provided for a context specifier object to +define a runtime context: -The \dfn{context protocol} consists of a single method that needs -to be provided for an object to define a runtime context: - -\begin{methoddesc}[context]{__context__}{} +\begin{methoddesc}[context specifier]{__context__}{} Return a context manager object. The object is required to support the context management protocol described below. If an object supports different kinds of runtime context, additional methods can @@ -1787,27 +1788,29 @@ \begin{methoddesc}[context manager]{__context__}{} Return the context manager object itself. This is required to - allow both contexts and context managers to be used with the - \keyword{with} statement. + allow both context specifiers and context managers to be used with + the \keyword{with} statement. \end{methoddesc} \begin{methoddesc}[context manager]{__enter__}{} - Set up the runtime context and return either the defining context - object or another object related to the runtime context. The value + Enter the runtime context and return either the defining context + specifier or another object related to the runtime context. The value returned by this method is bound to the identifier in the \keyword{as} clause of \keyword{with} statements using this context. (An example of a context with a context manager that returns the - original context object is file objects, which are returned from + original context specifier is file objects, which are returned from __enter__() to allow \function{open()} to be used directly in a with statement. An example of a context manager that returns a related - object is \code{decimal.Context} which returns a copy of the original - context to allow changes to be made to the current decimal context - without affecting code outside the \keyword{with} statement). + object is \code{decimal.Context} which sets the active decimal + context to a copy of the context specifier and then returns the copy + to allow changes to be made to the current decimal context in the + body of the \keyword{with} statement) without affecting code outside + the \keyword{with} statement). \end{methoddesc} \begin{methoddesc}[context manager]{__exit__}{exc_type, exc_val, exc_tb} - Tear down the runtime context and return a Boolean flag indicating if - an expection that occurred should be suppressed. If an exception + Exit the runtime context and return a Boolean flag indicating if any + expection that occurred should be suppressed. If an exception occurred while executing the body of the \keyword{with} statement, the arguments contain the exception type, value and traceback information. Otherwise, all three arguments are \var{None}. @@ -1826,20 +1829,21 @@ \method{__exit__()} method has actually failed. \end{methoddesc} -Python defines several context objects to support easy thread -synchronisation, prompt closure of files or other objects, and -thread-safe manipulation of the decimal arithmetic context. The -specific types are not important beyond their implementation of -the context protocol. +Python defines several context specifiers and managers to support +easy thread synchronisation, prompt closure of files or other +objects, and thread-safe manipulation of the decimal arithmetic +context. The specific types are not important beyond their +implementation of the context specification and context +management protocols. Python's generators and the \code{contextlib.contextmanager} decorator provide a convenient way to implement the context -and context management protocols. If a context object's -\method{__context__()} method is implemented as a generator decorated -with the \code{contextlib.contextmanager} decorator, it will -automatically return a context manager object supplying the -necessary \method{__context__()}, \method{__enter__()} and -\method{__exit__()} methods. +specification and context management protocols. If a context +specifier's \method{__context__()} method is implemented as a +generator decorated with the \code{contextlib.contextmanager} +decorator, it will automatically return a context manager +object supplying the necessary \method{__context__()}, +\method{__enter__()} and \method{__exit__()} methods. Note that there is no specific slot for any of these methods in the type structure for Python objects in the Python/C API. Extension Modified: python/trunk/Doc/ref/ref3.tex ============================================================================== --- python/trunk/Doc/ref/ref3.tex (original) +++ python/trunk/Doc/ref/ref3.tex Mon Apr 24 06:17:02 2006 @@ -2112,47 +2112,45 @@ \end{itemize} -\subsection{Context Managers and Contexts\label{context-managers}} +\subsection{Context Specifiers and Managers\label{context-managers}} \versionadded{2.5} -A \dfn{context object} is an object that defines the runtime context -to be established when executing a \keyword{with} statement. The -context object provides a \dfn{context manager} which manages the -entry into, and the exit from, the desired runtime context for the -execution of the block of code. Context managers are normally -invoked using the \keyword{with} statement (described in -section~\ref{with}), but can also be used by directly invoking -their methods. +A \dfn{context specifier} is an object that defines the runtime +context to be established when executing a \keyword{with} +statement. The context specifier provides a \dfn{context manager} +which manages the entry into, and the exit from, the desired +runtime context for the execution of the block of code. Context +managers are normally invoked using the \keyword{with} statement +(described in section~\ref{with}), but can also be used by +directly invoking their methods. \stindex{with} \index{context manager} -\index{context object} +\index{context specifier} -Typical uses of contexts and context managers include saving and +Typical uses of context specifiers and managers include saving and restoring various kinds of global state, locking and unlocking resources, closing opened files, etc. -For more information on contexts and context manager objects, see -``\ulink{Context Types}{../lib/typecontext.html}'' in the +For more information on context specifiers and context manager objects, +see ``\ulink{Context Types}{../lib/typecontext.html}'' in the \citetitle[../lib/lib.html]{Python Library Reference}. -\begin{methoddesc}[context]{__context__}{self} +\begin{methoddesc}[context specifier]{__context__}{self} Invoked when the object is used as the context expression of a -\keyword{with} statement. The return value must implement -\method{__enter__()} and \method{__exit__()} methods. Simple contexts -may be able to implement \method{__enter__()} and \method{__exit__()} -directly without requiring a separate context manager object and -should just return \var{self}. +\keyword{with} statement. The returned object must implement +\method{__enter__()} and \method{__exit__()} methods. -Context objects written in Python can also implement this method using -a generator function decorated with the +Context specifiers written in Python can also implement this method +using a generator function decorated with the \function{contextlib.contextmanager} decorator, as this can be simpler than writing individual \method{__enter__()} and \method{__exit__()} methods on a separate object when the state to be managed is complex. Context manager objects also need to implement this method; they are -required to return themselves. +required to return themselves (that is, this method will simply +return \var{self}). \end{methoddesc} \begin{methoddesc}[context manager]{__enter__}{self} Modified: python/trunk/Doc/ref/ref7.tex ============================================================================== --- python/trunk/Doc/ref/ref7.tex (original) +++ python/trunk/Doc/ref/ref7.tex Mon Apr 24 06:17:02 2006 @@ -315,10 +315,10 @@ \versionadded{2.5} The \keyword{with} statement is used to wrap the execution of a block -with methods defined by a context manager (see +with methods defined by a context specifier or manager (see section~\ref{context-managers}). This allows common \keyword{try}...\keyword{except}...\keyword{finally} usage patterns to -be encapsulated as context managers for convenient reuse. +be encapsulated as context specifiers or managers for convenient reuse. \begin{productionlist} \production{with_stmt} @@ -329,10 +329,10 @@ \begin{enumerate} -\item The expression is evaluated, to obtain a context object. +\item The expression is evaluated, to obtain a context specifier. -\item The context object's \method{__context__()} method is invoked to -obtain a context manager object. +\item The context specifier's \method{__context__()} method is +invoked to obtain a context manager object. \item The context manager's \method{__enter__()} method is invoked. From python-checkins at python.org Mon Apr 24 06:32:49 2006 From: python-checkins at python.org (nick.coghlan) Date: Mon, 24 Apr 2006 06:32:49 +0200 (CEST) Subject: [Python-checkins] r45682 - in python/trunk: Lib/test/test_cmd_line.py Misc/NEWS Message-ID: <20060424043249.8BFAB1E4002@bag.python.org> Author: nick.coghlan Date: Mon Apr 24 06:32:47 2006 New Revision: 45682 Modified: python/trunk/Lib/test/test_cmd_line.py python/trunk/Misc/NEWS Log: Add unit tests for the -m and -c command line switches Modified: python/trunk/Lib/test/test_cmd_line.py ============================================================================== --- python/trunk/Lib/test/test_cmd_line.py (original) +++ python/trunk/Lib/test/test_cmd_line.py Mon Apr 24 06:32:47 2006 @@ -18,6 +18,11 @@ def exit_code(self, cmd_line): return subprocess.call([sys.executable, cmd_line], stderr=subprocess.PIPE) + def popen_python(self, *args): + cmd_line = [sys.executable] + cmd_line.extend(args) + return subprocess.Popen(cmd_line, stdout=subprocess.PIPE, stderr=subprocess.PIPE) + def test_directories(self): self.assertNotEqual(self.exit_code('.'), 0) self.assertNotEqual(self.exit_code('< .'), 0) @@ -50,6 +55,56 @@ version = 'Python %d.%d' % sys.version_info[:2] self.assertTrue(self.start_python('-V').startswith(version)) + def test_run_module(self): + # Test expected operation of the '-m' switch + # Switch needs an argument + result = self.popen_python('-m') + exit_code = result.wait() + self.assertNotEqual(exit_code, 0) + err_details = result.stderr.read() + self.assertTrue(err_details.startswith('Argument expected')) + # Check we get an import error for a nonexistent module + result = self.popen_python('-m', 'fnord43520xyz') + exit_code = result.wait() + self.assertNotEqual(exit_code, 0) + err_details = result.stderr.read() + self.assertTrue('ImportError' in err_details) + # Traceback shown if the requested module is located for execution + # and subsequently fails (even if that module is runpy) + result = self.popen_python('-m', 'runpy', 'fnord') + exit_code = result.wait() + self.assertNotEqual(exit_code, 0) + err_details = result.stderr.read() + self.assertTrue(err_details.startswith('Traceback')) + # Silence if module is located and run successfully + result = self.popen_python('-m', 'timeit', '-n', '1') + exit_code = result.wait() + self.assertEqual(exit_code, 0) + err_details = result.stderr.read() + self.assertTrue(err_details in ('', '\n')) + + def test_run_code(self): + # Test expected operation of the '-c' switch + # Switch needs an argument + result = self.popen_python('-c') + exit_code = result.wait() + self.assertNotEqual(exit_code, 0) + err_details = result.stderr.read() + self.assertTrue(err_details.startswith('Argument expected')) + # Traceback shown for uncaught exceptions + result = self.popen_python('-c', 'raise Exception') + exit_code = result.wait() + self.assertNotEqual(exit_code, 0) + err_details = result.stderr.read() + self.assertTrue(err_details.startswith('Traceback')) + # Silence if execution is successful + result = self.popen_python('-c', '""') + exit_code = result.wait() + self.assertEqual(exit_code, 0) + err_details = result.stderr.read() + self.assertTrue(err_details in ('', '\n')) + + def test_main(): test.test_support.run_unittest(CmdLineTest) Modified: python/trunk/Misc/NEWS ============================================================================== --- python/trunk/Misc/NEWS (original) +++ python/trunk/Misc/NEWS Mon Apr 24 06:32:47 2006 @@ -158,6 +158,8 @@ Tests ----- +- test_cmd_line now checks operation of the -m and -c command switches + - The test_contextlib test in 2.5a1 wasn't actually run unless you ran it separately and by hand. It also wasn't cleaning up its changes to the current Decimal context. From python-checkins at python.org Mon Apr 24 06:37:16 2006 From: python-checkins at python.org (nick.coghlan) Date: Mon, 24 Apr 2006 06:37:16 +0200 (CEST) Subject: [Python-checkins] r45683 - in python/trunk: Lib/contextlib.py Lib/test/test_contextlib.py Misc/NEWS Message-ID: <20060424043716.1463F1E4024@bag.python.org> Author: nick.coghlan Date: Mon Apr 24 06:37:15 2006 New Revision: 45683 Modified: python/trunk/Lib/contextlib.py python/trunk/Lib/test/test_contextlib.py python/trunk/Misc/NEWS Log: Fix contextlib.nested to cope with exit methods raising and handling exceptions Modified: python/trunk/Lib/contextlib.py ============================================================================== --- python/trunk/Lib/contextlib.py (original) +++ python/trunk/Lib/contextlib.py Mon Apr 24 06:37:15 2006 @@ -127,7 +127,10 @@ except: exc = sys.exc_info() if exc != (None, None, None): - raise + # Don't rely on sys.exc_info() still containing + # the right information. Another exception may + # have been raised and caught by an exit method + raise exc[0], exc[1], exc[2] @contextmanager Modified: python/trunk/Lib/test/test_contextlib.py ============================================================================== --- python/trunk/Lib/test/test_contextlib.py (original) +++ python/trunk/Lib/test/test_contextlib.py Mon Apr 24 06:37:15 2006 @@ -146,6 +146,29 @@ else: self.fail("Didn't raise ZeroDivisionError") + def test_nested_right_exception(self): + state = [] + @contextmanager + def a(): + yield 1 + class b(object): + def __enter__(self): + return 2 + def __exit__(self, *exc_info): + try: + raise Exception() + except: + pass + try: + with nested(a(), b()) as (x, y): + 1/0 + except ZeroDivisionError: + self.assertEqual((x, y), (1, 2)) + except Exception: + self.fail("Reraised wrong exception") + else: + self.fail("Didn't raise ZeroDivisionError") + def test_nested_b_swallows(self): @contextmanager def a(): Modified: python/trunk/Misc/NEWS ============================================================================== --- python/trunk/Misc/NEWS (original) +++ python/trunk/Misc/NEWS Mon Apr 24 06:37:15 2006 @@ -86,6 +86,9 @@ Library ------- +- Fixed contextlib.nested to cope with exceptions being raised and + caught inside exit handlers. + - Updated optparse module to Optik 1.5.1 (allow numeric constants in hex, octal, or binary; add ``append_const`` action; keep going if gettext cannot be imported; added ``OptionParser.destroy()`` method; @@ -158,6 +161,9 @@ Tests ----- +- test_contextlib now checks contextlib.nested can cope with exceptions + being raised and caught inside exit handlers. + - test_cmd_line now checks operation of the -m and -c command switches - The test_contextlib test in 2.5a1 wasn't actually run unless you ran From python-checkins at python.org Mon Apr 24 06:41:45 2006 From: python-checkins at python.org (nick.coghlan) Date: Mon, 24 Apr 2006 06:41:45 +0200 (CEST) Subject: [Python-checkins] r45684 - peps/trunk/pep-0343.txt Message-ID: <20060424044145.F1F171E4002@bag.python.org> Author: nick.coghlan Date: Mon Apr 24 06:41:45 2006 New Revision: 45684 Modified: peps/trunk/pep-0343.txt Log: Correct a bug in the nested() example Modified: peps/trunk/pep-0343.txt ============================================================================== --- peps/trunk/pep-0343.txt (original) +++ peps/trunk/pep-0343.txt Mon Apr 24 06:41:45 2006 @@ -956,7 +956,10 @@ else: exc = (None, None, None) if exc != (None, None, None): - raise + # sys.exc_info() may have been + # changed by one of the exit methods + # so provide explicit exception info + raise exc[0], exc[1], exc[2] Sample usage: From buildbot at python.org Mon Apr 24 06:58:29 2006 From: buildbot at python.org (buildbot at python.org) Date: Mon, 24 Apr 2006 04:58:29 +0000 Subject: [Python-checkins] buildbot warnings in amd64 gentoo trunk Message-ID: <20060424045829.B6B021E4002@bag.python.org> The Buildbot has detected a new failure of amd64 gentoo trunk. Full details are available at: http://www.python.org/dev/buildbot/all/amd64%2520gentoo%2520trunk/builds/624 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: nick.coghlan Build Had Warnings: warnings test sincerely, -The Buildbot From buildbot at python.org Mon Apr 24 06:59:28 2006 From: buildbot at python.org (buildbot at python.org) Date: Mon, 24 Apr 2006 04:59:28 +0000 Subject: [Python-checkins] buildbot warnings in x86 gentoo trunk Message-ID: <20060424045928.564E31E4002@bag.python.org> The Buildbot has detected a new failure of x86 gentoo trunk. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520gentoo%2520trunk/builds/604 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: nick.coghlan Build Had Warnings: warnings test sincerely, -The Buildbot From python-checkins at python.org Mon Apr 24 06:59:29 2006 From: python-checkins at python.org (nick.coghlan) Date: Mon, 24 Apr 2006 06:59:29 +0200 (CEST) Subject: [Python-checkins] r45685 - python/trunk/Lib/test/test_contextlib.py Message-ID: <20060424045929.55D301E4002@bag.python.org> Author: nick.coghlan Date: Mon Apr 24 06:59:28 2006 New Revision: 45685 Modified: python/trunk/Lib/test/test_contextlib.py Log: Fix broken contextlib test from last checkin (I'd've sworn I tested that before checking it in. . .) Modified: python/trunk/Lib/test/test_contextlib.py ============================================================================== --- python/trunk/Lib/test/test_contextlib.py (original) +++ python/trunk/Lib/test/test_contextlib.py Mon Apr 24 06:59:28 2006 @@ -152,6 +152,8 @@ def a(): yield 1 class b(object): + def __context__(self): + return self def __enter__(self): return 2 def __exit__(self, *exc_info): From buildbot at python.org Mon Apr 24 07:17:14 2006 From: buildbot at python.org (buildbot at python.org) Date: Mon, 24 Apr 2006 05:17:14 +0000 Subject: [Python-checkins] buildbot warnings in ppc Debian unstable trunk Message-ID: <20060424051714.3BEAA1E4002@bag.python.org> The Buildbot has detected a new failure of ppc Debian unstable trunk. Full details are available at: http://www.python.org/dev/buildbot/all/ppc%2520Debian%2520unstable%2520trunk/builds/268 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: nick.coghlan Build Had Warnings: warnings test sincerely, -The Buildbot From buildbot at python.org Mon Apr 24 07:22:35 2006 From: buildbot at python.org (buildbot at python.org) Date: Mon, 24 Apr 2006 05:22:35 +0000 Subject: [Python-checkins] buildbot warnings in x86 XP trunk Message-ID: <20060424052235.A20811E4002@bag.python.org> The Buildbot has detected a new failure of x86 XP trunk. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520XP%2520trunk/builds/555 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: nick.coghlan Build Had Warnings: warnings test sincerely, -The Buildbot From python-checkins at python.org Mon Apr 24 07:24:27 2006 From: python-checkins at python.org (nick.coghlan) Date: Mon, 24 Apr 2006 07:24:27 +0200 (CEST) Subject: [Python-checkins] r45686 - python/trunk/Lib/test/test_cmd_line.py Message-ID: <20060424052427.688591E4002@bag.python.org> Author: nick.coghlan Date: Mon Apr 24 07:24:26 2006 New Revision: 45686 Modified: python/trunk/Lib/test/test_cmd_line.py Log: Back out new command line tests (broke buildbot) Modified: python/trunk/Lib/test/test_cmd_line.py ============================================================================== --- python/trunk/Lib/test/test_cmd_line.py (original) +++ python/trunk/Lib/test/test_cmd_line.py Mon Apr 24 07:24:26 2006 @@ -18,11 +18,6 @@ def exit_code(self, cmd_line): return subprocess.call([sys.executable, cmd_line], stderr=subprocess.PIPE) - def popen_python(self, *args): - cmd_line = [sys.executable] - cmd_line.extend(args) - return subprocess.Popen(cmd_line, stdout=subprocess.PIPE, stderr=subprocess.PIPE) - def test_directories(self): self.assertNotEqual(self.exit_code('.'), 0) self.assertNotEqual(self.exit_code('< .'), 0) @@ -55,56 +50,6 @@ version = 'Python %d.%d' % sys.version_info[:2] self.assertTrue(self.start_python('-V').startswith(version)) - def test_run_module(self): - # Test expected operation of the '-m' switch - # Switch needs an argument - result = self.popen_python('-m') - exit_code = result.wait() - self.assertNotEqual(exit_code, 0) - err_details = result.stderr.read() - self.assertTrue(err_details.startswith('Argument expected')) - # Check we get an import error for a nonexistent module - result = self.popen_python('-m', 'fnord43520xyz') - exit_code = result.wait() - self.assertNotEqual(exit_code, 0) - err_details = result.stderr.read() - self.assertTrue('ImportError' in err_details) - # Traceback shown if the requested module is located for execution - # and subsequently fails (even if that module is runpy) - result = self.popen_python('-m', 'runpy', 'fnord') - exit_code = result.wait() - self.assertNotEqual(exit_code, 0) - err_details = result.stderr.read() - self.assertTrue(err_details.startswith('Traceback')) - # Silence if module is located and run successfully - result = self.popen_python('-m', 'timeit', '-n', '1') - exit_code = result.wait() - self.assertEqual(exit_code, 0) - err_details = result.stderr.read() - self.assertTrue(err_details in ('', '\n')) - - def test_run_code(self): - # Test expected operation of the '-c' switch - # Switch needs an argument - result = self.popen_python('-c') - exit_code = result.wait() - self.assertNotEqual(exit_code, 0) - err_details = result.stderr.read() - self.assertTrue(err_details.startswith('Argument expected')) - # Traceback shown for uncaught exceptions - result = self.popen_python('-c', 'raise Exception') - exit_code = result.wait() - self.assertNotEqual(exit_code, 0) - err_details = result.stderr.read() - self.assertTrue(err_details.startswith('Traceback')) - # Silence if execution is successful - result = self.popen_python('-c', '""') - exit_code = result.wait() - self.assertEqual(exit_code, 0) - err_details = result.stderr.read() - self.assertTrue(err_details in ('', '\n')) - - def test_main(): test.test_support.run_unittest(CmdLineTest) From buildbot at python.org Mon Apr 24 07:27:30 2006 From: buildbot at python.org (buildbot at python.org) Date: Mon, 24 Apr 2006 05:27:30 +0000 Subject: [Python-checkins] buildbot warnings in g4 osx.4 trunk Message-ID: <20060424052730.BEC431E4002@bag.python.org> The Buildbot has detected a new failure of g4 osx.4 trunk. Full details are available at: http://www.python.org/dev/buildbot/all/g4%2520osx.4%2520trunk/builds/532 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: nick.coghlan Build Had Warnings: warnings test sincerely, -The Buildbot From buildbot at python.org Mon Apr 24 07:33:26 2006 From: buildbot at python.org (buildbot at python.org) Date: Mon, 24 Apr 2006 05:33:26 +0000 Subject: [Python-checkins] buildbot warnings in sparc solaris10 gcc trunk Message-ID: <20060424053326.6F4D31E400D@bag.python.org> The Buildbot has detected a new failure of sparc solaris10 gcc trunk. Full details are available at: http://www.python.org/dev/buildbot/all/sparc%2520solaris10%2520gcc%2520trunk/builds/538 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: nick.coghlan Build Had Warnings: warnings test sincerely, -The Buildbot From buildbot at python.org Mon Apr 24 07:41:31 2006 From: buildbot at python.org (buildbot at python.org) Date: Mon, 24 Apr 2006 05:41:31 +0000 Subject: [Python-checkins] buildbot warnings in ia64 Debian unstable trunk Message-ID: <20060424054131.C7D171E4002@bag.python.org> The Buildbot has detected a new failure of ia64 Debian unstable trunk. Full details are available at: http://www.python.org/dev/buildbot/all/ia64%2520Debian%2520unstable%2520trunk/builds/258 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: nick.coghlan Build Had Warnings: warnings test sincerely, -The Buildbot From python-checkins at python.org Mon Apr 24 07:52:15 2006 From: python-checkins at python.org (nick.coghlan) Date: Mon, 24 Apr 2006 07:52:15 +0200 (CEST) Subject: [Python-checkins] r45687 - python/trunk/Lib/test/test_cmd_line.py Message-ID: <20060424055215.D51401E4002@bag.python.org> Author: nick.coghlan Date: Mon Apr 24 07:52:15 2006 New Revision: 45687 Modified: python/trunk/Lib/test/test_cmd_line.py Log: More reliable version of new command line tests that just checks the exit codes Modified: python/trunk/Lib/test/test_cmd_line.py ============================================================================== --- python/trunk/Lib/test/test_cmd_line.py (original) +++ python/trunk/Lib/test/test_cmd_line.py Mon Apr 24 07:52:15 2006 @@ -15,8 +15,11 @@ popen2._cleanup() return data - def exit_code(self, cmd_line): - return subprocess.call([sys.executable, cmd_line], stderr=subprocess.PIPE) + def exit_code(self, *args): + cmd_line = [sys.executable] + cmd_line.extend(args) + return subprocess.call(cmd_line, stdout=subprocess.PIPE, + stderr=subprocess.PIPE) def test_directories(self): self.assertNotEqual(self.exit_code('.'), 0) @@ -50,6 +53,38 @@ version = 'Python %d.%d' % sys.version_info[:2] self.assertTrue(self.start_python('-V').startswith(version)) + def test_run_module(self): + # Test expected operation of the '-m' switch + # Switch needs an argument + self.assertNotEqual(self.exit_code('-m'), 0) + # Check we get an error for a nonexistent module + self.assertNotEqual( + self.exit_code('-m', 'fnord43520xyz'), + 0) + # Check the runpy module also gives an error for + # a nonexistent module + self.assertNotEqual( + self.exit_code('-m', 'runpy', 'fnord43520xyz'), + 0) + # All good if module is located and run successfully + self.assertEqual( + self.exit_code('-m', 'timeit', '-n', '1'), + 0) + + def test_run_code(self): + # Test expected operation of the '-c' switch + # Switch needs an argument + self.assertNotEqual(self.exit_code('-c'), 0) + # Check we get an error for an uncaught exception + self.assertNotEqual( + self.exit_code('-c', 'raise Exception'), + 0) + # All good if execution is successful + self.assertEqual( + self.exit_code('-c', 'pass'), + 0) + + def test_main(): test.test_support.run_unittest(CmdLineTest) From buildbot at python.org Mon Apr 24 08:06:34 2006 From: buildbot at python.org (buildbot at python.org) Date: Mon, 24 Apr 2006 06:06:34 +0000 Subject: [Python-checkins] buildbot warnings in x86 Ubuntu dapper (icc) trunk Message-ID: <20060424060634.295501E4002@bag.python.org> The Buildbot has detected a new failure of x86 Ubuntu dapper (icc) trunk. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520Ubuntu%2520dapper%2520%2528icc%2529%2520trunk/builds/237 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: nick.coghlan Build Had Warnings: warnings test sincerely, -The Buildbot From buildbot at python.org Mon Apr 24 09:03:01 2006 From: buildbot at python.org (buildbot at python.org) Date: Mon, 24 Apr 2006 07:03:01 +0000 Subject: [Python-checkins] buildbot warnings in hppa Ubuntu dapper trunk Message-ID: <20060424070301.752C11E4002@bag.python.org> The Buildbot has detected a new failure of hppa Ubuntu dapper trunk. Full details are available at: http://www.python.org/dev/buildbot/all/hppa%2520Ubuntu%2520dapper%2520trunk/builds/277 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: nick.coghlan Build Had Warnings: warnings test sincerely, -The Buildbot From buildbot at python.org Mon Apr 24 09:23:01 2006 From: buildbot at python.org (buildbot at python.org) Date: Mon, 24 Apr 2006 07:23:01 +0000 Subject: [Python-checkins] buildbot warnings in alpha Debian trunk Message-ID: <20060424072301.20B911E400C@bag.python.org> The Buildbot has detected a new failure of alpha Debian trunk. Full details are available at: http://www.python.org/dev/buildbot/all/alpha%2520Debian%2520trunk/builds/3 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: nick.coghlan Build Had Warnings: warnings test sincerely, -The Buildbot From ncoghlan at gmail.com Mon Apr 24 10:05:21 2006 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 24 Apr 2006 18:05:21 +1000 Subject: [Python-checkins] buildbot warnings in alpha Debian trunk In-Reply-To: <20060424072301.20B911E400C@bag.python.org> References: <20060424072301.20B911E400C@bag.python.org> Message-ID: <444C86C1.1040806@gmail.com> buildbot at python.org wrote: > The Buildbot has detected a new failure of alpha Debian trunk. > Full details are available at: > http://www.python.org/dev/buildbot/all/alpha%2520Debian%2520trunk/builds/3 > > Buildbot URL: http://www.python.org/dev/buildbot/all/ > > Build Reason: > Build Source Stamp: [branch trunk] HEAD > Blamelist: nick.coghlan Tweren't me, yer Honour! This particular failure was due to test_compiler tripping over the 30 minute time limit without producing any output. The sparc Ubuntu dapper run failed for the same reason. Wasn't someone saying something about this problem recently? Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From buildbot at python.org Mon Apr 24 10:35:19 2006 From: buildbot at python.org (buildbot at python.org) Date: Mon, 24 Apr 2006 08:35:19 +0000 Subject: [Python-checkins] buildbot warnings in MIPS Debian 2.4 Message-ID: <20060424083519.BE4B21E400B@bag.python.org> The Buildbot has detected a new failure of MIPS Debian 2.4. Full details are available at: http://www.python.org/dev/buildbot/all/MIPS%2520Debian%25202.4/builds/0 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch branches/release24-maint] HEAD Blamelist: tim.peters Build Had Warnings: warnings failed slave lost sincerely, -The Buildbot From python-checkins at python.org Mon Apr 24 13:37:13 2006 From: python-checkins at python.org (thomas.wouters) Date: Mon, 24 Apr 2006 13:37:13 +0200 (CEST) Subject: [Python-checkins] r45688 - python/trunk/Lib/test/test_tcl.py Message-ID: <20060424113713.C88E21E400A@bag.python.org> Author: thomas.wouters Date: Mon Apr 24 13:37:13 2006 New Revision: 45688 Modified: python/trunk/Lib/test/test_tcl.py Log: Stop test_tcl's testLoadTk from leaking the Tk commands 'loadtk' registers. Modified: python/trunk/Lib/test/test_tcl.py ============================================================================== --- python/trunk/Lib/test/test_tcl.py (original) +++ python/trunk/Lib/test/test_tcl.py Mon Apr 24 13:37:13 2006 @@ -124,6 +124,7 @@ self.assertRaises(TclError,tcl.winfo_geometry) tcl.loadtk() self.assertEqual('1x1+0+0', tcl.winfo_geometry()) + tcl.destroy() def testLoadTkFailure(self): import os From buildbot at python.org Mon Apr 24 13:56:06 2006 From: buildbot at python.org (buildbot at python.org) Date: Mon, 24 Apr 2006 11:56:06 +0000 Subject: [Python-checkins] buildbot failure in MIPS Debian trunk Message-ID: <20060424115606.40F531E400A@bag.python.org> The Buildbot has detected a new failure of MIPS Debian trunk. Full details are available at: http://www.python.org/dev/buildbot/all/MIPS%2520Debian%2520trunk/builds/2 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: thomas.wouters BUILD FAILED: failed compile sincerely, -The Buildbot From python-checkins at python.org Mon Apr 24 16:30:47 2006 From: python-checkins at python.org (andrew.kuchling) Date: Mon, 24 Apr 2006 16:30:47 +0200 (CEST) Subject: [Python-checkins] r45690 - python/trunk/Doc/whatsnew/whatsnew25.tex Message-ID: <20060424143047.DB69A1E4018@bag.python.org> Author: andrew.kuchling Date: Mon Apr 24 16:30:47 2006 New Revision: 45690 Modified: python/trunk/Doc/whatsnew/whatsnew25.tex Log: Edits, using the new term 'context specifier' in a few places Modified: python/trunk/Doc/whatsnew/whatsnew25.tex ============================================================================== --- python/trunk/Doc/whatsnew/whatsnew25.tex (original) +++ python/trunk/Doc/whatsnew/whatsnew25.tex Mon Apr 24 16:30:47 2006 @@ -579,13 +579,12 @@ %====================================================================== \section{PEP 343: The 'with' statement\label{pep-343}} -The '\keyword{with}' statement allows a clearer version of code that -uses \code{try...finally} blocks to ensure that clean-up code is -executed. - -In this section, I'll discuss the statement as it will commonly be -used. In the next section, I'll examine the implementation details -and show how to write objects for use with this statement. +The '\keyword{with}' statement clarifies code that previously would +use \code{try...finally} blocks to ensure that clean-up code is +executed. In this section, I'll discuss the statement as it will +commonly be used. In the next section, I'll examine the +implementation details and show how to write objects for use with this +statement. The '\keyword{with}' statement is a new control-flow structure whose basic structure is: @@ -660,21 +659,22 @@ Under the hood, the '\keyword{with}' statement is fairly complicated. Most people will only use '\keyword{with}' in company with existing -objects and don't need to know these details, so you can skip the -following section if you like. Authors of new objects will need to -understand the details of the underlying implementation. +objects and don't need to know these details, so you can skip the rest +of this section if you like. Authors of new objects will need to +understand the details of the underlying implementation and should +keep reading. A high-level explanation of the context management protocol is: \begin{itemize} \item The expression is evaluated and should result in an object -with a \method{__context__()} method. +with a \method{__context__()} method (called a ``context specifier''). -\item This object's \method{__context__()} method is called, and must -return another object that has \method{__enter__()} and -\method{__exit__()}. +\item The context specifier's \method{__context__()} method is called, +and must return another object (called a ``context manager'') that has +\method{__enter__()} and \method{__exit__()} methods. -\item This object's \method{__enter__()} method is called. The value +\item The context manager's \method{__enter__()} method is called. The value returned is assigned to \var{VAR}. If no \code{'as \var{VAR}'} clause is present, the value is simply discarded. @@ -725,14 +725,14 @@ method. Sometimes an object can simply return \code{self}; the \module{threading} module's lock objects do this, for example. For our database example, though, we need to create a new object; I'll -call this class \class{DatabaseContext}. Our \method{__context__()} +call this class \class{DatabaseContextMgr}. Our \method{__context__()} method must therefore look like this: \begin{verbatim} class DatabaseConnection: ... def __context__ (self): - return DatabaseContext(self) + return DatabaseContextMgr(self) # Database interface def cursor (self): @@ -743,12 +743,12 @@ "Rolls back current transaction" \end{verbatim} -Instance of \class{DatabaseContext} need the connection object so that +Instance of \class{DatabaseContextMgr} need the connection object so that the connection object's \method{commit()} or \method{rollback()} methods can be called: \begin{verbatim} -class DatabaseContext: +class DatabaseContextMgr: def __init__ (self, connection): self.connection = connection \end{verbatim} @@ -760,7 +760,7 @@ the cursor to a variable name. \begin{verbatim} -class DatabaseContext: +class DatabaseContextMgr: ... def __enter__ (self): # Code to start a new transaction @@ -772,13 +772,15 @@ where most of the work has to be done. The method has to check if an exception occurred. If there was no exception, the transaction is committed. The transaction is rolled back if there was an exception. -Here the code will just fall off the end of the function, returning -the default value of \code{None}. \code{None} is false, so the exception -will be re-raised automatically. If you wished, you could be more explicit -and add a \keyword{return} at the marked location. + +In the code below, execution will just fall off the end of the +function, returning the default value of \code{None}. \code{None} is +false, so the exception will be re-raised automatically. If you +wished, you could be more explicit and add a \keyword{return} +statement at the marked location. \begin{verbatim} -class DatabaseContext: +class DatabaseContextMgr: ... def __exit__ (self, type, value, tb): if tb is None: @@ -830,8 +832,8 @@ \end{verbatim} You can also use this decorator to write the \method{__context__()} -method for a class without creating a new class to act as the context -manager: +method for a class without having to create a new class representing +the context manager: \begin{verbatim} class DatabaseConnection: @@ -1262,8 +1264,8 @@ module's interface, will continue to be maintained in future versions of Python. (Contributed by Armin Rigo.) -Also, the \module{pstats} module used to analyze the data measured by -the profiler now supports directing the output to any file stream +Also, the \module{pstats} module for analyzing the data measured by +the profiler now supports directing the output to any file object by supplying a \var{stream} argument to the \class{Stats} constructor. (Contributed by Skip Montanaro.) From buildbot at python.org Mon Apr 24 17:38:38 2006 From: buildbot at python.org (buildbot at python.org) Date: Mon, 24 Apr 2006 15:38:38 +0000 Subject: [Python-checkins] buildbot warnings in x86 Ubuntu dapper (icc) trunk Message-ID: <20060424153838.91DC21E400A@bag.python.org> The Buildbot has detected a new failure of x86 Ubuntu dapper (icc) trunk. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520Ubuntu%2520dapper%2520%2528icc%2529%2520trunk/builds/240 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: andrew.kuchling Build Had Warnings: warnings test sincerely, -The Buildbot From python-checkins at python.org Mon Apr 24 18:55:08 2006 From: python-checkins at python.org (kurt.kaiser) Date: Mon, 24 Apr 2006 18:55:08 +0200 (CEST) Subject: [Python-checkins] r45691 - python/branches/IDLE-syntax-branch/Lib/idlelib/CallTipWindow.py python/branches/IDLE-syntax-branch/Lib/idlelib/EditorWindow.py python/branches/IDLE-syntax-branch/Lib/idlelib/IOBinding.py python/branches/IDLE-syntax-branch/Lib/idlelib/NEWS.txt python/branches/IDLE-syntax-branch/Lib/idlelib/ToolTip.py python/branches/IDLE-syntax-branch/Lib/idlelib/config-extensions.def python/branches/IDLE-syntax-branch/Lib/idlelib/idlever.py python/branches/IDLE-syntax-branch/Lib/idlelib/keybindingDialog.py Message-ID: <20060424165508.EDDCA1E400A@bag.python.org> Author: kurt.kaiser Date: Mon Apr 24 18:55:07 2006 New Revision: 45691 Modified: python/branches/IDLE-syntax-branch/Lib/idlelib/CallTipWindow.py python/branches/IDLE-syntax-branch/Lib/idlelib/EditorWindow.py python/branches/IDLE-syntax-branch/Lib/idlelib/IOBinding.py python/branches/IDLE-syntax-branch/Lib/idlelib/NEWS.txt python/branches/IDLE-syntax-branch/Lib/idlelib/ToolTip.py python/branches/IDLE-syntax-branch/Lib/idlelib/config-extensions.def python/branches/IDLE-syntax-branch/Lib/idlelib/idlever.py python/branches/IDLE-syntax-branch/Lib/idlelib/keybindingDialog.py Log: Merge back changes to trunk since this branch was merged to trunk. svn merge -r 41480:45690 svn+ssh://pythondev at svn.python.org/python/trunk/Lib/idlelib M ToolTip.py M keybindingDialog.py M EditorWindow.py M IOBinding.py M CallTipWindow.py M idlever.py M NEWS.txt M config-extensions.def Modified: python/branches/IDLE-syntax-branch/Lib/idlelib/CallTipWindow.py ============================================================================== --- python/branches/IDLE-syntax-branch/Lib/idlelib/CallTipWindow.py (original) +++ python/branches/IDLE-syntax-branch/Lib/idlelib/CallTipWindow.py Mon Apr 24 18:55:07 2006 @@ -6,7 +6,7 @@ """ from Tkinter import * -HIDE_VIRTUAL_EVENT_NAME = "<>" +HIDE_VIRTUAL_EVENT_NAME = "<>" HIDE_SEQUENCES = ("", "") CHECKHIDE_VIRTUAL_EVENT_NAME = "<>" CHECKHIDE_SEQUENCES = ("", "") @@ -60,10 +60,7 @@ self.tipwindow = tw = Toplevel(self.widget) self.position_window() - # XXX 12 Dec 2002 KBK The following command has two effects: It removes - # the calltip window border (good) but also causes (at least on - # Linux) the calltip to show as a top level window, burning through - # any other window dragged over it. Also, shows on all viewports! + # remove border on calltip window tw.wm_overrideredirect(1) try: # This command is only needed and available on Tk >= 8.4.0 for OSX Modified: python/branches/IDLE-syntax-branch/Lib/idlelib/EditorWindow.py ============================================================================== --- python/branches/IDLE-syntax-branch/Lib/idlelib/EditorWindow.py (original) +++ python/branches/IDLE-syntax-branch/Lib/idlelib/EditorWindow.py Mon Apr 24 18:55:07 2006 @@ -42,7 +42,7 @@ from Percolator import Percolator from ColorDelegator import ColorDelegator from UndoDelegator import UndoDelegator - from IOBinding import IOBinding + from IOBinding import IOBinding, filesystemencoding, encoding import Bindings from Tkinter import Toplevel from MultiStatusBar import MultiStatusBar @@ -256,6 +256,21 @@ self.askinteger = tkSimpleDialog.askinteger self.showerror = tkMessageBox.showerror + def _filename_to_unicode(self, filename): + """convert filename to unicode in order to display it in Tk""" + if isinstance(filename, unicode) or not filename: + return filename + else: + try: + return filename.decode(self.filesystemencoding) + except UnicodeDecodeError: + # XXX + try: + return filename.decode(self.encoding) + except UnicodeDecodeError: + # byte-to-byte conversion + return filename.decode('iso8859-1') + def new_callback(self, event): dirname, basename = self.io.defaultfilename() self.flist.new(dirname) @@ -566,47 +581,46 @@ def RemoveKeybindings(self): "Remove the keybindings before they are changed." # Called from configDialog.py - self.Bindings.default_keydefs=idleConf.GetCurrentKeySet() - keydefs = self.Bindings.default_keydefs + self.Bindings.default_keydefs = keydefs = idleConf.GetCurrentKeySet() for event, keylist in keydefs.items(): self.text.event_delete(event, *keylist) for extensionName in self.get_standard_extension_names(): - keydefs = idleConf.GetExtensionBindings(extensionName) - if keydefs: - for event, keylist in keydefs.items(): + xkeydefs = idleConf.GetExtensionBindings(extensionName) + if xkeydefs: + for event, keylist in xkeydefs.items(): self.text.event_delete(event, *keylist) def ApplyKeybindings(self): "Update the keybindings after they are changed" # Called from configDialog.py - self.Bindings.default_keydefs=idleConf.GetCurrentKeySet() + self.Bindings.default_keydefs = keydefs = idleConf.GetCurrentKeySet() self.apply_bindings() for extensionName in self.get_standard_extension_names(): - keydefs = idleConf.GetExtensionBindings(extensionName) - if keydefs: - self.apply_bindings(keydefs) + xkeydefs = idleConf.GetExtensionBindings(extensionName) + if xkeydefs: + self.apply_bindings(xkeydefs) #update menu accelerators - menuEventDict={} + menuEventDict = {} for menu in self.Bindings.menudefs: - menuEventDict[menu[0]]={} + menuEventDict[menu[0]] = {} for item in menu[1]: if item: - menuEventDict[menu[0]][prepstr(item[0])[1]]=item[1] + menuEventDict[menu[0]][prepstr(item[0])[1]] = item[1] for menubarItem in self.menudict.keys(): - menu=self.menudict[menubarItem] - end=menu.index(END)+1 - for index in range(0,end): - if menu.type(index)=='command': - accel=menu.entrycget(index,'accelerator') + menu = self.menudict[menubarItem] + end = menu.index(END) + 1 + for index in range(0, end): + if menu.type(index) == 'command': + accel = menu.entrycget(index, 'accelerator') if accel: - itemName=menu.entrycget(index,'label') - event='' + itemName = menu.entrycget(index, 'label') + event = '' if menuEventDict.has_key(menubarItem): if menuEventDict[menubarItem].has_key(itemName): - event=menuEventDict[menubarItem][itemName] + event = menuEventDict[menubarItem][itemName] if event: - accel=get_accelerator(keydefs, event) - menu.entryconfig(index,accelerator=accel) + accel = get_accelerator(keydefs, event) + menu.entryconfig(index, accelerator=accel) def set_notabs_indentwidth(self): "Update the indentwidth if changed and not using tabs in this window" @@ -676,8 +690,10 @@ menu.delete(1, END) # clear, and rebuild: for i, file in zip(count(), rf_list): file_name = file[0:-1] # zap \n + # make unicode string to display non-ASCII chars correctly + ufile_name = self._filename_to_unicode(file_name) callback = instance.__recent_file_callback(file_name) - menu.add_command(label=ulchars[i] + " " + file_name, + menu.add_command(label=ulchars[i] + " " + ufile_name, command=callback, underline=0) @@ -717,10 +733,12 @@ filename = self.io.filename if filename: filename = os.path.basename(filename) - return filename + # return unicode string to display non-ASCII chars correctly + return self._filename_to_unicode(filename) def long_title(self): - return self.io.filename or "" + # return unicode string to display non-ASCII chars correctly + return self._filename_to_unicode(self.io.filename or "") def center_insert_event(self, event): self.center() Modified: python/branches/IDLE-syntax-branch/Lib/idlelib/IOBinding.py ============================================================================== --- python/branches/IDLE-syntax-branch/Lib/idlelib/IOBinding.py (original) +++ python/branches/IDLE-syntax-branch/Lib/idlelib/IOBinding.py Mon Apr 24 18:55:07 2006 @@ -32,6 +32,9 @@ except (ImportError, locale.Error): pass +# Encoding for file names +filesystemencoding = sys.getfilesystemencoding() + encoding = "ascii" if sys.platform == 'win32': # On Windows, we could use "mbcs". However, to give the user @@ -374,6 +377,7 @@ try: f = open(filename, "wb") f.write(chars) + f.flush() f.close() return True except IOError, msg: @@ -517,7 +521,10 @@ if not self.opendialog: self.opendialog = tkFileDialog.Open(master=self.text, filetypes=self.filetypes) - return self.opendialog.show(initialdir=dir, initialfile=base) + filename = self.opendialog.show(initialdir=dir, initialfile=base) + if isinstance(filename, unicode): + filename = filename.encode(filesystemencoding) + return filename def defaultfilename(self, mode="open"): if self.filename: @@ -536,7 +543,10 @@ if not self.savedialog: self.savedialog = tkFileDialog.SaveAs(master=self.text, filetypes=self.filetypes) - return self.savedialog.show(initialdir=dir, initialfile=base) + filename = self.savedialog.show(initialdir=dir, initialfile=base) + if isinstance(filename, unicode): + filename = filename.encode(filesystemencoding) + return filename def updaterecentfileslist(self,filename): "Update recent file list on all editor windows" Modified: python/branches/IDLE-syntax-branch/Lib/idlelib/NEWS.txt ============================================================================== --- python/branches/IDLE-syntax-branch/Lib/idlelib/NEWS.txt (original) +++ python/branches/IDLE-syntax-branch/Lib/idlelib/NEWS.txt Mon Apr 24 18:55:07 2006 @@ -1,7 +1,20 @@ -What's New in IDLE 1.2a0? -======================= +What's New in IDLE 1.2a1? +========================= + +*Release date: 05-APR-2006* + +- Source file f.flush() after writing; trying to avoid lossage if user + kills GUI. + +- Options / Keys / Advanced dialog made functional. Also, allow binding + of 'movement' keys. + +- 'syntax' patch adds improved calltips and a new class attribute listbox. + MultiCall module allows binding multiple actions to an event. + Patch 906702 Noam Raphael -*Release date: XX-XXX-2005* +- Better indentation after first line of string continuation. + IDLEfork Patch 681992, Noam Raphael - Fixed CodeContext alignment problem, following suggestion from Tal Einat. @@ -60,7 +73,7 @@ - Improve error handling when .idlerc can't be created (warn and exit). -- The GUI was hanging if the shell window was closed while a raw_input() +- The GUI was hanging if the shell window was closed while a raw_input() was pending. Restored the quit() of the readline() mainloop(). http://mail.python.org/pipermail/idle-dev/2004-December/002307.html Modified: python/branches/IDLE-syntax-branch/Lib/idlelib/ToolTip.py ============================================================================== --- python/branches/IDLE-syntax-branch/Lib/idlelib/ToolTip.py (original) +++ python/branches/IDLE-syntax-branch/Lib/idlelib/ToolTip.py Mon Apr 24 18:55:07 2006 @@ -83,7 +83,7 @@ b.pack() root.update() tip = ListboxToolTip(b, ["Hello", "world"]) + root.mainloop() - # root.mainloop() # not in idle - -main() +if __name__ == '__main__': + main() Modified: python/branches/IDLE-syntax-branch/Lib/idlelib/config-extensions.def ============================================================================== --- python/branches/IDLE-syntax-branch/Lib/idlelib/config-extensions.def (original) +++ python/branches/IDLE-syntax-branch/Lib/idlelib/config-extensions.def Mon Apr 24 18:55:07 2006 @@ -70,7 +70,7 @@ [AutoComplete] enable=1 -popupwait=0 +popupwait=2000 [AutoComplete_cfgBindings] force-open-completions= [AutoComplete_bindings] Modified: python/branches/IDLE-syntax-branch/Lib/idlelib/idlever.py ============================================================================== --- python/branches/IDLE-syntax-branch/Lib/idlelib/idlever.py (original) +++ python/branches/IDLE-syntax-branch/Lib/idlelib/idlever.py Mon Apr 24 18:55:07 2006 @@ -1 +1 @@ -IDLE_VERSION = "1.2a0" +IDLE_VERSION = "1.2a1" Modified: python/branches/IDLE-syntax-branch/Lib/idlelib/keybindingDialog.py ============================================================================== --- python/branches/IDLE-syntax-branch/Lib/idlelib/keybindingDialog.py (original) +++ python/branches/IDLE-syntax-branch/Lib/idlelib/keybindingDialog.py Mon Apr 24 18:55:07 2006 @@ -26,12 +26,13 @@ self.result='' self.keyString=StringVar(self) self.keyString.set('') - self.SetModifiersForPlatform() + self.SetModifiersForPlatform() # set self.modifiers, self.modifier_label self.modifier_vars = [] for modifier in self.modifiers: variable = StringVar(self) variable.set('') self.modifier_vars.append(variable) + self.advanced = False self.CreateWidgets() self.LoadFinalKeyList() self.withdraw() #hide while setting geometry @@ -136,7 +137,7 @@ self.modifiers = ['Shift', 'Control', 'Option', 'Command'] else: self.modifiers = ['Control', 'Alt', 'Shift'] - self.modifier_label = {'Control': 'Ctrl'} + self.modifier_label = {'Control': 'Ctrl'} # short name def ToggleLevel(self): if self.buttonLevel.cget('text')[:8]=='Advanced': @@ -145,11 +146,13 @@ self.frameKeySeqAdvanced.lift() self.frameHelpAdvanced.lift() self.entryKeysAdvanced.focus_set() + self.advanced = True else: self.ClearKeySeq() self.buttonLevel.config(text='Advanced Key Binding Entry >>') self.frameKeySeqBasic.lift() self.frameControlsBasic.lift() + self.advanced = False def FinalKeySelected(self,event): self.BuildKeyString() @@ -208,7 +211,7 @@ return key def OK(self, event=None): - if self.KeysOK(): + if self.advanced or self.KeysOK(): # doesn't check advanced string yet self.result=self.keyString.get() self.destroy() @@ -217,7 +220,12 @@ self.destroy() def KeysOK(self): - "Validity check on user's keybinding selection" + '''Validity check on user's 'basic' keybinding selection. + + Doesn't check the string produced by the advanced dialog because + 'modifiers' isn't set. + + ''' keys = self.keyString.get() keys.strip() finalKey = self.listKeysFinal.get(ANCHOR) @@ -232,20 +240,19 @@ elif not keys.endswith('>'): tkMessageBox.showerror(title=title, parent=self, message='Missing the final Key') - elif not modifiers and finalKey not in self.functionKeys: + elif (not modifiers + and finalKey not in self.functionKeys + self.moveKeys): tkMessageBox.showerror(title=title, parent=self, message='No modifier key(s) specified.') elif (modifiers == ['Shift']) \ and (finalKey not in - self.functionKeys + ('Tab', 'Space')): - msg = 'The shift modifier by itself may not be used with' \ - ' this key symbol; only with F1-F12, Tab, or Space' - tkMessageBox.showerror(title=title, parent=self, - message=msg) + self.functionKeys + self.moveKeys + ('Tab', 'Space')): + msg = 'The shift modifier by itself may not be used with'\ + ' this key symbol.' + tkMessageBox.showerror(title=title, parent=self, message=msg) elif keySequence in self.currentKeySequences: msg = 'This key combination is already in use.' - tkMessageBox.showerror(title=title, parent=self, - message=msg) + tkMessageBox.showerror(title=title, parent=self, message=msg) else: keysOK = True return keysOK From python-checkins at python.org Mon Apr 24 20:52:09 2006 From: python-checkins at python.org (guido.van.rossum) Date: Mon, 24 Apr 2006 20:52:09 +0200 (CEST) Subject: [Python-checkins] r45692 - peps/trunk/pep-0007.txt peps/trunk/pep-3100.txt Message-ID: <20060424185209.48A961E400C@bag.python.org> Author: guido.van.rossum Date: Mon Apr 24 20:52:08 2006 New Revision: 45692 Modified: peps/trunk/pep-0007.txt peps/trunk/pep-3100.txt Log: Record that the Py3k C coding style will use 4 spaces, no tabs. And record when it's okay to reformat a file to conform. Modified: peps/trunk/pep-0007.txt ============================================================================== --- peps/trunk/pep-0007.txt (original) +++ peps/trunk/pep-0007.txt Mon Apr 24 20:52:08 2006 @@ -28,6 +28,9 @@ C dialect - Use ANSI/ISO standard C (the 1989 version of the standard). + This means (amongst many other things) that all declarations + must be at the top of a block (not necessarily at the top of + function). - Don't use GCC extensions (e.g. don't write multi-line strings without trailing backslashes). @@ -44,6 +47,7 @@ Code lay-out - Use single-tab indents, where a tab is worth 8 spaces. + (For Python 3000, see the section Python 3000 below.) - No line should be longer than 79 characters. If this and the previous rule together don't give you enough room to code, your @@ -189,6 +193,13 @@ not all do; the MSVC compiler is known to complain about this. +Python 3000 + + In Python 3000, we'll switch to a different indentation style: + 4 spaces per indent, all spaces (no tabs in any file). The + rest will remain the same. + + References [1] PEP 8, Style Guide for Python Code, van Rossum, Warsaw Modified: peps/trunk/pep-3100.txt ============================================================================== --- peps/trunk/pep-3100.txt (original) +++ peps/trunk/pep-3100.txt Mon Apr 24 20:52:08 2006 @@ -50,6 +50,16 @@ * PEP 352 (Required Superclass for Exceptions) [#pep352]_ +Style changes +============= + +* The C style guide will be updated to use 4-space indents, never tabs. + This style should be used for all new files; existing files can be + updated only if there is no hope to ever merge a particular file from + the Python 2 HEAD. Within a file, the indentation style should be + consistent. No other style guide changes are planned ATM. + + Core language ============= From python-checkins at python.org Mon Apr 24 22:22:35 2006 From: python-checkins at python.org (phillip.eby) Date: Mon, 24 Apr 2006 22:22:35 +0200 (CEST) Subject: [Python-checkins] r45694 - sandbox/trunk/setuptools/setuptools/command/bdist_egg.py sandbox/trunk/setuptools/setuptools/command/easy_install.py Message-ID: <20060424202235.B950F1E400A@bag.python.org> Author: phillip.eby Date: Mon Apr 24 22:22:32 2006 New Revision: 45694 Modified: sandbox/trunk/setuptools/setuptools/command/bdist_egg.py sandbox/trunk/setuptools/setuptools/command/easy_install.py Log: Strip 'module' from the end of compiled extension modules when computing the name of a .py loader/wrapper. Python's import machinery ignores this suffix when searching for an extension module. Modified: sandbox/trunk/setuptools/setuptools/command/bdist_egg.py ============================================================================== --- sandbox/trunk/setuptools/setuptools/command/bdist_egg.py (original) +++ sandbox/trunk/setuptools/setuptools/command/bdist_egg.py Mon Apr 24 22:22:32 2006 @@ -12,6 +12,13 @@ from types import CodeType from setuptools.extension import Library +def strip_module(filename): + if '.' in filename: + filename = os.path.splitext(filename)[0] + if filename.endswith('module'): + filename = filename[:-6] + return filename + def write_stub(resource, pyfile): f = open(pyfile,'w') f.write('\n'.join([ @@ -32,13 +39,6 @@ - - - - - - - class bdist_egg(Command): description = "create an \"egg\" distribution" @@ -179,7 +179,7 @@ to_compile = [] for (p,ext_name) in enumerate(ext_outputs): filename,ext = os.path.splitext(ext_name) - pyfile = os.path.join(self.bdist_dir, filename + '.py') + pyfile = os.path.join(self.bdist_dir, strip_module(filename)+'.py') self.stubs.append(pyfile) log.info("creating stub loader for %s" % ext_name) if not self.dry_run: Modified: sandbox/trunk/setuptools/setuptools/command/easy_install.py ============================================================================== --- sandbox/trunk/setuptools/setuptools/command/easy_install.py (original) +++ sandbox/trunk/setuptools/setuptools/command/easy_install.py Mon Apr 24 22:22:32 2006 @@ -745,7 +745,6 @@ to_compile = [] native_libs = [] top_level = {} - def process(src,dst): for old,new in prefixes: if src.startswith(old): @@ -754,6 +753,7 @@ dst = os.path.join(egg_tmp, *parts) dl = dst.lower() if dl.endswith('.pyd') or dl.endswith('.dll'): + parts[-1] = bdist_egg.strip_module(parts[-1]) top_level[os.path.splitext(parts[0])[0]] = 1 native_libs.append(src) elif dl.endswith('.py') and old!='SCRIPTS/': @@ -770,11 +770,11 @@ for res in native_libs: if res.lower().endswith('.pyd'): # create stubs for .pyd's parts = res.split('/') - resource, parts[-1] = parts[-1], parts[-1][:-1] + resource = parts[-1] + parts[-1] = bdist_egg.strip_module(parts[-1])+'.py' pyfile = os.path.join(egg_tmp, *parts) to_compile.append(pyfile); stubs.append(pyfile) bdist_egg.write_stub(resource, pyfile) - self.byte_compile(to_compile) # compile .py's bdist_egg.write_safety_flag(os.path.join(egg_tmp,'EGG-INFO'), bdist_egg.analyze_egg(egg_tmp, stubs)) # write zip-safety flag From python-checkins at python.org Mon Apr 24 22:50:19 2006 From: python-checkins at python.org (phillip.eby) Date: Mon, 24 Apr 2006 22:50:19 +0200 (CEST) Subject: [Python-checkins] r45695 - sandbox/trunk/setuptools/setuptools/dist.py Message-ID: <20060424205019.625FB1E401B@bag.python.org> Author: phillip.eby Date: Mon Apr 24 22:50:18 2006 New Revision: 45695 Modified: sandbox/trunk/setuptools/setuptools/dist.py Log: Catch 'module' names in internal module name list as well. Modified: sandbox/trunk/setuptools/setuptools/dist.py ============================================================================== --- sandbox/trunk/setuptools/setuptools/dist.py (original) +++ sandbox/trunk/setuptools/setuptools/dist.py Mon Apr 24 22:50:18 2006 @@ -624,10 +624,12 @@ for ext in self.ext_modules or (): if isinstance(ext,tuple): - name,buildinfo = ext - yield name + name, buildinfo = ext else: - yield ext.name + name = ext.name + if name.endswith('module'): + name = name[:-6] + yield name # Install it throughout the distutils for module in distutils.dist, distutils.core, distutils.cmd: @@ -652,8 +654,6 @@ - - class Feature: """A subset of the distribution that can be excluded if unneeded/wanted From python-checkins at python.org Mon Apr 24 22:53:00 2006 From: python-checkins at python.org (phillip.eby) Date: Mon, 24 Apr 2006 22:53:00 +0200 (CEST) Subject: [Python-checkins] r45696 - in sandbox/branches/setuptools-0.6/setuptools: command/bdist_egg.py command/easy_install.py dist.py Message-ID: <20060424205300.347461E400B@bag.python.org> Author: phillip.eby Date: Mon Apr 24 22:52:59 2006 New Revision: 45696 Modified: sandbox/branches/setuptools-0.6/setuptools/command/bdist_egg.py sandbox/branches/setuptools-0.6/setuptools/command/easy_install.py sandbox/branches/setuptools-0.6/setuptools/dist.py Log: Backport 'module' fixes to 0.6 Modified: sandbox/branches/setuptools-0.6/setuptools/command/bdist_egg.py ============================================================================== --- sandbox/branches/setuptools-0.6/setuptools/command/bdist_egg.py (original) +++ sandbox/branches/setuptools-0.6/setuptools/command/bdist_egg.py Mon Apr 24 22:52:59 2006 @@ -12,6 +12,13 @@ from types import CodeType from setuptools.extension import Library +def strip_module(filename): + if '.' in filename: + filename = os.path.splitext(filename)[0] + if filename.endswith('module'): + filename = filename[:-6] + return filename + def write_stub(resource, pyfile): f = open(pyfile,'w') f.write('\n'.join([ @@ -32,13 +39,6 @@ - - - - - - - class bdist_egg(Command): description = "create an \"egg\" distribution" @@ -179,7 +179,7 @@ to_compile = [] for (p,ext_name) in enumerate(ext_outputs): filename,ext = os.path.splitext(ext_name) - pyfile = os.path.join(self.bdist_dir, filename + '.py') + pyfile = os.path.join(self.bdist_dir, strip_module(filename)+'.py') self.stubs.append(pyfile) log.info("creating stub loader for %s" % ext_name) if not self.dry_run: Modified: sandbox/branches/setuptools-0.6/setuptools/command/easy_install.py ============================================================================== --- sandbox/branches/setuptools-0.6/setuptools/command/easy_install.py (original) +++ sandbox/branches/setuptools-0.6/setuptools/command/easy_install.py Mon Apr 24 22:52:59 2006 @@ -745,7 +745,6 @@ to_compile = [] native_libs = [] top_level = {} - def process(src,dst): for old,new in prefixes: if src.startswith(old): @@ -754,6 +753,7 @@ dst = os.path.join(egg_tmp, *parts) dl = dst.lower() if dl.endswith('.pyd') or dl.endswith('.dll'): + parts[-1] = bdist_egg.strip_module(parts[-1]) top_level[os.path.splitext(parts[0])[0]] = 1 native_libs.append(src) elif dl.endswith('.py') and old!='SCRIPTS/': @@ -770,11 +770,11 @@ for res in native_libs: if res.lower().endswith('.pyd'): # create stubs for .pyd's parts = res.split('/') - resource, parts[-1] = parts[-1], parts[-1][:-1] + resource = parts[-1] + parts[-1] = bdist_egg.strip_module(parts[-1])+'.py' pyfile = os.path.join(egg_tmp, *parts) to_compile.append(pyfile); stubs.append(pyfile) bdist_egg.write_stub(resource, pyfile) - self.byte_compile(to_compile) # compile .py's bdist_egg.write_safety_flag(os.path.join(egg_tmp,'EGG-INFO'), bdist_egg.analyze_egg(egg_tmp, stubs)) # write zip-safety flag Modified: sandbox/branches/setuptools-0.6/setuptools/dist.py ============================================================================== --- sandbox/branches/setuptools-0.6/setuptools/dist.py (original) +++ sandbox/branches/setuptools-0.6/setuptools/dist.py Mon Apr 24 22:52:59 2006 @@ -624,10 +624,12 @@ for ext in self.ext_modules or (): if isinstance(ext,tuple): - name,buildinfo = ext - yield name + name, buildinfo = ext else: - yield ext.name + name = ext.name + if name.endswith('module'): + name = name[:-6] + yield name # Install it throughout the distutils for module in distutils.dist, distutils.core, distutils.cmd: @@ -652,8 +654,6 @@ - - class Feature: """A subset of the distribution that can be excluded if unneeded/wanted From python-checkins at python.org Mon Apr 24 22:53:14 2006 From: python-checkins at python.org (phillip.eby) Date: Mon, 24 Apr 2006 22:53:14 +0200 (CEST) Subject: [Python-checkins] r45697 - in python/trunk: Lib/easy_install.py Lib/pkg_resources.py Lib/setuptools Lib/setuptools.egg-info Lib/test/test_setuptools.py Misc/NEWS Message-ID: <20060424205314.50EC61E400B@bag.python.org> Author: phillip.eby Date: Mon Apr 24 22:53:13 2006 New Revision: 45697 Removed: python/trunk/Lib/easy_install.py python/trunk/Lib/pkg_resources.py python/trunk/Lib/setuptools/ python/trunk/Lib/setuptools.egg-info/ python/trunk/Lib/test/test_setuptools.py Modified: python/trunk/Misc/NEWS Log: Revert addition of setuptools Deleted: /python/trunk/Lib/easy_install.py ============================================================================== --- /python/trunk/Lib/easy_install.py Mon Apr 24 22:53:13 2006 +++ (empty file) @@ -1,5 +0,0 @@ -"""Run the EasyInstall command""" - -if __name__ == '__main__': - from setuptools.command.easy_install import main - main() Deleted: /python/trunk/Lib/pkg_resources.py ============================================================================== --- /python/trunk/Lib/pkg_resources.py Mon Apr 24 22:53:13 2006 +++ (empty file) @@ -1,2377 +0,0 @@ -"""Package resource API --------------------- - -A resource is a logical file contained within a package, or a logical -subdirectory thereof. The package resource API expects resource names -to have their path parts separated with ``/``, *not* whatever the local -path separator is. Do not use os.path operations to manipulate resource -names being passed into the API. - -The package resource API is designed to work with normal filesystem packages, -.egg files, and unpacked .egg files. It can also work in a limited way with -.zip files and with custom PEP 302 loaders that support the ``get_data()`` -method. -""" - -import sys, os, zipimport, time, re, imp, new, pkgutil # XXX -from sets import ImmutableSet -from os import utime, rename, unlink # capture these to bypass sandboxing -from os import open as os_open - -def get_supported_platform(): - """Return this platform's maximum compatible version. - - distutils.util.get_platform() normally reports the minimum version - of Mac OS X that would be required to *use* extensions produced by - distutils. But what we want when checking compatibility is to know the - version of Mac OS X that we are *running*. To allow usage of packages that - explicitly require a newer version of Mac OS X, we must also know the - current version of the OS. - - If this condition occurs for any other platform with a version in its - platform strings, this function should be extended accordingly. - """ - plat = get_build_platform(); m = macosVersionString.match(plat) - if m is not None and sys.platform == "darwin": - try: - plat = 'macosx-%s-%s' % ('.'.join(_macosx_vers()[:2]), m.group(3)) - except ValueError: - pass # not Mac OS X - return plat - -__all__ = [ - # Basic resource access and distribution/entry point discovery - 'require', 'run_script', 'get_provider', 'get_distribution', - 'load_entry_point', 'get_entry_map', 'get_entry_info', 'iter_entry_points', - 'resource_string', 'resource_stream', 'resource_filename', - 'resource_listdir', 'resource_exists', 'resource_isdir', - - # Environmental control - 'declare_namespace', 'working_set', 'add_activation_listener', - 'find_distributions', 'set_extraction_path', 'cleanup_resources', - 'get_default_cache', - - # Primary implementation classes - 'Environment', 'WorkingSet', 'ResourceManager', - 'Distribution', 'Requirement', 'EntryPoint', - - # Exceptions - 'ResolutionError','VersionConflict','DistributionNotFound','UnknownExtra', - 'ExtractionError', - - # Parsing functions and string utilities - 'parse_requirements', 'parse_version', 'safe_name', 'safe_version', - 'get_platform', 'compatible_platforms', 'yield_lines', 'split_sections', - 'safe_extra', 'to_filename', - - # filesystem utilities - 'ensure_directory', 'normalize_path', - - # Distribution "precedence" constants - 'EGG_DIST', 'BINARY_DIST', 'SOURCE_DIST', 'CHECKOUT_DIST', 'DEVELOP_DIST', - - # "Provider" interfaces, implementations, and registration/lookup APIs - 'IMetadataProvider', 'IResourceProvider', 'FileMetadata', - 'PathMetadata', 'EggMetadata', 'EmptyProvider', 'empty_provider', - 'NullProvider', 'EggProvider', 'DefaultProvider', 'ZipProvider', - 'register_finder', 'register_namespace_handler', 'register_loader_type', - 'fixup_namespace_packages', 'get_importer', - - # Deprecated/backward compatibility only - 'run_main', 'AvailableDistributions', -] -class ResolutionError(Exception): - """Abstract base for dependency resolution errors""" - def __repr__(self): - return self.__class__.__name__+repr(self.args) - -class VersionConflict(ResolutionError): - """An already-installed version conflicts with the requested version""" - -class DistributionNotFound(ResolutionError): - """A requested distribution was not found""" - -class UnknownExtra(ResolutionError): - """Distribution doesn't have an "extra feature" of the given name""" - -_provider_factories = {} -PY_MAJOR = sys.version[:3] -EGG_DIST = 3 -BINARY_DIST = 2 -SOURCE_DIST = 1 -CHECKOUT_DIST = 0 -DEVELOP_DIST = -1 - -def register_loader_type(loader_type, provider_factory): - """Register `provider_factory` to make providers for `loader_type` - - `loader_type` is the type or class of a PEP 302 ``module.__loader__``, - and `provider_factory` is a function that, passed a *module* object, - returns an ``IResourceProvider`` for that module. - """ - _provider_factories[loader_type] = provider_factory - -def get_provider(moduleOrReq): - """Return an IResourceProvider for the named module or requirement""" - if isinstance(moduleOrReq,Requirement): - return working_set.find(moduleOrReq) or require(str(moduleOrReq))[0] - try: - module = sys.modules[moduleOrReq] - except KeyError: - __import__(moduleOrReq) - module = sys.modules[moduleOrReq] - loader = getattr(module, '__loader__', None) - return _find_adapter(_provider_factories, loader)(module) - -def _macosx_vers(_cache=[]): - if not _cache: - info = os.popen('/usr/bin/sw_vers').read().splitlines() - for line in info: - key, value = line.split(None, 1) - if key == 'ProductVersion:': - _cache.append(value.strip().split(".")) - break - else: - raise ValueError, "What?!" - return _cache[0] - -def _macosx_arch(machine): - return {'PowerPC':'ppc', 'Power_Macintosh':'ppc'}.get(machine,machine) - -def get_build_platform(): - """Return this platform's string for platform-specific distributions - - XXX Currently this is the same as ``distutils.util.get_platform()``, but it - needs some hacks for Linux and Mac OS X. - """ - from distutils.util import get_platform - plat = get_platform() - if sys.platform == "darwin" and not plat.startswith('macosx-'): - try: - version = _macosx_vers() - machine = os.uname()[4].replace(" ", "_") - return "macosx-%d.%d-%s" % (int(version[0]), int(version[1]), - _macosx_arch(machine)) - except ValueError: - # if someone is running a non-Mac darwin system, this will fall - # through to the default implementation - pass - return plat - -macosVersionString = re.compile(r"macosx-(\d+)\.(\d+)-(.*)") -darwinVersionString = re.compile(r"darwin-(\d+)\.(\d+)\.(\d+)-(.*)") -get_platform = get_build_platform # XXX backward compat - -def compatible_platforms(provided,required): - """Can code for the `provided` platform run on the `required` platform? - - Returns true if either platform is ``None``, or the platforms are equal. - - XXX Needs compatibility checks for Linux and other unixy OSes. - """ - if provided is None or required is None or provided==required: - return True # easy case - - # Mac OS X special cases - reqMac = macosVersionString.match(required) - if reqMac: - provMac = macosVersionString.match(provided) - - # is this a Mac package? - if not provMac: - # this is backwards compatibility for packages built before - # setuptools 0.6. All packages built after this point will - # use the new macosx designation. - provDarwin = darwinVersionString.match(provided) - if provDarwin: - dversion = int(provDarwin.group(1)) - macosversion = "%s.%s" % (reqMac.group(1), reqMac.group(2)) - if dversion == 7 and macosversion >= "10.3" or \ - dversion == 8 and macosversion >= "10.4": - - #import warnings - #warnings.warn("Mac eggs should be rebuilt to " - # "use the macosx designation instead of darwin.", - # category=DeprecationWarning) - return True - return False # egg isn't macosx or legacy darwin - - # are they the same major version and machine type? - if provMac.group(1) != reqMac.group(1) or \ - provMac.group(3) != reqMac.group(3): - return False - - - - # is the required OS major update >= the provided one? - if int(provMac.group(2)) > int(reqMac.group(2)): - return False - - return True - - # XXX Linux and other platforms' special cases should go here - return False - - -def run_script(dist_spec, script_name): - """Locate distribution `dist_spec` and run its `script_name` script""" - ns = sys._getframe(1).f_globals - name = ns['__name__'] - ns.clear() - ns['__name__'] = name - require(dist_spec)[0].run_script(script_name, ns) - -run_main = run_script # backward compatibility - -def get_distribution(dist): - """Return a current distribution object for a Requirement or string""" - if isinstance(dist,basestring): dist = Requirement.parse(dist) - if isinstance(dist,Requirement): dist = get_provider(dist) - if not isinstance(dist,Distribution): - raise TypeError("Expected string, Requirement, or Distribution", dist) - return dist - -def load_entry_point(dist, group, name): - """Return `name` entry point of `group` for `dist` or raise ImportError""" - return get_distribution(dist).load_entry_point(group, name) - -def get_entry_map(dist, group=None): - """Return the entry point map for `group`, or the full entry map""" - return get_distribution(dist).get_entry_map(group) - -def get_entry_info(dist, group, name): - """Return the EntryPoint object for `group`+`name`, or ``None``""" - return get_distribution(dist).get_entry_info(group, name) - - -try: - from pkgutil import get_importer -except ImportError: - import _pkgutil as pkgutil - get_importer = pkgutil.get_importer -else: - import pkgutil - - -class IMetadataProvider: - - def has_metadata(name): - """Does the package's distribution contain the named metadata?""" - - def get_metadata(name): - """The named metadata resource as a string""" - - def get_metadata_lines(name): - """Yield named metadata resource as list of non-blank non-comment lines - - Leading and trailing whitespace is stripped from each line, and lines - with ``#`` as the first non-blank character are omitted.""" - - def metadata_isdir(name): - """Is the named metadata a directory? (like ``os.path.isdir()``)""" - - def metadata_listdir(name): - """List of metadata names in the directory (like ``os.listdir()``)""" - - def run_script(script_name, namespace): - """Execute the named script in the supplied namespace dictionary""" - - - - - - - - - - -class IResourceProvider(IMetadataProvider): - """An object that provides access to package resources""" - - def get_resource_filename(manager, resource_name): - """Return a true filesystem path for `resource_name` - - `manager` must be an ``IResourceManager``""" - - def get_resource_stream(manager, resource_name): - """Return a readable file-like object for `resource_name` - - `manager` must be an ``IResourceManager``""" - - def get_resource_string(manager, resource_name): - """Return a string containing the contents of `resource_name` - - `manager` must be an ``IResourceManager``""" - - def has_resource(resource_name): - """Does the package contain the named resource?""" - - def resource_isdir(resource_name): - """Is the named resource a directory? (like ``os.path.isdir()``)""" - - def resource_listdir(resource_name): - """List of resource names in the directory (like ``os.listdir()``)""" - - - - - - - - - - - - - - - -class WorkingSet(object): - """A collection of active distributions on sys.path (or a similar list)""" - - def __init__(self, entries=None): - """Create working set from list of path entries (default=sys.path)""" - self.entries = [] - self.entry_keys = {} - self.by_key = {} - self.callbacks = [] - - if entries is None: - entries = sys.path - - for entry in entries: - self.add_entry(entry) - - - def add_entry(self, entry): - """Add a path item to ``.entries``, finding any distributions on it - - ``find_distributions(entry,False)`` is used to find distributions - corresponding to the path entry, and they are added. `entry` is - always appended to ``.entries``, even if it is already present. - (This is because ``sys.path`` can contain the same value more than - once, and the ``.entries`` of the ``sys.path`` WorkingSet should always - equal ``sys.path``.) - """ - self.entry_keys.setdefault(entry, []) - self.entries.append(entry) - for dist in find_distributions(entry, True): - self.add(dist, entry, False) - - - def __contains__(self,dist): - """True if `dist` is the active distribution for its project""" - return self.by_key.get(dist.key) == dist - - - - - - def find(self, req): - """Find a distribution matching requirement `req` - - If there is an active distribution for the requested project, this - returns it as long as it meets the version requirement specified by - `req`. But, if there is an active distribution for the project and it - does *not* meet the `req` requirement, ``VersionConflict`` is raised. - If there is no active distribution for the requested project, ``None`` - is returned. - """ - dist = self.by_key.get(req.key) - if dist is not None and dist not in req: - raise VersionConflict(dist,req) # XXX add more info - else: - return dist - - def iter_entry_points(self, group, name=None): - """Yield entry point objects from `group` matching `name` - - If `name` is None, yields all entry points in `group` from all - distributions in the working set, otherwise only ones matching - both `group` and `name` are yielded (in distribution order). - """ - for dist in self: - entries = dist.get_entry_map(group) - if name is None: - for ep in entries.values(): - yield ep - elif name in entries: - yield entries[name] - - def run_script(self, requires, script_name): - """Locate distribution for `requires` and run `script_name` script""" - ns = sys._getframe(1).f_globals - name = ns['__name__'] - ns.clear() - ns['__name__'] = name - self.require(requires)[0].run_script(script_name, ns) - - - - def __iter__(self): - """Yield distributions for non-duplicate projects in the working set - - The yield order is the order in which the items' path entries were - added to the working set. - """ - seen = {} - for item in self.entries: - for key in self.entry_keys[item]: - if key not in seen: - seen[key]=1 - yield self.by_key[key] - - def add(self, dist, entry=None, insert=True): - """Add `dist` to working set, associated with `entry` - - If `entry` is unspecified, it defaults to the ``.location`` of `dist`. - On exit from this routine, `entry` is added to the end of the working - set's ``.entries`` (if it wasn't already present). - - `dist` is only added to the working set if it's for a project that - doesn't already have a distribution in the set. If it's added, any - callbacks registered with the ``subscribe()`` method will be called. - """ - if insert: - dist.insert_on(self.entries, entry) - - if entry is None: - entry = dist.location - keys = self.entry_keys.setdefault(entry,[]) - - if dist.key in self.by_key: - return # ignore hidden distros - - self.by_key[dist.key] = dist - if dist.key not in keys: - keys.append(dist.key) - - self._added_new(dist) - - - def resolve(self, requirements, env=None, installer=None): - """List all distributions needed to (recursively) meet `requirements` - - `requirements` must be a sequence of ``Requirement`` objects. `env`, - if supplied, should be an ``Environment`` instance. If - not supplied, it defaults to all distributions available within any - entry or distribution in the working set. `installer`, if supplied, - will be invoked with each requirement that cannot be met by an - already-installed distribution; it should return a ``Distribution`` or - ``None``. - """ - - requirements = list(requirements)[::-1] # set up the stack - processed = {} # set of processed requirements - best = {} # key -> dist - to_activate = [] - - while requirements: - req = requirements.pop(0) # process dependencies breadth-first - if req in processed: - # Ignore cyclic or redundant dependencies - continue - dist = best.get(req.key) - if dist is None: - # Find the best distribution and add it to the map - dist = self.by_key.get(req.key) - if dist is None: - if env is None: - env = Environment(self.entries) - dist = best[req.key] = env.best_match(req, self, installer) - if dist is None: - raise DistributionNotFound(req) # XXX put more info here - to_activate.append(dist) - if dist not in req: - # Oops, the "best" so far conflicts with a dependency - raise VersionConflict(dist,req) # XXX put more info here - requirements.extend(dist.requires(req.extras)[::-1]) - processed[req] = True - - return to_activate # return list of distros to activate - - def find_plugins(self, - plugin_env, full_env=None, installer=None, fallback=True - ): - """Find all activatable distributions in `plugin_env` - - Example usage:: - - distributions, errors = working_set.find_plugins( - Environment(plugin_dirlist) - ) - map(working_set.add, distributions) # add plugins+libs to sys.path - print "Couldn't load", errors # display errors - - The `plugin_env` should be an ``Environment`` instance that contains - only distributions that are in the project's "plugin directory" or - directories. The `full_env`, if supplied, should be an ``Environment`` - contains all currently-available distributions. If `full_env` is not - supplied, one is created automatically from the ``WorkingSet`` this - method is called on, which will typically mean that every directory on - ``sys.path`` will be scanned for distributions. - - `installer` is a standard installer callback as used by the - ``resolve()`` method. The `fallback` flag indicates whether we should - attempt to resolve older versions of a plugin if the newest version - cannot be resolved. - - This method returns a 2-tuple: (`distributions`, `error_info`), where - `distributions` is a list of the distributions found in `plugin_env` - that were loadable, along with any other distributions that are needed - to resolve their dependencies. `error_info` is a dictionary mapping - unloadable plugin distributions to an exception instance describing the - error that occurred. Usually this will be a ``DistributionNotFound`` or - ``VersionConflict`` instance. - """ - - plugin_projects = list(plugin_env) - plugin_projects.sort() # scan project names in alphabetic order - - error_info = {} - distributions = {} - - if full_env is None: - env = Environment(self.entries) - env += plugin_env - else: - env = full_env + plugin_env - - shadow_set = self.__class__([]) - map(shadow_set.add, self) # put all our entries in shadow_set - - for project_name in plugin_projects: - - for dist in plugin_env[project_name]: - - req = [dist.as_requirement()] - - try: - resolvees = shadow_set.resolve(req, env, installer) - - except ResolutionError,v: - error_info[dist] = v # save error info - if fallback: - continue # try the next older version of project - else: - break # give up on this project, keep going - - else: - map(shadow_set.add, resolvees) - distributions.update(dict.fromkeys(resolvees)) - - # success, no need to try any more versions of this project - break - - distributions = list(distributions) - distributions.sort() - - return distributions, error_info - - - - - - def require(self, *requirements): - """Ensure that distributions matching `requirements` are activated - - `requirements` must be a string or a (possibly-nested) sequence - thereof, specifying the distributions and versions required. The - return value is a sequence of the distributions that needed to be - activated to fulfill the requirements; all relevant distributions are - included, even if they were already activated in this working set. - """ - - needed = self.resolve(parse_requirements(requirements)) - - for dist in needed: - self.add(dist) - - return needed - - - def subscribe(self, callback): - """Invoke `callback` for all distributions (including existing ones)""" - if callback in self.callbacks: - return - self.callbacks.append(callback) - for dist in self: - callback(dist) - - - def _added_new(self, dist): - for callback in self.callbacks: - callback(dist) - - - - - - - - - - - -class Environment(object): - """Searchable snapshot of distributions on a search path""" - - def __init__(self, search_path=None, platform=get_supported_platform(), python=PY_MAJOR): - """Snapshot distributions available on a search path - - Any distributions found on `search_path` are added to the environment. - `search_path` should be a sequence of ``sys.path`` items. If not - supplied, ``sys.path`` is used. - - `platform` is an optional string specifying the name of the platform - that platform-specific distributions must be compatible with. If - unspecified, it defaults to the current platform. `python` is an - optional string naming the desired version of Python (e.g. ``'2.4'``); - it defaults to the current version. - - You may explicitly set `platform` (and/or `python`) to ``None`` if you - wish to map *all* distributions, not just those compatible with the - running platform or Python version. - """ - self._distmap = {} - self._cache = {} - self.platform = platform - self.python = python - self.scan(search_path) - - def can_add(self, dist): - """Is distribution `dist` acceptable for this environment? - - The distribution must match the platform and python version - requirements specified when this environment was created, or False - is returned. - """ - return (self.python is None or dist.py_version is None - or dist.py_version==self.python) \ - and compatible_platforms(dist.platform,self.platform) - - def remove(self, dist): - """Remove `dist` from the environment""" - self._distmap[dist.key].remove(dist) - - def scan(self, search_path=None): - """Scan `search_path` for distributions usable in this environment - - Any distributions found are added to the environment. - `search_path` should be a sequence of ``sys.path`` items. If not - supplied, ``sys.path`` is used. Only distributions conforming to - the platform/python version defined at initialization are added. - """ - if search_path is None: - search_path = sys.path - - for item in search_path: - for dist in find_distributions(item): - self.add(dist) - - def __getitem__(self,project_name): - """Return a newest-to-oldest list of distributions for `project_name` - """ - try: - return self._cache[project_name] - except KeyError: - project_name = project_name.lower() - if project_name not in self._distmap: - return [] - - if project_name not in self._cache: - dists = self._cache[project_name] = self._distmap[project_name] - _sort_dists(dists) - - return self._cache[project_name] - - def add(self,dist): - """Add `dist` if we ``can_add()`` it and it isn't already added""" - if self.can_add(dist) and dist.has_version(): - dists = self._distmap.setdefault(dist.key,[]) - if dist not in dists: - dists.append(dist) - if dist.key in self._cache: - _sort_dists(self._cache[dist.key]) - - - def best_match(self, req, working_set, installer=None): - """Find distribution best matching `req` and usable on `working_set` - - This calls the ``find(req)`` method of the `working_set` to see if a - suitable distribution is already active. (This may raise - ``VersionConflict`` if an unsuitable version of the project is already - active in the specified `working_set`.) If a suitable distribution - isn't active, this method returns the newest distribution in the - environment that meets the ``Requirement`` in `req`. If no suitable - distribution is found, and `installer` is supplied, then the result of - calling the environment's ``obtain(req, installer)`` method will be - returned. - """ - dist = working_set.find(req) - if dist is not None: - return dist - for dist in self[req.key]: - if dist in req: - return dist - return self.obtain(req, installer) # try and download/install - - def obtain(self, requirement, installer=None): - """Obtain a distribution matching `requirement` (e.g. via download) - - Obtain a distro that matches requirement (e.g. via download). In the - base ``Environment`` class, this routine just returns - ``installer(requirement)``, unless `installer` is None, in which case - None is returned instead. This method is a hook that allows subclasses - to attempt other ways of obtaining a distribution before falling back - to the `installer` argument.""" - if installer is not None: - return installer(requirement) - - def __iter__(self): - """Yield the unique project names of the available distributions""" - for key in self._distmap.keys(): - if self[key]: yield key - - - - - def __iadd__(self, other): - """In-place addition of a distribution or environment""" - if isinstance(other,Distribution): - self.add(other) - elif isinstance(other,Environment): - for project in other: - for dist in other[project]: - self.add(dist) - else: - raise TypeError("Can't add %r to environment" % (other,)) - return self - - def __add__(self, other): - """Add an environment or distribution to an environment""" - new = self.__class__([], platform=None, python=None) - for env in self, other: - new += env - return new - - -AvailableDistributions = Environment # XXX backward compatibility - - -class ExtractionError(RuntimeError): - """An error occurred extracting a resource - - The following attributes are available from instances of this exception: - - manager - The resource manager that raised this exception - - cache_path - The base directory for resource extraction - - original_error - The exception instance that caused extraction to fail - """ - - - - -class ResourceManager: - """Manage resource extraction and packages""" - extraction_path = None - - def __init__(self): - self.cached_files = {} - - def resource_exists(self, package_or_requirement, resource_name): - """Does the named resource exist?""" - return get_provider(package_or_requirement).has_resource(resource_name) - - def resource_isdir(self, package_or_requirement, resource_name): - """Is the named resource an existing directory?""" - return get_provider(package_or_requirement).resource_isdir( - resource_name - ) - - def resource_filename(self, package_or_requirement, resource_name): - """Return a true filesystem path for specified resource""" - return get_provider(package_or_requirement).get_resource_filename( - self, resource_name - ) - - def resource_stream(self, package_or_requirement, resource_name): - """Return a readable file-like object for specified resource""" - return get_provider(package_or_requirement).get_resource_stream( - self, resource_name - ) - - def resource_string(self, package_or_requirement, resource_name): - """Return specified resource as a string""" - return get_provider(package_or_requirement).get_resource_string( - self, resource_name - ) - - def resource_listdir(self, package_or_requirement, resource_name): - """List the contents of the named resource directory""" - return get_provider(package_or_requirement).resource_listdir( - resource_name - ) - - def extraction_error(self): - """Give an error message for problems extracting file(s)""" - - old_exc = sys.exc_info()[1] - cache_path = self.extraction_path or get_default_cache() - - err = ExtractionError("""Can't extract file(s) to egg cache - -The following error occurred while trying to extract file(s) to the Python egg -cache: - - %s - -The Python egg cache directory is currently set to: - - %s - -Perhaps your account does not have write access to this directory? You can -change the cache directory by setting the PYTHON_EGG_CACHE environment -variable to point to an accessible directory. -""" % (old_exc, cache_path) - ) - err.manager = self - err.cache_path = cache_path - err.original_error = old_exc - raise err - - - - - - - - - - - - - - - - def get_cache_path(self, archive_name, names=()): - """Return absolute location in cache for `archive_name` and `names` - - The parent directory of the resulting path will be created if it does - not already exist. `archive_name` should be the base filename of the - enclosing egg (which may not be the name of the enclosing zipfile!), - including its ".egg" extension. `names`, if provided, should be a - sequence of path name parts "under" the egg's extraction location. - - This method should only be called by resource providers that need to - obtain an extraction location, and only for names they intend to - extract, as it tracks the generated names for possible cleanup later. - """ - extract_path = self.extraction_path or get_default_cache() - target_path = os.path.join(extract_path, archive_name+'-tmp', *names) - try: - ensure_directory(target_path) - except: - self.extraction_error() - - self.cached_files[target_path] = 1 - return target_path - - - def postprocess(self, tempname, filename): - """Perform any platform-specific postprocessing of `tempname` - - This is where Mac header rewrites should be done; other platforms don't - have anything special they should do. - - Resource providers should call this method ONLY after successfully - extracting a compressed resource. They must NOT call it on resources - that are already in the filesystem. - - `tempname` is the current (temporary) name of the file, and `filename` - is the name it will be renamed to by the caller after this routine - returns. - """ - # XXX - - - def set_extraction_path(self, path): - """Set the base path where resources will be extracted to, if needed. - - If you do not call this routine before any extractions take place, the - path defaults to the return value of ``get_default_cache()``. (Which - is based on the ``PYTHON_EGG_CACHE`` environment variable, with various - platform-specific fallbacks. See that routine's documentation for more - details.) - - Resources are extracted to subdirectories of this path based upon - information given by the ``IResourceProvider``. You may set this to a - temporary directory, but then you must call ``cleanup_resources()`` to - delete the extracted files when done. There is no guarantee that - ``cleanup_resources()`` will be able to remove all extracted files. - - (Note: you may not change the extraction path for a given resource - manager once resources have been extracted, unless you first call - ``cleanup_resources()``.) - """ - if self.cached_files: - raise ValueError( - "Can't change extraction path, files already extracted" - ) - - self.extraction_path = path - - def cleanup_resources(self, force=False): - """ - Delete all extracted resource files and directories, returning a list - of the file and directory names that could not be successfully removed. - This function does not have any concurrency protection, so it should - generally only be called when the extraction path is a temporary - directory exclusive to a single process. This method is not - automatically called; you must call it explicitly or register it as an - ``atexit`` function if you wish to ensure cleanup of a temporary - directory used for extractions. - """ - # XXX - - - -def get_default_cache(): - """Determine the default cache location - - This returns the ``PYTHON_EGG_CACHE`` environment variable, if set. - Otherwise, on Windows, it returns a "Python-Eggs" subdirectory of the - "Application Data" directory. On all other systems, it's "~/.python-eggs". - """ - try: - return os.environ['PYTHON_EGG_CACHE'] - except KeyError: - pass - - if os.name!='nt': - return os.path.expanduser('~/.python-eggs') - - app_data = 'Application Data' # XXX this may be locale-specific! - app_homes = [ - (('APPDATA',), None), # best option, should be locale-safe - (('USERPROFILE',), app_data), - (('HOMEDRIVE','HOMEPATH'), app_data), - (('HOMEPATH',), app_data), - (('HOME',), None), - (('WINDIR',), app_data), # 95/98/ME - ] - - for keys, subdir in app_homes: - dirname = '' - for key in keys: - if key in os.environ: - dirname = os.path.join(os.environ[key]) - else: - break - else: - if subdir: - dirname = os.path.join(dirname,subdir) - return os.path.join(dirname, 'Python-Eggs') - else: - raise RuntimeError( - "Please set the PYTHON_EGG_CACHE enviroment variable" - ) - -def safe_name(name): - """Convert an arbitrary string to a standard distribution name - - Any runs of non-alphanumeric/. characters are replaced with a single '-'. - """ - return re.sub('[^A-Za-z0-9.]+', '-', name) - - -def safe_version(version): - """Convert an arbitrary string to a standard version string - - Spaces become dots, and all other non-alphanumeric characters become - dashes, with runs of multiple dashes condensed to a single dash. - """ - version = version.replace(' ','.') - return re.sub('[^A-Za-z0-9.]+', '-', version) - - -def safe_extra(extra): - """Convert an arbitrary string to a standard 'extra' name - - Any runs of non-alphanumeric characters are replaced with a single '_', - and the result is always lowercased. - """ - return re.sub('[^A-Za-z0-9.]+', '_', extra).lower() - - -def to_filename(name): - """Convert a project or version name to its filename-escaped form - - Any '-' characters are currently replaced with '_'. - """ - return name.replace('-','_') - - - - - - - - -class NullProvider: - """Try to implement resources and metadata for arbitrary PEP 302 loaders""" - - egg_name = None - egg_info = None - loader = None - - def __init__(self, module): - self.loader = getattr(module, '__loader__', None) - self.module_path = os.path.dirname(getattr(module, '__file__', '')) - - def get_resource_filename(self, manager, resource_name): - return self._fn(self.module_path, resource_name) - - def get_resource_stream(self, manager, resource_name): - return StringIO(self.get_resource_string(manager, resource_name)) - - def get_resource_string(self, manager, resource_name): - return self._get(self._fn(self.module_path, resource_name)) - - def has_resource(self, resource_name): - return self._has(self._fn(self.module_path, resource_name)) - - def has_metadata(self, name): - return self.egg_info and self._has(self._fn(self.egg_info,name)) - - def get_metadata(self, name): - if not self.egg_info: - return "" - return self._get(self._fn(self.egg_info,name)) - - def get_metadata_lines(self, name): - return yield_lines(self.get_metadata(name)) - - def resource_isdir(self,resource_name): - return self._isdir(self._fn(self.module_path, resource_name)) - - def metadata_isdir(self,name): - return self.egg_info and self._isdir(self._fn(self.egg_info,name)) - - - def resource_listdir(self,resource_name): - return self._listdir(self._fn(self.module_path,resource_name)) - - def metadata_listdir(self,name): - if self.egg_info: - return self._listdir(self._fn(self.egg_info,name)) - return [] - - def run_script(self,script_name,namespace): - script = 'scripts/'+script_name - if not self.has_metadata(script): - raise ResolutionError("No script named %r" % script_name) - script_text = self.get_metadata(script).replace('\r\n','\n') - script_text = script_text.replace('\r','\n') - script_filename = self._fn(self.egg_info,script) - namespace['__file__'] = script_filename - if os.path.exists(script_filename): - execfile(script_filename, namespace, namespace) - else: - from linecache import cache - cache[script_filename] = ( - len(script_text), 0, script_text.split('\n'), script_filename - ) - script_code = compile(script_text,script_filename,'exec') - exec script_code in namespace, namespace - - def _has(self, path): - raise NotImplementedError( - "Can't perform this operation for unregistered loader type" - ) - - def _isdir(self, path): - raise NotImplementedError( - "Can't perform this operation for unregistered loader type" - ) - - def _listdir(self, path): - raise NotImplementedError( - "Can't perform this operation for unregistered loader type" - ) - - def _fn(self, base, resource_name): - return os.path.join(base, *resource_name.split('/')) - - def _get(self, path): - if hasattr(self.loader, 'get_data'): - return self.loader.get_data(path) - raise NotImplementedError( - "Can't perform this operation for loaders without 'get_data()'" - ) - -register_loader_type(object, NullProvider) - - -class EggProvider(NullProvider): - """Provider based on a virtual filesystem""" - - def __init__(self,module): - NullProvider.__init__(self,module) - self._setup_prefix() - - def _setup_prefix(self): - # we assume here that our metadata may be nested inside a "basket" - # of multiple eggs; that's why we use module_path instead of .archive - path = self.module_path - old = None - while path!=old: - if path.lower().endswith('.egg'): - self.egg_name = os.path.basename(path) - self.egg_info = os.path.join(path, 'EGG-INFO') - self.egg_root = path - break - old = path - path, base = os.path.split(path) - - - - - - - - -class DefaultProvider(EggProvider): - """Provides access to package resources in the filesystem""" - - def _has(self, path): - return os.path.exists(path) - - def _isdir(self,path): - return os.path.isdir(path) - - def _listdir(self,path): - return os.listdir(path) - - def get_resource_stream(self, manager, resource_name): - return open(self._fn(self.module_path, resource_name), 'rb') - - def _get(self, path): - stream = open(path, 'rb') - try: - return stream.read() - finally: - stream.close() - -register_loader_type(type(None), DefaultProvider) - - -class EmptyProvider(NullProvider): - """Provider that returns nothing for all requests""" - - _isdir = _has = lambda self,path: False - _get = lambda self,path: '' - _listdir = lambda self,path: [] - module_path = None - - def __init__(self): - pass - -empty_provider = EmptyProvider() - - - - -class ZipProvider(EggProvider): - """Resource support for zips and eggs""" - - eagers = None - - def __init__(self, module): - EggProvider.__init__(self,module) - self.zipinfo = zipimport._zip_directory_cache[self.loader.archive] - self.zip_pre = self.loader.archive+os.sep - - def _zipinfo_name(self, fspath): - # Convert a virtual filename (full path to file) into a zipfile subpath - # usable with the zipimport directory cache for our target archive - if fspath.startswith(self.zip_pre): - return fspath[len(self.zip_pre):] - raise AssertionError( - "%s is not a subpath of %s" % (fspath,self.zip_pre) - ) - - def _parts(self,zip_path): - # Convert a zipfile subpath into an egg-relative path part list - fspath = self.zip_pre+zip_path # pseudo-fs path - if fspath.startswith(self.egg_root+os.sep): - return fspath[len(self.egg_root)+1:].split(os.sep) - raise AssertionError( - "%s is not a subpath of %s" % (fspath,self.egg_root) - ) - - def get_resource_filename(self, manager, resource_name): - if not self.egg_name: - raise NotImplementedError( - "resource_filename() only supported for .egg, not .zip" - ) - # no need to lock for extraction, since we use temp names - zip_path = self._resource_to_zip(resource_name) - eagers = self._get_eager_resources() - if '/'.join(self._parts(zip_path)) in eagers: - for name in eagers: - self._extract_resource(manager, self._eager_to_zip(name)) - return self._extract_resource(manager, zip_path) - - def _extract_resource(self, manager, zip_path): - - if zip_path in self._index(): - for name in self._index()[zip_path]: - last = self._extract_resource( - manager, os.path.join(zip_path, name) - ) - return os.path.dirname(last) # return the extracted directory name - - zip_stat = self.zipinfo[zip_path] - t,d,size = zip_stat[5], zip_stat[6], zip_stat[3] - date_time = ( - (d>>9)+1980, (d>>5)&0xF, d&0x1F, # ymd - (t&0xFFFF)>>11, (t>>5)&0x3F, (t&0x1F) * 2, 0, 0, -1 # hms, etc. - ) - timestamp = time.mktime(date_time) - - try: - real_path = manager.get_cache_path( - self.egg_name, self._parts(zip_path) - ) - - if os.path.isfile(real_path): - stat = os.stat(real_path) - if stat.st_size==size and stat.st_mtime==timestamp: - # size and stamp match, don't bother extracting - return real_path - - outf, tmpnam = _mkstemp(".$extract", dir=os.path.dirname(real_path)) - os.write(outf, self.loader.get_data(zip_path)) - os.close(outf) - utime(tmpnam, (timestamp,timestamp)) - manager.postprocess(tmpnam, real_path) - - try: - rename(tmpnam, real_path) - - except os.error: - if os.path.isfile(real_path): - stat = os.stat(real_path) - - if stat.st_size==size and stat.st_mtime==timestamp: - # size and stamp match, somebody did it just ahead of - # us, so we're done - return real_path - elif os.name=='nt': # Windows, del old file and retry - unlink(real_path) - rename(tmpnam, real_path) - return real_path - raise - - except os.error: - manager.extraction_error() # report a user-friendly error - - return real_path - - def _get_eager_resources(self): - if self.eagers is None: - eagers = [] - for name in ('native_libs.txt', 'eager_resources.txt'): - if self.has_metadata(name): - eagers.extend(self.get_metadata_lines(name)) - self.eagers = eagers - return self.eagers - - def _index(self): - try: - return self._dirindex - except AttributeError: - ind = {} - for path in self.zipinfo: - parts = path.split(os.sep) - while parts: - parent = os.sep.join(parts[:-1]) - if parent in ind: - ind[parent].append(parts[-1]) - break - else: - ind[parent] = [parts.pop()] - self._dirindex = ind - return ind - - def _has(self, fspath): - zip_path = self._zipinfo_name(fspath) - return zip_path in self.zipinfo or zip_path in self._index() - - def _isdir(self,fspath): - return self._zipinfo_name(fspath) in self._index() - - def _listdir(self,fspath): - return list(self._index().get(self._zipinfo_name(fspath), ())) - - def _eager_to_zip(self,resource_name): - return self._zipinfo_name(self._fn(self.egg_root,resource_name)) - - def _resource_to_zip(self,resource_name): - return self._zipinfo_name(self._fn(self.module_path,resource_name)) - -register_loader_type(zipimport.zipimporter, ZipProvider) - - - - - - - - - - - - - - - - - - - - - - - - -class FileMetadata(EmptyProvider): - """Metadata handler for standalone PKG-INFO files - - Usage:: - - metadata = FileMetadata("/path/to/PKG-INFO") - - This provider rejects all data and metadata requests except for PKG-INFO, - which is treated as existing, and will be the contents of the file at - the provided location. - """ - - def __init__(self,path): - self.path = path - - def has_metadata(self,name): - return name=='PKG-INFO' - - def get_metadata(self,name): - if name=='PKG-INFO': - return open(self.path,'rU').read() - raise KeyError("No metadata except PKG-INFO is available") - - def get_metadata_lines(self,name): - return yield_lines(self.get_metadata(name)) - - - - - - - - - - - - - - - - -class PathMetadata(DefaultProvider): - """Metadata provider for egg directories - - Usage:: - - # Development eggs: - - egg_info = "/path/to/PackageName.egg-info" - base_dir = os.path.dirname(egg_info) - metadata = PathMetadata(base_dir, egg_info) - dist_name = os.path.splitext(os.path.basename(egg_info))[0] - dist = Distribution(basedir,project_name=dist_name,metadata=metadata) - - # Unpacked egg directories: - - egg_path = "/path/to/PackageName-ver-pyver-etc.egg" - metadata = PathMetadata(egg_path, os.path.join(egg_path,'EGG-INFO')) - dist = Distribution.from_filename(egg_path, metadata=metadata) - """ - def __init__(self, path, egg_info): - self.module_path = path - self.egg_info = egg_info - - -class EggMetadata(ZipProvider): - """Metadata provider for .egg files""" - - def __init__(self, importer): - """Create a metadata provider from a zipimporter""" - - self.zipinfo = zipimport._zip_directory_cache[importer.archive] - self.zip_pre = importer.archive+os.sep - self.loader = importer - if importer.prefix: - self.module_path = os.path.join(importer.archive, importer.prefix) - else: - self.module_path = importer.archive - self._setup_prefix() - - - -_distribution_finders = {} - -def register_finder(importer_type, distribution_finder): - """Register `distribution_finder` to find distributions in sys.path items - - `importer_type` is the type or class of a PEP 302 "Importer" (sys.path item - handler), and `distribution_finder` is a callable that, passed a path - item and the importer instance, yields ``Distribution`` instances found on - that path item. See ``pkg_resources.find_on_path`` for an example.""" - _distribution_finders[importer_type] = distribution_finder - - -def find_distributions(path_item, only=False): - """Yield distributions accessible via `path_item`""" - importer = get_importer(path_item) - finder = _find_adapter(_distribution_finders, importer) - return finder(importer, path_item, only) - -def find_in_zip(importer, path_item, only=False): - metadata = EggMetadata(importer) - if metadata.has_metadata('PKG-INFO'): - yield Distribution.from_filename(path_item, metadata=metadata) - if only: - return # don't yield nested distros - for subitem in metadata.resource_listdir('/'): - if subitem.endswith('.egg'): - subpath = os.path.join(path_item, subitem) - for dist in find_in_zip(zipimport.zipimporter(subpath), subpath): - yield dist - -register_finder(zipimport.zipimporter, find_in_zip) - -def StringIO(*args, **kw): - """Thunk to load the real StringIO on demand""" - global StringIO - try: - from cStringIO import StringIO - except ImportError: - from StringIO import StringIO - return StringIO(*args,**kw) - -def find_nothing(importer, path_item, only=False): - return () -register_finder(object,find_nothing) - -def find_on_path(importer, path_item, only=False): - """Yield distributions accessible on a sys.path directory""" - path_item = _normalize_cached(path_item) - - if os.path.isdir(path_item): - if path_item.lower().endswith('.egg'): - # unpacked egg - yield Distribution.from_filename( - path_item, metadata=PathMetadata( - path_item, os.path.join(path_item,'EGG-INFO') - ) - ) - else: - # scan for .egg and .egg-info in directory - for entry in os.listdir(path_item): - lower = entry.lower() - if lower.endswith('.egg-info'): - fullpath = os.path.join(path_item, entry) - if os.path.isdir(fullpath): - # egg-info directory, allow getting metadata - metadata = PathMetadata(path_item, fullpath) - else: - metadata = FileMetadata(fullpath) - yield Distribution.from_location( - path_item,entry,metadata,precedence=DEVELOP_DIST - ) - elif not only and lower.endswith('.egg'): - for dist in find_distributions(os.path.join(path_item, entry)): - yield dist - elif not only and lower.endswith('.egg-link'): - for line in file(os.path.join(path_item, entry)): - if not line.strip(): continue - for item in find_distributions(line.rstrip()): - yield item - -register_finder(pkgutil.ImpImporter, find_on_path) - -_namespace_handlers = {} -_namespace_packages = {} - -def register_namespace_handler(importer_type, namespace_handler): - """Register `namespace_handler` to declare namespace packages - - `importer_type` is the type or class of a PEP 302 "Importer" (sys.path item - handler), and `namespace_handler` is a callable like this:: - - def namespace_handler(importer,path_entry,moduleName,module): - # return a path_entry to use for child packages - - Namespace handlers are only called if the importer object has already - agreed that it can handle the relevant path item, and they should only - return a subpath if the module __path__ does not already contain an - equivalent subpath. For an example namespace handler, see - ``pkg_resources.file_ns_handler``. - """ - _namespace_handlers[importer_type] = namespace_handler - -def _handle_ns(packageName, path_item): - """Ensure that named package includes a subpath of path_item (if needed)""" - importer = get_importer(path_item) - if importer is None: - return None - loader = importer.find_module(packageName) - if loader is None: - return None - module = sys.modules.get(packageName) - if module is None: - module = sys.modules[packageName] = new.module(packageName) - module.__path__ = []; _set_parent_ns(packageName) - elif not hasattr(module,'__path__'): - raise TypeError("Not a package:", packageName) - handler = _find_adapter(_namespace_handlers, importer) - subpath = handler(importer,path_item,packageName,module) - if subpath is not None: - path = module.__path__; path.append(subpath) - loader.load_module(packageName); module.__path__ = path - return subpath - -def declare_namespace(packageName): - """Declare that package 'packageName' is a namespace package""" - - imp.acquire_lock() - try: - if packageName in _namespace_packages: - return - - path, parent = sys.path, None - if '.' in packageName: - parent = '.'.join(packageName.split('.')[:-1]) - declare_namespace(parent) - __import__(parent) - try: - path = sys.modules[parent].__path__ - except AttributeError: - raise TypeError("Not a package:", parent) - - # Track what packages are namespaces, so when new path items are added, - # they can be updated - _namespace_packages.setdefault(parent,[]).append(packageName) - _namespace_packages.setdefault(packageName,[]) - - for path_item in path: - # Ensure all the parent's path items are reflected in the child, - # if they apply - _handle_ns(packageName, path_item) - - finally: - imp.release_lock() - -def fixup_namespace_packages(path_item, parent=None): - """Ensure that previously-declared namespace packages include path_item""" - imp.acquire_lock() - try: - for package in _namespace_packages.get(parent,()): - subpath = _handle_ns(package, path_item) - if subpath: fixup_namespace_packages(subpath,package) - finally: - imp.release_lock() - -def file_ns_handler(importer, path_item, packageName, module): - """Compute an ns-package subpath for a filesystem or zipfile importer""" - - subpath = os.path.join(path_item, packageName.split('.')[-1]) - normalized = _normalize_cached(subpath) - for item in module.__path__: - if _normalize_cached(item)==normalized: - break - else: - # Only return the path if it's not already there - return subpath - -register_namespace_handler(pkgutil.ImpImporter, file_ns_handler) -register_namespace_handler(zipimport.zipimporter, file_ns_handler) - - -def null_ns_handler(importer, path_item, packageName, module): - return None - -register_namespace_handler(object,null_ns_handler) - - -def normalize_path(filename): - """Normalize a file/dir name for comparison purposes""" - return os.path.normcase(os.path.realpath(filename)) - -def _normalize_cached(filename,_cache={}): - try: - return _cache[filename] - except KeyError: - _cache[filename] = result = normalize_path(filename) - return result - -def _set_parent_ns(packageName): - parts = packageName.split('.') - name = parts.pop() - if parts: - parent = '.'.join(parts) - setattr(sys.modules[parent], name, sys.modules[packageName]) - - -def yield_lines(strs): - """Yield non-empty/non-comment lines of a ``basestring`` or sequence""" - if isinstance(strs,basestring): - for s in strs.splitlines(): - s = s.strip() - if s and not s.startswith('#'): # skip blank lines/comments - yield s - else: - for ss in strs: - for s in yield_lines(ss): - yield s - -LINE_END = re.compile(r"\s*(#.*)?$").match # whitespace and comment -CONTINUE = re.compile(r"\s*\\\s*(#.*)?$").match # line continuation -DISTRO = re.compile(r"\s*((\w|[-.])+)").match # Distribution or extra -VERSION = re.compile(r"\s*(<=?|>=?|==|!=)\s*((\w|[-.])+)").match # ver. info -COMMA = re.compile(r"\s*,").match # comma between items -OBRACKET = re.compile(r"\s*\[").match -CBRACKET = re.compile(r"\s*\]").match -MODULE = re.compile(r"\w+(\.\w+)*$").match -EGG_NAME = re.compile( - r"(?P[^-]+)" - r"( -(?P[^-]+) (-py(?P[^-]+) (-(?P.+))? )? )?", - re.VERBOSE | re.IGNORECASE -).match - -component_re = re.compile(r'(\d+ | [a-z]+ | \.| -)', re.VERBOSE) -replace = {'pre':'c', 'preview':'c','-':'final-','rc':'c'}.get - -def _parse_version_parts(s): - for part in component_re.split(s): - part = replace(part,part) - if not part or part=='.': - continue - if part[:1] in '0123456789': - yield part.zfill(8) # pad for numeric comparison - else: - yield '*'+part - - yield '*final' # ensure that alpha/beta/candidate are before final - -def parse_version(s): - """Convert a version string to a chronologically-sortable key - - This is a rough cross between distutils' StrictVersion and LooseVersion; - if you give it versions that would work with StrictVersion, then it behaves - the same; otherwise it acts like a slightly-smarter LooseVersion. It is - *possible* to create pathological version coding schemes that will fool - this parser, but they should be very rare in practice. - - The returned value will be a tuple of strings. Numeric portions of the - version are padded to 8 digits so they will compare numerically, but - without relying on how numbers compare relative to strings. Dots are - dropped, but dashes are retained. Trailing zeros between alpha segments - or dashes are suppressed, so that e.g. "2.4.0" is considered the same as - "2.4". Alphanumeric parts are lower-cased. - - The algorithm assumes that strings like "-" and any alpha string that - alphabetically follows "final" represents a "patch level". So, "2.4-1" - is assumed to be a branch or patch of "2.4", and therefore "2.4.1" is - considered newer than "2.4-1", whic in turn is newer than "2.4". - - Strings like "a", "b", "c", "alpha", "beta", "candidate" and so on (that - come before "final" alphabetically) are assumed to be pre-release versions, - so that the version "2.4" is considered newer than "2.4a1". - - Finally, to handle miscellaneous cases, the strings "pre", "preview", and - "rc" are treated as if they were "c", i.e. as though they were release - candidates, and therefore are not as new as a version string that does not - contain them. - """ - parts = [] - for part in _parse_version_parts(s.lower()): - if part.startswith('*'): - if part<'*final': # remove '-' before a prerelease tag - while parts and parts[-1]=='*final-': parts.pop() - # remove trailing zeros from each series of numeric parts - while parts and parts[-1]=='00000000': - parts.pop() - parts.append(part) - return tuple(parts) - -class EntryPoint(object): - """Object representing an advertised importable object""" - - def __init__(self, name, module_name, attrs=(), extras=(), dist=None): - if not MODULE(module_name): - raise ValueError("Invalid module name", module_name) - self.name = name - self.module_name = module_name - self.attrs = tuple(attrs) - self.extras = Requirement.parse(("x[%s]" % ','.join(extras))).extras - self.dist = dist - - def __str__(self): - s = "%s = %s" % (self.name, self.module_name) - if self.attrs: - s += ':' + '.'.join(self.attrs) - if self.extras: - s += ' [%s]' % ','.join(self.extras) - return s - - def __repr__(self): - return "EntryPoint.parse(%r)" % str(self) - - def load(self, require=True, env=None, installer=None): - if require: self.require(env, installer) - entry = __import__(self.module_name, globals(),globals(), ['__name__']) - for attr in self.attrs: - try: - entry = getattr(entry,attr) - except AttributeError: - raise ImportError("%r has no %r attribute" % (entry,attr)) - return entry - - def require(self, env=None, installer=None): - if self.extras and not self.dist: - raise UnknownExtra("Can't require() without a distribution", self) - map(working_set.add, - working_set.resolve(self.dist.requires(self.extras),env,installer)) - - - - #@classmethod - def parse(cls, src, dist=None): - """Parse a single entry point from string `src` - - Entry point syntax follows the form:: - - name = some.module:some.attr [extra1,extra2] - - The entry name and module name are required, but the ``:attrs`` and - ``[extras]`` parts are optional - """ - try: - attrs = extras = () - name,value = src.split('=',1) - if '[' in value: - value,extras = value.split('[',1) - req = Requirement.parse("x["+extras) - if req.specs: raise ValueError - extras = req.extras - if ':' in value: - value,attrs = value.split(':',1) - if not MODULE(attrs.rstrip()): - raise ValueError - attrs = attrs.rstrip().split('.') - except ValueError: - raise ValueError( - "EntryPoint must be in 'name=module:attrs [extras]' format", - src - ) - else: - return cls(name.strip(), value.lstrip(), attrs, extras, dist) - - parse = classmethod(parse) - - - - - - - - - #@classmethod - def parse_group(cls, group, lines, dist=None): - """Parse an entry point group""" - if not MODULE(group): - raise ValueError("Invalid group name", group) - this = {} - for line in yield_lines(lines): - ep = cls.parse(line, dist) - if ep.name in this: - raise ValueError("Duplicate entry point", group, ep.name) - this[ep.name]=ep - return this - - parse_group = classmethod(parse_group) - - #@classmethod - def parse_map(cls, data, dist=None): - """Parse a map of entry point groups""" - if isinstance(data,dict): - data = data.items() - else: - data = split_sections(data) - maps = {} - for group, lines in data: - if group is None: - if not lines: - continue - raise ValueError("Entry points must be listed in groups") - group = group.strip() - if group in maps: - raise ValueError("Duplicate group name", group) - maps[group] = cls.parse_group(group, lines, dist) - return maps - - parse_map = classmethod(parse_map) - - - - - - -class Distribution(object): - """Wrap an actual or potential sys.path entry w/metadata""" - def __init__(self, - location=None, metadata=None, project_name=None, version=None, - py_version=PY_MAJOR, platform=None, precedence = EGG_DIST - ): - self.project_name = safe_name(project_name or 'Unknown') - if version is not None: - self._version = safe_version(version) - self.py_version = py_version - self.platform = platform - self.location = location - self.precedence = precedence - self._provider = metadata or empty_provider - - #@classmethod - def from_location(cls,location,basename,metadata=None,**kw): - project_name, version, py_version, platform = [None]*4 - basename, ext = os.path.splitext(basename) - if ext.lower() in (".egg",".egg-info"): - match = EGG_NAME(basename) - if match: - project_name, version, py_version, platform = match.group( - 'name','ver','pyver','plat' - ) - return cls( - location, metadata, project_name=project_name, version=version, - py_version=py_version, platform=platform, **kw - ) - from_location = classmethod(from_location) - - hashcmp = property( - lambda self: ( - getattr(self,'parsed_version',()), self.precedence, self.key, - -len(self.location or ''), self.location, self.py_version, - self.platform - ) - ) - def __cmp__(self, other): return cmp(self.hashcmp, other) - def __hash__(self): return hash(self.hashcmp) - - # These properties have to be lazy so that we don't have to load any - # metadata until/unless it's actually needed. (i.e., some distributions - # may not know their name or version without loading PKG-INFO) - - #@property - def key(self): - try: - return self._key - except AttributeError: - self._key = key = self.project_name.lower() - return key - key = property(key) - - #@property - def parsed_version(self): - try: - return self._parsed_version - except AttributeError: - self._parsed_version = pv = parse_version(self.version) - return pv - - parsed_version = property(parsed_version) - - #@property - def version(self): - try: - return self._version - except AttributeError: - for line in self._get_metadata('PKG-INFO'): - if line.lower().startswith('version:'): - self._version = safe_version(line.split(':',1)[1].strip()) - return self._version - else: - raise ValueError( - "Missing 'Version:' header and/or PKG-INFO file", self - ) - version = property(version) - - - - - #@property - def _dep_map(self): - try: - return self.__dep_map - except AttributeError: - dm = self.__dep_map = {None: []} - for name in 'requires.txt', 'depends.txt': - for extra,reqs in split_sections(self._get_metadata(name)): - if extra: extra = safe_extra(extra) - dm.setdefault(extra,[]).extend(parse_requirements(reqs)) - return dm - _dep_map = property(_dep_map) - - def requires(self,extras=()): - """List of Requirements needed for this distro if `extras` are used""" - dm = self._dep_map - deps = [] - deps.extend(dm.get(None,())) - for ext in extras: - try: - deps.extend(dm[safe_extra(ext)]) - except KeyError: - raise UnknownExtra( - "%s has no such extra feature %r" % (self, ext) - ) - return deps - - def _get_metadata(self,name): - if self.has_metadata(name): - for line in self.get_metadata_lines(name): - yield line - - def activate(self,path=None): - """Ensure distribution is importable on `path` (default=sys.path)""" - if path is None: path = sys.path - self.insert_on(path) - if path is sys.path: - fixup_namespace_packages(self.location) - for pkg in self._get_metadata('namespace_packages.txt'): - if pkg in sys.modules: declare_namespace(pkg) - - def egg_name(self): - """Return what this distribution's standard .egg filename should be""" - filename = "%s-%s-py%s" % ( - to_filename(self.project_name), to_filename(self.version), - self.py_version or PY_MAJOR - ) - - if self.platform: - filename += '-'+self.platform - return filename - - def __repr__(self): - if self.location: - return "%s (%s)" % (self,self.location) - else: - return str(self) - - def __str__(self): - try: version = getattr(self,'version',None) - except ValueError: version = None - version = version or "[unknown version]" - return "%s %s" % (self.project_name,version) - - def __getattr__(self,attr): - """Delegate all unrecognized public attributes to .metadata provider""" - if attr.startswith('_'): - raise AttributeError,attr - return getattr(self._provider, attr) - - #@classmethod - def from_filename(cls,filename,metadata=None, **kw): - return cls.from_location( - _normalize_cached(filename), os.path.basename(filename), metadata, - **kw - ) - from_filename = classmethod(from_filename) - - def as_requirement(self): - """Return a ``Requirement`` that matches this distribution exactly""" - return Requirement.parse('%s==%s' % (self.project_name, self.version)) - - def load_entry_point(self, group, name): - """Return the `name` entry point of `group` or raise ImportError""" - ep = self.get_entry_info(group,name) - if ep is None: - raise ImportError("Entry point %r not found" % ((group,name),)) - return ep.load() - - def get_entry_map(self, group=None): - """Return the entry point map for `group`, or the full entry map""" - try: - ep_map = self._ep_map - except AttributeError: - ep_map = self._ep_map = EntryPoint.parse_map( - self._get_metadata('entry_points.txt'), self - ) - if group is not None: - return ep_map.get(group,{}) - return ep_map - - def get_entry_info(self, group, name): - """Return the EntryPoint object for `group`+`name`, or ``None``""" - return self.get_entry_map(group).get(name) - - def insert_on(self, path, loc = None): - """Insert self.location in path before its nearest parent directory""" - loc = loc or self.location - if not loc: return - if path is sys.path: - self.check_version_conflict() - best, pos = 0, -1 - for p,item in enumerate(path): - item = _normalize_cached(item) - if loc.startswith(item) and len(item)>best and loc<>item: - best, pos = len(item), p - if pos==-1: - if loc not in path: path.append(loc) - elif loc not in path[:pos+1]: - while loc in path: path.remove(loc) - path.insert(pos,loc) - - - def check_version_conflict(self): - if self.key=='setuptools': - return # ignore the inevitable setuptools self-conflicts :( - - nsp = dict.fromkeys(self._get_metadata('namespace_packages.txt')) - loc = normalize_path(self.location) - for modname in self._get_metadata('top_level.txt'): - if (modname not in sys.modules or modname in nsp - or modname in _namespace_packages - ): - continue - - fn = getattr(sys.modules[modname], '__file__', None) - if fn and normalize_path(fn).startswith(loc): - continue - issue_warning( - "Module %s was already imported from %s, but %s is being added" - " to sys.path" % (modname, fn, self.location), - ) - - def has_version(self): - try: - self.version - except ValueError: - issue_warning("Unbuilt egg for "+repr(self)) - return False - return True - - def clone(self,**kw): - """Copy this distribution, substituting in any changed keyword args""" - for attr in ( - 'project_name', 'version', 'py_version', 'platform', 'location', - 'precedence' - ): - kw.setdefault(attr, getattr(self,attr,None)) - kw.setdefault('metadata', self._provider) - return self.__class__(**kw) - - - - - #@property - def extras(self): - return [dep for dep in self._dep_map if dep] - extras = property(extras) - - -def issue_warning(*args,**kw): - level = 1 - g = globals() - try: - # find the first stack frame that is *not* code in - # the pkg_resources module, to use for the warning - while sys._getframe(level).f_globals is g: - level += 1 - except ValueError: - pass - from warnings import warn - warn(stacklevel = level+1, *args, **kw) - - - - - - - - - - - - - - - - - - - - - - - -def parse_requirements(strs): - """Yield ``Requirement`` objects for each specification in `strs` - - `strs` must be an instance of ``basestring``, or a (possibly-nested) - iterable thereof. - """ - # create a steppable iterator, so we can handle \-continuations - lines = iter(yield_lines(strs)) - - def scan_list(ITEM,TERMINATOR,line,p,groups,item_name): - - items = [] - - while not TERMINATOR(line,p): - if CONTINUE(line,p): - try: - line = lines.next(); p = 0 - except StopIteration: - raise ValueError( - "\\ must not appear on the last nonblank line" - ) - - match = ITEM(line,p) - if not match: - raise ValueError("Expected "+item_name+" in",line,"at",line[p:]) - - items.append(match.group(*groups)) - p = match.end() - - match = COMMA(line,p) - if match: - p = match.end() # skip the comma - elif not TERMINATOR(line,p): - raise ValueError( - "Expected ',' or end-of-list in",line,"at",line[p:] - ) - - match = TERMINATOR(line,p) - if match: p = match.end() # skip the terminator, if any - return line, p, items - - for line in lines: - match = DISTRO(line) - if not match: - raise ValueError("Missing distribution spec", line) - project_name = match.group(1) - p = match.end() - extras = [] - - match = OBRACKET(line,p) - if match: - p = match.end() - line, p, extras = scan_list( - DISTRO, CBRACKET, line, p, (1,), "'extra' name" - ) - - line, p, specs = scan_list(VERSION,LINE_END,line,p,(1,2),"version spec") - specs = [(op,safe_version(val)) for op,val in specs] - yield Requirement(project_name, specs, extras) - - -def _sort_dists(dists): - tmp = [(dist.hashcmp,dist) for dist in dists] - tmp.sort() - dists[::-1] = [d for hc,d in tmp] - - - - - - - - - - - - - - - - - -class Requirement: - def __init__(self, project_name, specs, extras): - """DO NOT CALL THIS UNDOCUMENTED METHOD; use Requirement.parse()!""" - self.unsafe_name, project_name = project_name, safe_name(project_name) - self.project_name, self.key = project_name, project_name.lower() - index = [(parse_version(v),state_machine[op],op,v) for op,v in specs] - index.sort() - self.specs = [(op,ver) for parsed,trans,op,ver in index] - self.index, self.extras = index, tuple(map(safe_extra,extras)) - self.hashCmp = ( - self.key, tuple([(op,parsed) for parsed,trans,op,ver in index]), - ImmutableSet(self.extras) - ) - self.__hash = hash(self.hashCmp) - - def __str__(self): - specs = ','.join([''.join(s) for s in self.specs]) - extras = ','.join(self.extras) - if extras: extras = '[%s]' % extras - return '%s%s%s' % (self.project_name, extras, specs) - - def __eq__(self,other): - return isinstance(other,Requirement) and self.hashCmp==other.hashCmp - - def __contains__(self,item): - if isinstance(item,Distribution): - if item.key <> self.key: return False - if self.index: item = item.parsed_version # only get if we need it - elif isinstance(item,basestring): - item = parse_version(item) - last = None - for parsed,trans,op,ver in self.index: - action = trans[cmp(item,parsed)] - if action=='F': return False - elif action=='T': return True - elif action=='+': last = True - elif action=='-' or last is None: last = False - if last is None: last = True # no rules encountered - return last - - - def __hash__(self): - return self.__hash - - def __repr__(self): return "Requirement.parse(%r)" % str(self) - - #@staticmethod - def parse(s): - reqs = list(parse_requirements(s)) - if reqs: - if len(reqs)==1: - return reqs[0] - raise ValueError("Expected only one requirement", s) - raise ValueError("No requirements found", s) - - parse = staticmethod(parse) - -state_machine = { - # =>< - '<' : '--T', - '<=': 'T-T', - '>' : 'F+F', - '>=': 'T+F', - '==': 'T..', - '!=': 'F++', -} - - -def _get_mro(cls): - """Get an mro for a type or classic class""" - if not isinstance(cls,type): - class cls(cls,object): pass - return cls.__mro__[1:] - return cls.__mro__ - -def _find_adapter(registry, ob): - """Return an adapter factory for `ob` from `registry`""" - for t in _get_mro(getattr(ob, '__class__', type(ob))): - if t in registry: - return registry[t] - - -def ensure_directory(path): - """Ensure that the parent directory of `path` exists""" - dirname = os.path.dirname(path) - if not os.path.isdir(dirname): - os.makedirs(dirname) - -def split_sections(s): - """Split a string or iterable thereof into (section,content) pairs - - Each ``section`` is a stripped version of the section header ("[section]") - and each ``content`` is a list of stripped lines excluding blank lines and - comment-only lines. If there are any such lines before the first section - header, they're returned in a first ``section`` of ``None``. - """ - section = None - content = [] - for line in yield_lines(s): - if line.startswith("["): - if line.endswith("]"): - if section or content: - yield section, content - section = line[1:-1].strip() - content = [] - else: - raise ValueError("Invalid section heading", line) - else: - content.append(line) - - # wrap up last segment - yield section, content - -def _mkstemp(*args,**kw): - from tempfile import mkstemp - old_open = os.open - try: - os.open = os_open # temporarily bypass sandboxing - return mkstemp(*args,**kw) - finally: - os.open = old_open # and then put it back - - -# Set up global resource manager -_manager = ResourceManager() -def _initialize(g): - for name in dir(_manager): - if not name.startswith('_'): - g[name] = getattr(_manager, name) -_initialize(globals()) - -# Prepare the master working set and make the ``require()`` API available -working_set = WorkingSet() -try: - # Does the main program list any requirements? - from __main__ import __requires__ -except ImportError: - pass # No: just use the default working set based on sys.path -else: - # Yes: ensure the requirements are met, by prefixing sys.path if necessary - try: - working_set.require(__requires__) - except VersionConflict: # try it without defaults already on sys.path - working_set = WorkingSet([]) # by starting with an empty path - for dist in working_set.resolve( - parse_requirements(__requires__), Environment() - ): - working_set.add(dist) - for entry in sys.path: # add any missing entries from sys.path - if entry not in working_set.entries: - working_set.add_entry(entry) - sys.path[:] = working_set.entries # then copy back to sys.path - -require = working_set.require -iter_entry_points = working_set.iter_entry_points -add_activation_listener = working_set.subscribe -run_script = working_set.run_script -run_main = run_script # backward compatibility -# Activate all distributions already on sys.path, and ensure that -# all distributions added to the working set in the future (e.g. by -# calling ``require()``) will get activated as well. -add_activation_listener(lambda dist: dist.activate()) -working_set.entries=[]; map(working_set.add_entry,sys.path) # match order Deleted: /python/trunk/Lib/test/test_setuptools.py ============================================================================== --- /python/trunk/Lib/test/test_setuptools.py Mon Apr 24 22:53:13 2006 +++ (empty file) @@ -1,16 +0,0 @@ -"""Tests for setuptools. - -The tests for setuptools are defined in the setuptools.tests package; -this runs them from there. -""" - -import test.test_support -from setuptools.command.test import ScanningLoader - -def test_main(): - test.test_support.run_suite( - ScanningLoader().loadTestsFromName('setuptools.tests') - ) - -if __name__ == "__main__": - test_main() Modified: python/trunk/Misc/NEWS ============================================================================== --- python/trunk/Misc/NEWS (original) +++ python/trunk/Misc/NEWS Mon Apr 24 22:53:13 2006 @@ -97,9 +97,6 @@ - Bug #1473760: ``tempfile.TemporaryFile()`` could hang on Windows, when called from a thread spawned as a side effect of importing a module. -- New modules: setuptools, easy_install, and pkg_resources, to support - building, installing, and using Python eggs, respectively. - - The pydoc module now supports documenting packages contained in .zip or .egg files. From buildbot at python.org Mon Apr 24 23:37:40 2006 From: buildbot at python.org (buildbot at python.org) Date: Mon, 24 Apr 2006 21:37:40 +0000 Subject: [Python-checkins] buildbot warnings in hppa Ubuntu dapper trunk Message-ID: <20060424213740.98E121E4010@bag.python.org> The Buildbot has detected a new failure of hppa Ubuntu dapper trunk. Full details are available at: http://www.python.org/dev/buildbot/all/hppa%2520Ubuntu%2520dapper%2520trunk/builds/281 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: phillip.eby Build Had Warnings: warnings test sincerely, -The Buildbot From python-checkins at python.org Tue Apr 25 00:45:14 2006 From: python-checkins at python.org (tim.peters) Date: Tue, 25 Apr 2006 00:45:14 +0200 (CEST) Subject: [Python-checkins] r45698 - python/trunk/Lib/trace.py Message-ID: <20060424224514.6978D1E400A@bag.python.org> Author: tim.peters Date: Tue Apr 25 00:45:13 2006 New Revision: 45698 Modified: python/trunk/Lib/trace.py Log: Whitespace normalization. Modified: python/trunk/Lib/trace.py ============================================================================== --- python/trunk/Lib/trace.py (original) +++ python/trunk/Lib/trace.py Tue Apr 25 00:45:13 2006 @@ -40,7 +40,7 @@ # create a Trace object, telling it what to ignore, and whether to # do tracing or line-counting or both. tracer = trace.Trace(ignoredirs=[sys.prefix, sys.exec_prefix,], trace=0, - count=1) + count=1) # run the new command using the given tracer tracer.run('main()') # make a report, placing output in /tmp From buildbot at python.org Tue Apr 25 01:52:55 2006 From: buildbot at python.org (buildbot at python.org) Date: Mon, 24 Apr 2006 23:52:55 +0000 Subject: [Python-checkins] buildbot warnings in alpha Debian trunk Message-ID: <20060424235255.D69981E400B@bag.python.org> The Buildbot has detected a new failure of alpha Debian trunk. Full details are available at: http://www.python.org/dev/buildbot/all/alpha%2520Debian%2520trunk/builds/7 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: tim.peters Build Had Warnings: warnings test sincerely, -The Buildbot From python-checkins at python.org Tue Apr 25 02:34:36 2006 From: python-checkins at python.org (trent.mick) Date: Tue, 25 Apr 2006 02:34:36 +0200 (CEST) Subject: [Python-checkins] r45699 - python/branches/release24-maint/Lib/distutils/msvccompiler.py Message-ID: <20060425003436.085D11E400A@bag.python.org> Author: trent.mick Date: Tue Apr 25 02:34:35 2006 New Revision: 45699 Modified: python/branches/release24-maint/Lib/distutils/msvccompiler.py Log: Put break at correct level so *all* root HKEYs acutally get checked for an installed VC6. Otherwise only the first such tree gets checked and this warning doesn't get displayed. Modified: python/branches/release24-maint/Lib/distutils/msvccompiler.py ============================================================================== --- python/branches/release24-maint/Lib/distutils/msvccompiler.py (original) +++ python/branches/release24-maint/Lib/distutils/msvccompiler.py Tue Apr 25 02:34:35 2006 @@ -580,7 +580,7 @@ "but the expected registry settings are not present.\n" "You must at least run the Visual Studio GUI once " "so that these entries are created.") - break + break return [] def set_path_env_var(self, name): From python-checkins at python.org Tue Apr 25 02:34:50 2006 From: python-checkins at python.org (trent.mick) Date: Tue, 25 Apr 2006 02:34:50 +0200 (CEST) Subject: [Python-checkins] r45700 - python/trunk/Lib/distutils/msvccompiler.py Message-ID: <20060425003450.AB39D1E400A@bag.python.org> Author: trent.mick Date: Tue Apr 25 02:34:50 2006 New Revision: 45700 Modified: python/trunk/Lib/distutils/msvccompiler.py Log: Put break at correct level so *all* root HKEYs acutally get checked for an installed VC6. Otherwise only the first such tree gets checked and this warning doesn't get displayed. Modified: python/trunk/Lib/distutils/msvccompiler.py ============================================================================== --- python/trunk/Lib/distutils/msvccompiler.py (original) +++ python/trunk/Lib/distutils/msvccompiler.py Tue Apr 25 02:34:50 2006 @@ -618,7 +618,7 @@ "but the expected registry settings are not present.\n" "You must at least run the Visual Studio GUI once " "so that these entries are created.") - break + break return [] def set_path_env_var(self, name): From buildbot at python.org Tue Apr 25 02:55:09 2006 From: buildbot at python.org (buildbot at python.org) Date: Tue, 25 Apr 2006 00:55:09 +0000 Subject: [Python-checkins] buildbot warnings in x86 OpenBSD 2.4 Message-ID: <20060425005510.119681E400A@bag.python.org> The Buildbot has detected a new failure of x86 OpenBSD 2.4. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520OpenBSD%25202.4/builds/74 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch branches/release24-maint] HEAD Blamelist: trent.mick Build Had Warnings: warnings test sincerely, -The Buildbot From buildbot at python.org Tue Apr 25 02:55:38 2006 From: buildbot at python.org (buildbot at python.org) Date: Tue, 25 Apr 2006 00:55:38 +0000 Subject: [Python-checkins] buildbot failure in x86 XP-2 trunk Message-ID: <20060425005538.6232B1E400A@bag.python.org> The Buildbot has detected a new failure of x86 XP-2 trunk. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520XP-2%2520trunk/builds/366 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: trent.mick BUILD FAILED: failed failed slave lost sincerely, -The Buildbot From buildbot at python.org Tue Apr 25 04:03:11 2006 From: buildbot at python.org (buildbot at python.org) Date: Tue, 25 Apr 2006 02:03:11 +0000 Subject: [Python-checkins] buildbot warnings in hppa Ubuntu dapper 2.4 Message-ID: <20060425020311.77ACB1E400A@bag.python.org> The Buildbot has detected a new failure of hppa Ubuntu dapper 2.4. Full details are available at: http://www.python.org/dev/buildbot/all/hppa%2520Ubuntu%2520dapper%25202.4/builds/38 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch branches/release24-maint] HEAD Blamelist: trent.mick Build Had Warnings: warnings test sincerely, -The Buildbot From python-checkins at python.org Tue Apr 25 05:31:37 2006 From: python-checkins at python.org (tim.peters) Date: Tue, 25 Apr 2006 05:31:37 +0200 (CEST) Subject: [Python-checkins] r45701 - in python/trunk: Doc/lib/libdoctest.tex Lib/doctest.py Lib/test/test_doctest.py Misc/NEWS Message-ID: <20060425033137.D0AA81E400A@bag.python.org> Author: tim.peters Date: Tue Apr 25 05:31:36 2006 New Revision: 45701 Modified: python/trunk/Doc/lib/libdoctest.tex python/trunk/Lib/doctest.py python/trunk/Lib/test/test_doctest.py python/trunk/Misc/NEWS Log: Patch #1475231: add a new SKIP doctest option, thanks to Edward Loper. Modified: python/trunk/Doc/lib/libdoctest.tex ============================================================================== --- python/trunk/Doc/lib/libdoctest.tex (original) +++ python/trunk/Doc/lib/libdoctest.tex Tue Apr 25 05:31:36 2006 @@ -616,6 +616,20 @@ \end{datadesc} +\begin{datadesc}{SKIP} + + When specified, do not run the example at all. This can be useful + in contexts where doctest examples serve as both documentation and + test cases, and an example should be included for documentation + purposes, but should not be checked. E.g., the example's output + might be random; or the example might depend on resources which + would be unavailable to the test driver. + + The SKIP flag can also be used for temporarily "commenting out" + examples. + +\end{datadesc} + \begin{datadesc}{COMPARISON_FLAGS} A bitmask or'ing together all the comparison flags above. \end{datadesc} @@ -744,6 +758,7 @@ were added; by default \code{} in expected output matches an empty line in actual output; and doctest directives were added]{2.4} +\versionchanged[Constant \constant{SKIP} was added]{2.5} There's also a way to register new option flag names, although this isn't useful unless you intend to extend \refmodule{doctest} internals Modified: python/trunk/Lib/doctest.py ============================================================================== --- python/trunk/Lib/doctest.py (original) +++ python/trunk/Lib/doctest.py Tue Apr 25 05:31:36 2006 @@ -54,6 +54,7 @@ 'DONT_ACCEPT_BLANKLINE', 'NORMALIZE_WHITESPACE', 'ELLIPSIS', + 'SKIP', 'IGNORE_EXCEPTION_DETAIL', 'COMPARISON_FLAGS', 'REPORT_UDIFF', @@ -136,12 +137,14 @@ DONT_ACCEPT_BLANKLINE = register_optionflag('DONT_ACCEPT_BLANKLINE') NORMALIZE_WHITESPACE = register_optionflag('NORMALIZE_WHITESPACE') ELLIPSIS = register_optionflag('ELLIPSIS') +SKIP = register_optionflag('SKIP') IGNORE_EXCEPTION_DETAIL = register_optionflag('IGNORE_EXCEPTION_DETAIL') COMPARISON_FLAGS = (DONT_ACCEPT_TRUE_FOR_1 | DONT_ACCEPT_BLANKLINE | NORMALIZE_WHITESPACE | ELLIPSIS | + SKIP | IGNORE_EXCEPTION_DETAIL) REPORT_UDIFF = register_optionflag('REPORT_UDIFF') @@ -1233,6 +1236,10 @@ else: self.optionflags &= ~optionflag + # If 'SKIP' is set, then skip this example. + if self.optionflags & SKIP: + continue + # Record that we started this example. tries += 1 if not quiet: @@ -1792,6 +1799,7 @@ DONT_ACCEPT_BLANKLINE NORMALIZE_WHITESPACE ELLIPSIS + SKIP IGNORE_EXCEPTION_DETAIL REPORT_UDIFF REPORT_CDIFF @@ -1914,6 +1922,7 @@ DONT_ACCEPT_BLANKLINE NORMALIZE_WHITESPACE ELLIPSIS + SKIP IGNORE_EXCEPTION_DETAIL REPORT_UDIFF REPORT_CDIFF Modified: python/trunk/Lib/test/test_doctest.py ============================================================================== --- python/trunk/Lib/test/test_doctest.py (original) +++ python/trunk/Lib/test/test_doctest.py Tue Apr 25 05:31:36 2006 @@ -1079,6 +1079,25 @@ ... # doctest: +NORMALIZE_WHITESPACE [0, 1, ..., 18, 19] +The SKIP flag causes an example to be skipped entirely. I.e., the +example is not run. It can be useful in contexts where doctest +examples serve as both documentation and test cases, and an example +should be included for documentation purposes, but should not be +checked (e.g., because its output is random, or depends on resources +which would be unavailable.) The SKIP flag can also be used for +'commenting out' broken examples. + + >>> import unavailable_resource # doctest: +SKIP + >>> unavailable_resource.do_something() # doctest: +SKIP + >>> unavailable_resource.blow_up() # doctest: +SKIP + Traceback (most recent call last): + ... + UncheckedBlowUpError: Nobody checks me. + + >>> import random + >>> print random.random() # doctest: +SKIP + 0.721216923889 + The REPORT_UDIFF flag causes failures that involve multi-line expected and actual outputs to be displayed using a unified diff: Modified: python/trunk/Misc/NEWS ============================================================================== --- python/trunk/Misc/NEWS (original) +++ python/trunk/Misc/NEWS Tue Apr 25 05:31:36 2006 @@ -86,6 +86,10 @@ Library ------- +- Patch #1475231: ``doctest`` has a new ``SKIP`` option, which causes + a doctest to be skipped (the code is not run, and the expected output + or exception is ignored). + - Fixed contextlib.nested to cope with exceptions being raised and caught inside exit handlers. From buildbot at python.org Tue Apr 25 05:41:38 2006 From: buildbot at python.org (buildbot at python.org) Date: Tue, 25 Apr 2006 03:41:38 +0000 Subject: [Python-checkins] buildbot warnings in x86 XP trunk Message-ID: <20060425034138.9054B1E400A@bag.python.org> The Buildbot has detected a new failure of x86 XP trunk. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520XP%2520trunk/builds/564 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: tim.peters Build Had Warnings: warnings failed slave lost sincerely, -The Buildbot From nnorwitz at gmail.com Tue Apr 25 06:15:54 2006 From: nnorwitz at gmail.com (Neal Norwitz) Date: Mon, 24 Apr 2006 21:15:54 -0700 Subject: [Python-checkins] buildbot warnings in alpha Debian trunk In-Reply-To: <444C86C1.1040806@gmail.com> References: <20060424072301.20B911E400C@bag.python.org> <444C86C1.1040806@gmail.com> Message-ID: On 4/24/06, Nick Coghlan wrote: > buildbot at python.org wrote: > > > > Build Reason: > > Build Source Stamp: [branch trunk] HEAD > > Blamelist: nick.coghlan > > Tweren't me, yer Honour! > > This particular failure was due to test_compiler tripping over the 30 minute > time limit without producing any output. The sparc Ubuntu dapper run failed > for the same reason. > > Wasn't someone saying something about this problem recently? I checked in a change that printed out a message after each 5 minutes to prevent the problem you describe. So the only way it should occur if a single test around the loop takes at least 15 minutes. That seems unlikely, but who knows. I didn't look at this. If it continues to be a problem, I'll check it out. n From python-checkins at python.org Tue Apr 25 07:04:35 2006 From: python-checkins at python.org (neal.norwitz) Date: Tue, 25 Apr 2006 07:04:35 +0200 (CEST) Subject: [Python-checkins] r45702 - python/trunk/Doc/lib/libdoctest.tex Message-ID: <20060425050435.DB08A1E400B@bag.python.org> Author: neal.norwitz Date: Tue Apr 25 07:04:35 2006 New Revision: 45702 Modified: python/trunk/Doc/lib/libdoctest.tex Log: versionadded for SKIP Modified: python/trunk/Doc/lib/libdoctest.tex ============================================================================== --- python/trunk/Doc/lib/libdoctest.tex (original) +++ python/trunk/Doc/lib/libdoctest.tex Tue Apr 25 07:04:35 2006 @@ -628,6 +628,7 @@ The SKIP flag can also be used for temporarily "commenting out" examples. + \versionadded{2.5} \end{datadesc} \begin{datadesc}{COMPARISON_FLAGS} From python-checkins at python.org Tue Apr 25 07:05:03 2006 From: python-checkins at python.org (neal.norwitz) Date: Tue, 25 Apr 2006 07:05:03 +0200 (CEST) Subject: [Python-checkins] r45703 - python/trunk/Misc/NEWS Message-ID: <20060425050503.7F2781E400B@bag.python.org> Author: neal.norwitz Date: Tue Apr 25 07:05:03 2006 New Revision: 45703 Modified: python/trunk/Misc/NEWS Log: Restore Walters name Modified: python/trunk/Misc/NEWS ============================================================================== --- python/trunk/Misc/NEWS (original) +++ python/trunk/Misc/NEWS Tue Apr 25 07:05:03 2006 @@ -428,7 +428,7 @@ subclasses of str always behaved. int/long/float, conversion of an instance to the base class has been moved to the proper nb_* magic slot and out of PyNumber_*(). - Thanks Walter D?wald. + Thanks Walter D?rwald. - Descriptors defined in C with a PyGetSetDef structure, where the setter is NULL, now raise an AttributeError when attempting to set or delete the From buildbot at python.org Tue Apr 25 07:17:05 2006 From: buildbot at python.org (buildbot at python.org) Date: Tue, 25 Apr 2006 05:17:05 +0000 Subject: [Python-checkins] buildbot warnings in x86 XP-2 trunk Message-ID: <20060425051705.E277B1E400B@bag.python.org> The Buildbot has detected a new failure of x86 XP-2 trunk. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520XP-2%2520trunk/builds/369 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: neal.norwitz Build Had Warnings: warnings failed slave lost sincerely, -The Buildbot From tim.peters at gmail.com Tue Apr 25 07:45:07 2006 From: tim.peters at gmail.com (Tim Peters) Date: Tue, 25 Apr 2006 01:45:07 -0400 Subject: [Python-checkins] r45702 - python/trunk/Doc/lib/libdoctest.tex In-Reply-To: <20060425050435.DB08A1E400B@bag.python.org> References: <20060425050435.DB08A1E400B@bag.python.org> Message-ID: <1f7befae0604242245g7cc6b80eg16656ffa0216c644@mail.gmail.com> [neal.norwitz] > Modified: > python/trunk/Doc/lib/libdoctest.tex > Log: > versionadded for SKIP > > Modified: python/trunk/Doc/lib/libdoctest.tex > > --- python/trunk/Doc/lib/libdoctest.tex (original) > +++ python/trunk/Doc/lib/libdoctest.tex Tue Apr 25 07:04:35 2006 > @@ -628,6 +628,7 @@ > The SKIP flag can also be used for temporarily "commenting out" > examples. > > + \versionadded{2.5} There was already a \versionchanged blurb for SKIP in the original patch, down with the other \versionchanged blurbs for the other doctest options. From nnorwitz at gmail.com Tue Apr 25 07:49:08 2006 From: nnorwitz at gmail.com (Neal Norwitz) Date: Mon, 24 Apr 2006 22:49:08 -0700 Subject: [Python-checkins] r45702 - python/trunk/Doc/lib/libdoctest.tex In-Reply-To: <1f7befae0604242245g7cc6b80eg16656ffa0216c644@mail.gmail.com> References: <20060425050435.DB08A1E400B@bag.python.org> <1f7befae0604242245g7cc6b80eg16656ffa0216c644@mail.gmail.com> Message-ID: Ah, I missed it. I'll revert. On 4/24/06, Tim Peters wrote: > [neal.norwitz] > > Modified: > > python/trunk/Doc/lib/libdoctest.tex > > Log: > > versionadded for SKIP > > > > Modified: python/trunk/Doc/lib/libdoctest.tex > > > --- python/trunk/Doc/lib/libdoctest.tex (original) > > +++ python/trunk/Doc/lib/libdoctest.tex Tue Apr 25 07:04:35 2006 > > @@ -628,6 +628,7 @@ > > The SKIP flag can also be used for temporarily "commenting out" > > examples. > > > > + \versionadded{2.5} > > There was already a \versionchanged blurb for SKIP in the original > patch, down with the other \versionchanged blurbs for the other > doctest options. > _______________________________________________ > Python-checkins mailing list > Python-checkins at python.org > http://mail.python.org/mailman/listinfo/python-checkins > From python-checkins at python.org Tue Apr 25 07:49:44 2006 From: python-checkins at python.org (neal.norwitz) Date: Tue, 25 Apr 2006 07:49:44 +0200 (CEST) Subject: [Python-checkins] r45704 - python/trunk/Doc/lib/libdoctest.tex Message-ID: <20060425054944.13C7D1E4004@bag.python.org> Author: neal.norwitz Date: Tue Apr 25 07:49:42 2006 New Revision: 45704 Modified: python/trunk/Doc/lib/libdoctest.tex Log: Revert previous change, SKIP had a versionadded elsewhere Modified: python/trunk/Doc/lib/libdoctest.tex ============================================================================== --- python/trunk/Doc/lib/libdoctest.tex (original) +++ python/trunk/Doc/lib/libdoctest.tex Tue Apr 25 07:49:42 2006 @@ -628,7 +628,6 @@ The SKIP flag can also be used for temporarily "commenting out" examples. - \versionadded{2.5} \end{datadesc} \begin{datadesc}{COMPARISON_FLAGS} From buildbot at python.org Tue Apr 25 08:01:39 2006 From: buildbot at python.org (buildbot at python.org) Date: Tue, 25 Apr 2006 06:01:39 +0000 Subject: [Python-checkins] buildbot warnings in sparc Ubuntu dapper trunk Message-ID: <20060425060139.394FD1E4004@bag.python.org> The Buildbot has detected a new failure of sparc Ubuntu dapper trunk. Full details are available at: http://www.python.org/dev/buildbot/all/sparc%2520Ubuntu%2520dapper%2520trunk/builds/153 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: trent.mick Build Had Warnings: warnings test sincerely, -The Buildbot From python-checkins at python.org Tue Apr 25 08:19:09 2006 From: python-checkins at python.org (neal.norwitz) Date: Tue, 25 Apr 2006 08:19:09 +0200 (CEST) Subject: [Python-checkins] r45705 - peps/trunk/pep-0356.txt Message-ID: <20060425061909.A5B281E4004@bag.python.org> Author: neal.norwitz Date: Tue Apr 25 08:19:08 2006 New Revision: 45705 Modified: peps/trunk/pep-0356.txt Log: optparse was upgraded, new AST bug Modified: peps/trunk/pep-0356.txt ============================================================================== --- peps/trunk/pep-0356.txt (original) +++ peps/trunk/pep-0356.txt Tue Apr 25 08:19:08 2006 @@ -149,9 +149,6 @@ - Python core should compile cleanly with g++ (Owner: Anthony) - - optparse should be updated to the latest Optik codebase - (Owner: Anthony) - - upgrade stdlib to use newer hashlib module instead of older deprecated md5/sha modules. (Owner: Anthony) @@ -173,6 +170,8 @@ - AST compiler problems (Owner: Jeremy Hylton) + * http://python.org/sf/1474677 foo(b=1, 2) doesn't raise SyntaxError + * eval(str(-sys.maxint - 1)) should produce an int, not long. when a constant it should not be stored in co_consts From buildbot at python.org Tue Apr 25 08:49:47 2006 From: buildbot at python.org (buildbot at python.org) Date: Tue, 25 Apr 2006 06:49:47 +0000 Subject: [Python-checkins] buildbot warnings in x86 cygwin trunk Message-ID: <20060425064947.DD2AE1E4004@bag.python.org> The Buildbot has detected a new failure of x86 cygwin trunk. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520cygwin%2520trunk/builds/282 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: neal.norwitz Build Had Warnings: warnings test sincerely, -The Buildbot From buildbot at python.org Tue Apr 25 09:13:39 2006 From: buildbot at python.org (buildbot at python.org) Date: Tue, 25 Apr 2006 07:13:39 +0000 Subject: [Python-checkins] buildbot warnings in x86 Ubuntu dapper (icc) trunk Message-ID: <20060425071339.3184F1E4004@bag.python.org> The Buildbot has detected a new failure of x86 Ubuntu dapper (icc) trunk. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520Ubuntu%2520dapper%2520%2528icc%2529%2520trunk/builds/246 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: neal.norwitz Build Had Warnings: warnings test sincerely, -The Buildbot From python-checkins at python.org Tue Apr 25 12:56:53 2006 From: python-checkins at python.org (nick.coghlan) Date: Tue, 25 Apr 2006 12:56:53 +0200 (CEST) Subject: [Python-checkins] r45706 - in python/trunk: Doc/lib/libcontextlib.tex Doc/lib/libstdtypes.tex Doc/ref/ref3.tex Doc/ref/ref7.tex Lib/contextlib.py Lib/decimal.py Lib/test/test_contextlib.py Message-ID: <20060425105653.723561E4014@bag.python.org> Author: nick.coghlan Date: Tue Apr 25 12:56:51 2006 New Revision: 45706 Modified: python/trunk/Doc/lib/libcontextlib.tex python/trunk/Doc/lib/libstdtypes.tex python/trunk/Doc/ref/ref3.tex python/trunk/Doc/ref/ref7.tex python/trunk/Lib/contextlib.py python/trunk/Lib/decimal.py python/trunk/Lib/test/test_contextlib.py Log: Move the PEP 343 documentation and implementation closer to the terminology in the alpha 1 documentation. - "context manager" reverts to its alpha 1 definition - the term "context specifier" goes away entirely - contextlib.GeneratorContextManager is renamed GeneratorContext There are still a number of changes relative to alpha 1: - the expression in the with statement is explicitly called the "context expression" in the language reference - the terms 'with statement context', 'context object' or 'with statement context' are used in several places instead of a bare 'context'. The aim of this is to avoid ambiguity in relation to the runtime context set up when the block is executed, and the context objects that already exist in various application domains (such as decimal.Context) - contextlib.contextmanager is renamed to contextfactory This best reflects the nature of the function resulting from the use of that decorator - decimal.ContextManager is renamed to WithStatementContext Simple dropping the 'Manager' part wasn't possible due to the fact that decimal.Context already exists and means something different. WithStatementContext is ugly but workable. A technically unrelated change snuck into this commit: contextlib.closing now avoids the overhead of creating a generator, since it's trivial to implement that particular context manager directly. Modified: python/trunk/Doc/lib/libcontextlib.tex ============================================================================== --- python/trunk/Doc/lib/libcontextlib.tex (original) +++ python/trunk/Doc/lib/libcontextlib.tex Tue Apr 25 12:56:51 2006 @@ -11,18 +11,19 @@ Functions provided: -\begin{funcdesc}{contextmanager}{func} -This function is a decorator that can be used to define context managers -for use with the \keyword{with} statement, without needing to create a -class or separate \method{__enter__()} and \method{__exit__()} methods. +\begin{funcdesc}{context}func} +This function is a decorator that can be used to define a factory +function for \keyword{with} statement context objects, without +needing to create a class or separate \method{__enter__()} and +\method{__exit__()} methods. A simple example: \begin{verbatim} from __future__ import with_statement -from contextlib import contextmanager +from contextlib import contextfactory - at contextmanager + at contextfactory def tag(name): print "<%s>" % name yield @@ -36,9 +37,10 @@ \end{verbatim} -When called, the decorated function must return a generator-iterator. -This iterator must yield exactly one value, which will be bound to the -targets in the \keyword{with} statement's \keyword{as} clause, if any. +The function being decorated must return a generator-iterator when +called. This iterator must yield exactly one value, which will be +bound to the targets in the \keyword{with} statement's \keyword{as} +clause, if any. At the point where the generator yields, the block nested in the \keyword{with} statement is executed. The generator is then resumed @@ -53,20 +55,20 @@ treat the exception as having been handled, and resume execution with the statement immediately following the \keyword{with} statement. -Note that you can use \code{@contextmanager} to define a context -specifier's \method{__context__} method. This is usually more +Note that you can use \code{@contextfactory} to define a context +manager's \method{__context__} method. This is usually more convenient than creating another class just to serve as a context -manager. For example: +object. For example: \begin{verbatim} from __future__ import with_statement -from contextlib import contextmanager +from contextlib import contextfactory class Tag: def __init__(self, name): self.name = name - @contextmanager + @contextfactory def __context__(self): print "<%s>" % self.name yield self @@ -83,7 +85,7 @@ \end{funcdesc} \begin{funcdesc}{nested}{ctx1\optional{, ctx2\optional{, ...}}} -Combine multiple context specifiers into a single nested context manager. +Combine multiple context managers into a single nested context manager. Code like this: @@ -104,12 +106,12 @@ \end{verbatim} Note that if the \method{__exit__()} method of one of the nested -context managers indicates an exception should be suppressed, no +context objects indicates an exception should be suppressed, no exception information will be passed to any remaining outer context -managers. Similarly, if the \method{__exit__()} method of one of the -nested context managers raises an exception, any previous exception +objects. Similarly, if the \method{__exit__()} method of one of the +nested context objects raises an exception, any previous exception state will be lost; the new exception will be passed to the -\method{__exit__()} methods of any remaining outer context managers. +\method{__exit__()} methods of any remaining outer context objects. In general, \method{__exit__()} methods should avoid raising exceptions, and in particular they should not re-raise a passed-in exception. @@ -117,13 +119,13 @@ \label{context-closing} \begin{funcdesc}{closing}{thing} -Return a context manager that closes \var{thing} upon completion of the +Return a context that closes \var{thing} upon completion of the block. This is basically equivalent to: \begin{verbatim} -from contextlib import contextmanager +from contextlib import contextfactory - at contextmanager + at contextfactory def closing(thing): try: yield thing @@ -137,14 +139,33 @@ from contextlib import closing import codecs -with closing(urllib.urlopen('http://www.python.org')) as f: - for line in f: +with closing(urllib.urlopen('http://www.python.org')) as page: + for line in page: print line \end{verbatim} -without needing to explicitly close \code{f}. Even if an error occurs, -\code{f.close()} will be called when the \keyword{with} block is exited. +without needing to explicitly close \code{page}. Even if an error +occurs, \code{page.close()} will be called when the \keyword{with} +block is exited. + +Context managers with a close method can use this context factory +directly without needing to implement their own +\method{__context__()} method. +\begin{verbatim} +from __future__ import with_statement +from contextlib import closing +class MyClass: + def close(self): + print "Closing", self + __context__ = closing + +>>> with MyClass() as x: +... print "Hello from", x +... +Hello from <__main__.MyClass instance at 0xb7df02ec> +Closing <__main__.MyClass instance at 0xb7df02ec> +\end{verbatim} \end{funcdesc} \begin{seealso} Modified: python/trunk/Doc/lib/libstdtypes.tex ============================================================================== --- python/trunk/Doc/lib/libstdtypes.tex (original) +++ python/trunk/Doc/lib/libstdtypes.tex Tue Apr 25 12:56:51 2006 @@ -1756,59 +1756,59 @@ \subsection{Context Types \label{typecontext}} \versionadded{2.5} -\index{context specification protocol} +\index{with statement context protocol} \index{context management protocol} -\index{protocol!context specification} +\index{protocol!with statement context} \index{protocol!context management} Python's \keyword{with} statement supports the concept of a runtime -context defined by a context specifier. This is implemented using +context defined by a context manager. This is implemented using three distinct methods; these are used to allow user-defined -classes to define a context. +classes to define a runtime context. -The \dfn{context specification protocol} consists of a single -method that needs to be provided for a context specifier object to +The \dfn{context management protocol} consists of a single +method that needs to be provided for a context manager object to define a runtime context: -\begin{methoddesc}[context specifier]{__context__}{} - Return a context manager object. The object is required to support - the context management protocol described below. If an object - supports different kinds of runtime context, additional methods can - be provided to specifically request context managers for those - kinds of context. (An example of an object supporting multiple kinds - of context would be a synchronisation object which supported both - a locked context for normal thread synchronisation and an unlocked - context to temporarily release a held lock while performing a - potentially long running operation) +\begin{methoddesc}[context manager]{__context__}{} + Return a with statement context object. The object is required to + support the with statement context protocol described below. If an + object supports different kinds of runtime context, additional + methods can be provided to specifically request context objects for + those kinds of runtime context. (An example of an object supporting + multiple kinds of context would be a synchronisation object which + supported both a locked context for normal thread synchronisation + and an unlocked context to temporarily release a held lock while + performing a potentially long running operation) \end{methoddesc} -The context manager objects themselves are required to support the +The with statement context objects themselves are required to support the following three methods, which together form the -\dfn{context management protocol}: +\dfn{with statement context protocol}: -\begin{methoddesc}[context manager]{__context__}{} - Return the context manager object itself. This is required to - allow both context specifiers and context managers to be used with - the \keyword{with} statement. +\begin{methoddesc}[with statement context]{__context__}{} + Return the context object itself. This is required to allow both + context objects and context managers to be used in a \keyword{with} + statement. \end{methoddesc} -\begin{methoddesc}[context manager]{__enter__}{} +\begin{methoddesc}[with statement context]{__enter__}{} Enter the runtime context and return either the defining context - specifier or another object related to the runtime context. The value + manager or another object related to the runtime context. The value returned by this method is bound to the identifier in the \keyword{as} clause of \keyword{with} statements using this context. - (An example of a context with a context manager that returns the - original context specifier is file objects, which are returned from - __enter__() to allow \function{open()} to be used directly in a with - statement. An example of a context manager that returns a related + (An example of a context object that returns the original context + manager is file objects, which are returned from __enter__() to + allow \function{open()} to be used directly in a with + statement. An example of a context object that returns a related object is \code{decimal.Context} which sets the active decimal - context to a copy of the context specifier and then returns the copy - to allow changes to be made to the current decimal context in the - body of the \keyword{with} statement) without affecting code outside + context to a copy of the context manager and then returns the copy. + This allows changes to be made to the current decimal context in the + body of the \keyword{with} statement without affecting code outside the \keyword{with} statement). \end{methoddesc} -\begin{methoddesc}[context manager]{__exit__}{exc_type, exc_val, exc_tb} +\begin{methoddesc}[with statement context]{__exit__}{exc_type, exc_val, exc_tb} Exit the runtime context and return a Boolean flag indicating if any expection that occurred should be suppressed. If an exception occurred while executing the body of the \keyword{with} statement, the @@ -1829,19 +1829,18 @@ \method{__exit__()} method has actually failed. \end{methoddesc} -Python defines several context specifiers and managers to support +Python defines several context objects and managers to support easy thread synchronisation, prompt closure of files or other objects, and thread-safe manipulation of the decimal arithmetic context. The specific types are not important beyond their -implementation of the context specification and context -management protocols. +implementation of the context management and with statement context +protocols. -Python's generators and the \code{contextlib.contextmanager} -decorator provide a convenient way to implement the context -specification and context management protocols. If a context -specifier's \method{__context__()} method is implemented as a -generator decorated with the \code{contextlib.contextmanager} -decorator, it will automatically return a context manager +Python's generators and the \code{contextlib.contextfactory} decorator +provide a convenient way to implement these protocols. If a context +manager's \method{__context__()} method is implemented as a +generator decorated with the \code{contextlib.contextfactory} +decorator, it will automatically return a with statement context object supplying the necessary \method{__context__()}, \method{__enter__()} and \method{__exit__()} methods. Modified: python/trunk/Doc/ref/ref3.tex ============================================================================== --- python/trunk/Doc/ref/ref3.tex (original) +++ python/trunk/Doc/ref/ref3.tex Tue Apr 25 12:56:51 2006 @@ -2112,59 +2112,60 @@ \end{itemize} -\subsection{Context Specifiers and Managers\label{context-managers}} +\subsection{With Statement Contexts and Context Managers\label{context-managers}} \versionadded{2.5} -A \dfn{context specifier} is an object that defines the runtime +A \dfn{context manager} is an object that defines the runtime context to be established when executing a \keyword{with} -statement. The context specifier provides a \dfn{context manager} -which manages the entry into, and the exit from, the desired -runtime context for the execution of the block of code. Context -managers are normally invoked using the \keyword{with} statement -(described in section~\ref{with}), but can also be used by -directly invoking their methods. +statement. The context manager provides a +\dfn{with statement context object} which manages the entry into, +and the exit from, the desired runtime context for the execution +of the block of code. Context managers are normally invoked using +the \keyword{with} statement (described in section~\ref{with}), but +can also be used by directly invoking their methods. \stindex{with} \index{context manager} -\index{context specifier} +\index{context (with statement)} +\index{with statement context} -Typical uses of context specifiers and managers include saving and +Typical uses of context managers and contexts include saving and restoring various kinds of global state, locking and unlocking resources, closing opened files, etc. -For more information on context specifiers and context manager objects, +For more information on context managers and context objects, see ``\ulink{Context Types}{../lib/typecontext.html}'' in the \citetitle[../lib/lib.html]{Python Library Reference}. -\begin{methoddesc}[context specifier]{__context__}{self} +\begin{methoddesc}[context manager]{__context__}{self} Invoked when the object is used as the context expression of a \keyword{with} statement. The returned object must implement \method{__enter__()} and \method{__exit__()} methods. -Context specifiers written in Python can also implement this method +Context managers written in Python can also implement this method using a generator function decorated with the -\function{contextlib.contextmanager} decorator, as this can be simpler +\function{contextlib.contextfactory} decorator, as this can be simpler than writing individual \method{__enter__()} and \method{__exit__()} methods on a separate object when the state to be managed is complex. -Context manager objects also need to implement this method; they are -required to return themselves (that is, this method will simply +With statement context objects also need to implement this method; they +are required to return themselves (that is, this method will simply return \var{self}). \end{methoddesc} -\begin{methoddesc}[context manager]{__enter__}{self} -Enter the context managed by this object. The \keyword{with} statement -will bind this method's return value to the target(s) specified in the -\keyword{as} clause of the statement, if any. +\begin{methoddesc}[with statement context]{__enter__}{self} +Enter the runtime context related to this object. The \keyword{with} +statement will bind this method's return value to the target(s) +specified in the \keyword{as} clause of the statement, if any. \end{methoddesc} \begin{methoddesc}[context manager]{__exit__} {self, exc_type, exc_value, traceback} -Exit the context managed by this object. The parameters describe the -exception that caused the context to be exited. If the context was -exited without an exception, all three arguments will be -\constant{None}. +Exit the runtime context related to this object. The parameters +describe the exception that caused the context to be exited. If +the context was exited without an exception, all three arguments +will be \constant{None}. If an exception is supplied, and the method wishes to suppress the exception (i.e., prevent it from being propagated), it should return a Modified: python/trunk/Doc/ref/ref7.tex ============================================================================== --- python/trunk/Doc/ref/ref7.tex (original) +++ python/trunk/Doc/ref/ref7.tex Tue Apr 25 12:56:51 2006 @@ -315,10 +315,10 @@ \versionadded{2.5} The \keyword{with} statement is used to wrap the execution of a block -with methods defined by a context specifier or manager (see -section~\ref{context-managers}). This allows common +with methods defined by a context manager or with statement context +object (see section~\ref{context-managers}). This allows common \keyword{try}...\keyword{except}...\keyword{finally} usage patterns to -be encapsulated as context specifiers or managers for convenient reuse. +be encapsulated for convenient reuse. \begin{productionlist} \production{with_stmt} @@ -329,12 +329,12 @@ \begin{enumerate} -\item The expression is evaluated, to obtain a context specifier. +\item The context expression is evaluated, to obtain a context manager. -\item The context specifier's \method{__context__()} method is -invoked to obtain a context manager object. +\item The context manger's \method{__context__()} method is +invoked to obtain a with statement context object. -\item The context manager's \method{__enter__()} method is invoked. +\item The context object's \method{__enter__()} method is invoked. \item If a target list was included in the \keyword{with} statement, the return value from \method{__enter__()} is assigned to it. @@ -347,7 +347,7 @@ \item The suite is executed. -\item The context manager's \method{__exit__()} method is invoked. If +\item The context object's \method{__exit__()} method is invoked. If an exception caused the suite to be exited, its type, value, and traceback are passed as arguments to \method{__exit__()}. Otherwise, three \constant{None} arguments are supplied. Modified: python/trunk/Lib/contextlib.py ============================================================================== --- python/trunk/Lib/contextlib.py (original) +++ python/trunk/Lib/contextlib.py Tue Apr 25 12:56:51 2006 @@ -2,10 +2,10 @@ import sys -__all__ = ["contextmanager", "nested", "closing"] +__all__ = ["contextfactory", "nested", "closing"] -class GeneratorContextManager(object): - """Helper for @contextmanager decorator.""" +class GeneratorContext(object): + """Helper for @contextfactory decorator.""" def __init__(self, gen): self.gen = gen @@ -48,8 +48,8 @@ raise -def contextmanager(func): - """@contextmanager decorator. +def contextfactory(func): + """@contextfactory decorator. Typical usage: @@ -77,7 +77,7 @@ """ def helper(*args, **kwds): - return GeneratorContextManager(func(*args, **kwds)) + return GeneratorContext(func(*args, **kwds)) try: helper.__name__ = func.__name__ helper.__doc__ = func.__doc__ @@ -87,7 +87,7 @@ return helper - at contextmanager + at contextfactory def nested(*contexts): """Support multiple context managers in a single with-statement. @@ -133,9 +133,8 @@ raise exc[0], exc[1], exc[2] - at contextmanager -def closing(thing): - """Context manager to automatically close something at the end of a block. +class closing(object): + """Context to automatically close something at the end of a block. Code like this: @@ -151,7 +150,11 @@ f.close() """ - try: - yield thing - finally: - thing.close() + def __init__(self, thing): + self.thing = thing + def __context__(self): + return self + def __enter__(self): + return self.thing + def __exit__(self, *exc_info): + self.thing.close() Modified: python/trunk/Lib/decimal.py ============================================================================== --- python/trunk/Lib/decimal.py (original) +++ python/trunk/Lib/decimal.py Tue Apr 25 12:56:51 2006 @@ -2173,7 +2173,7 @@ del name, val, globalname, rounding_functions -class ContextManager(object): +class WithStatementContext(object): """Helper class to simplify Context management. Sample usage: @@ -2249,7 +2249,7 @@ return ', '.join(s) + ')' def __context__(self): - return ContextManager(self.copy()) + return WithStatementContext(self.copy()) def clear_flags(self): """Reset all flags to zero""" Modified: python/trunk/Lib/test/test_contextlib.py ============================================================================== --- python/trunk/Lib/test/test_contextlib.py (original) +++ python/trunk/Lib/test/test_contextlib.py Tue Apr 25 12:56:51 2006 @@ -13,9 +13,9 @@ class ContextManagerTestCase(unittest.TestCase): - def test_contextmanager_plain(self): + def test_contextfactory_plain(self): state = [] - @contextmanager + @contextfactory def woohoo(): state.append(1) yield 42 @@ -26,9 +26,9 @@ state.append(x) self.assertEqual(state, [1, 42, 999]) - def test_contextmanager_finally(self): + def test_contextfactory_finally(self): state = [] - @contextmanager + @contextfactory def woohoo(): state.append(1) try: @@ -47,8 +47,8 @@ self.fail("Expected ZeroDivisionError") self.assertEqual(state, [1, 42, 999]) - def test_contextmanager_no_reraise(self): - @contextmanager + def test_contextfactory_no_reraise(self): + @contextfactory def whee(): yield ctx = whee().__context__() @@ -56,8 +56,8 @@ # Calling __exit__ should not result in an exception self.failIf(ctx.__exit__(TypeError, TypeError("foo"), None)) - def test_contextmanager_trap_yield_after_throw(self): - @contextmanager + def test_contextfactory_trap_yield_after_throw(self): + @contextfactory def whoo(): try: yield @@ -69,9 +69,9 @@ RuntimeError, ctx.__exit__, TypeError, TypeError("foo"), None ) - def test_contextmanager_except(self): + def test_contextfactory_except(self): state = [] - @contextmanager + @contextfactory def woohoo(): state.append(1) try: @@ -86,14 +86,14 @@ raise ZeroDivisionError(999) self.assertEqual(state, [1, 42, 999]) - def test_contextmanager_attribs(self): + def test_contextfactory_attribs(self): def attribs(**kw): def decorate(func): for k,v in kw.items(): setattr(func,k,v) return func return decorate - @contextmanager + @contextfactory @attribs(foo='bar') def baz(spam): """Whee!""" @@ -106,13 +106,13 @@ # XXX This needs more work def test_nested(self): - @contextmanager + @contextfactory def a(): yield 1 - @contextmanager + @contextfactory def b(): yield 2 - @contextmanager + @contextfactory def c(): yield 3 with nested(a(), b(), c()) as (x, y, z): @@ -122,14 +122,14 @@ def test_nested_cleanup(self): state = [] - @contextmanager + @contextfactory def a(): state.append(1) try: yield 2 finally: state.append(3) - @contextmanager + @contextfactory def b(): state.append(4) try: @@ -148,7 +148,7 @@ def test_nested_right_exception(self): state = [] - @contextmanager + @contextfactory def a(): yield 1 class b(object): @@ -172,10 +172,10 @@ self.fail("Didn't raise ZeroDivisionError") def test_nested_b_swallows(self): - @contextmanager + @contextfactory def a(): yield - @contextmanager + @contextfactory def b(): try: yield @@ -189,7 +189,7 @@ self.fail("Didn't swallow ZeroDivisionError") def test_nested_break(self): - @contextmanager + @contextfactory def a(): yield state = 0 @@ -201,7 +201,7 @@ self.assertEqual(state, 1) def test_nested_continue(self): - @contextmanager + @contextfactory def a(): yield state = 0 @@ -213,7 +213,7 @@ self.assertEqual(state, 3) def test_nested_return(self): - @contextmanager + @contextfactory def a(): try: yield From python-checkins at python.org Tue Apr 25 13:05:57 2006 From: python-checkins at python.org (nick.coghlan) Date: Tue, 25 Apr 2006 13:05:57 +0200 (CEST) Subject: [Python-checkins] r45707 - python/trunk/Doc/lib/libcontextlib.tex Message-ID: <20060425110557.8D26D1E4004@bag.python.org> Author: nick.coghlan Date: Tue Apr 25 13:05:56 2006 New Revision: 45707 Modified: python/trunk/Doc/lib/libcontextlib.tex Log: Fix latex typo Modified: python/trunk/Doc/lib/libcontextlib.tex ============================================================================== --- python/trunk/Doc/lib/libcontextlib.tex (original) +++ python/trunk/Doc/lib/libcontextlib.tex Tue Apr 25 13:05:56 2006 @@ -11,7 +11,7 @@ Functions provided: -\begin{funcdesc}{context}func} +\begin{funcdesc}{context}{func} This function is a decorator that can be used to define a factory function for \keyword{with} statement context objects, without needing to create a class or separate \method{__enter__()} and From buildbot at python.org Tue Apr 25 14:04:41 2006 From: buildbot at python.org (buildbot at python.org) Date: Tue, 25 Apr 2006 12:04:41 +0000 Subject: [Python-checkins] buildbot warnings in alpha Debian trunk Message-ID: <20060425120441.950331E4004@bag.python.org> The Buildbot has detected a new failure of alpha Debian trunk. Full details are available at: http://www.python.org/dev/buildbot/all/alpha%2520Debian%2520trunk/builds/12 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: nick.coghlan Build Had Warnings: warnings test sincerely, -The Buildbot From python-checkins at python.org Tue Apr 25 14:28:57 2006 From: python-checkins at python.org (thomas.wouters) Date: Tue, 25 Apr 2006 14:28:57 +0200 (CEST) Subject: [Python-checkins] r45708 - python/trunk/Doc/lib/libunittest.tex Message-ID: <20060425122857.57D9F1E4004@bag.python.org> Author: thomas.wouters Date: Tue Apr 25 14:28:56 2006 New Revision: 45708 Modified: python/trunk/Doc/lib/libunittest.tex Log: Fix markup glitch in unittest docs. Will backport. Modified: python/trunk/Doc/lib/libunittest.tex ============================================================================== --- python/trunk/Doc/lib/libunittest.tex (original) +++ python/trunk/Doc/lib/libunittest.tex Tue Apr 25 14:28:56 2006 @@ -226,7 +226,7 @@ identify the test case as a \dfn{failure}. Other exceptions that do not arise from checks made through the \method{assert*()} and \method{fail*()} methods are identified by the testing framework as -dfn{errors}. +\dfn{errors}. The way to run a test case will be described later. For now, note that to construct an instance of such a test case, we call its From python-checkins at python.org Tue Apr 25 14:29:43 2006 From: python-checkins at python.org (thomas.wouters) Date: Tue, 25 Apr 2006 14:29:43 +0200 (CEST) Subject: [Python-checkins] r45709 - python/branches/release24-maint/Doc/lib/libunittest.tex Message-ID: <20060425122943.EC8E31E4004@bag.python.org> Author: thomas.wouters Date: Tue Apr 25 14:29:43 2006 New Revision: 45709 Modified: python/branches/release24-maint/Doc/lib/libunittest.tex Log: Backport trunk's 45708: Fix markup glitch in unittest docs. Modified: python/branches/release24-maint/Doc/lib/libunittest.tex ============================================================================== --- python/branches/release24-maint/Doc/lib/libunittest.tex (original) +++ python/branches/release24-maint/Doc/lib/libunittest.tex Tue Apr 25 14:29:43 2006 @@ -226,7 +226,7 @@ identify the test case as a \dfn{failure}. Other exceptions that do not arise from checks made through the \method{assert*()} and \method{fail*()} methods are identified by the testing framework as -dfn{errors}. +\dfn{errors}. The way to run a test case will be described later. For now, note that to construct an instance of such a test case, we call its From python-checkins at python.org Tue Apr 25 14:31:38 2006 From: python-checkins at python.org (andrew.kuchling) Date: Tue, 25 Apr 2006 14:31:38 +0200 (CEST) Subject: [Python-checkins] r45710 - python/trunk/Doc/whatsnew/whatsnew25.tex Message-ID: <20060425123138.A32881E4020@bag.python.org> Author: andrew.kuchling Date: Tue Apr 25 14:31:38 2006 New Revision: 45710 Modified: python/trunk/Doc/whatsnew/whatsnew25.tex Log: Add two items; easy_install is now off the table, though pkgutil still is Modified: python/trunk/Doc/whatsnew/whatsnew25.tex ============================================================================== --- python/trunk/Doc/whatsnew/whatsnew25.tex (original) +++ python/trunk/Doc/whatsnew/whatsnew25.tex Tue Apr 25 14:31:38 2006 @@ -2,7 +2,6 @@ \usepackage{distutils} % $Id$ -% The easy_install stuff % Describe the pkgutil module % Fix XXX comments % Count up the patches and bugs @@ -1293,6 +1292,11 @@ '%H:%M:%S %Y-%m-%d') \end{verbatim} +\item The \module{doctest} module gained a \code{SKIP} option that +keeps an example from being executed at all. This is intended for +code snippets that are usage examples intended for the reader and +aren't actually test cases. + \item The \module{fileinput} module was made more flexible. Unicode filenames are now supported, and a \var{mode} parameter that defaults to \code{"r"} was added to the @@ -1416,6 +1420,15 @@ (Contributed by Antti Louko and Diego Petten\`o.) % (Patch 1180695, 1212117) +\item The Python debugger provided by the \module{pdb} module +can now store lists of commands to execute when a breakpoint is +reached and execution stops. Once breakpoint #1 has been created, +enter \samp{commands 1} and enter a series of commands to be executed, +finishing the list with \samp{end}. The command list can include +commands that resume execution, such as \samp{continue} or +\samp{next}. (Contributed by Gr\'egoire Dooms.) +% Patch 790710 + \item The \module{pickle} and \module{cPickle} modules no longer accept a return value of \code{None} from the \method{__reduce__()} method; the method must return a tuple of From buildbot at python.org Tue Apr 25 14:31:40 2006 From: buildbot at python.org (buildbot at python.org) Date: Tue, 25 Apr 2006 12:31:40 +0000 Subject: [Python-checkins] buildbot warnings in sparc Ubuntu dapper trunk Message-ID: <20060425123140.6AAFE1E4020@bag.python.org> The Buildbot has detected a new failure of sparc Ubuntu dapper trunk. Full details are available at: http://www.python.org/dev/buildbot/all/sparc%2520Ubuntu%2520dapper%2520trunk/builds/155 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: nick.coghlan Build Had Warnings: warnings test sincerely, -The Buildbot From python-checkins at python.org Tue Apr 25 14:47:26 2006 From: python-checkins at python.org (andrew.kuchling) Date: Tue, 25 Apr 2006 14:47:26 +0200 (CEST) Subject: [Python-checkins] r45711 - python/trunk/Doc/whatsnew/whatsnew25.tex Message-ID: <20060425124726.C0FFE1E4020@bag.python.org> Author: andrew.kuchling Date: Tue Apr 25 14:47:25 2006 New Revision: 45711 Modified: python/trunk/Doc/whatsnew/whatsnew25.tex Log: Rework context terminology Modified: python/trunk/Doc/whatsnew/whatsnew25.tex ============================================================================== --- python/trunk/Doc/whatsnew/whatsnew25.tex (original) +++ python/trunk/Doc/whatsnew/whatsnew25.tex Tue Apr 25 14:47:25 2006 @@ -667,13 +667,13 @@ \begin{itemize} \item The expression is evaluated and should result in an object -with a \method{__context__()} method (called a ``context specifier''). +with a \method{__context__()} method (called a ``context manager''). \item The context specifier's \method{__context__()} method is called, -and must return another object (called a ``context manager'') that has +and must return another object (called a ``with-statement context object'') that has \method{__enter__()} and \method{__exit__()} methods. -\item The context manager's \method{__enter__()} method is called. The value +\item The context object's \method{__enter__()} method is called. The value returned is assigned to \var{VAR}. If no \code{'as \var{VAR}'} clause is present, the value is simply discarded. @@ -685,7 +685,8 @@ \function{sys.exc_info()}. The method's return value controls whether the exception is re-raised: any false value re-raises the exception, and \code{True} will result in suppressing it. You'll only rarely -want to suppress the exception; the author of the code containing the +want to suppress the exception, because if you do +the author of the code containing the '\keyword{with}' statement will never realize anything went wrong. \item If \var{BLOCK} didn't raise an exception, @@ -724,14 +725,14 @@ method. Sometimes an object can simply return \code{self}; the \module{threading} module's lock objects do this, for example. For our database example, though, we need to create a new object; I'll -call this class \class{DatabaseContextMgr}. Our \method{__context__()} +call this class \class{DatabaseContext}. Our \method{__context__()} method must therefore look like this: \begin{verbatim} class DatabaseConnection: ... def __context__ (self): - return DatabaseContextMgr(self) + return DatabaseContext(self) # Database interface def cursor (self): @@ -742,12 +743,12 @@ "Rolls back current transaction" \end{verbatim} -Instance of \class{DatabaseContextMgr} need the connection object so that +Instances of \class{DatabaseContext} need the connection object so that the connection object's \method{commit()} or \method{rollback()} methods can be called: \begin{verbatim} -class DatabaseContextMgr: +class DatabaseContext: def __init__ (self, connection): self.connection = connection \end{verbatim} @@ -759,7 +760,7 @@ the cursor to a variable name. \begin{verbatim} -class DatabaseContextMgr: +class DatabaseContext: ... def __enter__ (self): # Code to start a new transaction @@ -779,7 +780,7 @@ statement at the marked location. \begin{verbatim} -class DatabaseContextMgr: +class DatabaseContext: ... def __exit__ (self, type, value, tb): if tb is None: @@ -798,8 +799,8 @@ decorator that are useful for writing objects for use with the '\keyword{with}' statement. -The decorator is called \function{contextmanager}, and lets you write -a simple context manager as a generator function. The generator +The decorator is called \function{contextfactory}, and lets you write +a single generator function instead of defining a new class. The generator should yield exactly one value. The code up to the \keyword{yield} will be executed as the \method{__enter__()} method, and the value yielded will be the method's return value that will get bound to the @@ -812,9 +813,9 @@ using this decorator as: \begin{verbatim} -from contextlib import contextmanager +from contextlib import contextfactory - at contextmanager + at contextfactory def db_transaction (connection): cursor = connection.cursor() try: @@ -831,13 +832,12 @@ \end{verbatim} You can also use this decorator to write the \method{__context__()} -method for a class without having to create a new class representing -the context manager: +method for a class: \begin{verbatim} class DatabaseConnection: - @contextmanager + @contextfactory def __context__ (self): cursor = self.cursor() try: @@ -850,10 +850,11 @@ \end{verbatim} -There's a \function{nested(\var{mgr1}, \var{mgr2}, ...)} function that -combines a number of contexts so you don't need to write -nested '\keyword{with}' statements. This example statement does two -things, starting a database transaction and acquiring a thread lock: +The \module{contextlib} module also has a \function{nested(\var{mgr1}, +\var{mgr2}, ...)} function that combines a number of contexts so you +don't need to write nested '\keyword{with}' statements. In this +example, the single '\keyword{with}' statement both starts a database +transaction and acquires a thread lock: \begin{verbatim} lock = threading.Lock() From buildbot at python.org Tue Apr 25 14:51:13 2006 From: buildbot at python.org (buildbot at python.org) Date: Tue, 25 Apr 2006 12:51:13 +0000 Subject: [Python-checkins] buildbot warnings in hppa Ubuntu dapper trunk Message-ID: <20060425125113.2CE5A1E400F@bag.python.org> The Buildbot has detected a new failure of hppa Ubuntu dapper trunk. Full details are available at: http://www.python.org/dev/buildbot/all/hppa%2520Ubuntu%2520dapper%2520trunk/builds/288 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: nick.coghlan Build Had Warnings: warnings failed slave lost sincerely, -The Buildbot From buildbot at python.org Tue Apr 25 14:58:45 2006 From: buildbot at python.org (buildbot at python.org) Date: Tue, 25 Apr 2006 12:58:45 +0000 Subject: [Python-checkins] buildbot warnings in ia64 Debian unstable 2.4 Message-ID: <20060425125846.0A8781E4004@bag.python.org> The Buildbot has detected a new failure of ia64 Debian unstable 2.4. Full details are available at: http://www.python.org/dev/buildbot/all/ia64%2520Debian%2520unstable%25202.4/builds/39 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch branches/release24-maint] HEAD Blamelist: thomas.wouters Build Had Warnings: warnings failed slave lost sincerely, -The Buildbot From ncoghlan at gmail.com Tue Apr 25 15:08:36 2006 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 25 Apr 2006 23:08:36 +1000 Subject: [Python-checkins] buildbot warnings in sparc Ubuntu dapper trunk In-Reply-To: <20060425123140.6AAFE1E4020@bag.python.org> References: <20060425123140.6AAFE1E4020@bag.python.org> Message-ID: <444E1F54.3060102@gmail.com> buildbot at python.org wrote: > The Buildbot has detected a new failure of sparc Ubuntu dapper trunk. > Full details are available at: > http://www.python.org/dev/buildbot/all/sparc%2520Ubuntu%2520dapper%2520trunk/builds/155 > > Buildbot URL: http://www.python.org/dev/buildbot/all/ > > Build Reason: > Build Source Stamp: [branch trunk] HEAD > Blamelist: nick.coghlan > > Build Had Warnings: warnings test The sparc Ubuntu and alpha Debian buildbot slaves both died again with the 30 minute timeout in test_compiler, even after Neal's fix the other day. I have no idea how the architecture could make a difference, but there's clearly something strange going on, since the other slaves are all fine, and these two *were* green on their previous run. . . The only thing that stood out for me is that in the cases I've looked at where it has broken test_compiler ran after test_curses, but for the two recent cases where it worked properly, test_compiler ran before test_curses. But I only noticed that because the test_curses gobbledygook is rather distinctive, so it may not actually have anything to do with the failures :) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From python-checkins at python.org Tue Apr 25 15:53:24 2006 From: python-checkins at python.org (thomas.wouters) Date: Tue, 25 Apr 2006 15:53:24 +0200 (CEST) Subject: [Python-checkins] r45712 - in python/trunk: Lib/test/test_pty.py Misc/ACKS Modules/fcntlmodule.c Message-ID: <20060425135324.234391E400B@bag.python.org> Author: thomas.wouters Date: Tue Apr 25 15:53:23 2006 New Revision: 45712 Modified: python/trunk/Lib/test/test_pty.py python/trunk/Misc/ACKS python/trunk/Modules/fcntlmodule.c Log: SF bug/patch #1433877: string parameter to ioctl not null terminated The new char-array used in ioctl calls wasn't explicitly NUL-terminated; quite probably the cause for the test_pty failures on Solaris that we circumvented earlier. (I wasn't able to reproduce it with this patch, but it has been somewhat elusive to start with.) Modified: python/trunk/Lib/test/test_pty.py ============================================================================== --- python/trunk/Lib/test/test_pty.py (original) +++ python/trunk/Lib/test/test_pty.py Tue Apr 25 15:53:23 2006 @@ -4,13 +4,6 @@ TEST_STRING_1 = "I wish to buy a fish license.\n" TEST_STRING_2 = "For my pet fish, Eric.\n" -# Solaris (at least 2.9 and 2.10) seem to have a fickle isatty(). The first -# test below, testing the result of os.openpty() for tty-ness, sometimes -# (but not always) fails. The second isatty test, in the sub-process, always -# works. Allow that fickle first test to fail on these platforms, since it -# doesn't actually affect functionality. -fickle_isatty = ["sunos5"] - if verbose: def debug(msg): print msg @@ -54,7 +47,7 @@ # " An optional feature could not be imported " ... ? raise TestSkipped, "Pseudo-terminals (seemingly) not functional." - if not os.isatty(slave_fd) and sys.platform not in fickle_isatty: + if not os.isatty(slave_fd): raise TestFailed, "slave_fd is not a tty" debug("Writing to slave_fd") Modified: python/trunk/Misc/ACKS ============================================================================== --- python/trunk/Misc/ACKS (original) +++ python/trunk/Misc/ACKS Tue Apr 25 15:53:23 2006 @@ -33,6 +33,7 @@ Luigi Ballabio Michael J. Barber Chris Barker +Quentin Barnes Cesar Eduardo Barros Des Barry Ulf Bartelt Modified: python/trunk/Modules/fcntlmodule.c ============================================================================== --- python/trunk/Modules/fcntlmodule.c (original) +++ python/trunk/Modules/fcntlmodule.c Tue Apr 25 15:53:23 2006 @@ -95,6 +95,7 @@ static PyObject * fcntl_ioctl(PyObject *self, PyObject *args) { +#define IOCTL_BUFSZ 1024 int fd; /* In PyArg_ParseTuple below, use the unsigned int 'I' format for the signed int 'code' variable, because Python turns 0x8000000 @@ -106,7 +107,7 @@ char *str; Py_ssize_t len; int mutate_arg = 1; - char buf[1024]; + char buf[IOCTL_BUFSZ+1]; /* argument plus NUL byte */ if (PyArg_ParseTuple(args, "O&Iw#|i:ioctl", conv_descriptor, &fd, &code, @@ -114,8 +115,9 @@ char *arg; if (mutate_arg) { - if (len <= sizeof buf) { + if (len <= IOCTL_BUFSZ) { memcpy(buf, str, len); + buf[len] = '\0'; arg = buf; } else { @@ -123,13 +125,14 @@ } } else { - if (len > sizeof buf) { + if (len > IOCTL_BUFSZ) { PyErr_SetString(PyExc_ValueError, "ioctl string arg too long"); return NULL; } else { memcpy(buf, str, len); + buf[len] = '\0'; arg = buf; } } @@ -141,7 +144,7 @@ else { ret = ioctl(fd, code, arg); } - if (mutate_arg && (len < sizeof buf)) { + if (mutate_arg && (len < IOCTL_BUFSZ)) { memcpy(str, buf, len); } if (ret < 0) { @@ -159,12 +162,13 @@ PyErr_Clear(); if (PyArg_ParseTuple(args, "O&Is#:ioctl", conv_descriptor, &fd, &code, &str, &len)) { - if (len > sizeof buf) { + if (len > IOCTL_BUFSZ) { PyErr_SetString(PyExc_ValueError, "ioctl string arg too long"); return NULL; } memcpy(buf, str, len); + buf[len] = '\0'; Py_BEGIN_ALLOW_THREADS ret = ioctl(fd, code, buf); Py_END_ALLOW_THREADS @@ -195,6 +199,7 @@ return NULL; } return PyInt_FromLong((long)ret); +#undef IOCTL_BUFSZ } PyDoc_STRVAR(ioctl_doc, From python-checkins at python.org Tue Apr 25 16:10:00 2006 From: python-checkins at python.org (george.yoshida) Date: Tue, 25 Apr 2006 16:10:00 +0200 (CEST) Subject: [Python-checkins] r45713 - python/trunk/Doc/whatsnew/whatsnew25.tex Message-ID: <20060425141000.3E1071E4004@bag.python.org> Author: george.yoshida Date: Tue Apr 25 16:09:58 2006 New Revision: 45713 Modified: python/trunk/Doc/whatsnew/whatsnew25.tex Log: minor tweak Modified: python/trunk/Doc/whatsnew/whatsnew25.tex ============================================================================== --- python/trunk/Doc/whatsnew/whatsnew25.tex (original) +++ python/trunk/Doc/whatsnew/whatsnew25.tex Tue Apr 25 16:09:58 2006 @@ -1423,7 +1423,7 @@ \item The Python debugger provided by the \module{pdb} module can now store lists of commands to execute when a breakpoint is -reached and execution stops. Once breakpoint #1 has been created, +reached and execution stops. Once breakpoint \#1 has been created, enter \samp{commands 1} and enter a series of commands to be executed, finishing the list with \samp{end}. The command list can include commands that resume execution, such as \samp{continue} or From buildbot at python.org Tue Apr 25 16:49:30 2006 From: buildbot at python.org (buildbot at python.org) Date: Tue, 25 Apr 2006 14:49:30 +0000 Subject: [Python-checkins] buildbot warnings in sparc solaris10 gcc trunk Message-ID: <20060425144930.3FCC01E4004@bag.python.org> The Buildbot has detected a new failure of sparc solaris10 gcc trunk. Full details are available at: http://www.python.org/dev/buildbot/all/sparc%2520solaris10%2520gcc%2520trunk/builds/553 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: thomas.wouters Build Had Warnings: warnings test sincerely, -The Buildbot From python-checkins at python.org Tue Apr 25 17:08:11 2006 From: python-checkins at python.org (thomas.wouters) Date: Tue, 25 Apr 2006 17:08:11 +0200 (CEST) Subject: [Python-checkins] r45714 - python/trunk/Modules/socketmodule.c Message-ID: <20060425150811.0EDE31E4004@bag.python.org> Author: thomas.wouters Date: Tue Apr 25 17:08:10 2006 New Revision: 45714 Modified: python/trunk/Modules/socketmodule.c Log: Fix SF bug #1476111: SystemError in socket sendto. The AF_INET6 and AF_PACKET cases in getsockaddrarg were missing their own checks for tuple-ness of the address argument, which means a confusing SystemError was raised by PyArg_ParseTuple instead. Modified: python/trunk/Modules/socketmodule.c ============================================================================== --- python/trunk/Modules/socketmodule.c (original) +++ python/trunk/Modules/socketmodule.c Tue Apr 25 17:08:10 2006 @@ -1217,6 +1217,14 @@ int port, flowinfo, scope_id, result; addr = (struct sockaddr_in6*)&(s->sock_addr).in6; flowinfo = scope_id = 0; + if (!PyTuple_Check(args)) { + PyErr_Format( + PyExc_TypeError, + "getsockaddrarg: " + "AF_INET6 address must be tuple, not %.500s", + args->ob_type->tp_name); + return 0; + } if (!PyArg_ParseTuple(args, "eti|ii", "idna", &host, &port, &flowinfo, &scope_id)) { @@ -1319,6 +1327,14 @@ char *haddr = NULL; unsigned int halen = 0; + if (!PyTuple_Check(args)) { + PyErr_Format( + PyExc_TypeError, + "getsockaddrarg: " + "AF_PACKET address must be tuple, not %.500s", + args->ob_type->tp_name); + return 0; + } if (!PyArg_ParseTuple(args, "si|iis#", &interfaceName, &protoNumber, &pkttype, &hatype, &haddr, &halen)) From python-checkins at python.org Tue Apr 25 17:29:49 2006 From: python-checkins at python.org (thomas.wouters) Date: Tue, 25 Apr 2006 17:29:49 +0200 (CEST) Subject: [Python-checkins] r45715 - in python/trunk: Include/osdefs.h Modules/posixmodule.c Python/getcwd.c Message-ID: <20060425152949.847011E4004@bag.python.org> Author: thomas.wouters Date: Tue Apr 25 17:29:46 2006 New Revision: 45715 Modified: python/trunk/Include/osdefs.h python/trunk/Modules/posixmodule.c python/trunk/Python/getcwd.c Log: Define MAXPATHLEN to be at least PATH_MAX, if that's defined. Python uses MAXPATHLEN-sized buffers for various output-buffers (like to realpath()), and that's correct on BSD platforms, but not Linux (which uses PATH_MAX, and does not define MAXPATHLEN.) Cursory googling suggests Linux is following a newer standard than BSD, but in cases like this, who knows. Using the greater of PATH_MAX and 1024 as a fallback for MAXPATHLEN seems to be the most portable solution. Modified: python/trunk/Include/osdefs.h ============================================================================== --- python/trunk/Include/osdefs.h (original) +++ python/trunk/Include/osdefs.h Tue Apr 25 17:29:46 2006 @@ -37,8 +37,12 @@ /* Max pathname length */ #ifndef MAXPATHLEN +#if defined(PATH_MAX) && PATH_MAX > 1024 +#define MAXPATHLEN PATH_MAX +#else #define MAXPATHLEN 1024 #endif +#endif /* Search path entry delimiter */ #ifndef DELIM Modified: python/trunk/Modules/posixmodule.c ============================================================================== --- python/trunk/Modules/posixmodule.c (original) +++ python/trunk/Modules/posixmodule.c Tue Apr 25 17:29:46 2006 @@ -262,7 +262,11 @@ #endif /* OS2 */ #ifndef MAXPATHLEN +#if defined(PATH_MAX) && PATH_MAX > 1024 +#define MAXPATHLEN PATH_MAX +#else #define MAXPATHLEN 1024 +#endif #endif /* MAXPATHLEN */ #ifdef UNION_WAIT Modified: python/trunk/Python/getcwd.c ============================================================================== --- python/trunk/Python/getcwd.c (original) +++ python/trunk/Python/getcwd.c Tue Apr 25 17:29:46 2006 @@ -14,8 +14,12 @@ #endif #ifndef MAXPATHLEN +#if defined(PATH_MAX) && PATH_MAX > 1024 +#define MAXPATHLEN PATH_MAX +#else #define MAXPATHLEN 1024 #endif +#endif extern char *getwd(char *); From python-checkins at python.org Tue Apr 25 17:33:49 2006 From: python-checkins at python.org (thomas.wouters) Date: Tue, 25 Apr 2006 17:33:49 +0200 (CEST) Subject: [Python-checkins] r45716 - in python/branches/release24-maint: Include/osdefs.h Modules/posixmodule.c Python/getcwd.c Message-ID: <20060425153349.0DF091E4004@bag.python.org> Author: thomas.wouters Date: Tue Apr 25 17:33:48 2006 New Revision: 45716 Modified: python/branches/release24-maint/Include/osdefs.h python/branches/release24-maint/Modules/posixmodule.c python/branches/release24-maint/Python/getcwd.c Log: Backport trunk's r45715: Define MAXPATHLEN to be at least PATH_MAX, if that's defined. Python uses MAXPATHLEN-sized buffers for various output-buffers (like to realpath()), and that's correct on BSD platforms, but not Linux (which uses PATH_MAX, and does not define MAXPATHLEN.) Cursory googling suggests Linux is following a newer standard than BSD, but in cases like this, who knows. Using the greater of PATH_MAX and 1024 as a fallback for MAXPATHLEN seems to be the most portable solution. Modified: python/branches/release24-maint/Include/osdefs.h ============================================================================== --- python/branches/release24-maint/Include/osdefs.h (original) +++ python/branches/release24-maint/Include/osdefs.h Tue Apr 25 17:33:48 2006 @@ -37,8 +37,12 @@ /* Max pathname length */ #ifndef MAXPATHLEN +#if defined(PATH_MAX) && PATH_MAX > 1024 +#define MAXPATHLEN PATH_MAX +#else #define MAXPATHLEN 1024 #endif +#endif /* Search path entry delimiter */ #ifndef DELIM Modified: python/branches/release24-maint/Modules/posixmodule.c ============================================================================== --- python/branches/release24-maint/Modules/posixmodule.c (original) +++ python/branches/release24-maint/Modules/posixmodule.c Tue Apr 25 17:33:48 2006 @@ -244,7 +244,11 @@ #endif /* OS2 */ #ifndef MAXPATHLEN +#if defined(PATH_MAX) && PATH_MAX > 1024 +#define MAXPATHLEN PATH_MAX +#else #define MAXPATHLEN 1024 +#endif #endif /* MAXPATHLEN */ #ifdef UNION_WAIT Modified: python/branches/release24-maint/Python/getcwd.c ============================================================================== --- python/branches/release24-maint/Python/getcwd.c (original) +++ python/branches/release24-maint/Python/getcwd.c Tue Apr 25 17:33:48 2006 @@ -14,8 +14,12 @@ #endif #ifndef MAXPATHLEN +#if defined(PATH_MAX) && PATH_MAX > 1024 +#define MAXPATHLEN PATH_MAX +#else #define MAXPATHLEN 1024 #endif +#endif extern char *getwd(char *); From buildbot at python.org Tue Apr 25 17:41:43 2006 From: buildbot at python.org (buildbot at python.org) Date: Tue, 25 Apr 2006 15:41:43 +0000 Subject: [Python-checkins] buildbot warnings in x86 XP trunk Message-ID: <20060425154143.6C6B41E4004@bag.python.org> The Buildbot has detected a new failure of x86 XP trunk. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520XP%2520trunk/builds/573 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: thomas.wouters Build Had Warnings: warnings failed slave lost sincerely, -The Buildbot From buildbot at python.org Tue Apr 25 17:57:53 2006 From: buildbot at python.org (buildbot at python.org) Date: Tue, 25 Apr 2006 15:57:53 +0000 Subject: [Python-checkins] buildbot warnings in alpha Debian trunk Message-ID: <20060425155753.A18951E4004@bag.python.org> The Buildbot has detected a new failure of alpha Debian trunk. Full details are available at: http://www.python.org/dev/buildbot/all/alpha%2520Debian%2520trunk/builds/15 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: george.yoshida,thomas.wouters Build Had Warnings: warnings test sincerely, -The Buildbot From buildbot at python.org Tue Apr 25 19:50:49 2006 From: buildbot at python.org (buildbot at python.org) Date: Tue, 25 Apr 2006 17:50:49 +0000 Subject: [Python-checkins] buildbot failure in sparc Ubuntu dapper 2.4 Message-ID: <20060425175049.EAE571E4004@bag.python.org> The Buildbot has detected a new failure of sparc Ubuntu dapper 2.4. Full details are available at: http://www.python.org/dev/buildbot/all/sparc%2520Ubuntu%2520dapper%25202.4/builds/38 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch branches/release24-maint] HEAD Blamelist: thomas.wouters BUILD FAILED: failed compile sincerely, -The Buildbot From python-checkins at python.org Tue Apr 25 20:26:09 2006 From: python-checkins at python.org (thomas.heller) Date: Tue, 25 Apr 2006 20:26:09 +0200 (CEST) Subject: [Python-checkins] r45717 - python/trunk/Modules/_ctypes/libffi/src/powerpc/ffi_darwin.c Message-ID: <20060425182609.B66CF1E4004@bag.python.org> Author: thomas.heller Date: Tue Apr 25 20:26:08 2006 New Revision: 45717 Modified: python/trunk/Modules/_ctypes/libffi/src/powerpc/ffi_darwin.c Log: Fix compiler warnings on Darwin. Patch by Brett Canon, see https://sourceforge.net/tracker/?func=detail&atid=532156&aid=1475959&group_id=71702 Modified: python/trunk/Modules/_ctypes/libffi/src/powerpc/ffi_darwin.c ============================================================================== --- python/trunk/Modules/_ctypes/libffi/src/powerpc/ffi_darwin.c (original) +++ python/trunk/Modules/_ctypes/libffi/src/powerpc/ffi_darwin.c Tue Apr 25 20:26:08 2006 @@ -380,18 +380,18 @@ extern void ffi_call_AIX(/*@out@*/ extended_cif *, unsigned, unsigned, /*@out@*/ unsigned *, - void (*fn)(), - void (*fn2)()); + void (*fn)(void), + void (*fn2)(extended_cif *, unsigned *const)); extern void ffi_call_DARWIN(/*@out@*/ extended_cif *, unsigned, unsigned, /*@out@*/ unsigned *, - void (*fn)(), - void (*fn2)()); + void (*fn)(void), + void (*fn2)(extended_cif *, unsigned *const)); /*@=declundef@*/ /*@=exportheader@*/ void ffi_call(/*@dependent@*/ ffi_cif *cif, - void (*fn)(), + void (*fn)(void), /*@out@*/ void *rvalue, /*@dependent@*/ void **avalue) { From buildbot at python.org Tue Apr 25 21:43:37 2006 From: buildbot at python.org (buildbot at python.org) Date: Tue, 25 Apr 2006 19:43:37 +0000 Subject: [Python-checkins] buildbot warnings in alpha Debian trunk Message-ID: <20060425194337.DA6351E4004@bag.python.org> The Buildbot has detected a new failure of alpha Debian trunk. Full details are available at: http://www.python.org/dev/buildbot/all/alpha%2520Debian%2520trunk/builds/17 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: thomas.heller Build Had Warnings: warnings test sincerely, -The Buildbot From python-checkins at python.org Tue Apr 25 22:12:45 2006 From: python-checkins at python.org (guido.van.rossum) Date: Tue, 25 Apr 2006 22:12:45 +0200 (CEST) Subject: [Python-checkins] r45718 - python/trunk/Lib/threading.py Message-ID: <20060425201245.D9AD61E402A@bag.python.org> Author: guido.van.rossum Date: Tue Apr 25 22:12:45 2006 New Revision: 45718 Modified: python/trunk/Lib/threading.py Log: Implement MvL's improvement on __context__ in Condition; this can just call __context__ on the underlying lock. (The same change for Semaphore does *not* work!) Modified: python/trunk/Lib/threading.py ============================================================================== --- python/trunk/Lib/threading.py (original) +++ python/trunk/Lib/threading.py Tue Apr 25 22:12:45 2006 @@ -164,7 +164,6 @@ self.__lock = lock # Export the lock's acquire() and release() methods self.acquire = lock.acquire - self.__enter__ = self.acquire self.release = lock.release # If the lock defines _release_save() and/or _acquire_restore(), # these override the default implementations (which just call @@ -184,10 +183,7 @@ self.__waiters = [] def __context__(self): - return self - - def __exit__(self, t, v, tb): - self.release() + return self.__lock.__context__() def __repr__(self): return "" % (self.__lock, len(self.__waiters)) From neal at metaslash.com Tue Apr 25 22:13:24 2006 From: neal at metaslash.com (Neal Norwitz) Date: Tue, 25 Apr 2006 16:13:24 -0400 Subject: [Python-checkins] Python Regression Test Failures basics (1) Message-ID: <20060425201324.GA5063@python.psfb.org> case $MAKEFLAGS in \ *-s*) CC='gcc -pthread' LDSHARED='gcc -pthread -shared' OPT='-g -Wall -Wstrict-prototypes' ./python -E ./setup.py -q build;; \ *) CC='gcc -pthread' LDSHARED='gcc -pthread -shared' OPT='-g -Wall -Wstrict-prototypes' ./python -E ./setup.py build;; \ esac running build running build_ext db.h: found (4, 1) in /usr/include db lib: using (4, 1) db-4.1 sqlite: found /usr/include/sqlite3.h /usr/include/sqlite3.h: version 3.2.1 INFO: Can't locate Tcl/Tk libs and/or headers running build_scripts [33726 refs] ./python -E -c 'import sys ; from distutils.util import get_platform ; print get_platform()+"-"+sys.version[0:3]' >platform [8656 refs] find ./Lib -name '*.py[co]' -print | xargs rm -f ./python -E -tt ./Lib/test/regrtest.py -l test_grammar test_opcodes test_operations test_builtin test_exceptions test_types test_MimeWriter test_StringIO test___all__ test___future__ test__locale test_aepack test_aepack skipped -- No module named aepack test_al test_al skipped -- No module named al test_anydbm test_applesingle test_applesingle skipped -- No module named macostools test_array test_ast test_asynchat test_atexit test_audioop test_augassign test_base64 test_bastion test_binascii test_binhex test_binop test_bisect test_bool test_bsddb test_bsddb185 test_bsddb185 skipped -- No module named bsddb185 test_bsddb3 test_bsddb3 skipped -- Use of the `bsddb' resource not enabled test_bufio test_bz2 test_cProfile test_calendar test_call test_capi test_cd test_cd skipped -- No module named cd test_cfgparser test_cgi test_charmapcodec test_cl test_cl skipped -- No module named cl test_class test_cmath test_cmd_line test_code test_codeccallbacks test_codecencodings_cn test_codecencodings_hk test_codecencodings_jp test_codecencodings_kr test_codecencodings_tw test_codecmaps_cn test_codecmaps_hk test_codecmaps_jp test_codecmaps_kr test_codecmaps_tw test_codecs test_codeop test_coding test_coercion test_colorsys test_commands test_compare test_compile test_compiler test_complex test_contains test_contextlib test_cookie test_cookielib test_copy test_copy_reg test_cpickle test_crypt test_csv test_ctypes test_curses test_curses skipped -- Use of the `curses' resource not enabled test_datetime test_dbm test_decimal test_decorators test_defaultdict test_deque test_descr test_descrtut test_dict test_difflib test_dircache test_dis test_distutils test_dl test_doctest test_doctest2 test_dumbdbm test_dummy_thread test_dummy_threading test_email test_email_codecs test_email_renamed test_enumerate test_eof test_errno test_exception_variations test_extcall test_fcntl test_file test_filecmp test_fileinput test_float test_fnmatch test_fork1 test_format test_fpformat test_frozen test_funcattrs test_functional test_future test_gc test_gdbm test_generators test_genexps test_getargs test_getargs2 test_getopt test_gettext test_gl test_gl skipped -- No module named gl test_glob test_global test_grp test_gzip test_hash test_hashlib test_heapq test_hexoct test_hmac test_hotshot test_htmllib test_htmlparser test_httplib test_imageop test_imaplib test_imgfile test_imgfile skipped -- No module named imgfile test_imp test_import test_importhooks test_index test_inspect test_ioctl test_ioctl skipped -- Unable to open /dev/tty test_isinstance test_iter test_iterlen test_itertools test_largefile test_linuxaudiodev test_linuxaudiodev skipped -- Use of the `audio' resource not enabled test_list test_locale test_logging test_long test_long_future test_longexp test_macfs test_macfs skipped -- No module named macfs test_macostools test_macostools skipped -- No module named macostools test_macpath test_mailbox test_marshal test_math test_md5 test_mhlib test_mimetools test_mimetypes test_minidom test_mmap test_module test_multibytecodec test_multibytecodec_support test_multifile test_mutants test_netrc test_new test_nis test_nis skipped -- Local domain name not set test_normalization test_ntpath test_old_mailbox test_openpty test_operator test_optparse test_os test_ossaudiodev test_ossaudiodev skipped -- Use of the `audio' resource not enabled test_parser test_peepholer test_pep247 test_pep263 test_pep277 test_pep277 skipped -- test works only on NT+ test_pep292 test_pep352 test_pickle test_pickletools test_pkg test_pkgimport test_platform test_plistlib test_plistlib skipped -- No module named plistlib test_poll test_popen [8661 refs] [8661 refs] [8661 refs] test_popen2 test_posix test_posixpath test_pow test_pprint test_profile test_profilehooks test_pty test_pwd test_pyclbr test_pyexpat test_queue test_quopri [8997 refs] [8997 refs] test_random test_re test_repr test_resource test_rfc822 test_rgbimg test_richcmp test_robotparser test_runpy test_sax test_scope test_scriptpackages test_scriptpackages skipped -- No module named aetools test_select test_set test_sets test_sgmllib test_sha test_shelve test_shlex test_shutil test_signal test_site test_slice test_socket test_socket_ssl test_socket_ssl skipped -- Use of the `network' resource not enabled test_socketserver test_socketserver skipped -- Use of the `network' resource not enabled test_softspace test_sort test_sqlite test_startfile test_startfile skipped -- cannot import name startfile test_str test_strftime test_string test_stringprep test_strop test_strptime test_struct test_structseq test_subprocess [8656 refs] [8656 refs] [8656 refs] [8656 refs] [8656 refs] [8656 refs] [8656 refs] [8656 refs] [8656 refs] [8656 refs] [8656 refs] [8656 refs] [8656 refs] [8656 refs] [8656 refs] [8656 refs] [8656 refs] [8656 refs] [8656 refs] [8656 refs] this bit of output is from a test of stdout in a different process ... [8656 refs] [8656 refs] [8656 refs] test_sunaudiodev test_sunaudiodev skipped -- No module named sunaudiodev test_sundry test_symtable test_syntax test_sys [8656 refs] [8656 refs] test_tarfile test_tcl test_tcl skipped -- No module named _tkinter test_tempfile [8656 refs] test_textwrap test_thread test_threaded_import test_threadedtempfile test_threading test_threading_local test_threadsignals test_time test_timeout test_timeout skipped -- Use of the `network' resource not enabled test_tokenize test_trace test_traceback test_transformer test_tuple test_ucn test_unary test_unicode test_unicode_file test_unicode_file skipped -- No Unicode filesystem semantics on this platform. test_unicodedata test_unittest test_univnewlines test_unpack test_urllib test_urllib2 test_urllib2net test_urllib2net skipped -- Use of the `network' resource not enabled test_urllibnet test_urllibnet skipped -- Use of the `network' resource not enabled test_urlparse test_userdict test_userlist test_userstring test_uu test_wait3 test_wait4 test_warnings test_wave test_weakref test_whichdb test_winreg test_winreg skipped -- No module named _winreg test_winsound test_winsound skipped -- No module named winsound test_with test_with skipped -- cannot import name GeneratorContextManager test_xdrlib test_xml_etree test_xml_etree_c test_xmllib test_xmlrpc test_xpickle test_xrange test_zipfile test_zipimport test_zlib 283 tests OK. 31 tests skipped: test_aepack test_al test_applesingle test_bsddb185 test_bsddb3 test_cd test_cl test_curses test_gl test_imgfile test_ioctl test_linuxaudiodev test_macfs test_macostools test_nis test_ossaudiodev test_pep277 test_plistlib test_scriptpackages test_socket_ssl test_socketserver test_startfile test_sunaudiodev test_tcl test_timeout test_unicode_file test_urllib2net test_urllibnet test_winreg test_winsound test_with 2 skips unexpected on linux2: test_ioctl test_with [413972 refs] ./python -E -tt ./Lib/test/regrtest.py -l test_grammar test_opcodes test_operations test_builtin test_exceptions test_types test_MimeWriter test_StringIO test___all__ test___future__ test__locale test_aepack test_aepack skipped -- No module named aepack test_al test_al skipped -- No module named al test_anydbm test_applesingle test_applesingle skipped -- No module named macostools test_array test_ast test_asynchat test_atexit test_audioop test_augassign test_base64 test_bastion test_binascii test_binhex test_binop test_bisect test_bool test_bsddb test_bsddb185 test_bsddb185 skipped -- No module named bsddb185 test_bsddb3 test_bsddb3 skipped -- Use of the `bsddb' resource not enabled test_bufio test_bz2 test_cProfile test_calendar test_call test_capi test_cd test_cd skipped -- No module named cd test_cfgparser test_cgi test_charmapcodec test_cl test_cl skipped -- No module named cl test_class test_cmath test_cmd_line test_code test_codeccallbacks test_codecencodings_cn test_codecencodings_hk test_codecencodings_jp test_codecencodings_kr test_codecencodings_tw test_codecmaps_cn test_codecmaps_hk test_codecmaps_jp test_codecmaps_kr test_codecmaps_tw test_codecs test_codeop test_coding test_coercion test_colorsys test_commands test_compare test_compile test_compiler test_complex test_contains test_contextlib test_cookie test_cookielib test_copy test_copy_reg test_cpickle test_crypt test_csv test_ctypes test test_ctypes failed -- Traceback (most recent call last): File "/home/neal/python/trunk/Lib/ctypes/test/test_python_api.py", line 47, in test_PyInt_Long self.failUnlessEqual(grc(res), ref42 + 1) AssertionError: 338 != 339 test_curses test_curses skipped -- Use of the `curses' resource not enabled test_datetime test_dbm test_decimal test_decorators test_defaultdict test_deque test_descr test_descrtut test_dict test_difflib test_dircache test_dis test_distutils test_dl test_doctest test_doctest2 test_dumbdbm test_dummy_thread test_dummy_threading test_email test_email_codecs test_email_renamed test_enumerate test_eof test_errno test_exception_variations test_extcall test_fcntl test_file test_filecmp test_fileinput test_float test_fnmatch test_fork1 test_format test_fpformat test_frozen test_funcattrs test_functional test_future test_gc test_gdbm test_generators test_genexps test_getargs test_getargs2 test_getopt test_gettext test_gl test_gl skipped -- No module named gl test_glob test_global test_grp test_gzip test_hash test_hashlib test_heapq test_hexoct test_hmac test_hotshot test_htmllib test_htmlparser test_httplib test_imageop test_imaplib test_imgfile test_imgfile skipped -- No module named imgfile test_imp test_import test_importhooks test_index test_inspect test_ioctl test_ioctl skipped -- Unable to open /dev/tty test_isinstance test_iter test_iterlen test_itertools test_largefile test_linuxaudiodev test_linuxaudiodev skipped -- Use of the `audio' resource not enabled test_list test_locale test_logging test_long test_long_future test_longexp test_macfs test_macfs skipped -- No module named macfs test_macostools test_macostools skipped -- No module named macostools test_macpath test_mailbox test_marshal test_math test_md5 test_mhlib test_mimetools test_mimetypes test_minidom test_mmap test_module test_multibytecodec test_multibytecodec_support test_multifile test_mutants test_netrc test_new test_nis test_nis skipped -- Local domain name not set test_normalization test_ntpath test_old_mailbox test_openpty test_operator test_optparse test_os test_ossaudiodev test_ossaudiodev skipped -- Use of the `audio' resource not enabled test_parser test_peepholer test_pep247 test_pep263 test_pep277 test_pep277 skipped -- test works only on NT+ test_pep292 test_pep352 test_pickle test_pickletools test_pkg test_pkgimport test_platform test_plistlib test_plistlib skipped -- No module named plistlib test_poll test_popen [8661 refs] [8661 refs] [8661 refs] test_popen2 test_posix test_posixpath test_pow test_pprint test_profile test_profilehooks test_pty test_pwd test_pyclbr test_pyexpat test_queue test_quopri [8997 refs] [8997 refs] test_random test_re test_repr test_resource test_rfc822 test_rgbimg test_richcmp test_robotparser test_runpy test_sax test_scope test_scriptpackages test_scriptpackages skipped -- No module named aetools test_select test_set test_sets test_sgmllib test_sha test_shelve test_shlex test_shutil test_signal test_site test_slice test_socket test_socket_ssl test_socket_ssl skipped -- Use of the `network' resource not enabled test_socketserver test_socketserver skipped -- Use of the `network' resource not enabled test_softspace test_sort test_sqlite test_startfile test_startfile skipped -- cannot import name startfile test_str test_strftime test_string test_stringprep test_strop test_strptime test_struct test_structseq test_subprocess [8656 refs] [8656 refs] [8656 refs] [8656 refs] [8656 refs] [8656 refs] [8656 refs] [8656 refs] [8656 refs] [8656 refs] [8656 refs] [8656 refs] [8656 refs] [8656 refs] [8656 refs] [8656 refs] [8656 refs] [8656 refs] [8656 refs] [8656 refs] this bit of output is from a test of stdout in a different process ... [8656 refs] [8656 refs] [8656 refs] test_sunaudiodev test_sunaudiodev skipped -- No module named sunaudiodev test_sundry test_symtable test_syntax test_sys [8656 refs] [8656 refs] test_tarfile test_tcl test_tcl skipped -- No module named _tkinter test_tempfile [8656 refs] test_textwrap test_thread test_threaded_import test_threadedtempfile test_threading test_threading_local test_threadsignals test_time test_timeout test_timeout skipped -- Use of the `network' resource not enabled test_tokenize test_trace test_traceback test_transformer test_tuple test_ucn test_unary test_unicode test_unicode_file test_unicode_file skipped -- No Unicode filesystem semantics on this platform. test_unicodedata test_unittest test_univnewlines test_unpack test_urllib test_urllib2 test_urllib2net test_urllib2net skipped -- Use of the `network' resource not enabled test_urllibnet test_urllibnet skipped -- Use of the `network' resource not enabled test_urlparse test_userdict test_userlist test_userstring test_uu test_wait3 test_wait4 test_warnings test_wave test_weakref test_whichdb test_winreg test_winreg skipped -- No module named _winreg test_winsound test_winsound skipped -- No module named winsound test_with test_with skipped -- cannot import name GeneratorContextManager test_xdrlib test_xml_etree test_xml_etree_c test_xmllib test_xmlrpc test_xpickle test_xrange test_zipfile test_zipimport test_zlib 282 tests OK. 1 test failed: test_ctypes 31 tests skipped: test_aepack test_al test_applesingle test_bsddb185 test_bsddb3 test_cd test_cl test_curses test_gl test_imgfile test_ioctl test_linuxaudiodev test_macfs test_macostools test_nis test_ossaudiodev test_pep277 test_plistlib test_scriptpackages test_socket_ssl test_socketserver test_startfile test_sunaudiodev test_tcl test_timeout test_unicode_file test_urllib2net test_urllibnet test_winreg test_winsound test_with 2 skips unexpected on linux2: test_ioctl test_with [413377 refs] make: *** [test] Error 1 From buildbot at python.org Tue Apr 25 23:12:47 2006 From: buildbot at python.org (buildbot at python.org) Date: Tue, 25 Apr 2006 21:12:47 +0000 Subject: [Python-checkins] buildbot warnings in x86 Ubuntu dapper (icc) trunk Message-ID: <20060425211247.9F8891E4009@bag.python.org> The Buildbot has detected a new failure of x86 Ubuntu dapper (icc) trunk. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520Ubuntu%2520dapper%2520%2528icc%2529%2520trunk/builds/254 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: guido.van.rossum Build Had Warnings: warnings test sincerely, -The Buildbot From python-checkins at python.org Wed Apr 26 00:32:07 2006 From: python-checkins at python.org (phillip.eby) Date: Wed, 26 Apr 2006 00:32:07 +0200 (CEST) Subject: [Python-checkins] r45719 - sandbox/trunk/setuptools/doc sandbox/trunk/setuptools/doc/formats.txt Message-ID: <20060425223207.BD0B21E4009@bag.python.org> Author: phillip.eby Date: Wed Apr 26 00:32:07 2006 New Revision: 45719 Added: sandbox/trunk/setuptools/doc/ sandbox/trunk/setuptools/doc/formats.txt (contents, props changed) Log: Add "internals" documentation describing file formats, sys.path manipulation, zip file issues, file naming, etc., mostly by reference to other documentation and a few distutils-sig posts. Added: sandbox/trunk/setuptools/doc/formats.txt ============================================================================== --- (empty file) +++ sandbox/trunk/setuptools/doc/formats.txt Wed Apr 26 00:32:07 2006 @@ -0,0 +1,645 @@ +===================================== +The Internal Structure of Python Eggs +===================================== + +STOP! This is not the first document you should read! + +This document assumes you have at least some passing familiarity with +*using* setuptools, the ``pkg_resources`` module, and EasyInstall. It +does not attempt to explain basic concepts like inter-project +dependencies, nor does it contain detailed lexical syntax for most +file formats. Neither does it explain concepts like "namespace +packages" or "resources" in any detail, as all of these subjects are +covered at length in the setuptools developer's guide and the +``pkg_resources`` reference manual. + +Instead, this is **internal** documentation for how those concepts and +features are *implemented* in concrete terms. It is intended for people +who are working on the setuptools code base, who want to be able to +troubleshoot setuptools problems, want to write code that reads the file +formats involved, or want to otherwise tinker with setuptools-generated +files and directories. + +Note, however, that these are all internal implementation details and +are therefore subject to change; stick to the published API if you don't +want to be responsible for keeping your code from breaking when +setuptools changes. You have been warned. + + +.. contents:: **Table of Contents** + + +---------------------- +Eggs and their Formats +---------------------- + +A "Python egg" is a logical structure embodying the release of a +specific version of a Python project, comprising its code, resources, +and metadata. There are multiple formats that can be used to physically +encode a Python egg, and others can be developed. However, a key +principle of Python eggs is that they should be discoverable and +importable. That is, it should be possible for a Python application to +easily and efficiently find out what eggs are present on a system, and +to ensure that the desired eggs' contents are importable. + +There are two basic formats currently implemented for Python eggs: + +1. ``.egg`` format: a directory or zipfile *containing* the project's + code and resources, along with an ``EGG-INFO`` subdirectory that + contains the project's metadata + +2. ``.egg-info`` format: a file or directory placed *adjacent* to the + project's code and resources, that directly contains the project's + metadata. + +Both formats can include arbitrary Python code and resources, including +static data files, package and non-package directories, Python +modules, C extension modules, and so on. But each format is optimized +for different purposes. + +The ``.egg`` format is well-suited to distribution and the easy +uninstallation or upgrades of code, since the project is essentially +self-contained within a single directory or file, unmingled with any +other projects' code or resources. It also makes it possible to have +multiple versions of a project simultaneously installed, such that +individual programs can select the versions they wish to use. + +The ``.egg-info`` format, on the other hand, was created to support +backward-compatibility, performance, and ease of installation for system +packaging tools that expect to install all projects' code and resources +to a single directory (e.g. ``site-packages``). Placing the metadata +in that same directory simplifies the installation process, since it +isn't necessary to create ``.pth`` files or otherwise modify +``sys.path`` to include each installed egg. + +Its disadvantage, however, is that it provides no support for clean +uninstallation or upgrades, and of course only a single version of a +project can be installed to a given directory. Thus, support from a +package management tool is required. (This is why setuptools' "install" +command refers to this type of egg installation as "single-version, +externally managed".) Also, they lack sufficient data to allow them to +be copied from their installation source. easy_install can "ship" an +application by copying ``.egg`` files or directories to a target +location, but it cannot do this for ``.egg-info`` installs, because +there is no way to tell what code and resources belong to a particular +egg -- there may be several eggs "scrambled" together in a single +installation location, and the ``.egg-info`` format does not currently +include a way to list the files that were installed. (This may change +in a future version.) + + +Code and Resources +================== + +The layout of the code and resources is dictated by Python's normal +import layout, relative to the egg's "base location". + +For the ``.egg`` format, the base location is the ``.egg`` itself. That +is, adding the ``.egg`` filename or directory name to ``sys.path`` +makes its contents importable. + +For the ``.egg-info`` format, however, the base location is the +directory that *contains* the ``.egg-info``, and thus it is the +directory that must be added to ``sys.path`` to make the egg importable. +(Note that this means that the "normal" installation of a package to a +``sys.path`` directory is sufficient to make it an "egg" if it has an +``.egg-info`` file or directory installed alongside of it.) + + +Project Metadata +================= + +If eggs contained only code and resources, there would of course be +no difference between them and any other directory or zip file on +``sys.path``. Thus, metadata must also be included, using a metadata +file or directory. + +For the ``.egg`` format, the metadata is placed in an ``EGG-INFO`` +subdirectory, directly within the ``.egg`` file or directory. For the +``.egg-info`` format, metadata is stored directly within the +``.egg-info`` directory itself. + +The minimum project metadata that all eggs must have is a standard +Python ``PKG-INFO`` file, named ``PKG-INFO`` and placed within the +metadata directory appropriate to the format. Because it's possible for +this to be the only metadata file included, ``.egg-info`` format eggs +are not required to be a directory; they can just be a ``.egg-info`` +file that directly contains the ``PKG-INFO`` metadata. This eliminates +the need to create a directory just to store one file. This option is +*not* available for ``.egg`` formats, since setuptools always includes +other metadata. (In fact, setuptools itself never generates +``.egg-info`` files, either; the support for using files was added so +that the requirement could easily be satisfied by other tools, such +as the distutils in Python 2.5). + +In addition to the ``PKG-INFO`` file, an egg's metadata directory may +also include files and directories representing various forms of +optional standard metadata (see the section on `Standard Metadata`_, +below) or user-defined metadata required by the project. For example, +some projects may define a metadata format to describe their application +plugins, and metadata in this format would then be included by plugin +creators in their projects' metadata directories. + + +Filename-Embedded Metadata +========================== + +To allow introspection of installed projects and runtime resolution of +inter-project dependencies, a certain amount of information is embedded +in egg filenames. At a minimum, this includes the project name, and +ideally will also include the project version number. Optionally, it +can also include the target Python version and required runtime +platform if platform-specific C code is included. The syntax of an +egg filename is as follows:: + + name [ "-" version [ "-py" pyver [ "-" required_platform]]] "." ext + +The "name" and "version" should be escaped using the ``to_filename()`` +function provided by ``pkg_resources``, after first processing them with +``safe_name()`` and ``safe_version()`` respectively. These latter two +functions can also be used to later "unescape" these parts of the +filename. (For a detailed description of these transformations, please +see the "Parsing Utilities" section of the ``pkg_resources`` manual.) + +The "pyver" string is the Python major version, as found in the first +3 characters of ``sys.version``. "required_platform" is essentially +a distutils ``get_platform()`` string, but with enhancements to properly +distinguish Mac OS versions. (See the ``get_build_platform()`` +documentation in the "Platform Utilities" section of the +``pkg_resources`` manual for more details.) + +Finally, the "ext" is either ``.egg`` or ``.egg-info``, as appropriate +for the egg's format. + +Normally, an egg's filename should include at least the project name and +version, as this allows the runtime system to find desired project +versions without having to read the egg's PKG-INFO to determine its +version number. + +Setuptools, however, only includes the version number in the filename +when an ``.egg`` file is built using the ``bdist_egg`` command, or when +an ``.egg-info`` directory is being installed by the +``install_egg_info`` command. When generating metadata for use with the +original source tree, it only includes the project name, so that the +directory will not have to be renamed each time the project's version +changes. + +This is especially important when version numbers change frequently, and +the source metadata directory is kept under version control with the +rest of the project. (As would be the case when the project's source +includes project-defined metadata that is not generated from by +setuptools from data in the setup script.) + + +Egg Links +========= + +In addition to the ``.egg`` and ``.egg-info`` formats, there is a third +egg-related extension that you may encounter on occasion: ``.egg-link`` +files. + +These files are not eggs, strictly speaking. They simply provide a way +to reference an egg that is not physically installed in the desired +location. They exist primarily as a cross-platform alternative to +symbolic links, to support "installing" code that is being developed in +a different location than the desired installation location. For +example, if a user is developing an application plugin in their home +directory, but the plugin needs to be "installed" in an application +plugin directory, running "setup.py develop -md /path/to/app/plugins" +will install an ``.egg-link`` file in ``/path/to/app/plugins``, that +tells the egg runtime system where to find the actual egg (the user's +project source directory and its ``.egg-info`` subdirectory). + +``.egg-link`` files are named following the format for ``.egg`` and +``.egg-info`` names, but only the project name is included; no version, +Python version, or platform information is included. When the runtime +searches for available eggs, ``.egg-link`` files are opened and the +actual egg file/directory name is read from them. + +Each ``.egg-link`` file should contain a single file or directory name, +with no newlines. This filename should be the base location of one or +more eggs. That is, the name must either end in ``.egg``, or else it +should be the parent directory of one or more ``.egg-info`` format eggs. + + + +----------------- +Standard Metadata +----------------- + +In addition to the minimum required ``PKG-INFO`` metadata, projects can +include a variety of standard metadata files or directories, as +described below. Except as otherwise noted, these files and directories +are automatically generated by setuptools, based on information supplied +in the setup script or through analysis of the project's code and +resources. + +Most of these files and directories are generated via "egg-info +writers" during execution of the setuptools ``egg_info`` command, and +are listed in the ``egg_info.writers`` entry point group defined by +setuptools' own ``setup.py`` file. + +Project authors can register their own metadata writers as entry points +in this group (as described in the setuptools manual under "Adding new +EGG-INFO Files") to cause setuptools to generate project-specific +metadata files or directories during execution of the ``egg_info`` +command. It is up to project authors to document these new metadata +formats, if they create any. + + +``.txt`` File Formats +===================== + +Files described in this section that have ``.txt`` extensions have a +simple lexical format consisting of a sequence of text lines, each line +terminated by a linefeed character (regardless of platform). Leading +and trailing whitespace on each line is ignored, as are blank lines and +lines whose first nonblank character is a ``#`` (comment symbol). (This +is the parsing format defined by the ``yield_lines()`` function of +the ``pkg_resources`` module.) + +All ``.txt`` files defined by this section follow this format, but some +are also "sectioned" files, meaning that their contents are divided into +sections, using square-bracketed section headers akin to Windows +``.ini`` format. Note that this does *not* imply that the lines within +the sections follow an ``.ini`` format, however. Please see an +individual metadata file's documentation for a description of what the +lines and section names mean in that particular file. + +Sectioned files can be parsed using the ``split_sections()`` function; +see the "Parsing Utilities" section of the ``pkg_resources`` manual for +for details. + + +Dependency Metadata +=================== + + +``requires.txt`` +---------------- + +This is a "sectioned" text file. Each section is a sequence of +"requirements", as parsed by the ``parse_requirements()`` function; +please see the ``pkg_resources`` manual for the complete requirement +parsing syntax. + +The first, unnamed section (i.e., before the first section header) in +this file is the project's core requirements, which must be installed +for the project to function. (Specified using the ``install_requires`` +keyword to ``setup()``). + +The remaining (named) sections describe the project's "extra" +requirements, as specified using the ``extras_require`` keyword to +``setup()``. The section name is the name of the optional feature, and +the section body lists that feature's dependencies. + +Note that it is not normally necessary to inspect this file directly; +``pkg_resources.Distribution`` objects have a ``requires()`` method +that can be used to obtain ``Requirement`` objects describing the +project's core and optional dependencies. + + + +``dependency_links.txt`` +------------------------ + +A list of dependency URLs, one per line, as specified using the +``dependency_links`` keyword to ``setup()``. These may be direct +download URLs, or the URLs of web pages containing direct download +links, and will be used by EasyInstall to find dependencies, as though +the user had manually provided them via the ``--find-links`` command +line option. Please see the setuptools manual and EasyInstall manual +for more information on specifying this option, and for information on +how EasyInstall processes ``--find-links`` URLs. + + +``depends.txt`` -- Obsolete, do not create! +------------------------------------------- + +This file follows an identical format to ``requires.txt``, but is +obsolete and should not be used. The earliest versions of setuptools +required users to manually create and maintain this file, so the runtime +still supports reading it, if it exists. The new filename was created +so that it could be automatically generated from ``setup()`` information +without overwriting an existing hand-created ``depends.txt``, if one +was already present in the project's source ``.egg-info`` directory. + + +``namespace_packages.txt`` -- Namespace Package Metadata +======================================================== + +A list of namespace package names, one per line, as supplied to the +``namespace_packages`` keyword to ``setup()``. Please see the manuals +for setuptools and ``pkg_resources`` for more information about +namespace packages. + + +``entry_points.txt`` -- "Entry Point"/Plugin Metadata +===================================================== + +This is a "sectioned" text file, whose contents encode the +``entry_points`` keyword supplied to ``setup()``. All sections are +named, as the section names specify the entry point groups in which the +corresponding section's entry points are registered. + +Each section is a sequence of "entry point" lines, each parseable using +the ``EntryPoint.parse`` classmethod; please see the ``pkg_resources`` +manual for the complete entry point parsing syntax. + +Note that it is not necessary to parse this file directly; the +``pkg_resources`` module provides a variety of APIs to locate and load +entry points automatically. Please see the setuptools and +``pkg_resources`` manuals for details on the nature and uses of entry +points. + + +The ``scripts`` Subdirectory +============================ + +This directory is currently only created for ``.egg`` files built by +the setuptools ``bdist_egg`` command. It will contain copies of all +of the project's "traditional" scripts (i.e., those specified using the +``scripts`` keyword to ``setup()``). This is so that they can be +reconstituted when an ``.egg`` file is installed. + +The scripts are placed here using the disutils' standard +``install_scripts`` command, so any ``#!`` lines reflect the Python +installation where the egg was built. But instead of copying the +scripts to the local script installation directory, EasyInstall writes +short wrapper scripts that invoke the original scripts from inside the +egg, after ensuring that sys.path includes the egg and any eggs it +depends on. For more about script wrappers, see the section below on +`Installation and Path Management Issues`_. + + +Zip Support Metadata +==================== + + +``native_libs.txt`` +------------------- + +A list of C extensions and other dynamic link libraries contained in +the egg, one per line. Paths are ``/``-separated and relative to the +egg's base location. + +This file is generated as part of ``bdist_egg`` processing, and as such +only appears in ``.egg`` files (and ``.egg`` directories created by +unpacking them). It is used to ensure that all libraries are extracted +from a zipped egg at the same time, in case there is any direct linkage +between them. Please see the `Zip File Issues`_ section below for more +information on library and resource extraction from ``.egg`` files. + + +``eager_resources.txt`` +----------------------- + +A list of resource files and/or directories, one per line, as specified +via the ``eager_resources`` keyword to ``setup()``. Paths are +``/``-separated and relative to the egg's base location. + +Resource files or directories listed here will be extracted +simultaneously, if any of the named resources are extracted, or if any +native libraries listed in ``native_libs.txt`` are extracted. Please +see the setuptools manual for details on what this feature is used for +and how it works, as well as the `Zip File Issues`_ section below. + + +``zip-safe`` and ``not-zip-safe`` +--------------------------------- + +These are zero-length files, and either one or the other should exist. +If ``zip-safe`` exists, it means that the project will work properly +when installedas an ``.egg`` zipfile, and conversely the existence of +``not-zip-safe`` means the project should not be installed as an +``.egg`` file. The ``zip_safe`` option to setuptools' ``setup()`` +determines which file will be written. If the option isn't provided, +setuptools attempts to make its own assessment of whether the package +can work, based on code and content analysis. + +If neither file is present at installation time, EasyInstall defaults +to assuming that the project should be unzipped. (Command-line options +to EasyInstall, however, take precedence even over an existing +``zip-safe`` or ``not-zip-safe`` file.) + +Note that these flag files appear only in ``.egg`` files generated by +``bdist_egg``, and in ``.egg`` directories created by unpacking such an +``.egg`` file. + + + +``top_level.txt`` -- Conflict Management Metadata +================================================= + +This file is a list of the top-level module or package names provided +by the project, one Python identifier per line. + +Subpackages are not included; a project containing both a ``foo.bar`` +and a ``foo.baz`` would include only one line, ``foo``, in its +``top_level.txt``. + +This data is used by ``pkg_resources`` at runtime to issue a warning if +an egg is added to ``sys.path`` when its contained packages may have +already been imported. + +(It was also once used to detect conflicts with non-egg packages at +installation time, but in more recent versions, setuptools installs eggs +in such a way that they always override non-egg packages, thus +preventing a problem from arising.) + + +``SOURCES.txt`` -- Source Files Manifest +======================================== + +This file is roughly equivalent to the distutils' ``MANIFEST`` file. +The differences are as follows: + +* The filenames always use ``/`` as a path separator, which must be + converted back to a platform-specific path whenever they are read. + +* The file is automatically generated by setuptools whenever the + ``egg_info`` or ``sdist`` commands are run, and it is *not* + user-editable. + +Although this metadata is included with distributed eggs, it is not +actually used at runtime for any purpose. Its function is to ensure +that setuptools-built *source* distributions can correctly discover +what files are part of the project's source, even if the list had been +generated using revision control metadata on the original author's +system. + +In other words, ``SOURCES.txt`` has little or no runtime value for being +included in distributed eggs, and it is possible that future versions of +the ``bdist_egg`` and ``install_egg_info`` commands will strip it before +installation or distribution. Therefore, do not rely on its being +available outside of an original source directory or source +distribution. + + +------------------------------ +Other Technical Considerations +------------------------------ + + +Zip File Issues +=============== + +Although zip files resemble directories, they are not fully +substitutable for them. Most platforms do not support loading dynamic +link libraries contained in zipfiles, so it is not possible to directly +import C extensions from ``.egg`` zipfiles. Similarly, there are many +existing libraries -- whether in Python or C -- that require actual +operating system filenames, and do not work with arbitrary "file-like" +objects or in-memory strings, and thus cannot operate directly on the +contents of zip files. + +To address these issues, the ``pkg_resources`` module provides a +"resource API" to support obtaining either the contents of a resource, +or a true operating system filename for the resource. If the egg +containing the resource is a directory, the resource's real filename +is simply returned. However, if the egg is a zipfile, then the +resource is first extracted to a cache directory, and the filename +within the cache is returned. + +The cache directory is determined by the ``pkg_resources`` API; please +see the ``set_cache_path()`` and ``get_default_cache()`` documentation +for details. + + +The Extraction Process +---------------------- + +Resources are extracted to a cache subdirectory whose name is based +on the enclosing ``.egg`` filename and the path to the resource. If +there is already a file of the correct name, size, and timestamp, its +filename is returned to the requester. Otherwise, the desired file is +extracted first to a temporary name generated using +``mkstemp(".$extract",target_dir)``, and then its timestamp is set to +match the one in the zip file, before renaming it to its final name. +(Some collision detection and resolution code is used to handle the +fact that Windows doesn't overwrite files when renaming.) + +If a resource directory is requested, all of its contents are +recursively extracted in this fashion, to ensure that the directory +name can be used as if it were valid all along. + +If the resource requested for extraction is listed in the +``native_libs.txt`` or ``eager_resources.txt`` metadata files, then +*all* resources listed in *either* file will be extracted before the +requested resource's filename is returned, thus ensuring that all +C extensions and data used by them will be simultaneously available. + + +Extension Import Wrappers +------------------------- + +Since Python's built-in zip import feature does not support loading +C extension modules from zipfiles, the setuptools ``bdist_egg`` command +generates special import wrappers to make it work. + +The wrappers are ``.py`` files (along with corresponding ``.pyc`` +and/or ``.pyo`` files) that have the same module name as the +corresponding C extension. These wrappers are located in the same +package directory (or top-level directory) within the zipfile, so that +say, ``foomodule.so`` will get a corresponding ``foo.py``, while +``bar/baz.pyd`` will get a corresponding ``bar/baz.py``. + +These wrapper files contain a short stanza of Python code that asks +``pkg_resources`` for the filename of the corresponding C extension, +then reloads the module using the obtained filename. This will cause +``pkg_resources`` to first ensure that all of the egg's C extensions +(and any accompanying "eager resources") are extracted to the cache +before attempting to link to the C library. + +Note, by the way, that ``.egg`` directories will also contain these +wrapper files. However, Python's default import priority is such that +C extensions take precedence over same-named Python modules, so the +import wrappers are ignored unless the egg is a zipfile. + + +Installation and Path Management Issues +======================================= + +Python's initial setup of ``sys.path`` is very dependent on the Python +version and installation platform, as well as how Python was started +(i.e., script vs. ``-c`` vs. ``-m`` vs. interactive interpreter). +In fact, Python also provides only two relatively robust ways to affect +``sys.path`` outside of direct manipulation in code: the ``PYTHONPATH`` +environment variable, and ``.pth`` files. + +However, with no cross-platform way to safely and persistently change +environment variables, this leaves ``.pth`` files as EasyInstall's only +real option for persistent configuration of ``sys.path``. + +But ``.pth`` files are rather strictly limited in what they are allowed +to do normally. They add directories only to the *end* of ``sys.path``, +after any locally-installed ``site-packages`` directory, and they are +only processed *in* the ``site-packages`` directory to start with. + +This is a double whammy for users who lack write access to that +directory, because they can't create a ``.pth`` file that Python will +read, and even if a sympathetic system administrator adds one for them +that calls ``site.addsitedir()`` to allow some other directory to +contain ``.pth`` files, they won't be able to install newer versions of +anything that's installed in the systemwide ``site-packages``, because +their paths will still be added *after* ``site-packages``. + +So EasyInstall applies two workarounds to solve these problems. + +The first is that EasyInstall leverages ``.pth`` files' "import" feature +to manipulate ``sys.path`` and ensure that anything EasyInstall adds +to a ``.pth`` file will always appear before both the standard library +and the local ``site-packages`` directories. Thus, it is always +possible for a user who can write a Python-read ``.pth`` file to ensure +that their packages come first in their own environment. + +Second, when installing to a ``PYTHONPATH`` directory (as opposed to +a "site" directory like ``site-packages``) EasyInstall will also install +a special version of the ``site`` module. Because it's in a +``PYTHONPATH`` directory, this module will get control before the +standard library version of ``site`` does. It will record the state of +``sys.path`` before invoking the "real" ``site`` module, and then +afterwards it processes any ``.pth`` files found in ``PYTHONPATH`` +directories, including all the fixups needed to ensure that eggs always +appear before the standard library in sys.path, but are in a relative +order to one another that is defined by their ``PYTHONPATH`` and +``.pth``-prescribed sequence. + +The net result of these changes is that ``sys.path`` order will be +as follows at runtime: + +1. The ``sys.argv[0]`` directory, or an emtpy string if no script + is being executed. + +2. All eggs installed by EasyInstall in any ``.pth`` file in each + ``PYTHONPATH`` directory, in order first by ``PYTHONPATH`` order, + then normal ``.pth`` processing order (which is to say alphabetical + by ``.pth`` filename, then by the order of listing within each + ``.pth`` file). + +3. All eggs installed by EasyInstall in any ``.pth`` file in each "site" + directory (such as ``site-packages``), following the same ordering + rules as for the ones on ``PYTHONPATH``. + +4. The ``PYTHONPATH`` directories themselves, in their original order + +5. Any paths from ``.pth`` files found on ``PYTHONPATH`` that were *not* + eggs installed by EasyInstall, again following the same relative + ordering rules. + +6. The standard library and "site" directories, along with the contents + of any ``.pth`` files found in the "site" directories. + +Notice that sections 1, 4, and 6 comprise the "normal" Python setup for +``sys.path``. Sections 2 and 3 are inserted to support eggs, and +section 5 emulates what the "normal" semantics of ``.pth`` files on +``PYTHONPATH`` would be if Python natively supported them. + +For further discussion of the tradeoffs that went into this design, as +well as notes on the actual magic inserted into ``.pth`` files to make +them do these things, please see also the following messages to the +distutils-SIG mailing list: + +* http://mail.python.org/pipermail/distutils-sig/2006-February/006026.html +* http://mail.python.org/pipermail/distutils-sig/2006-March/006123.html + From python-checkins at python.org Wed Apr 26 02:10:54 2006 From: python-checkins at python.org (phillip.eby) Date: Wed, 26 Apr 2006 02:10:54 +0200 (CEST) Subject: [Python-checkins] r45720 - sandbox/trunk/setuptools/doc/formats.txt Message-ID: <20060426001054.D24BA1E401C@bag.python.org> Author: phillip.eby Date: Wed Apr 26 02:10:53 2006 New Revision: 45720 Modified: sandbox/trunk/setuptools/doc/formats.txt Log: Forgot to document script wrappers Modified: sandbox/trunk/setuptools/doc/formats.txt ============================================================================== --- sandbox/trunk/setuptools/doc/formats.txt (original) +++ sandbox/trunk/setuptools/doc/formats.txt Wed Apr 26 02:10:53 2006 @@ -152,7 +152,7 @@ platform if platform-specific C code is included. The syntax of an egg filename is as follows:: - name [ "-" version [ "-py" pyver [ "-" required_platform]]] "." ext + name ["-" version ["-py" pyver ["-" required_platform]]] "." ext The "name" and "version" should be escaped using the ``to_filename()`` function provided by ``pkg_resources``, after first processing them with @@ -368,7 +368,7 @@ scripts to the local script installation directory, EasyInstall writes short wrapper scripts that invoke the original scripts from inside the egg, after ensuring that sys.path includes the egg and any eggs it -depends on. For more about script wrappers, see the section below on +depends on. For more about `script wrappers`_, see the section below on `Installation and Path Management Issues`_. @@ -643,3 +643,47 @@ * http://mail.python.org/pipermail/distutils-sig/2006-February/006026.html * http://mail.python.org/pipermail/distutils-sig/2006-March/006123.html + +Script Wrappers +--------------- + +EasyInstall never directly installs a project's original scripts to +a script installation directory. Instead, it writes short wrapper +scripts that first ensure that the project's dependencies are active +on sys.path, before invoking the original script. These wrappers +have a #! line that points to the version of Python that was used to +install them, and their second line is always a comment that indicates +the type of script wrapper, the project version required for the script +to run, and information identifying the script to be invoked. + +The format of this marker line is:: + + "# EASY-INSTALL-" script_type ": " tuple_of_strings "\n" + +The ``script_type`` is one of ``SCRIPT``, ``DEV-SCRIPT``, or +``ENTRY-SCRIPT``. The ``tuple_of_strings`` is a comma-separated +sequence of Python string constants. For ``SCRIPT`` and ``DEV-SCRIPT`` +wrappers, there are two strings: the project version requirement, and +the script name (as a filename within the ``scripts`` metadata +directory). For ``ENTRY-SCRIPT`` wrappers, there are three: +the project version requirement, the entry point group name, and the +entry point name. (See the "Automatic Script Creation" section in the +setuptools manual for more information about entry point scripts.) + +In each case, the project version requirement string will be a string +parseable with the ``pkg_resources`` modules' ``Requirement.parse()`` +classmethod. The only difference between a ``SCRIPT`` wrapper and a +``DEV-SCRIPT`` is that a ``DEV-SCRIPT`` actually executes the original +source script in the project's source tree, and is created when the +"setup.py develop" command is run. A ``SCRIPT`` wrapper, on the other +hand, uses the "installed" script written to the ``EGG-INFO/scripts`` +subdirectory of the corresponding ``.egg`` zipfile or directory. +(``.egg-info`` eggs do not have script wrappers associated with them, +except in the "setup.py develop" case.) + +The purpose of including the marker line in generated script wrappers is +to facilitate introspection of installed scripts, and their relationship +to installed eggs. For example, an uninstallation tool could use this +data to identify what scripts can safely be removed, and/or identify +what scripts would stop working if a particular egg is uninstalled. + From python-checkins at python.org Wed Apr 26 03:15:53 2006 From: python-checkins at python.org (tim.peters) Date: Wed, 26 Apr 2006 03:15:53 +0200 (CEST) Subject: [Python-checkins] r45721 - python/trunk/Lib/test/test_with.py Message-ID: <20060426011553.D37D51E400C@bag.python.org> Author: tim.peters Date: Wed Apr 26 03:15:53 2006 New Revision: 45721 Modified: python/trunk/Lib/test/test_with.py Log: Rev 45706 renamed stuff in contextlib.py, but didn't rename uses of it in test_with.py. As a result, test_with has been skipped (due to failing imports) on all buildbot boxes since. Alas, that's not a test failure -- you have to pay attention to the 1 skip unexpected on PLATFORM: test_with kinds of output at the ends of test runs to notice that this got broken. It's likely that more renaming in test_with.py would be desirable. Modified: python/trunk/Lib/test/test_with.py ============================================================================== --- python/trunk/Lib/test/test_with.py (original) +++ python/trunk/Lib/test/test_with.py Wed Apr 26 03:15:53 2006 @@ -10,13 +10,13 @@ import sys import unittest from collections import deque -from contextlib import GeneratorContextManager, contextmanager +from contextlib import GeneratorContext, contextfactory from test.test_support import run_unittest -class MockContextManager(GeneratorContextManager): +class MockContextManager(GeneratorContext): def __init__(self, gen): - GeneratorContextManager.__init__(self, gen) + GeneratorContext.__init__(self, gen) self.context_called = False self.enter_called = False self.exit_called = False @@ -24,16 +24,16 @@ def __context__(self): self.context_called = True - return GeneratorContextManager.__context__(self) + return GeneratorContext.__context__(self) def __enter__(self): self.enter_called = True - return GeneratorContextManager.__enter__(self) + return GeneratorContext.__enter__(self) def __exit__(self, type, value, traceback): self.exit_called = True self.exit_args = (type, value, traceback) - return GeneratorContextManager.__exit__(self, type, value, traceback) + return GeneratorContext.__exit__(self, type, value, traceback) def mock_contextmanager(func): @@ -495,7 +495,7 @@ self.assertAfterWithGeneratorInvariantsNoError(self.bar) def testRaisedStopIteration1(self): - @contextmanager + @contextfactory def cm(): yield @@ -523,7 +523,7 @@ self.assertRaises(StopIteration, shouldThrow) def testRaisedGeneratorExit1(self): - @contextmanager + @contextfactory def cm(): yield From buildbot at python.org Wed Apr 26 04:29:07 2006 From: buildbot at python.org (buildbot at python.org) Date: Wed, 26 Apr 2006 02:29:07 +0000 Subject: [Python-checkins] buildbot warnings in alpha Debian trunk Message-ID: <20060426022907.77AD41E400B@bag.python.org> The Buildbot has detected a new failure of alpha Debian trunk. Full details are available at: http://www.python.org/dev/buildbot/all/alpha%2520Debian%2520trunk/builds/19 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: tim.peters Build Had Warnings: warnings test sincerely, -The Buildbot From nnorwitz at gmail.com Wed Apr 26 05:21:03 2006 From: nnorwitz at gmail.com (Neal Norwitz) Date: Tue, 25 Apr 2006 20:21:03 -0700 Subject: [Python-checkins] r45712 - in python/trunk: Lib/test/test_pty.py Misc/ACKS Modules/fcntlmodule.c In-Reply-To: <20060425135324.234391E400B@bag.python.org> References: <20060425135324.234391E400B@bag.python.org> Message-ID: Should this be backported to 2.4? On 4/25/06, thomas.wouters wrote: > Author: thomas.wouters > Date: Tue Apr 25 15:53:23 2006 > New Revision: 45712 > > Modified: > python/trunk/Lib/test/test_pty.py > python/trunk/Misc/ACKS > python/trunk/Modules/fcntlmodule.c > Log: > > SF bug/patch #1433877: string parameter to ioctl not null terminated > > The new char-array used in ioctl calls wasn't explicitly NUL-terminated; > quite probably the cause for the test_pty failures on Solaris that we > circumvented earlier. (I wasn't able to reproduce it with this patch, but it > has been somewhat elusive to start with.) > > > > Modified: python/trunk/Lib/test/test_pty.py > ============================================================================== > --- python/trunk/Lib/test/test_pty.py (original) > +++ python/trunk/Lib/test/test_pty.py Tue Apr 25 15:53:23 2006 > @@ -4,13 +4,6 @@ > TEST_STRING_1 = "I wish to buy a fish license.\n" > TEST_STRING_2 = "For my pet fish, Eric.\n" > > -# Solaris (at least 2.9 and 2.10) seem to have a fickle isatty(). The first > -# test below, testing the result of os.openpty() for tty-ness, sometimes > -# (but not always) fails. The second isatty test, in the sub-process, always > -# works. Allow that fickle first test to fail on these platforms, since it > -# doesn't actually affect functionality. > -fickle_isatty = ["sunos5"] > - > if verbose: > def debug(msg): > print msg > @@ -54,7 +47,7 @@ > # " An optional feature could not be imported " ... ? > raise TestSkipped, "Pseudo-terminals (seemingly) not functional." > > - if not os.isatty(slave_fd) and sys.platform not in fickle_isatty: > + if not os.isatty(slave_fd): > raise TestFailed, "slave_fd is not a tty" > > debug("Writing to slave_fd") > > Modified: python/trunk/Misc/ACKS > ============================================================================== > --- python/trunk/Misc/ACKS (original) > +++ python/trunk/Misc/ACKS Tue Apr 25 15:53:23 2006 > @@ -33,6 +33,7 @@ > Luigi Ballabio > Michael J. Barber > Chris Barker > +Quentin Barnes > Cesar Eduardo Barros > Des Barry > Ulf Bartelt > > Modified: python/trunk/Modules/fcntlmodule.c > ============================================================================== > --- python/trunk/Modules/fcntlmodule.c (original) > +++ python/trunk/Modules/fcntlmodule.c Tue Apr 25 15:53:23 2006 > @@ -95,6 +95,7 @@ > static PyObject * > fcntl_ioctl(PyObject *self, PyObject *args) > { > +#define IOCTL_BUFSZ 1024 > int fd; > /* In PyArg_ParseTuple below, use the unsigned int 'I' format for > the signed int 'code' variable, because Python turns 0x8000000 > @@ -106,7 +107,7 @@ > char *str; > Py_ssize_t len; > int mutate_arg = 1; > - char buf[1024]; > + char buf[IOCTL_BUFSZ+1]; /* argument plus NUL byte */ > > if (PyArg_ParseTuple(args, "O&Iw#|i:ioctl", > conv_descriptor, &fd, &code, > @@ -114,8 +115,9 @@ > char *arg; > > if (mutate_arg) { > - if (len <= sizeof buf) { > + if (len <= IOCTL_BUFSZ) { > memcpy(buf, str, len); > + buf[len] = '\0'; > arg = buf; > } > else { > @@ -123,13 +125,14 @@ > } > } > else { > - if (len > sizeof buf) { > + if (len > IOCTL_BUFSZ) { > PyErr_SetString(PyExc_ValueError, > "ioctl string arg too long"); > return NULL; > } > else { > memcpy(buf, str, len); > + buf[len] = '\0'; > arg = buf; > } > } > @@ -141,7 +144,7 @@ > else { > ret = ioctl(fd, code, arg); > } > - if (mutate_arg && (len < sizeof buf)) { > + if (mutate_arg && (len < IOCTL_BUFSZ)) { > memcpy(str, buf, len); > } > if (ret < 0) { > @@ -159,12 +162,13 @@ > PyErr_Clear(); > if (PyArg_ParseTuple(args, "O&Is#:ioctl", > conv_descriptor, &fd, &code, &str, &len)) { > - if (len > sizeof buf) { > + if (len > IOCTL_BUFSZ) { > PyErr_SetString(PyExc_ValueError, > "ioctl string arg too long"); > return NULL; > } > memcpy(buf, str, len); > + buf[len] = '\0'; > Py_BEGIN_ALLOW_THREADS > ret = ioctl(fd, code, buf); > Py_END_ALLOW_THREADS > @@ -195,6 +199,7 @@ > return NULL; > } > return PyInt_FromLong((long)ret); > +#undef IOCTL_BUFSZ > } > > PyDoc_STRVAR(ioctl_doc, > _______________________________________________ > Python-checkins mailing list > Python-checkins at python.org > http://mail.python.org/mailman/listinfo/python-checkins > From python-checkins at python.org Wed Apr 26 07:15:42 2006 From: python-checkins at python.org (fred.drake) Date: Wed, 26 Apr 2006 07:15:42 +0200 (CEST) Subject: [Python-checkins] r45722 - in python/trunk/Doc: Makefile.deps lib/libtrace.tex Message-ID: <20060426051542.5E8081E400B@bag.python.org> Author: fred.drake Date: Wed Apr 26 07:15:41 2006 New Revision: 45722 Modified: python/trunk/Doc/Makefile.deps python/trunk/Doc/lib/libtrace.tex Log: markup fixes, cleanup Modified: python/trunk/Doc/Makefile.deps ============================================================================== --- python/trunk/Doc/Makefile.deps (original) +++ python/trunk/Doc/Makefile.deps Wed Apr 26 07:15:41 2006 @@ -180,6 +180,7 @@ lib/libprofile.tex \ lib/libhotshot.tex \ lib/libtimeit.tex \ + lib/libtrace.tex \ lib/libcgi.tex \ lib/libcgitb.tex \ lib/liburllib.tex \ Modified: python/trunk/Doc/lib/libtrace.tex ============================================================================== --- python/trunk/Doc/lib/libtrace.tex (original) +++ python/trunk/Doc/lib/libtrace.tex Wed Apr 26 07:15:41 2006 @@ -9,7 +9,7 @@ list functions executed during a program run. It can be used in another program or from the command line. -\subsection{Command Line Usage} +\subsection{Command Line Usage\label{trace-cli}} The \module{trace} module can be invoked from the command line. It can be as simple as @@ -19,39 +19,62 @@ \end{verbatim} The above will generate annotated listings of all Python modules imported -during the execution of \code{somefile.py}. +during the execution of \file{somefile.py}. -\subsection{Command Line Arguments} +The following command-line arguments are supported: \begin{description} -\item[--trace, -t]{Display lines as they are executed.} -\item[--count, -c]{Produce a set of annotated listing files upon program -completion that shows how many times each statement was executed.} -\item[--report, -r]{Produce an annotated list from an earlier program run that -used the \code{--count} and \code{--file} arguments.} -\item[--no-report, -R]{Do not generate annotated listings. This is useful -if you intend to make several runs with \code{--count} then produce a single -set of annotated listings at the end.} -\item[--listfuncs, -l]{List the functions executed by running the program.} -\item[--trackcalls, -T]{Generate calling relationships exposed by running the -program.} -\item[--file, -f]{Name a file containing (or to contain) counts.} -\item[--coverdir, -C]{Name a directory in which to save annotated listing -files.} -\item[--missing, -m]{When generating annotated listings, mark lines which -were not executed with \code{>>>>>>}.} -\item[--summary -s]{When using \code{--count} or \code{--report}, write a -brief summary to stdout for each file processed.} -\item[--ignore-module]{Ignore the named module and its submodules (if it is -a package). May be given multiple times.} -\item[--ignore-dir]{Ignore all modules and packages in the named directory -and subdirectories. May be given multiple times.} -\end{description} +\item[\longprogramopt{trace}, \programopt{-t}] +Display lines as they are executed. + +\item[\longprogramopt{count}, \programopt{-c}] +Produce a set of annotated listing files upon program +completion that shows how many times each statement was executed. + +\item[\longprogramopt{report}, \programopt{-r}] +Produce an annotated list from an earlier program run that +used the \longprogramopt{count} and \longprogramopt{file} arguments. + +\item[\longprogramopt{no-report}, \programopt{-R}] +Do not generate annotated listings. This is useful if you intend to make +several runs with \longprogramopt{count} then produce a single set +of annotated listings at the end. + +\item[\longprogramopt{listfuncs}, \programopt{-l}] +List the functions executed by running the program. + +\item[\longprogramopt{trackcalls}, \programopt{-T}] +Generate calling relationships exposed by running the program. + +\item[\longprogramopt{file}, \programopt{-f}] +Name a file containing (or to contain) counts. + +\item[\longprogramopt{coverdir}, \programopt{-C}] +Name a directory in which to save annotated listing files. -\subsection{Program Usage} +\item[\longprogramopt{missing}, \programopt{-m}] +When generating annotated listings, mark lines which +were not executed with \code{>}\code{>}\code{>}\code{>}\code{>}\code{>}. -\begin{classdesc}{Trace}{\optional{count=1\optional{,trace=1\optional{,countfuncs=0\optional{,countcallers=0\optional{,ignoremods=()\optional{,ignoredirs=()\optional{,infile=None\optional{,outfile=None}}}}}}}}} +\item[\longprogramopt{summary}, \programopt{-s}] +When using \longprogramopt{count} or \longprogramopt{report}, write a +brief summary to stdout for each file processed. +\item[\longprogramopt{ignore-module}] +Ignore the named module and its submodules (if it is +a package). May be given multiple times. + +\item[\longprogramopt{ignore-dir}] +Ignore all modules and packages in the named directory +and subdirectories. May be given multiple times. +\end{description} + +\subsection{Programming Interface\label{trace-api}} + +\begin{classdesc}{Trace}{\optional{count=1\optional{, trace=1\optional{, + countfuncs=0\optional{, countcallers=0\optional{, + ignoremods=()\optional{, ignoredirs=()\optional{, + infile=None\optional{, outfile=None}}}}}}}}} Create an object to trace execution of a single statement or expression. All parameters are optional. \var{count} enables counting of line numbers. \var{trace} enables line execution tracing. \var{countfuncs} enables @@ -61,36 +84,41 @@ or packages should be ignored. \var{infile} is the file from which to read stored count information. \var{outfile} is a file in which to write updated count information. - \end{classdesc} \begin{methoddesc}[Trace]{run}{cmd} -Run \code{cmd} under control of the Trace object with the current tracing +Run \var{cmd} under control of the Trace object with the current tracing parameters. \end{methoddesc} -\begin{methoddesc}[Trace]{runctx}{cmd\optional{,globals=None\optional{,locals=None}}} -Run \code{cmd} under control of the Trace object with the current tracing +\begin{methoddesc}[Trace]{runctx}{cmd\optional{, globals=None\optional{, + locals=None}}} +Run \var{cmd} under control of the Trace object with the current tracing parameters in the defined global and local environments. If not defined, -\code{globals} and \code{locals} default to empty dictionaries. +\var{globals} and \var{locals} default to empty dictionaries. \end{methoddesc} \begin{methoddesc}[Trace]{runfunc}{func, *args, **kwds} -Call \code{function} with the given arguments under control of the Trace -object with the current tracing parameters. +Call \var{func} with the given arguments under control of the +\class{Trace} object with the current tracing parameters. \end{methoddesc} -\subsubsection{Example} +This is a simple example showing the use of this module: \begin{verbatim} import sys +import trace # create a Trace object, telling it what to ignore, and whether to # do tracing or line-counting or both. -tracer = trace.Trace(ignoredirs=[sys.prefix, sys.exec_prefix,], trace=0, - count=1) +tracer = trace.Trace( + ignoredirs=[sys.prefix, sys.exec_prefix], + trace=0, + count=1) + # run the new command using the given tracer tracer.run('main()') + # make a report, placing output in /tmp r = tracer.results() r.write_results(show_missing=True, coverdir="/tmp") From python-checkins at python.org Wed Apr 26 07:19:40 2006 From: python-checkins at python.org (fred.drake) Date: Wed, 26 Apr 2006 07:19:40 +0200 (CEST) Subject: [Python-checkins] r45723 - python/trunk/Doc/tut/tut.tex Message-ID: <20060426051940.0EE8A1E400B@bag.python.org> Author: fred.drake Date: Wed Apr 26 07:19:39 2006 New Revision: 45723 Modified: python/trunk/Doc/tut/tut.tex Log: minor adjustment suggested by Peter Gephardt Modified: python/trunk/Doc/tut/tut.tex ============================================================================== --- python/trunk/Doc/tut/tut.tex (original) +++ python/trunk/Doc/tut/tut.tex Wed Apr 26 07:19:39 2006 @@ -4771,7 +4771,7 @@ \begin{verbatim} >>> import urllib2 >>> for line in urllib2.urlopen('http://tycho.usno.navy.mil/cgi-bin/timer.pl'): -... if 'EST' in line: # look for Eastern Standard Time +... if 'EST' in line or 'EDT' in line: # look for Eastern Time ... print line
Nov. 25, 09:43:32 PM EST From python-checkins at python.org Wed Apr 26 07:34:04 2006 From: python-checkins at python.org (neal.norwitz) Date: Wed, 26 Apr 2006 07:34:04 +0200 (CEST) Subject: [Python-checkins] r45724 - python/trunk/Modules/gcmodule.c Message-ID: <20060426053404.BB60C1E400B@bag.python.org> Author: neal.norwitz Date: Wed Apr 26 07:34:03 2006 New Revision: 45724 Modified: python/trunk/Modules/gcmodule.c Log: Patch from Aldo Cortesi (OpenBSD buildbot owner). After the patch (45590) to add extra debug stats to the gc module, Python was crashing on OpenBSD due to: Fatal Python error: Interpreter not initialized (version mismatch?) This seems to occur due to calling collect() when initialized (in pythonrun.c) is set to 0. Now, the import will occur in the init function which shouldn't suffer this problem. Modified: python/trunk/Modules/gcmodule.c ============================================================================== --- python/trunk/Modules/gcmodule.c (original) +++ python/trunk/Modules/gcmodule.c Wed Apr 26 07:34:03 2006 @@ -75,6 +75,7 @@ DEBUG_OBJECTS | \ DEBUG_SAVEALL static int debug; +static PyObject *tmod = NULL; /*-------------------------------------------------------------------------- gc_refs values. @@ -734,7 +735,6 @@ PyGC_Head unreachable; /* non-problematic unreachable trash */ PyGC_Head finalizers; /* objects with, & reachable from, __del__ */ PyGC_Head *gc; - static PyObject *tmod = NULL; double t1 = 0.0; if (delstr == NULL) { @@ -743,12 +743,6 @@ Py_FatalError("gc couldn't allocate \"__del__\""); } - if (tmod == NULL) { - tmod = PyImport_ImportModule("time"); - if (tmod == NULL) - PyErr_Clear(); - } - if (debug & DEBUG_STATS) { if (tmod != NULL) { PyObject *f = PyObject_CallMethod(tmod, "time", NULL); @@ -1233,6 +1227,19 @@ Py_INCREF(garbage); if (PyModule_AddObject(m, "garbage", garbage) < 0) return; + + /* Importing can't be done in collect() because collect() + * can be called via PyGC_Collect() in Py_Finalize(). + * This wouldn't be a problem, except that is + * reset to 0 before calling collect which trips up + * the import and triggers an assertion. + */ + if (tmod == NULL) { + tmod = PyImport_ImportModule("time"); + if (tmod == NULL) + PyErr_Clear(); + } + #define ADD_INT(NAME) if (PyModule_AddIntConstant(m, #NAME, NAME) < 0) return ADD_INT(DEBUG_STATS); ADD_INT(DEBUG_COLLECTABLE); From python-checkins at python.org Wed Apr 26 08:26:14 2006 From: python-checkins at python.org (neal.norwitz) Date: Wed, 26 Apr 2006 08:26:14 +0200 (CEST) Subject: [Python-checkins] r45725 - python/trunk/Lib/test/test_pty.py Message-ID: <20060426062614.32EC91E400B@bag.python.org> Author: neal.norwitz Date: Wed Apr 26 08:26:12 2006 New Revision: 45725 Modified: python/trunk/Lib/test/test_pty.py Log: Fix this test on Solaris. There can be embedded \r, so don't just replace the one at the end. Modified: python/trunk/Lib/test/test_pty.py ============================================================================== --- python/trunk/Lib/test/test_pty.py (original) +++ python/trunk/Lib/test/test_pty.py Wed Apr 26 08:26:12 2006 @@ -24,11 +24,11 @@ # OSF/1 (Tru64) apparently turns \n into \r\r\n. if data.endswith('\r\r\n'): - return data[:-3] + '\n' + return data.replace('\r\r\n', '\n') # IRIX apparently turns \n into \r\n. if data.endswith('\r\n'): - return data[:-2] + '\n' + return data.replace('\r\n', '\n') return data From buildbot at python.org Wed Apr 26 10:45:18 2006 From: buildbot at python.org (buildbot at python.org) Date: Wed, 26 Apr 2006 08:45:18 +0000 Subject: [Python-checkins] buildbot warnings in hppa Ubuntu dapper trunk Message-ID: <20060426084518.EA2691E4014@bag.python.org> The Buildbot has detected a new failure of hppa Ubuntu dapper trunk. Full details are available at: http://www.python.org/dev/buildbot/all/hppa%2520Ubuntu%2520dapper%2520trunk/builds/298 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: neal.norwitz Build Had Warnings: warnings test sincerely, -The Buildbot From thomas at python.org Wed Apr 26 10:58:38 2006 From: thomas at python.org (Thomas Wouters) Date: Wed, 26 Apr 2006 10:58:38 +0200 Subject: [Python-checkins] r45712 - in python/trunk: Lib/test/test_pty.py Misc/ACKS Modules/fcntlmodule.c In-Reply-To: References: <20060425135324.234391E400B@bag.python.org> Message-ID: <9e804ac0604260158x1a3b37c9j398e02406d297ced@mail.gmail.com> On 4/26/06, Neal Norwitz wrote: > > Should this be backported to 2.4? On 4/25/06, thomas.wouters wrote: > > SF bug/patch #1433877: string parameter to ioctl not null terminated > > > > The new char-array used in ioctl calls wasn't explicitly NUL-terminated; > > quite probably the cause for the test_pty failures on Solaris that we > > circumvented earlier. (I wasn't able to reproduce it with this patch, > but it > > has been somewhat elusive to start with.) > > I think so, yes. The reason I haven't is that I want to make sure it doesn't break, say, the Solaris tests, before I did the actual backport. It hasn't so far, but given the elusiveness of the pty bug I'll wait until some more testruns have happened. -- Thomas Wouters Hi! I'm a .signature virus! copy me into your .signature file to help me spread! -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-checkins/attachments/20060426/bc29c4f4/attachment.html From python-checkins at python.org Wed Apr 26 12:20:29 2006 From: python-checkins at python.org (andrew.kuchling) Date: Wed, 26 Apr 2006 12:20:29 +0200 (CEST) Subject: [Python-checkins] r45726 - peps/trunk/pep-0000.txt peps/trunk/pep-0321.txt Message-ID: <20060426102029.488401E400C@bag.python.org> Author: andrew.kuchling Date: Wed Apr 26 12:20:27 2006 New Revision: 45726 Modified: peps/trunk/pep-0000.txt peps/trunk/pep-0321.txt Log: Withdraw PEP 321: datetime objects have a strptime() method now Modified: peps/trunk/pep-0000.txt ============================================================================== --- peps/trunk/pep-0000.txt (original) +++ peps/trunk/pep-0000.txt Wed Apr 26 12:20:27 2006 @@ -91,7 +91,6 @@ S 302 New Import Hooks JvR, Moore S 304 Controlling Generation of Bytecode Files Montanaro S 314 Metadata for Python Software Packages v1.1 Kuchling, Jones - S 321 Date/Time Parsing and Formatting Kuchling S 323 Copyable Iterators Martelli S 331 Locale-Independent Float/String Conversions Reis S 334 Simple Coroutines via SuspendIteration Evans @@ -220,6 +219,7 @@ SD 316 Programming by Contract for Python Way SR 317 Eliminate Implicit Exception Instantiation Taschuk SR 319 Python Synchronize/Asynchronize Block Pelletier + SR 321 Date/Time Parsing and Formatting Kuchling SR 325 Resource-Release Support for Generators Pedroni SR 326 A Case for Top and Bottom Values Carlson, Reedy SR 329 Treating Builtins as Constants in the Standard Library Hettinger @@ -379,7 +379,7 @@ SF 318 Decorators for Functions and Methods Smith, et al SR 319 Python Synchronize/Asynchronize Block Pelletier IF 320 Python 2.4 Release Schedule Warsaw, et al - S 321 Date/Time Parsing and Formatting Kuchling + SR 321 Date/Time Parsing and Formatting Kuchling SF 322 Reverse Iteration Hettinger S 323 Copyable Iterators Martelli SF 324 subprocess - New POSIX process module Astrand Modified: peps/trunk/pep-0321.txt ============================================================================== --- peps/trunk/pep-0321.txt (original) +++ peps/trunk/pep-0321.txt Wed Apr 26 12:20:27 2006 @@ -3,7 +3,7 @@ Version: $Revision$ Last-Modified: $Date$ Author: A.M. Kuchling -Status: Draft +Status: Withdrawn Type: Standards Track Content-Type: text/x-rst Python-Version: 2.4 From buildbot at python.org Wed Apr 26 12:28:39 2006 From: buildbot at python.org (buildbot at python.org) Date: Wed, 26 Apr 2006 10:28:39 +0000 Subject: [Python-checkins] buildbot warnings in sparc Ubuntu dapper trunk Message-ID: <20060426102840.0315D1E400B@bag.python.org> The Buildbot has detected a new failure of sparc Ubuntu dapper trunk. Full details are available at: http://www.python.org/dev/buildbot/all/sparc%2520Ubuntu%2520dapper%2520trunk/builds/163 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: neal.norwitz Build Had Warnings: warnings test sincerely, -The Buildbot From anthony at interlink.com.au Wed Apr 26 13:15:35 2006 From: anthony at interlink.com.au (Anthony Baxter) Date: Wed, 26 Apr 2006 21:15:35 +1000 Subject: [Python-checkins] TRUNK FREEZE from 00:00 UTC, 27th April 2006 for 2.5a2 Message-ID: <200604262115.39655.anthony@interlink.com.au> I'm going to be cutting 2.5a2 tomorrow. The trunk should be considered FROZEN from 00:00 UTC on the 27th of April (in about 12-13 hours time). Unless you're one of the release team, please don't make checkins to the trunk until the release is done. I'll send another message when it's done. Thanks, Anthony -- Anthony Baxter It's never too late to have a happy childhood. From python-checkins at python.org Wed Apr 26 13:50:05 2006 From: python-checkins at python.org (nick.coghlan) Date: Wed, 26 Apr 2006 13:50:05 +0200 (CEST) Subject: [Python-checkins] r45727 - python/trunk/Doc/lib/libcontextlib.tex Message-ID: <20060426115005.3311A1E400B@bag.python.org> Author: nick.coghlan Date: Wed Apr 26 13:50:04 2006 New Revision: 45727 Modified: python/trunk/Doc/lib/libcontextlib.tex Log: Fix an error in the last contextlib.closing example Modified: python/trunk/Doc/lib/libcontextlib.tex ============================================================================== --- python/trunk/Doc/lib/libcontextlib.tex (original) +++ python/trunk/Doc/lib/libcontextlib.tex Wed Apr 26 13:50:04 2006 @@ -149,8 +149,7 @@ block is exited. Context managers with a close method can use this context factory -directly without needing to implement their own -\method{__context__()} method. +to easily implement their own \method{__context__()} method. \begin{verbatim} from __future__ import with_statement from contextlib import closing @@ -158,7 +157,8 @@ class MyClass: def close(self): print "Closing", self - __context__ = closing + def __context__(self): + return closing(self) >>> with MyClass() as x: ... print "Hello from", x From ncoghlan at gmail.com Wed Apr 26 14:11:39 2006 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 26 Apr 2006 22:11:39 +1000 Subject: [Python-checkins] r45721 - python/trunk/Lib/test/test_with.py In-Reply-To: <20060426011553.D37D51E400C@bag.python.org> References: <20060426011553.D37D51E400C@bag.python.org> Message-ID: <444F637B.6080402@gmail.com> tim.peters wrote: > Author: tim.peters > Date: Wed Apr 26 03:15:53 2006 > New Revision: 45721 > > Modified: > python/trunk/Lib/test/test_with.py > Log: > Rev 45706 renamed stuff in contextlib.py, but didn't rename > uses of it in test_with.py. As a result, test_with has been skipped > (due to failing imports) on all buildbot boxes since. Alas, that's > not a test failure -- you have to pay attention to the > > 1 skip unexpected on PLATFORM: > test_with > > kinds of output at the ends of test runs to notice that this got > broken. That would be my fault - I've got about four unexpected skips I actually expect because I don't have the relevant external modules built. I must have missed this new one joining the list. > It's likely that more renaming in test_with.py would be desirable. Yep. I can see why you deferred fixing it, though :) I'm also curious as to why that test uses its own version of Nested rather than the one in contextlib. (granted, you can't inherit from the contextlib version, but containment should work fine) Anyway, I've created SF bug #1476845 to keep track of the things that need to be done to finish the PEP 343 terminology cleanup. (What we have now should be fine for alpha 2) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From python-checkins at python.org Wed Apr 26 14:21:07 2006 From: python-checkins at python.org (andrew.kuchling) Date: Wed, 26 Apr 2006 14:21:07 +0200 (CEST) Subject: [Python-checkins] r45728 - python/trunk/Doc/whatsnew/whatsnew25.tex Message-ID: <20060426122107.4048B1E400B@bag.python.org> Author: andrew.kuchling Date: Wed Apr 26 14:21:06 2006 New Revision: 45728 Modified: python/trunk/Doc/whatsnew/whatsnew25.tex Log: [Bug #1475080] Fix example Modified: python/trunk/Doc/whatsnew/whatsnew25.tex ============================================================================== --- python/trunk/Doc/whatsnew/whatsnew25.tex (original) +++ python/trunk/Doc/whatsnew/whatsnew25.tex Wed Apr 26 14:21:06 2006 @@ -1827,7 +1827,7 @@ # Do this instead t = (symbol,) -c.execute('select * from stocks where symbol=?', ('IBM',)) +c.execute('select * from stocks where symbol=?', t) # Larger example for t in (('2006-03-28', 'BUY', 'IBM', 1000, 45.00), @@ -2065,6 +2065,6 @@ The author would like to thank the following people for offering suggestions, corrections and assistance with various drafts of this article: Phillip J. Eby, Kent Johnson, Martin von~L\"owis, Gustavo -Niemeyer, Mike Rovner, Thomas Wouters. +Niemeyer, James Pryor, Mike Rovner, Thomas Wouters. \end{document} From python-checkins at python.org Wed Apr 26 14:23:40 2006 From: python-checkins at python.org (andrew.kuchling) Date: Wed, 26 Apr 2006 14:23:40 +0200 (CEST) Subject: [Python-checkins] r45729 - python/trunk/Doc/whatsnew/whatsnew25.tex Message-ID: <20060426122340.D2AA41E400B@bag.python.org> Author: andrew.kuchling Date: Wed Apr 26 14:23:39 2006 New Revision: 45729 Modified: python/trunk/Doc/whatsnew/whatsnew25.tex Log: Add labels to all sections Modified: python/trunk/Doc/whatsnew/whatsnew25.tex ============================================================================== --- python/trunk/Doc/whatsnew/whatsnew25.tex (original) +++ python/trunk/Doc/whatsnew/whatsnew25.tex Wed Apr 26 14:23:39 2006 @@ -1063,7 +1063,7 @@ %====================================================================== -\section{Other Language Changes} +\section{Other Language Changes\label{other-lang}} Here are all of the changes that Python 2.5 makes to the core Python language. @@ -1139,7 +1139,7 @@ %====================================================================== -\subsection{Interactive Interpreter Changes} +\subsection{Interactive Interpreter Changes\label{interactive}} In the interactive interpreter, \code{quit} and \code{exit} have long been strings so that new users get a somewhat helpful message @@ -1157,7 +1157,7 @@ %====================================================================== -\subsection{Optimizations} +\subsection{Optimizations\label{opts}} \begin{itemize} @@ -1184,7 +1184,7 @@ %====================================================================== -\section{New, Improved, and Removed Modules} +\section{New, Improved, and Removed Modules\label{modules}} The standard library received many enhancements and bug fixes in Python 2.5. Here's a partial list of the most notable changes, sorted @@ -1521,7 +1521,7 @@ %====================================================================== -\subsection{The ctypes package} +\subsection{The ctypes package\label{module-ctypes}} The \module{ctypes} package, written by Thomas Heller, has been added to the standard library. \module{ctypes} lets you call arbitrary functions @@ -1603,7 +1603,7 @@ %====================================================================== -\subsection{The ElementTree package} +\subsection{The ElementTree package\label{module-etree}} A subset of Fredrik Lundh's ElementTree library for processing XML has been added to the standard library as \module{xmlcore.etree}. The @@ -1714,7 +1714,7 @@ %====================================================================== -\subsection{The hashlib package} +\subsection{The hashlib package\label{module-hashlib}} A new \module{hashlib} module, written by Gregory P. Smith, has been added to replace the @@ -1762,7 +1762,7 @@ %====================================================================== -\subsection{The sqlite3 package} +\subsection{The sqlite3 package\label{module-sqlite}} The pysqlite module (\url{http://www.pysqlite.org}), a wrapper for the SQLite embedded database, has been added to the standard library under @@ -1876,7 +1876,7 @@ % ====================================================================== -\section{Build and C API Changes} +\section{Build and C API Changes\label{build-api}} Changes to Python's build process and to the C API include: @@ -1954,7 +1954,7 @@ %====================================================================== -\subsection{Port-Specific Changes} +\subsection{Port-Specific Changes\label{ports}} \begin{itemize} @@ -2018,7 +2018,7 @@ %====================================================================== -\section{Porting to Python 2.5} +\section{Porting to Python 2.5\label{porting}} This section lists previously described changes that may require changes to your code: From buildbot at python.org Wed Apr 26 15:10:16 2006 From: buildbot at python.org (buildbot at python.org) Date: Wed, 26 Apr 2006 13:10:16 +0000 Subject: [Python-checkins] buildbot warnings in x86 Ubuntu dapper (icc) trunk Message-ID: <20060426131016.A2A681E4012@bag.python.org> The Buildbot has detected a new failure of x86 Ubuntu dapper (icc) trunk. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520Ubuntu%2520dapper%2520%2528icc%2529%2520trunk/builds/259 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: nick.coghlan Build Had Warnings: warnings test sincerely, -The Buildbot From buildbot at python.org Wed Apr 26 15:54:04 2006 From: buildbot at python.org (buildbot at python.org) Date: Wed, 26 Apr 2006 13:54:04 +0000 Subject: [Python-checkins] buildbot warnings in hppa Ubuntu dapper trunk Message-ID: <20060426135405.193111E4021@bag.python.org> The Buildbot has detected a new failure of hppa Ubuntu dapper trunk. Full details are available at: http://www.python.org/dev/buildbot/all/hppa%2520Ubuntu%2520dapper%2520trunk/builds/300 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: andrew.kuchling Build Had Warnings: warnings failed slave lost sincerely, -The Buildbot From buildbot at python.org Wed Apr 26 16:08:24 2006 From: buildbot at python.org (buildbot at python.org) Date: Wed, 26 Apr 2006 14:08:24 +0000 Subject: [Python-checkins] buildbot warnings in alpha Debian trunk Message-ID: <20060426140824.E543A1E400F@bag.python.org> The Buildbot has detected a new failure of alpha Debian trunk. Full details are available at: http://www.python.org/dev/buildbot/all/alpha%2520Debian%2520trunk/builds/23 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: andrew.kuchling Build Had Warnings: warnings test sincerely, -The Buildbot From python-checkins at python.org Wed Apr 26 17:53:31 2006 From: python-checkins at python.org (thomas.wouters) Date: Wed, 26 Apr 2006 17:53:31 +0200 (CEST) Subject: [Python-checkins] r45730 - in python/trunk: Lib/test/regrtest.py Lib/test/test_bigmem.py Lib/test/test_support.py Misc/NEWS Message-ID: <20060426155331.606FA1E400B@bag.python.org> Author: thomas.wouters Date: Wed Apr 26 17:53:30 2006 New Revision: 45730 Added: python/trunk/Lib/test/test_bigmem.py Modified: python/trunk/Lib/test/regrtest.py python/trunk/Lib/test/test_support.py python/trunk/Misc/NEWS Log: The result of SF patch #1471578: big-memory tests for strings, lists and tuples. Lots to be added, still, but this will give big-memory people something to play with in 2.5 alpha 2, and hopefully get more people to write these tests. Modified: python/trunk/Lib/test/regrtest.py ============================================================================== --- python/trunk/Lib/test/regrtest.py (original) +++ python/trunk/Lib/test/regrtest.py Wed Apr 26 17:53:30 2006 @@ -25,6 +25,7 @@ -N: nocoverdir -- Put coverage files alongside modules -L: runleaks -- run the leaks(1) command just before exit -R: huntrleaks -- search for reference leaks (needs debug build, v. slow) +-M: memlimit -- run very large memory-consuming tests If non-option arguments are present, they are names for tests to run, unless -x is given, in which case they are names for tests not to run. @@ -63,6 +64,19 @@ reports are written to. These parameters all have defaults (5, 4 and "reflog.txt" respectively), so the minimal invocation is '-R ::'. +-M runs tests that require an exorbitant amount of memory. These tests +typically try to ascertain containers keep working when containing more than +2 bilion objects, and only work on 64-bit systems. The passed-in memlimit, +which is a string in the form of '2.5Gb', determines howmuch memory the +tests will limit themselves to (but they may go slightly over.) The number +shouldn't be more memory than the machine has (including swap memory). You +should also keep in mind that swap memory is generally much, much slower +than RAM, and setting memlimit to all available RAM or higher will heavily +tax the machine. On the other hand, it is no use running these tests with a +limit of less than 2.5Gb, and many require more than 20Gb. Tests that expect +to use more than memlimit memory will be skipped. The big-memory tests +generally run very, very long. + -u is used to specify which special resource intensive tests to run, such as those requiring large file support or network connectivity. The argument is a comma-separated list of words indicating the @@ -180,12 +194,12 @@ test_support.record_original_stdout(sys.stdout) try: - opts, args = getopt.getopt(sys.argv[1:], 'hvgqxsrf:lu:t:TD:NLR:w', + opts, args = getopt.getopt(sys.argv[1:], 'hvgqxsrf:lu:t:TD:NLR:wM:', ['help', 'verbose', 'quiet', 'generate', 'exclude', 'single', 'random', 'fromfile', 'findleaks', 'use=', 'threshold=', 'trace', 'coverdir=', 'nocoverdir', 'runleaks', - 'huntrleaks=', 'verbose2', + 'huntrleaks=', 'verbose2', 'memlimit=', ]) except getopt.error, msg: usage(2, msg) @@ -241,6 +255,8 @@ huntrleaks[1] = int(huntrleaks[1]) if len(huntrleaks[2]) == 0: huntrleaks[2] = "reflog.txt" + elif o in ('-M', '--memlimit'): + test_support.set_memlimit(a) elif o in ('-u', '--use'): u = [x.lower() for x in a.split(',')] for r in u: Added: python/trunk/Lib/test/test_bigmem.py ============================================================================== --- (empty file) +++ python/trunk/Lib/test/test_bigmem.py Wed Apr 26 17:53:30 2006 @@ -0,0 +1,921 @@ +from test import test_support +from test.test_support import bigmemtest, _1G, _2G + +import unittest +import operator +import string +import sys + +# Bigmem testing houserules: +# +# - Try not to allocate too many large objects. It's okay to rely on +# refcounting semantics, but don't forget that 's = create_largestring()' +# doesn't release the old 's' (if it exists) until well after its new +# value has been created. Use 'del s' before the create_largestring call. +# +# - Do *not* compare large objects using assertEquals or similar. It's a +# lengty operation and the errormessage will be utterly useless due to +# its size. To make sure whether a result has the right contents, better +# to use the strip or count methods, or compare meaningful slices. +# +# - Don't forget to test for large indices, offsets and results and such, +# in addition to large sizes. +# +# - When repeating an object (say, a substring, or a small list) to create +# a large object, make the subobject of a length that is not a power of +# 2. That way, int-wrapping problems are more easily detected. +# +# - While the bigmemtest decorator speaks of 'minsize', all tests will +# actually be called with a much smaller number too, in the normal +# test run (5Kb currently.) This is so the tests themselves get frequent +# testing Consequently, always make all large allocations based on the +# passed-in 'size', and don't rely on the size being very large. Also, +# memuse-per-size should remain sane (less than a few thousand); if your +# test uses more, adjust 'size' upward, instead. + +class StrTest(unittest.TestCase): + @bigmemtest(minsize=_2G, memuse=2) + def test_capitalize(self, size): + SUBSTR = ' abc def ghi' + s = '-' * size + SUBSTR + caps = s.capitalize() + self.assertEquals(caps[-len(SUBSTR):], + SUBSTR.capitalize()) + self.assertEquals(caps.lstrip('-'), SUBSTR) + + @bigmemtest(minsize=_2G + 10, memuse=1) + def test_center(self, size): + SUBSTR = ' abc def ghi' + s = SUBSTR.center(size) + self.assertEquals(len(s), size) + lpadsize = rpadsize = (len(s) - len(SUBSTR)) // 2 + if len(s) % 2: + lpadsize += 1 + self.assertEquals(s[lpadsize:-rpadsize], SUBSTR) + self.assertEquals(s.strip(), SUBSTR.strip()) + + @bigmemtest(minsize=_2G, memuse=2) + def test_count(self, size): + SUBSTR = ' abc def ghi' + s = '.' * size + SUBSTR + self.assertEquals(s.count('.'), size) + s += '.' + self.assertEquals(s.count('.'), size + 1) + self.assertEquals(s.count(' '), 3) + self.assertEquals(s.count('i'), 1) + self.assertEquals(s.count('j'), 0) + + @bigmemtest(minsize=0, memuse=1) + def test_decode(self, size): + pass + + @bigmemtest(minsize=0, memuse=1) + def test_encode(self, size): + pass + + @bigmemtest(minsize=_2G, memuse=2) + def test_endswith(self, size): + SUBSTR = ' abc def ghi' + s = '-' * size + SUBSTR + self.failUnless(s.endswith(SUBSTR)) + self.failUnless(s.endswith(s)) + s2 = '...' + s + self.failUnless(s2.endswith(s)) + self.failIf(s.endswith('a' + SUBSTR)) + self.failIf(SUBSTR.endswith(s)) + + @bigmemtest(minsize=_2G + 10, memuse=2) + def test_expandtabs(self, size): + s = '-' * size + tabsize = 8 + self.assertEquals(s.expandtabs(), s) + del s + slen, remainder = divmod(size, tabsize) + s = ' \t' * slen + s = s.expandtabs(tabsize) + self.assertEquals(len(s), size - remainder) + self.assertEquals(len(s.strip(' ')), 0) + + @bigmemtest(minsize=_2G, memuse=2) + def test_find(self, size): + SUBSTR = ' abc def ghi' + sublen = len(SUBSTR) + s = ''.join([SUBSTR, '-' * size, SUBSTR]) + self.assertEquals(s.find(' '), 0) + self.assertEquals(s.find(SUBSTR), 0) + self.assertEquals(s.find(' ', sublen), sublen + size) + self.assertEquals(s.find(SUBSTR, len(SUBSTR)), sublen + size) + self.assertEquals(s.find('i'), SUBSTR.find('i')) + self.assertEquals(s.find('i', sublen), + sublen + size + SUBSTR.find('i')) + self.assertEquals(s.find('i', size), + sublen + size + SUBSTR.find('i')) + self.assertEquals(s.find('j'), -1) + + @bigmemtest(minsize=_2G, memuse=2) + def test_index(self, size): + SUBSTR = ' abc def ghi' + sublen = len(SUBSTR) + s = ''.join([SUBSTR, '-' * size, SUBSTR]) + self.assertEquals(s.index(' '), 0) + self.assertEquals(s.index(SUBSTR), 0) + self.assertEquals(s.index(' ', sublen), sublen + size) + self.assertEquals(s.index(SUBSTR, sublen), sublen + size) + self.assertEquals(s.index('i'), SUBSTR.index('i')) + self.assertEquals(s.index('i', sublen), + sublen + size + SUBSTR.index('i')) + self.assertEquals(s.index('i', size), + sublen + size + SUBSTR.index('i')) + self.assertRaises(ValueError, s.index, 'j') + + @bigmemtest(minsize=_2G, memuse=2) + def test_isalnum(self, size): + SUBSTR = '123456' + s = 'a' * size + SUBSTR + self.failUnless(s.isalnum()) + s += '.' + self.failIf(s.isalnum()) + + @bigmemtest(minsize=_2G, memuse=2) + def test_isalpha(self, size): + SUBSTR = 'zzzzzzz' + s = 'a' * size + SUBSTR + self.failUnless(s.isalpha()) + s += '.' + self.failIf(s.isalpha()) + + @bigmemtest(minsize=_2G, memuse=2) + def test_isdigit(self, size): + SUBSTR = '123456' + s = '9' * size + SUBSTR + self.failUnless(s.isdigit()) + s += 'z' + self.failIf(s.isdigit()) + + @bigmemtest(minsize=_2G, memuse=2) + def test_islower(self, size): + chars = ''.join([ chr(c) for c in range(255) if not chr(c).isupper() ]) + repeats = size // len(chars) + 2 + s = chars * repeats + self.failUnless(s.islower()) + s += 'A' + self.failIf(s.islower()) + + @bigmemtest(minsize=_2G, memuse=2) + def test_isspace(self, size): + whitespace = ' \f\n\r\t\v' + repeats = size // len(whitespace) + 2 + s = whitespace * repeats + self.failUnless(s.isspace()) + s += 'j' + self.failIf(s.isspace()) + + @bigmemtest(minsize=_2G, memuse=2) + def test_istitle(self, size): + SUBSTR = '123456' + s = ''.join(['A', 'a' * size, SUBSTR]) + self.failUnless(s.istitle()) + s += 'A' + self.failUnless(s.istitle()) + s += 'aA' + self.failIf(s.istitle()) + + @bigmemtest(minsize=_2G, memuse=2) + def test_isupper(self, size): + chars = ''.join([ chr(c) for c in range(255) if not chr(c).islower() ]) + repeats = size // len(chars) + 2 + s = chars * repeats + self.failUnless(s.isupper()) + s += 'a' + self.failIf(s.isupper()) + + @bigmemtest(minsize=_2G, memuse=2) + def test_join(self, size): + s = 'A' * size + x = s.join(['aaaaa', 'bbbbb']) + self.assertEquals(x.count('a'), 5) + self.assertEquals(x.count('b'), 5) + self.failUnless(x.startswith('aaaaaA')) + self.failUnless(x.endswith('Abbbbb')) + + @bigmemtest(minsize=_2G + 10, memuse=1) + def test_ljust(self, size): + SUBSTR = ' abc def ghi' + s = SUBSTR.ljust(size) + self.failUnless(s.startswith(SUBSTR + ' ')) + self.assertEquals(len(s), size) + self.assertEquals(s.strip(), SUBSTR.strip()) + + @bigmemtest(minsize=_2G + 10, memuse=2) + def test_lower(self, size): + s = 'A' * size + s = s.lower() + self.assertEquals(len(s), size) + self.assertEquals(s.count('a'), size) + + @bigmemtest(minsize=_2G + 10, memuse=1) + def test_lstrip(self, size): + SUBSTR = 'abc def ghi' + s = SUBSTR.rjust(size) + self.assertEquals(len(s), size) + self.assertEquals(s.lstrip(), SUBSTR.lstrip()) + del s + s = SUBSTR.ljust(size) + self.assertEquals(len(s), size) + stripped = s.lstrip() + self.failUnless(stripped is s) + + @bigmemtest(minsize=_2G + 10, memuse=2) + def test_replace(self, size): + replacement = 'a' + s = ' ' * size + s = s.replace(' ', replacement) + self.assertEquals(len(s), size) + self.assertEquals(s.count(replacement), size) + s = s.replace(replacement, ' ', size - 4) + self.assertEquals(len(s), size) + self.assertEquals(s.count(replacement), 4) + self.assertEquals(s[-10:], ' aaaa') + + @bigmemtest(minsize=_2G, memuse=2) + def test_rfind(self, size): + SUBSTR = ' abc def ghi' + sublen = len(SUBSTR) + s = ''.join([SUBSTR, '-' * size, SUBSTR]) + self.assertEquals(s.rfind(' '), sublen + size + SUBSTR.rfind(' ')) + self.assertEquals(s.rfind(SUBSTR), sublen + size) + self.assertEquals(s.rfind(' ', 0, size), SUBSTR.rfind(' ')) + self.assertEquals(s.rfind(SUBSTR, 0, sublen + size), 0) + self.assertEquals(s.rfind('i'), sublen + size + SUBSTR.rfind('i')) + self.assertEquals(s.rfind('i', 0, sublen), SUBSTR.rfind('i')) + self.assertEquals(s.rfind('i', 0, sublen + size), + SUBSTR.rfind('i')) + self.assertEquals(s.rfind('j'), -1) + + @bigmemtest(minsize=_2G, memuse=2) + def test_rindex(self, size): + SUBSTR = ' abc def ghi' + sublen = len(SUBSTR) + s = ''.join([SUBSTR, '-' * size, SUBSTR]) + self.assertEquals(s.rindex(' '), + sublen + size + SUBSTR.rindex(' ')) + self.assertEquals(s.rindex(SUBSTR), sublen + size) + self.assertEquals(s.rindex(' ', 0, sublen + size - 1), + SUBSTR.rindex(' ')) + self.assertEquals(s.rindex(SUBSTR, 0, sublen + size), 0) + self.assertEquals(s.rindex('i'), + sublen + size + SUBSTR.rindex('i')) + self.assertEquals(s.rindex('i', 0, sublen), SUBSTR.rindex('i')) + self.assertEquals(s.rindex('i', 0, sublen + size), + SUBSTR.rindex('i')) + self.assertRaises(ValueError, s.rindex, 'j') + + @bigmemtest(minsize=_2G + 10, memuse=1) + def test_rjust(self, size): + SUBSTR = ' abc def ghi' + s = SUBSTR.ljust(size) + self.failUnless(s.startswith(SUBSTR + ' ')) + self.assertEquals(len(s), size) + self.assertEquals(s.strip(), SUBSTR.strip()) + + @bigmemtest(minsize=_2G + 10, memuse=1) + def test_rstrip(self, size): + SUBSTR = ' abc def ghi' + s = SUBSTR.ljust(size) + self.assertEquals(len(s), size) + self.assertEquals(s.rstrip(), SUBSTR.rstrip()) + del s + s = SUBSTR.rjust(size) + self.assertEquals(len(s), size) + stripped = s.rstrip() + self.failUnless(stripped is s) + + # The test takes about size bytes to build a string, and then about + # sqrt(size) substrings of sqrt(size) in size and a list to + # hold sqrt(size) items. It's close but just over 2x size. + @bigmemtest(minsize=_2G, memuse=2.1) + def test_split_small(self, size): + # Crudely calculate an estimate so that the result of s.split won't + # take up an inordinate amount of memory + chunksize = int(size ** 0.5 + 2) + SUBSTR = 'a' + ' ' * chunksize + s = SUBSTR * chunksize + l = s.split() + self.assertEquals(len(l), chunksize) + self.assertEquals(set(l), set(['a'])) + del l + l = s.split('a') + self.assertEquals(len(l), chunksize + 1) + self.assertEquals(set(l), set(['', ' ' * chunksize])) + + # Allocates a string of twice size (and briefly two) and a list of + # size. Because of internal affairs, the s.split() call produces a + # list of size times the same one-character string, so we only + # suffer for the list size. (Otherwise, it'd cost another 48 times + # size in bytes!) Nevertheless, a list of size takes + # 8*size bytes. + @bigmemtest(minsize=_2G + 5, memuse=10) + def test_split_large(self, size): + s = ' a' * size + ' ' + l = s.split() + self.assertEquals(len(l), size) + self.assertEquals(set(l), set(['a'])) + del l + l = s.split('a') + self.assertEquals(len(l), size + 1) + self.assertEquals(set(l), set([' '])) + + @bigmemtest(minsize=_2G, memuse=2.1) + def test_splitlines(self, size): + # Crudely calculate an estimate so that the result of s.split won't + # take up an inordinate amount of memory + chunksize = int(size ** 0.5 + 2) // 2 + SUBSTR = ' ' * chunksize + '\n' + ' ' * chunksize + '\r\n' + s = SUBSTR * chunksize + l = s.splitlines() + self.assertEquals(len(l), chunksize * 2) + self.assertEquals(set(l), set([' ' * chunksize])) + + @bigmemtest(minsize=_2G, memuse=2) + def test_startswith(self, size): + SUBSTR = ' abc def ghi' + s = '-' * size + SUBSTR + self.failUnless(s.startswith(s)) + self.failUnless(s.startswith('-' * size)) + self.failIf(s.startswith(SUBSTR)) + + @bigmemtest(minsize=_2G, memuse=1) + def test_strip(self, size): + SUBSTR = ' abc def ghi ' + s = SUBSTR.rjust(size) + self.assertEquals(len(s), size) + self.assertEquals(s.strip(), SUBSTR.strip()) + del s + s = SUBSTR.ljust(size) + self.assertEquals(len(s), size) + self.assertEquals(s.strip(), SUBSTR.strip()) + + @bigmemtest(minsize=_2G, memuse=2) + def test_swapcase(self, size): + SUBSTR = "aBcDeFG12.'\xa9\x00" + sublen = len(SUBSTR) + repeats = size // sublen + 2 + s = SUBSTR * repeats + s = s.swapcase() + self.assertEquals(len(s), sublen * repeats) + self.assertEquals(s[:sublen * 3], SUBSTR.swapcase() * 3) + self.assertEquals(s[-sublen * 3:], SUBSTR.swapcase() * 3) + + @bigmemtest(minsize=_2G, memuse=2) + def test_title(self, size): + SUBSTR = 'SpaaHAaaAaham' + s = SUBSTR * (size // len(SUBSTR) + 2) + s = s.title() + self.failUnless(s.startswith((SUBSTR * 3).title())) + self.failUnless(s.endswith(SUBSTR.lower() * 3)) + + @bigmemtest(minsize=_2G, memuse=2) + def test_translate(self, size): + trans = string.maketrans('.aZ', '-!$') + SUBSTR = 'aZz.z.Aaz.' + sublen = len(SUBSTR) + repeats = size // sublen + 2 + s = SUBSTR * repeats + s = s.translate(trans) + self.assertEquals(len(s), repeats * sublen) + self.assertEquals(s[:sublen], SUBSTR.translate(trans)) + self.assertEquals(s[-sublen:], SUBSTR.translate(trans)) + self.assertEquals(s.count('.'), 0) + self.assertEquals(s.count('!'), repeats * 2) + self.assertEquals(s.count('z'), repeats * 3) + + @bigmemtest(minsize=_2G + 5, memuse=2) + def test_upper(self, size): + s = 'a' * size + s = s.upper() + self.assertEquals(len(s), size) + self.assertEquals(s.count('A'), size) + + @bigmemtest(minsize=_2G + 20, memuse=1) + def test_zfill(self, size): + SUBSTR = '-568324723598234' + s = SUBSTR.zfill(size) + self.failUnless(s.endswith('0' + SUBSTR[1:])) + self.failUnless(s.startswith('-0')) + self.assertEquals(len(s), size) + self.assertEquals(s.count('0'), size - len(SUBSTR)) + + @bigmemtest(minsize=_2G + 10, memuse=2) + def test_format(self, size): + s = '-' * size + sf = '%s' % (s,) + self.failUnless(s == sf) + del sf + sf = '..%s..' % (s,) + self.assertEquals(len(sf), len(s) + 4) + self.failUnless(sf.startswith('..-')) + self.failUnless(sf.endswith('-..')) + del s, sf + + size = (size // 2) + edge = '-' * size + s = ''.join([edge, '%s', edge]) + del edge + s = s % '...' + self.assertEquals(len(s), size * 2 + 3) + self.assertEquals(s.count('.'), 3) + self.assertEquals(s.count('-'), size * 2) + + @bigmemtest(minsize=_2G + 10, memuse=2) + def test_repr_small(self, size): + s = '-' * size + s = repr(s) + self.assertEquals(len(s), size + 2) + self.assertEquals(s[0], "'") + self.assertEquals(s[-1], "'") + self.assertEquals(s.count('-'), size) + del s + # repr() will create a string four times as large as this 'binary + # string', but we don't want to allocate much more than twice + # size in total. (We do extra testing in test_repr_large()) + size = size // 5 * 2 + s = '\x00' * size + s = repr(s) + self.assertEquals(len(s), size * 4 + 2) + self.assertEquals(s[0], "'") + self.assertEquals(s[-1], "'") + self.assertEquals(s.count('\\'), size) + self.assertEquals(s.count('0'), size * 2) + + @bigmemtest(minsize=_2G + 10, memuse=5) + def test_repr_large(self, size): + s = '\x00' * size + s = repr(s) + self.assertEquals(len(s), size * 4 + 2) + self.assertEquals(s[0], "'") + self.assertEquals(s[-1], "'") + self.assertEquals(s.count('\\'), size) + self.assertEquals(s.count('0'), size * 2) + + # This test is meaningful even with size < 2G, as long as the + # doubled string is > 2G (but it tests more if both are > 2G :) + @bigmemtest(minsize=_1G + 2, memuse=3) + def test_concat(self, size): + s = '.' * size + self.assertEquals(len(s), size) + s = s + s + self.assertEquals(len(s), size * 2) + self.assertEquals(s.count('.'), size * 2) + + # This test is meaningful even with size < 2G, as long as the + # repeated string is > 2G (but it tests more if both are > 2G :) + @bigmemtest(minsize=_1G + 2, memuse=3) + def test_repeat(self, size): + s = '.' * size + self.assertEquals(len(s), size) + s = s * 2 + self.assertEquals(len(s), size * 2) + self.assertEquals(s.count('.'), size * 2) + + @bigmemtest(minsize=_2G + 20, memuse=1) + def test_slice_and_getitem(self, size): + SUBSTR = '0123456789' + sublen = len(SUBSTR) + s = SUBSTR * (size // sublen) + stepsize = len(s) // 100 + stepsize = stepsize - (stepsize % sublen) + for i in range(0, len(s) - stepsize, stepsize): + self.assertEquals(s[i], SUBSTR[0]) + self.assertEquals(s[i:i + sublen], SUBSTR) + self.assertEquals(s[i:i + sublen:2], SUBSTR[::2]) + if i > 0: + self.assertEquals(s[i + sublen - 1:i - 1:-3], + SUBSTR[sublen::-3]) + # Make sure we do some slicing and indexing near the end of the + # string, too. + self.assertEquals(s[len(s) - 1], SUBSTR[-1]) + self.assertEquals(s[-1], SUBSTR[-1]) + self.assertEquals(s[len(s) - 10], SUBSTR[0]) + self.assertEquals(s[-sublen], SUBSTR[0]) + self.assertEquals(s[len(s):], '') + self.assertEquals(s[len(s) - 1:], SUBSTR[-1]) + self.assertEquals(s[-1:], SUBSTR[-1]) + self.assertEquals(s[len(s) - sublen:], SUBSTR) + self.assertEquals(s[-sublen:], SUBSTR) + self.assertEquals(len(s[:]), len(s)) + self.assertEquals(len(s[:len(s) - 5]), len(s) - 5) + self.assertEquals(len(s[5:-5]), len(s) - 10) + + self.assertRaises(IndexError, operator.getitem, s, len(s)) + self.assertRaises(IndexError, operator.getitem, s, len(s) + 1) + self.assertRaises(IndexError, operator.getitem, s, len(s) + 1<<31) + + @bigmemtest(minsize=_2G, memuse=2) + def test_contains(self, size): + SUBSTR = '0123456789' + edge = '-' * (size // 2) + s = ''.join([edge, SUBSTR, edge]) + del edge + self.failUnless(SUBSTR in s) + self.failIf(SUBSTR * 2 in s) + self.failUnless('-' in s) + self.failIf('a' in s) + s += 'a' + self.failUnless('a' in s) + + @bigmemtest(minsize=_2G + 10, memuse=2) + def test_compare(self, size): + s1 = '-' * size + s2 = '-' * size + self.failUnless(s1 == s2) + del s2 + s2 = s1 + 'a' + self.failIf(s1 == s2) + del s2 + s2 = '.' * size + self.failIf(s1 == s2) + + @bigmemtest(minsize=_2G + 10, memuse=1) + def test_hash(self, size): + # Not sure if we can do any meaningful tests here... Even if we + # start relying on the exact algorithm used, the result will be + # different depending on the size of the C 'long int'. Even this + # test is dodgy (there's no *guarantee* that the two things should + # have a different hash, even if they, in the current + # implementation, almost always do.) + s = '\x00' * size + h1 = hash(s) + del s + s = '\x00' * (size + 1) + self.failIf(h1 == hash(s)) + +class TupleTest(unittest.TestCase): + + # Tuples have a small, fixed-sized head and an array of pointers to + # data. Since we're testing 64-bit addressing, we can assume that the + # pointers are 8 bytes, and that thus that the tuples take up 8 bytes + # per size. + + # As a side-effect of testing long tuples, these tests happen to test + # having more than 2<<31 references to any given object. Hence the + # use of different types of objects as contents in different tests. + + @bigmemtest(minsize=_2G + 2, memuse=16) + def test_compare(self, size): + t1 = (u'',) * size + t2 = (u'',) * size + self.failUnless(t1 == t2) + del t2 + t2 = (u'',) * (size + 1) + self.failIf(t1 == t2) + del t2 + t2 = (1,) * size + self.failIf(t1 == t2) + + # Test concatenating into a single tuple of more than 2G in length, + # and concatenating a tuple of more than 2G in length separately, so + # the smaller test still gets run even if there isn't memory for the + # larger test (but we still let the tester know the larger test is + # skipped, in verbose mode.) + def basic_concat_test(self, size): + t = ((),) * size + self.assertEquals(len(t), size) + t = t + t + self.assertEquals(len(t), size * 2) + + @bigmemtest(minsize=_2G // 2 + 2, memuse=24) + def test_concat_small(self, size): + return self.basic_concat_test(size) + + @bigmemtest(minsize=_2G + 2, memuse=24) + def test_concat_large(self, size): + return self.basic_concat_test(size) + + @bigmemtest(minsize=_2G // 5 + 10, memuse=8*5) + def test_contains(self, size): + t = (1, 2, 3, 4, 5) * size + self.assertEquals(len(t), size * 5) + self.failUnless(5 in t) + self.failIf((1, 2, 3, 4, 5) in t) + self.failIf(0 in t) + + @bigmemtest(minsize=_2G + 10, memuse=8) + def test_hash(self, size): + t1 = (0,) * size + h1 = hash(t1) + del t1 + t2 = (0,) * (size + 1) + self.failIf(h1 == hash(t2)) + + @bigmemtest(minsize=_2G + 10, memuse=8) + def test_index_and_slice(self, size): + t = (None,) * size + self.assertEquals(len(t), size) + self.assertEquals(t[-1], None) + self.assertEquals(t[5], None) + self.assertEquals(t[size - 1], None) + self.assertRaises(IndexError, operator.getitem, t, size) + self.assertEquals(t[:5], (None,) * 5) + self.assertEquals(t[-5:], (None,) * 5) + self.assertEquals(t[20:25], (None,) * 5) + self.assertEquals(t[-25:-20], (None,) * 5) + self.assertEquals(t[size - 5:], (None,) * 5) + self.assertEquals(t[size - 5:size], (None,) * 5) + self.assertEquals(t[size - 6:size - 2], (None,) * 4) + self.assertEquals(t[size:size], ()) + self.assertEquals(t[size:size+5], ()) + + # Like test_concat, split in two. + def basic_test_repeat(self, size): + t = ('',) * size + self.assertEquals(len(t), size) + t = t * 2 + self.assertEquals(len(t), size * 2) + + @bigmemtest(minsize=_2G // 2 + 2, memuse=24) + def test_repeat_small(self, size): + return self.basic_test_repeat(size) + + @bigmemtest(minsize=_2G + 2, memuse=24) + def test_repeat_large(self, size): + return self.basic_test_repeat(size) + + # Like test_concat, split in two. + def basic_test_repr(self, size): + t = (0,) * size + s = repr(t) + # The repr of a tuple of 0's is exactly three times the tuple length. + self.assertEquals(len(s), size * 3) + self.assertEquals(s[:5], '(0, 0') + self.assertEquals(s[-5:], '0, 0)') + self.assertEquals(s.count('0'), size) + + @bigmemtest(minsize=_2G // 3 + 2, memuse=8+3) + def test_repr_small(self, size): + return self.basic_test_repr(size) + + @bigmemtest(minsize=_2G + 2, memuse=8+3) + def test_repr_large(self, size): + return self.basic_test_repr(size) + +class ListTest(unittest.TestCase): + + # Like tuples, lists have a small, fixed-sized head and an array of + # pointers to data, so 8 bytes per size. Also like tuples, we make the + # lists hold references to various objects to test their refcount + # limits. + + @bigmemtest(minsize=_2G + 2, memuse=16) + def test_compare(self, size): + l1 = [u''] * size + l2 = [u''] * size + self.failUnless(l1 == l2) + del l2 + l2 = [u''] * (size + 1) + self.failIf(l1 == l2) + del l2 + l2 = [2] * size + self.failIf(l1 == l2) + + # Test concatenating into a single list of more than 2G in length, + # and concatenating a list of more than 2G in length separately, so + # the smaller test still gets run even if there isn't memory for the + # larger test (but we still let the tester know the larger test is + # skipped, in verbose mode.) + def basic_test_concat(self, size): + l = [[]] * size + self.assertEquals(len(l), size) + l = l + l + self.assertEquals(len(l), size * 2) + + @bigmemtest(minsize=_2G // 2 + 2, memuse=24) + def test_concat_small(self, size): + return self.basic_test_concat(size) + + @bigmemtest(minsize=_2G + 2, memuse=24) + def test_concat_large(self, size): + return self.basic_test_concat(size) + + @bigmemtest(minsize=_2G // 5 + 10, memuse=8*5) + def test_contains(self, size): + l = [1, 2, 3, 4, 5] * size + self.assertEquals(len(l), size * 5) + self.failUnless(5 in l) + self.failIf([1, 2, 3, 4, 5] in l) + self.failIf(0 in l) + + @bigmemtest(minsize=_2G + 10, memuse=8) + def test_hash(self, size): + l = [0] * size + self.failUnlessRaises(TypeError, hash, l) + + @bigmemtest(minsize=_2G + 10, memuse=8) + def test_index_and_slice(self, size): + l = [None] * size + self.assertEquals(len(l), size) + self.assertEquals(l[-1], None) + self.assertEquals(l[5], None) + self.assertEquals(l[size - 1], None) + self.assertRaises(IndexError, operator.getitem, l, size) + self.assertEquals(l[:5], [None] * 5) + self.assertEquals(l[-5:], [None] * 5) + self.assertEquals(l[20:25], [None] * 5) + self.assertEquals(l[-25:-20], [None] * 5) + self.assertEquals(l[size - 5:], [None] * 5) + self.assertEquals(l[size - 5:size], [None] * 5) + self.assertEquals(l[size - 6:size - 2], [None] * 4) + self.assertEquals(l[size:size], []) + self.assertEquals(l[size:size+5], []) + + l[size - 2] = 5 + self.assertEquals(len(l), size) + self.assertEquals(l[-3:], [None, 5, None]) + self.assertEquals(l.count(5), 1) + self.assertRaises(IndexError, operator.setitem, l, size, 6) + self.assertEquals(len(l), size) + + l[size - 7:] = [1, 2, 3, 4, 5] + size -= 2 + self.assertEquals(len(l), size) + self.assertEquals(l[-7:], [None, None, 1, 2, 3, 4, 5]) + + l[:7] = [1, 2, 3, 4, 5] + size -= 2 + self.assertEquals(len(l), size) + self.assertEquals(l[:7], [1, 2, 3, 4, 5, None, None]) + + del l[size - 1] + size -= 1 + self.assertEquals(len(l), size) + self.assertEquals(l[-1], 4) + + del l[-2:] + size -= 2 + self.assertEquals(len(l), size) + self.assertEquals(l[-1], 2) + + del l[0] + size -= 1 + self.assertEquals(len(l), size) + self.assertEquals(l[0], 2) + + del l[:2] + size -= 2 + self.assertEquals(len(l), size) + self.assertEquals(l[0], 4) + + # Like test_concat, split in two. + def basic_test_repeat(self, size): + l = [] * size + self.failIf(l) + l = [''] * size + self.assertEquals(len(l), size) + l = l * 2 + self.assertEquals(len(l), size * 2) + + @bigmemtest(minsize=_2G // 2 + 2, memuse=24) + def test_repeat_small(self, size): + return self.basic_test_repeat(size) + + @bigmemtest(minsize=_2G + 2, memuse=24) + def test_repeat_large(self, size): + return self.basic_test_repeat(size) + + # Test repr-result of >2G + def basic_test_repr(self, size): + l = [0] * size + s = repr(l) + # The repr of a list of 0's is exactly three times the list length. + self.assertEquals(len(s), size * 3) + self.assertEquals(s[:5], '[0, 0') + self.assertEquals(s[-5:], '0, 0]') + self.assertEquals(s.count('0'), size) + + @bigmemtest(minsize=_2G // 3 + 2, memuse=8 + 3) + def test_repr_small(self, size): + return self.basic_test_repr(size) + + @bigmemtest(minsize=_2G + 2, memuse=8 + 3) + def test_repr_large(self, size): + return self.basic_test_repr(size) + + + @bigmemtest(minsize=_2G, memuse=8) + def test_append(self, size): + l = [object()] * size + l.append(object()) + self.assertEquals(len(l), size+1) + self.failUnless(l[-3] is l[-2]) + self.failIf(l[-2] is l[-1]) + + @bigmemtest(minsize=_2G // 5 + 2, memuse=8 * 5) + def test_count(self, size): + l = [1, 2, 3, 4, 5] * size + self.assertEquals(l.count(1), size) + self.assertEquals(l.count("1"), 0) + + def basic_test_extend(self, size): + l = [file] * size + l.extend(l) + self.assertEquals(len(l), size * 2) + self.failUnless(l[0] is l[-1]) + self.failUnless(l[size - 1] is l[size + 1]) + + @bigmemtest(minsize=_2G // 2 + 2, memuse=8) + def test_extend_small(self, size): + return self.basic_test_extend(size) + + @bigmemtest(minsize=_2G + 2, memuse=8) + def test_extend_large(self, size): + return self.basic_test_extend(size) + + @bigmemtest(minsize=_2G + 10, memuse=8) + def test_index(self, size): + l = [1L, 2L, 3L, 4L, 5L] * (size // 5) + self.assertEquals(l.index(1), 0) + self.assertEquals(l.index(5, size - 5), size - 1) + self.assertEquals(l.index(5, size - 5, size), size - 1) + self.assertRaises(ValueError, l.index, 1, size - 4, size) + self.assertRaises(ValueError, l.index, 6L) + + @bigmemtest(minsize=_2G + 10, memuse=8) + def test_insert(self, size): + l = [1.0] * size + l.insert(size - 1, "A") + size += 1 + self.assertEquals(len(l), size) + self.assertEquals(l[-3:], [1.0, "A", 1.0]) + + l.insert(size + 1, "B") + size += 1 + self.assertEquals(len(l), size) + self.assertEquals(l[-3:], ["A", 1.0, "B"]) + + l.insert(1, "C") + size += 1 + self.assertEquals(len(l), size) + self.assertEquals(l[:3], [1.0, "C", 1.0]) + self.assertEquals(l[size - 3:], ["A", 1.0, "B"]) + + @bigmemtest(minsize=_2G + 20, memuse=8) + def test_pop(self, size): + l = [u"a", u"b", u"c", u"d", u"e"] * (size // 5) + self.assertEquals(len(l), size) + + item = l.pop() + size -= 1 + self.assertEquals(len(l), size) + self.assertEquals(item, u"e") + + item = l.pop(0) + size -= 1 + self.assertEquals(len(l), size) + self.assertEquals(item, u"a") + + item = l.pop(size - 2) + size -= 1 + self.assertEquals(len(l), size) + self.assertEquals(item, u"c") + + @bigmemtest(minsize=_2G + 10, memuse=8) + def test_remove(self, size): + l = [10] * size + self.assertEquals(len(l), size) + + l.remove(10) + size -= 1 + self.assertEquals(len(l), size) + + l.append(5) + size += 1 + self.assertEquals(len(l), size) + self.assertEquals(l[-2:], [10, 5]) + l.remove(5) + size -= 1 + self.assertEquals(len(l), size) + self.assertEquals(l[-2:], [10, 10]) + + @bigmemtest(minsize=_2G + 10, memuse=8) + def test_reverse(self, size): + l = [1, 2, 3, 4, 5] * (size // 5) + l.reverse() + self.assertEquals(len(l), size) + self.assertEquals(l[-5:], [5, 4, 3, 2, 1]) + self.assertEquals(l[:5], [5, 4, 3, 2, 1]) + + @bigmemtest(minsize=_2G + 10, memuse=8) + def test_sort(self, size): + l = [1, 2, 3, 4, 5] * (size // 5) + l.sort() + self.assertEquals(len(l), size) + self.assertEquals(l.count(1), size // 5) + self.assertEquals(l[:10], [1] * 10) + self.assertEquals(l[-10:], [5] * 10) + +def test_main(): + test_support.run_unittest(StrTest, TupleTest, ListTest) + +if __name__ == '__main__': + if len(sys.argv) > 1: + test_support.set_memlimit(sys.argv[1]) + test_main() Modified: python/trunk/Lib/test/test_support.py ============================================================================== --- python/trunk/Lib/test/test_support.py (original) +++ python/trunk/Lib/test/test_support.py Wed Apr 26 17:53:30 2006 @@ -30,7 +30,9 @@ """ verbose = 1 # Flag set to 0 by regrtest.py -use_resources = None # Flag set to [] by regrtest.py +use_resources = None # Flag set to [] by regrtest.py +max_memuse = 0 # Disable bigmem tests (they will still be run with + # small sizes, to make sure they work.) # _original_stdout is meant to hold stdout at the time regrtest began. # This may be "the real" stdout, or IDLE's emulation of stdout, or whatever. @@ -250,6 +252,72 @@ return open(fn) #======================================================================= +# Big-memory-test support. Separate from 'resources' because memory use should be configurable. + +# Some handy shorthands. Note that these are used for byte-limits as well +# as size-limits, in the various bigmem tests +_1M = 1024*1024 +_1G = 1024 * _1M +_2G = 2 * _1G + +def set_memlimit(limit): + import re + global max_memuse + sizes = { + 'k': 1024, + 'm': _1M, + 'g': _1G, + 't': 1024*_1G, + } + m = re.match(r'(\d+(\.\d+)?) (K|M|G|T)b?$', limit, + re.IGNORECASE | re.VERBOSE) + if m is None: + raise ValueError('Invalid memory limit %r' % (limit,)) + memlimit = int(float(m.group(1)) * sizes[m.group(3).lower()]) + if memlimit < 2.5*_1G: + raise ValueError('Memory limit %r too low to be useful' % (limit,)) + max_memuse = memlimit + +def bigmemtest(minsize, memuse, overhead=5*_1M): + """Decorator for bigmem tests. + + 'minsize' is the minimum useful size for the test (in arbitrary, + test-interpreted units.) 'memuse' is the number of 'bytes per size' for + the test, or a good estimate of it. 'overhead' specifies fixed overhead, + independant of the testsize, and defaults to 5Mb. + + The decorator tries to guess a good value for 'size' and passes it to + the decorated test function. If minsize * memuse is more than the + allowed memory use (as defined by max_memuse), the test is skipped. + Otherwise, minsize is adjusted upward to use up to max_memuse. + """ + def decorator(f): + def wrapper(self): + if not max_memuse: + # If max_memuse is 0 (the default), + # we still want to run the tests with size set to a few kb, + # to make sure they work. We still want to avoid using + # too much memory, though, but we do that noisily. + maxsize = 1024*5 + self.failIf(maxsize * memuse + overhead > 20 * _1M) + else: + maxsize = int((max_memuse - overhead) / memuse) + if maxsize < minsize: + # Really ought to print 'test skipped' or something + if verbose: + sys.stderr.write("Skipping %s because of memory " + "constraint\n" % (f.__name__,)) + return + # Try to keep some breathing room in memory use + maxsize = max(maxsize - 50 * _1M, minsize) + return f(self, maxsize) + wrapper.minsize = minsize + wrapper.memuse = memuse + wrapper.overhead = overhead + return wrapper + return decorator + +#======================================================================= # Preliminary PyUNIT integration. import unittest Modified: python/trunk/Misc/NEWS ============================================================================== --- python/trunk/Misc/NEWS (original) +++ python/trunk/Misc/NEWS Wed Apr 26 17:53:30 2006 @@ -171,6 +171,11 @@ it separately and by hand. It also wasn't cleaning up its changes to the current Decimal context. +- regrtest.py now has a -M option to run tests that test the new limits of + containers, on 64-bit architectures. Running these tests is only sensible + on 64-bit machines with more than two gigabytes of memory. The argument + passed is the maximum amount of memory for the tests to use. + Tools ----- From python-checkins at python.org Wed Apr 26 19:11:17 2006 From: python-checkins at python.org (tim.peters) Date: Wed, 26 Apr 2006 19:11:17 +0200 (CEST) Subject: [Python-checkins] r45731 - python/trunk/Lib/test/test_bigmem.py python/trunk/Lib/test/test_support.py Message-ID: <20060426171117.DD40C1E4010@bag.python.org> Author: tim.peters Date: Wed Apr 26 19:11:16 2006 New Revision: 45731 Modified: python/trunk/Lib/test/test_bigmem.py python/trunk/Lib/test/test_support.py Log: Whitespace normalization. Modified: python/trunk/Lib/test/test_bigmem.py ============================================================================== --- python/trunk/Lib/test/test_bigmem.py (original) +++ python/trunk/Lib/test/test_bigmem.py Wed Apr 26 19:11:16 2006 @@ -1,921 +1,921 @@ -from test import test_support -from test.test_support import bigmemtest, _1G, _2G - -import unittest -import operator -import string -import sys - -# Bigmem testing houserules: -# -# - Try not to allocate too many large objects. It's okay to rely on -# refcounting semantics, but don't forget that 's = create_largestring()' -# doesn't release the old 's' (if it exists) until well after its new -# value has been created. Use 'del s' before the create_largestring call. -# -# - Do *not* compare large objects using assertEquals or similar. It's a -# lengty operation and the errormessage will be utterly useless due to -# its size. To make sure whether a result has the right contents, better -# to use the strip or count methods, or compare meaningful slices. -# -# - Don't forget to test for large indices, offsets and results and such, -# in addition to large sizes. -# -# - When repeating an object (say, a substring, or a small list) to create -# a large object, make the subobject of a length that is not a power of -# 2. That way, int-wrapping problems are more easily detected. -# -# - While the bigmemtest decorator speaks of 'minsize', all tests will -# actually be called with a much smaller number too, in the normal -# test run (5Kb currently.) This is so the tests themselves get frequent -# testing Consequently, always make all large allocations based on the -# passed-in 'size', and don't rely on the size being very large. Also, -# memuse-per-size should remain sane (less than a few thousand); if your -# test uses more, adjust 'size' upward, instead. - -class StrTest(unittest.TestCase): - @bigmemtest(minsize=_2G, memuse=2) - def test_capitalize(self, size): - SUBSTR = ' abc def ghi' - s = '-' * size + SUBSTR - caps = s.capitalize() - self.assertEquals(caps[-len(SUBSTR):], - SUBSTR.capitalize()) - self.assertEquals(caps.lstrip('-'), SUBSTR) - - @bigmemtest(minsize=_2G + 10, memuse=1) - def test_center(self, size): - SUBSTR = ' abc def ghi' - s = SUBSTR.center(size) - self.assertEquals(len(s), size) - lpadsize = rpadsize = (len(s) - len(SUBSTR)) // 2 - if len(s) % 2: - lpadsize += 1 - self.assertEquals(s[lpadsize:-rpadsize], SUBSTR) - self.assertEquals(s.strip(), SUBSTR.strip()) - - @bigmemtest(minsize=_2G, memuse=2) - def test_count(self, size): - SUBSTR = ' abc def ghi' - s = '.' * size + SUBSTR - self.assertEquals(s.count('.'), size) - s += '.' - self.assertEquals(s.count('.'), size + 1) - self.assertEquals(s.count(' '), 3) - self.assertEquals(s.count('i'), 1) - self.assertEquals(s.count('j'), 0) - - @bigmemtest(minsize=0, memuse=1) - def test_decode(self, size): - pass - - @bigmemtest(minsize=0, memuse=1) - def test_encode(self, size): - pass - - @bigmemtest(minsize=_2G, memuse=2) - def test_endswith(self, size): - SUBSTR = ' abc def ghi' - s = '-' * size + SUBSTR - self.failUnless(s.endswith(SUBSTR)) - self.failUnless(s.endswith(s)) - s2 = '...' + s - self.failUnless(s2.endswith(s)) - self.failIf(s.endswith('a' + SUBSTR)) - self.failIf(SUBSTR.endswith(s)) - - @bigmemtest(minsize=_2G + 10, memuse=2) - def test_expandtabs(self, size): - s = '-' * size - tabsize = 8 - self.assertEquals(s.expandtabs(), s) - del s - slen, remainder = divmod(size, tabsize) - s = ' \t' * slen - s = s.expandtabs(tabsize) - self.assertEquals(len(s), size - remainder) - self.assertEquals(len(s.strip(' ')), 0) - - @bigmemtest(minsize=_2G, memuse=2) - def test_find(self, size): - SUBSTR = ' abc def ghi' - sublen = len(SUBSTR) - s = ''.join([SUBSTR, '-' * size, SUBSTR]) - self.assertEquals(s.find(' '), 0) - self.assertEquals(s.find(SUBSTR), 0) - self.assertEquals(s.find(' ', sublen), sublen + size) - self.assertEquals(s.find(SUBSTR, len(SUBSTR)), sublen + size) - self.assertEquals(s.find('i'), SUBSTR.find('i')) - self.assertEquals(s.find('i', sublen), - sublen + size + SUBSTR.find('i')) - self.assertEquals(s.find('i', size), - sublen + size + SUBSTR.find('i')) - self.assertEquals(s.find('j'), -1) - - @bigmemtest(minsize=_2G, memuse=2) - def test_index(self, size): - SUBSTR = ' abc def ghi' - sublen = len(SUBSTR) - s = ''.join([SUBSTR, '-' * size, SUBSTR]) - self.assertEquals(s.index(' '), 0) - self.assertEquals(s.index(SUBSTR), 0) - self.assertEquals(s.index(' ', sublen), sublen + size) - self.assertEquals(s.index(SUBSTR, sublen), sublen + size) - self.assertEquals(s.index('i'), SUBSTR.index('i')) - self.assertEquals(s.index('i', sublen), - sublen + size + SUBSTR.index('i')) - self.assertEquals(s.index('i', size), - sublen + size + SUBSTR.index('i')) - self.assertRaises(ValueError, s.index, 'j') - - @bigmemtest(minsize=_2G, memuse=2) - def test_isalnum(self, size): - SUBSTR = '123456' - s = 'a' * size + SUBSTR - self.failUnless(s.isalnum()) - s += '.' - self.failIf(s.isalnum()) - - @bigmemtest(minsize=_2G, memuse=2) - def test_isalpha(self, size): - SUBSTR = 'zzzzzzz' - s = 'a' * size + SUBSTR - self.failUnless(s.isalpha()) - s += '.' - self.failIf(s.isalpha()) - - @bigmemtest(minsize=_2G, memuse=2) - def test_isdigit(self, size): - SUBSTR = '123456' - s = '9' * size + SUBSTR - self.failUnless(s.isdigit()) - s += 'z' - self.failIf(s.isdigit()) - - @bigmemtest(minsize=_2G, memuse=2) - def test_islower(self, size): - chars = ''.join([ chr(c) for c in range(255) if not chr(c).isupper() ]) - repeats = size // len(chars) + 2 - s = chars * repeats - self.failUnless(s.islower()) - s += 'A' - self.failIf(s.islower()) - - @bigmemtest(minsize=_2G, memuse=2) - def test_isspace(self, size): - whitespace = ' \f\n\r\t\v' - repeats = size // len(whitespace) + 2 - s = whitespace * repeats - self.failUnless(s.isspace()) - s += 'j' - self.failIf(s.isspace()) - - @bigmemtest(minsize=_2G, memuse=2) - def test_istitle(self, size): - SUBSTR = '123456' - s = ''.join(['A', 'a' * size, SUBSTR]) - self.failUnless(s.istitle()) - s += 'A' - self.failUnless(s.istitle()) - s += 'aA' - self.failIf(s.istitle()) - - @bigmemtest(minsize=_2G, memuse=2) - def test_isupper(self, size): - chars = ''.join([ chr(c) for c in range(255) if not chr(c).islower() ]) - repeats = size // len(chars) + 2 - s = chars * repeats - self.failUnless(s.isupper()) - s += 'a' - self.failIf(s.isupper()) - - @bigmemtest(minsize=_2G, memuse=2) - def test_join(self, size): - s = 'A' * size - x = s.join(['aaaaa', 'bbbbb']) - self.assertEquals(x.count('a'), 5) - self.assertEquals(x.count('b'), 5) - self.failUnless(x.startswith('aaaaaA')) - self.failUnless(x.endswith('Abbbbb')) - - @bigmemtest(minsize=_2G + 10, memuse=1) - def test_ljust(self, size): - SUBSTR = ' abc def ghi' - s = SUBSTR.ljust(size) - self.failUnless(s.startswith(SUBSTR + ' ')) - self.assertEquals(len(s), size) - self.assertEquals(s.strip(), SUBSTR.strip()) - - @bigmemtest(minsize=_2G + 10, memuse=2) - def test_lower(self, size): - s = 'A' * size - s = s.lower() - self.assertEquals(len(s), size) - self.assertEquals(s.count('a'), size) - - @bigmemtest(minsize=_2G + 10, memuse=1) - def test_lstrip(self, size): - SUBSTR = 'abc def ghi' - s = SUBSTR.rjust(size) - self.assertEquals(len(s), size) - self.assertEquals(s.lstrip(), SUBSTR.lstrip()) - del s - s = SUBSTR.ljust(size) - self.assertEquals(len(s), size) - stripped = s.lstrip() - self.failUnless(stripped is s) - - @bigmemtest(minsize=_2G + 10, memuse=2) - def test_replace(self, size): - replacement = 'a' - s = ' ' * size - s = s.replace(' ', replacement) - self.assertEquals(len(s), size) - self.assertEquals(s.count(replacement), size) - s = s.replace(replacement, ' ', size - 4) - self.assertEquals(len(s), size) - self.assertEquals(s.count(replacement), 4) - self.assertEquals(s[-10:], ' aaaa') - - @bigmemtest(minsize=_2G, memuse=2) - def test_rfind(self, size): - SUBSTR = ' abc def ghi' - sublen = len(SUBSTR) - s = ''.join([SUBSTR, '-' * size, SUBSTR]) - self.assertEquals(s.rfind(' '), sublen + size + SUBSTR.rfind(' ')) - self.assertEquals(s.rfind(SUBSTR), sublen + size) - self.assertEquals(s.rfind(' ', 0, size), SUBSTR.rfind(' ')) - self.assertEquals(s.rfind(SUBSTR, 0, sublen + size), 0) - self.assertEquals(s.rfind('i'), sublen + size + SUBSTR.rfind('i')) - self.assertEquals(s.rfind('i', 0, sublen), SUBSTR.rfind('i')) - self.assertEquals(s.rfind('i', 0, sublen + size), - SUBSTR.rfind('i')) - self.assertEquals(s.rfind('j'), -1) - - @bigmemtest(minsize=_2G, memuse=2) - def test_rindex(self, size): - SUBSTR = ' abc def ghi' - sublen = len(SUBSTR) - s = ''.join([SUBSTR, '-' * size, SUBSTR]) - self.assertEquals(s.rindex(' '), - sublen + size + SUBSTR.rindex(' ')) - self.assertEquals(s.rindex(SUBSTR), sublen + size) - self.assertEquals(s.rindex(' ', 0, sublen + size - 1), - SUBSTR.rindex(' ')) - self.assertEquals(s.rindex(SUBSTR, 0, sublen + size), 0) - self.assertEquals(s.rindex('i'), - sublen + size + SUBSTR.rindex('i')) - self.assertEquals(s.rindex('i', 0, sublen), SUBSTR.rindex('i')) - self.assertEquals(s.rindex('i', 0, sublen + size), - SUBSTR.rindex('i')) - self.assertRaises(ValueError, s.rindex, 'j') - - @bigmemtest(minsize=_2G + 10, memuse=1) - def test_rjust(self, size): - SUBSTR = ' abc def ghi' - s = SUBSTR.ljust(size) - self.failUnless(s.startswith(SUBSTR + ' ')) - self.assertEquals(len(s), size) - self.assertEquals(s.strip(), SUBSTR.strip()) - - @bigmemtest(minsize=_2G + 10, memuse=1) - def test_rstrip(self, size): - SUBSTR = ' abc def ghi' - s = SUBSTR.ljust(size) - self.assertEquals(len(s), size) - self.assertEquals(s.rstrip(), SUBSTR.rstrip()) - del s - s = SUBSTR.rjust(size) - self.assertEquals(len(s), size) - stripped = s.rstrip() - self.failUnless(stripped is s) - - # The test takes about size bytes to build a string, and then about - # sqrt(size) substrings of sqrt(size) in size and a list to - # hold sqrt(size) items. It's close but just over 2x size. - @bigmemtest(minsize=_2G, memuse=2.1) - def test_split_small(self, size): - # Crudely calculate an estimate so that the result of s.split won't - # take up an inordinate amount of memory - chunksize = int(size ** 0.5 + 2) - SUBSTR = 'a' + ' ' * chunksize - s = SUBSTR * chunksize - l = s.split() - self.assertEquals(len(l), chunksize) - self.assertEquals(set(l), set(['a'])) - del l - l = s.split('a') - self.assertEquals(len(l), chunksize + 1) - self.assertEquals(set(l), set(['', ' ' * chunksize])) - - # Allocates a string of twice size (and briefly two) and a list of - # size. Because of internal affairs, the s.split() call produces a - # list of size times the same one-character string, so we only - # suffer for the list size. (Otherwise, it'd cost another 48 times - # size in bytes!) Nevertheless, a list of size takes - # 8*size bytes. - @bigmemtest(minsize=_2G + 5, memuse=10) - def test_split_large(self, size): - s = ' a' * size + ' ' - l = s.split() - self.assertEquals(len(l), size) - self.assertEquals(set(l), set(['a'])) - del l - l = s.split('a') - self.assertEquals(len(l), size + 1) - self.assertEquals(set(l), set([' '])) - - @bigmemtest(minsize=_2G, memuse=2.1) - def test_splitlines(self, size): - # Crudely calculate an estimate so that the result of s.split won't - # take up an inordinate amount of memory - chunksize = int(size ** 0.5 + 2) // 2 - SUBSTR = ' ' * chunksize + '\n' + ' ' * chunksize + '\r\n' - s = SUBSTR * chunksize - l = s.splitlines() - self.assertEquals(len(l), chunksize * 2) - self.assertEquals(set(l), set([' ' * chunksize])) - - @bigmemtest(minsize=_2G, memuse=2) - def test_startswith(self, size): - SUBSTR = ' abc def ghi' - s = '-' * size + SUBSTR - self.failUnless(s.startswith(s)) - self.failUnless(s.startswith('-' * size)) - self.failIf(s.startswith(SUBSTR)) - - @bigmemtest(minsize=_2G, memuse=1) - def test_strip(self, size): - SUBSTR = ' abc def ghi ' - s = SUBSTR.rjust(size) - self.assertEquals(len(s), size) - self.assertEquals(s.strip(), SUBSTR.strip()) - del s - s = SUBSTR.ljust(size) - self.assertEquals(len(s), size) - self.assertEquals(s.strip(), SUBSTR.strip()) - - @bigmemtest(minsize=_2G, memuse=2) - def test_swapcase(self, size): - SUBSTR = "aBcDeFG12.'\xa9\x00" - sublen = len(SUBSTR) - repeats = size // sublen + 2 - s = SUBSTR * repeats - s = s.swapcase() - self.assertEquals(len(s), sublen * repeats) - self.assertEquals(s[:sublen * 3], SUBSTR.swapcase() * 3) - self.assertEquals(s[-sublen * 3:], SUBSTR.swapcase() * 3) - - @bigmemtest(minsize=_2G, memuse=2) - def test_title(self, size): - SUBSTR = 'SpaaHAaaAaham' - s = SUBSTR * (size // len(SUBSTR) + 2) - s = s.title() - self.failUnless(s.startswith((SUBSTR * 3).title())) - self.failUnless(s.endswith(SUBSTR.lower() * 3)) - - @bigmemtest(minsize=_2G, memuse=2) - def test_translate(self, size): - trans = string.maketrans('.aZ', '-!$') - SUBSTR = 'aZz.z.Aaz.' - sublen = len(SUBSTR) - repeats = size // sublen + 2 - s = SUBSTR * repeats - s = s.translate(trans) - self.assertEquals(len(s), repeats * sublen) - self.assertEquals(s[:sublen], SUBSTR.translate(trans)) - self.assertEquals(s[-sublen:], SUBSTR.translate(trans)) - self.assertEquals(s.count('.'), 0) - self.assertEquals(s.count('!'), repeats * 2) - self.assertEquals(s.count('z'), repeats * 3) - - @bigmemtest(minsize=_2G + 5, memuse=2) - def test_upper(self, size): - s = 'a' * size - s = s.upper() - self.assertEquals(len(s), size) - self.assertEquals(s.count('A'), size) - - @bigmemtest(minsize=_2G + 20, memuse=1) - def test_zfill(self, size): - SUBSTR = '-568324723598234' - s = SUBSTR.zfill(size) - self.failUnless(s.endswith('0' + SUBSTR[1:])) - self.failUnless(s.startswith('-0')) - self.assertEquals(len(s), size) - self.assertEquals(s.count('0'), size - len(SUBSTR)) - - @bigmemtest(minsize=_2G + 10, memuse=2) - def test_format(self, size): - s = '-' * size - sf = '%s' % (s,) - self.failUnless(s == sf) - del sf - sf = '..%s..' % (s,) - self.assertEquals(len(sf), len(s) + 4) - self.failUnless(sf.startswith('..-')) - self.failUnless(sf.endswith('-..')) - del s, sf - - size = (size // 2) - edge = '-' * size - s = ''.join([edge, '%s', edge]) - del edge - s = s % '...' - self.assertEquals(len(s), size * 2 + 3) - self.assertEquals(s.count('.'), 3) - self.assertEquals(s.count('-'), size * 2) - - @bigmemtest(minsize=_2G + 10, memuse=2) - def test_repr_small(self, size): - s = '-' * size - s = repr(s) - self.assertEquals(len(s), size + 2) - self.assertEquals(s[0], "'") - self.assertEquals(s[-1], "'") - self.assertEquals(s.count('-'), size) - del s - # repr() will create a string four times as large as this 'binary - # string', but we don't want to allocate much more than twice - # size in total. (We do extra testing in test_repr_large()) - size = size // 5 * 2 - s = '\x00' * size - s = repr(s) - self.assertEquals(len(s), size * 4 + 2) - self.assertEquals(s[0], "'") - self.assertEquals(s[-1], "'") - self.assertEquals(s.count('\\'), size) - self.assertEquals(s.count('0'), size * 2) - - @bigmemtest(minsize=_2G + 10, memuse=5) - def test_repr_large(self, size): - s = '\x00' * size - s = repr(s) - self.assertEquals(len(s), size * 4 + 2) - self.assertEquals(s[0], "'") - self.assertEquals(s[-1], "'") - self.assertEquals(s.count('\\'), size) - self.assertEquals(s.count('0'), size * 2) - - # This test is meaningful even with size < 2G, as long as the - # doubled string is > 2G (but it tests more if both are > 2G :) - @bigmemtest(minsize=_1G + 2, memuse=3) - def test_concat(self, size): - s = '.' * size - self.assertEquals(len(s), size) - s = s + s - self.assertEquals(len(s), size * 2) - self.assertEquals(s.count('.'), size * 2) - - # This test is meaningful even with size < 2G, as long as the - # repeated string is > 2G (but it tests more if both are > 2G :) - @bigmemtest(minsize=_1G + 2, memuse=3) - def test_repeat(self, size): - s = '.' * size - self.assertEquals(len(s), size) - s = s * 2 - self.assertEquals(len(s), size * 2) - self.assertEquals(s.count('.'), size * 2) - - @bigmemtest(minsize=_2G + 20, memuse=1) - def test_slice_and_getitem(self, size): - SUBSTR = '0123456789' - sublen = len(SUBSTR) - s = SUBSTR * (size // sublen) - stepsize = len(s) // 100 - stepsize = stepsize - (stepsize % sublen) - for i in range(0, len(s) - stepsize, stepsize): - self.assertEquals(s[i], SUBSTR[0]) - self.assertEquals(s[i:i + sublen], SUBSTR) - self.assertEquals(s[i:i + sublen:2], SUBSTR[::2]) - if i > 0: - self.assertEquals(s[i + sublen - 1:i - 1:-3], - SUBSTR[sublen::-3]) - # Make sure we do some slicing and indexing near the end of the - # string, too. - self.assertEquals(s[len(s) - 1], SUBSTR[-1]) - self.assertEquals(s[-1], SUBSTR[-1]) - self.assertEquals(s[len(s) - 10], SUBSTR[0]) - self.assertEquals(s[-sublen], SUBSTR[0]) - self.assertEquals(s[len(s):], '') - self.assertEquals(s[len(s) - 1:], SUBSTR[-1]) - self.assertEquals(s[-1:], SUBSTR[-1]) - self.assertEquals(s[len(s) - sublen:], SUBSTR) - self.assertEquals(s[-sublen:], SUBSTR) - self.assertEquals(len(s[:]), len(s)) - self.assertEquals(len(s[:len(s) - 5]), len(s) - 5) - self.assertEquals(len(s[5:-5]), len(s) - 10) - - self.assertRaises(IndexError, operator.getitem, s, len(s)) - self.assertRaises(IndexError, operator.getitem, s, len(s) + 1) - self.assertRaises(IndexError, operator.getitem, s, len(s) + 1<<31) - - @bigmemtest(minsize=_2G, memuse=2) - def test_contains(self, size): - SUBSTR = '0123456789' - edge = '-' * (size // 2) - s = ''.join([edge, SUBSTR, edge]) - del edge - self.failUnless(SUBSTR in s) - self.failIf(SUBSTR * 2 in s) - self.failUnless('-' in s) - self.failIf('a' in s) - s += 'a' - self.failUnless('a' in s) - - @bigmemtest(minsize=_2G + 10, memuse=2) - def test_compare(self, size): - s1 = '-' * size - s2 = '-' * size - self.failUnless(s1 == s2) - del s2 - s2 = s1 + 'a' - self.failIf(s1 == s2) - del s2 - s2 = '.' * size - self.failIf(s1 == s2) - - @bigmemtest(minsize=_2G + 10, memuse=1) - def test_hash(self, size): - # Not sure if we can do any meaningful tests here... Even if we - # start relying on the exact algorithm used, the result will be - # different depending on the size of the C 'long int'. Even this - # test is dodgy (there's no *guarantee* that the two things should - # have a different hash, even if they, in the current - # implementation, almost always do.) - s = '\x00' * size - h1 = hash(s) - del s - s = '\x00' * (size + 1) - self.failIf(h1 == hash(s)) - -class TupleTest(unittest.TestCase): - - # Tuples have a small, fixed-sized head and an array of pointers to - # data. Since we're testing 64-bit addressing, we can assume that the - # pointers are 8 bytes, and that thus that the tuples take up 8 bytes - # per size. - - # As a side-effect of testing long tuples, these tests happen to test - # having more than 2<<31 references to any given object. Hence the - # use of different types of objects as contents in different tests. - - @bigmemtest(minsize=_2G + 2, memuse=16) - def test_compare(self, size): - t1 = (u'',) * size - t2 = (u'',) * size - self.failUnless(t1 == t2) - del t2 - t2 = (u'',) * (size + 1) - self.failIf(t1 == t2) - del t2 - t2 = (1,) * size - self.failIf(t1 == t2) - - # Test concatenating into a single tuple of more than 2G in length, - # and concatenating a tuple of more than 2G in length separately, so - # the smaller test still gets run even if there isn't memory for the - # larger test (but we still let the tester know the larger test is - # skipped, in verbose mode.) - def basic_concat_test(self, size): - t = ((),) * size - self.assertEquals(len(t), size) - t = t + t - self.assertEquals(len(t), size * 2) - - @bigmemtest(minsize=_2G // 2 + 2, memuse=24) - def test_concat_small(self, size): - return self.basic_concat_test(size) - - @bigmemtest(minsize=_2G + 2, memuse=24) - def test_concat_large(self, size): - return self.basic_concat_test(size) - - @bigmemtest(minsize=_2G // 5 + 10, memuse=8*5) - def test_contains(self, size): - t = (1, 2, 3, 4, 5) * size - self.assertEquals(len(t), size * 5) - self.failUnless(5 in t) - self.failIf((1, 2, 3, 4, 5) in t) - self.failIf(0 in t) - - @bigmemtest(minsize=_2G + 10, memuse=8) - def test_hash(self, size): - t1 = (0,) * size - h1 = hash(t1) - del t1 - t2 = (0,) * (size + 1) - self.failIf(h1 == hash(t2)) - - @bigmemtest(minsize=_2G + 10, memuse=8) - def test_index_and_slice(self, size): - t = (None,) * size - self.assertEquals(len(t), size) - self.assertEquals(t[-1], None) - self.assertEquals(t[5], None) - self.assertEquals(t[size - 1], None) - self.assertRaises(IndexError, operator.getitem, t, size) - self.assertEquals(t[:5], (None,) * 5) - self.assertEquals(t[-5:], (None,) * 5) - self.assertEquals(t[20:25], (None,) * 5) - self.assertEquals(t[-25:-20], (None,) * 5) - self.assertEquals(t[size - 5:], (None,) * 5) - self.assertEquals(t[size - 5:size], (None,) * 5) - self.assertEquals(t[size - 6:size - 2], (None,) * 4) - self.assertEquals(t[size:size], ()) - self.assertEquals(t[size:size+5], ()) - - # Like test_concat, split in two. - def basic_test_repeat(self, size): - t = ('',) * size - self.assertEquals(len(t), size) - t = t * 2 - self.assertEquals(len(t), size * 2) - - @bigmemtest(minsize=_2G // 2 + 2, memuse=24) - def test_repeat_small(self, size): - return self.basic_test_repeat(size) - - @bigmemtest(minsize=_2G + 2, memuse=24) - def test_repeat_large(self, size): - return self.basic_test_repeat(size) - - # Like test_concat, split in two. - def basic_test_repr(self, size): - t = (0,) * size - s = repr(t) - # The repr of a tuple of 0's is exactly three times the tuple length. - self.assertEquals(len(s), size * 3) - self.assertEquals(s[:5], '(0, 0') - self.assertEquals(s[-5:], '0, 0)') - self.assertEquals(s.count('0'), size) - - @bigmemtest(minsize=_2G // 3 + 2, memuse=8+3) - def test_repr_small(self, size): - return self.basic_test_repr(size) - - @bigmemtest(minsize=_2G + 2, memuse=8+3) - def test_repr_large(self, size): - return self.basic_test_repr(size) - -class ListTest(unittest.TestCase): - - # Like tuples, lists have a small, fixed-sized head and an array of - # pointers to data, so 8 bytes per size. Also like tuples, we make the - # lists hold references to various objects to test their refcount - # limits. - - @bigmemtest(minsize=_2G + 2, memuse=16) - def test_compare(self, size): - l1 = [u''] * size - l2 = [u''] * size - self.failUnless(l1 == l2) - del l2 - l2 = [u''] * (size + 1) - self.failIf(l1 == l2) - del l2 - l2 = [2] * size - self.failIf(l1 == l2) - - # Test concatenating into a single list of more than 2G in length, - # and concatenating a list of more than 2G in length separately, so - # the smaller test still gets run even if there isn't memory for the - # larger test (but we still let the tester know the larger test is - # skipped, in verbose mode.) - def basic_test_concat(self, size): - l = [[]] * size - self.assertEquals(len(l), size) - l = l + l - self.assertEquals(len(l), size * 2) - - @bigmemtest(minsize=_2G // 2 + 2, memuse=24) - def test_concat_small(self, size): - return self.basic_test_concat(size) - - @bigmemtest(minsize=_2G + 2, memuse=24) - def test_concat_large(self, size): - return self.basic_test_concat(size) - - @bigmemtest(minsize=_2G // 5 + 10, memuse=8*5) - def test_contains(self, size): - l = [1, 2, 3, 4, 5] * size - self.assertEquals(len(l), size * 5) - self.failUnless(5 in l) - self.failIf([1, 2, 3, 4, 5] in l) - self.failIf(0 in l) - - @bigmemtest(minsize=_2G + 10, memuse=8) - def test_hash(self, size): - l = [0] * size - self.failUnlessRaises(TypeError, hash, l) - - @bigmemtest(minsize=_2G + 10, memuse=8) - def test_index_and_slice(self, size): - l = [None] * size - self.assertEquals(len(l), size) - self.assertEquals(l[-1], None) - self.assertEquals(l[5], None) - self.assertEquals(l[size - 1], None) - self.assertRaises(IndexError, operator.getitem, l, size) - self.assertEquals(l[:5], [None] * 5) - self.assertEquals(l[-5:], [None] * 5) - self.assertEquals(l[20:25], [None] * 5) - self.assertEquals(l[-25:-20], [None] * 5) - self.assertEquals(l[size - 5:], [None] * 5) - self.assertEquals(l[size - 5:size], [None] * 5) - self.assertEquals(l[size - 6:size - 2], [None] * 4) - self.assertEquals(l[size:size], []) - self.assertEquals(l[size:size+5], []) - - l[size - 2] = 5 - self.assertEquals(len(l), size) - self.assertEquals(l[-3:], [None, 5, None]) - self.assertEquals(l.count(5), 1) - self.assertRaises(IndexError, operator.setitem, l, size, 6) - self.assertEquals(len(l), size) - - l[size - 7:] = [1, 2, 3, 4, 5] - size -= 2 - self.assertEquals(len(l), size) - self.assertEquals(l[-7:], [None, None, 1, 2, 3, 4, 5]) - - l[:7] = [1, 2, 3, 4, 5] - size -= 2 - self.assertEquals(len(l), size) - self.assertEquals(l[:7], [1, 2, 3, 4, 5, None, None]) - - del l[size - 1] - size -= 1 - self.assertEquals(len(l), size) - self.assertEquals(l[-1], 4) - - del l[-2:] - size -= 2 - self.assertEquals(len(l), size) - self.assertEquals(l[-1], 2) - - del l[0] - size -= 1 - self.assertEquals(len(l), size) - self.assertEquals(l[0], 2) - - del l[:2] - size -= 2 - self.assertEquals(len(l), size) - self.assertEquals(l[0], 4) - - # Like test_concat, split in two. - def basic_test_repeat(self, size): - l = [] * size - self.failIf(l) - l = [''] * size - self.assertEquals(len(l), size) - l = l * 2 - self.assertEquals(len(l), size * 2) - - @bigmemtest(minsize=_2G // 2 + 2, memuse=24) - def test_repeat_small(self, size): - return self.basic_test_repeat(size) - - @bigmemtest(minsize=_2G + 2, memuse=24) - def test_repeat_large(self, size): - return self.basic_test_repeat(size) - - # Test repr-result of >2G - def basic_test_repr(self, size): - l = [0] * size - s = repr(l) - # The repr of a list of 0's is exactly three times the list length. - self.assertEquals(len(s), size * 3) - self.assertEquals(s[:5], '[0, 0') - self.assertEquals(s[-5:], '0, 0]') - self.assertEquals(s.count('0'), size) - - @bigmemtest(minsize=_2G // 3 + 2, memuse=8 + 3) - def test_repr_small(self, size): - return self.basic_test_repr(size) - - @bigmemtest(minsize=_2G + 2, memuse=8 + 3) - def test_repr_large(self, size): - return self.basic_test_repr(size) - - - @bigmemtest(minsize=_2G, memuse=8) - def test_append(self, size): - l = [object()] * size - l.append(object()) - self.assertEquals(len(l), size+1) - self.failUnless(l[-3] is l[-2]) - self.failIf(l[-2] is l[-1]) - - @bigmemtest(minsize=_2G // 5 + 2, memuse=8 * 5) - def test_count(self, size): - l = [1, 2, 3, 4, 5] * size - self.assertEquals(l.count(1), size) - self.assertEquals(l.count("1"), 0) - - def basic_test_extend(self, size): - l = [file] * size - l.extend(l) - self.assertEquals(len(l), size * 2) - self.failUnless(l[0] is l[-1]) - self.failUnless(l[size - 1] is l[size + 1]) - - @bigmemtest(minsize=_2G // 2 + 2, memuse=8) - def test_extend_small(self, size): - return self.basic_test_extend(size) - - @bigmemtest(minsize=_2G + 2, memuse=8) - def test_extend_large(self, size): - return self.basic_test_extend(size) - - @bigmemtest(minsize=_2G + 10, memuse=8) - def test_index(self, size): - l = [1L, 2L, 3L, 4L, 5L] * (size // 5) - self.assertEquals(l.index(1), 0) - self.assertEquals(l.index(5, size - 5), size - 1) - self.assertEquals(l.index(5, size - 5, size), size - 1) - self.assertRaises(ValueError, l.index, 1, size - 4, size) - self.assertRaises(ValueError, l.index, 6L) - - @bigmemtest(minsize=_2G + 10, memuse=8) - def test_insert(self, size): - l = [1.0] * size - l.insert(size - 1, "A") - size += 1 - self.assertEquals(len(l), size) - self.assertEquals(l[-3:], [1.0, "A", 1.0]) - - l.insert(size + 1, "B") - size += 1 - self.assertEquals(len(l), size) - self.assertEquals(l[-3:], ["A", 1.0, "B"]) - - l.insert(1, "C") - size += 1 - self.assertEquals(len(l), size) - self.assertEquals(l[:3], [1.0, "C", 1.0]) - self.assertEquals(l[size - 3:], ["A", 1.0, "B"]) - - @bigmemtest(minsize=_2G + 20, memuse=8) - def test_pop(self, size): - l = [u"a", u"b", u"c", u"d", u"e"] * (size // 5) - self.assertEquals(len(l), size) - - item = l.pop() - size -= 1 - self.assertEquals(len(l), size) - self.assertEquals(item, u"e") - - item = l.pop(0) - size -= 1 - self.assertEquals(len(l), size) - self.assertEquals(item, u"a") - - item = l.pop(size - 2) - size -= 1 - self.assertEquals(len(l), size) - self.assertEquals(item, u"c") - - @bigmemtest(minsize=_2G + 10, memuse=8) - def test_remove(self, size): - l = [10] * size - self.assertEquals(len(l), size) - - l.remove(10) - size -= 1 - self.assertEquals(len(l), size) - - l.append(5) - size += 1 - self.assertEquals(len(l), size) - self.assertEquals(l[-2:], [10, 5]) - l.remove(5) - size -= 1 - self.assertEquals(len(l), size) - self.assertEquals(l[-2:], [10, 10]) - - @bigmemtest(minsize=_2G + 10, memuse=8) - def test_reverse(self, size): - l = [1, 2, 3, 4, 5] * (size // 5) - l.reverse() - self.assertEquals(len(l), size) - self.assertEquals(l[-5:], [5, 4, 3, 2, 1]) - self.assertEquals(l[:5], [5, 4, 3, 2, 1]) - - @bigmemtest(minsize=_2G + 10, memuse=8) - def test_sort(self, size): - l = [1, 2, 3, 4, 5] * (size // 5) - l.sort() - self.assertEquals(len(l), size) - self.assertEquals(l.count(1), size // 5) - self.assertEquals(l[:10], [1] * 10) - self.assertEquals(l[-10:], [5] * 10) - -def test_main(): - test_support.run_unittest(StrTest, TupleTest, ListTest) - -if __name__ == '__main__': - if len(sys.argv) > 1: - test_support.set_memlimit(sys.argv[1]) - test_main() +from test import test_support +from test.test_support import bigmemtest, _1G, _2G + +import unittest +import operator +import string +import sys + +# Bigmem testing houserules: +# +# - Try not to allocate too many large objects. It's okay to rely on +# refcounting semantics, but don't forget that 's = create_largestring()' +# doesn't release the old 's' (if it exists) until well after its new +# value has been created. Use 'del s' before the create_largestring call. +# +# - Do *not* compare large objects using assertEquals or similar. It's a +# lengty operation and the errormessage will be utterly useless due to +# its size. To make sure whether a result has the right contents, better +# to use the strip or count methods, or compare meaningful slices. +# +# - Don't forget to test for large indices, offsets and results and such, +# in addition to large sizes. +# +# - When repeating an object (say, a substring, or a small list) to create +# a large object, make the subobject of a length that is not a power of +# 2. That way, int-wrapping problems are more easily detected. +# +# - While the bigmemtest decorator speaks of 'minsize', all tests will +# actually be called with a much smaller number too, in the normal +# test run (5Kb currently.) This is so the tests themselves get frequent +# testing Consequently, always make all large allocations based on the +# passed-in 'size', and don't rely on the size being very large. Also, +# memuse-per-size should remain sane (less than a few thousand); if your +# test uses more, adjust 'size' upward, instead. + +class StrTest(unittest.TestCase): + @bigmemtest(minsize=_2G, memuse=2) + def test_capitalize(self, size): + SUBSTR = ' abc def ghi' + s = '-' * size + SUBSTR + caps = s.capitalize() + self.assertEquals(caps[-len(SUBSTR):], + SUBSTR.capitalize()) + self.assertEquals(caps.lstrip('-'), SUBSTR) + + @bigmemtest(minsize=_2G + 10, memuse=1) + def test_center(self, size): + SUBSTR = ' abc def ghi' + s = SUBSTR.center(size) + self.assertEquals(len(s), size) + lpadsize = rpadsize = (len(s) - len(SUBSTR)) // 2 + if len(s) % 2: + lpadsize += 1 + self.assertEquals(s[lpadsize:-rpadsize], SUBSTR) + self.assertEquals(s.strip(), SUBSTR.strip()) + + @bigmemtest(minsize=_2G, memuse=2) + def test_count(self, size): + SUBSTR = ' abc def ghi' + s = '.' * size + SUBSTR + self.assertEquals(s.count('.'), size) + s += '.' + self.assertEquals(s.count('.'), size + 1) + self.assertEquals(s.count(' '), 3) + self.assertEquals(s.count('i'), 1) + self.assertEquals(s.count('j'), 0) + + @bigmemtest(minsize=0, memuse=1) + def test_decode(self, size): + pass + + @bigmemtest(minsize=0, memuse=1) + def test_encode(self, size): + pass + + @bigmemtest(minsize=_2G, memuse=2) + def test_endswith(self, size): + SUBSTR = ' abc def ghi' + s = '-' * size + SUBSTR + self.failUnless(s.endswith(SUBSTR)) + self.failUnless(s.endswith(s)) + s2 = '...' + s + self.failUnless(s2.endswith(s)) + self.failIf(s.endswith('a' + SUBSTR)) + self.failIf(SUBSTR.endswith(s)) + + @bigmemtest(minsize=_2G + 10, memuse=2) + def test_expandtabs(self, size): + s = '-' * size + tabsize = 8 + self.assertEquals(s.expandtabs(), s) + del s + slen, remainder = divmod(size, tabsize) + s = ' \t' * slen + s = s.expandtabs(tabsize) + self.assertEquals(len(s), size - remainder) + self.assertEquals(len(s.strip(' ')), 0) + + @bigmemtest(minsize=_2G, memuse=2) + def test_find(self, size): + SUBSTR = ' abc def ghi' + sublen = len(SUBSTR) + s = ''.join([SUBSTR, '-' * size, SUBSTR]) + self.assertEquals(s.find(' '), 0) + self.assertEquals(s.find(SUBSTR), 0) + self.assertEquals(s.find(' ', sublen), sublen + size) + self.assertEquals(s.find(SUBSTR, len(SUBSTR)), sublen + size) + self.assertEquals(s.find('i'), SUBSTR.find('i')) + self.assertEquals(s.find('i', sublen), + sublen + size + SUBSTR.find('i')) + self.assertEquals(s.find('i', size), + sublen + size + SUBSTR.find('i')) + self.assertEquals(s.find('j'), -1) + + @bigmemtest(minsize=_2G, memuse=2) + def test_index(self, size): + SUBSTR = ' abc def ghi' + sublen = len(SUBSTR) + s = ''.join([SUBSTR, '-' * size, SUBSTR]) + self.assertEquals(s.index(' '), 0) + self.assertEquals(s.index(SUBSTR), 0) + self.assertEquals(s.index(' ', sublen), sublen + size) + self.assertEquals(s.index(SUBSTR, sublen), sublen + size) + self.assertEquals(s.index('i'), SUBSTR.index('i')) + self.assertEquals(s.index('i', sublen), + sublen + size + SUBSTR.index('i')) + self.assertEquals(s.index('i', size), + sublen + size + SUBSTR.index('i')) + self.assertRaises(ValueError, s.index, 'j') + + @bigmemtest(minsize=_2G, memuse=2) + def test_isalnum(self, size): + SUBSTR = '123456' + s = 'a' * size + SUBSTR + self.failUnless(s.isalnum()) + s += '.' + self.failIf(s.isalnum()) + + @bigmemtest(minsize=_2G, memuse=2) + def test_isalpha(self, size): + SUBSTR = 'zzzzzzz' + s = 'a' * size + SUBSTR + self.failUnless(s.isalpha()) + s += '.' + self.failIf(s.isalpha()) + + @bigmemtest(minsize=_2G, memuse=2) + def test_isdigit(self, size): + SUBSTR = '123456' + s = '9' * size + SUBSTR + self.failUnless(s.isdigit()) + s += 'z' + self.failIf(s.isdigit()) + + @bigmemtest(minsize=_2G, memuse=2) + def test_islower(self, size): + chars = ''.join([ chr(c) for c in range(255) if not chr(c).isupper() ]) + repeats = size // len(chars) + 2 + s = chars * repeats + self.failUnless(s.islower()) + s += 'A' + self.failIf(s.islower()) + + @bigmemtest(minsize=_2G, memuse=2) + def test_isspace(self, size): + whitespace = ' \f\n\r\t\v' + repeats = size // len(whitespace) + 2 + s = whitespace * repeats + self.failUnless(s.isspace()) + s += 'j' + self.failIf(s.isspace()) + + @bigmemtest(minsize=_2G, memuse=2) + def test_istitle(self, size): + SUBSTR = '123456' + s = ''.join(['A', 'a' * size, SUBSTR]) + self.failUnless(s.istitle()) + s += 'A' + self.failUnless(s.istitle()) + s += 'aA' + self.failIf(s.istitle()) + + @bigmemtest(minsize=_2G, memuse=2) + def test_isupper(self, size): + chars = ''.join([ chr(c) for c in range(255) if not chr(c).islower() ]) + repeats = size // len(chars) + 2 + s = chars * repeats + self.failUnless(s.isupper()) + s += 'a' + self.failIf(s.isupper()) + + @bigmemtest(minsize=_2G, memuse=2) + def test_join(self, size): + s = 'A' * size + x = s.join(['aaaaa', 'bbbbb']) + self.assertEquals(x.count('a'), 5) + self.assertEquals(x.count('b'), 5) + self.failUnless(x.startswith('aaaaaA')) + self.failUnless(x.endswith('Abbbbb')) + + @bigmemtest(minsize=_2G + 10, memuse=1) + def test_ljust(self, size): + SUBSTR = ' abc def ghi' + s = SUBSTR.ljust(size) + self.failUnless(s.startswith(SUBSTR + ' ')) + self.assertEquals(len(s), size) + self.assertEquals(s.strip(), SUBSTR.strip()) + + @bigmemtest(minsize=_2G + 10, memuse=2) + def test_lower(self, size): + s = 'A' * size + s = s.lower() + self.assertEquals(len(s), size) + self.assertEquals(s.count('a'), size) + + @bigmemtest(minsize=_2G + 10, memuse=1) + def test_lstrip(self, size): + SUBSTR = 'abc def ghi' + s = SUBSTR.rjust(size) + self.assertEquals(len(s), size) + self.assertEquals(s.lstrip(), SUBSTR.lstrip()) + del s + s = SUBSTR.ljust(size) + self.assertEquals(len(s), size) + stripped = s.lstrip() + self.failUnless(stripped is s) + + @bigmemtest(minsize=_2G + 10, memuse=2) + def test_replace(self, size): + replacement = 'a' + s = ' ' * size + s = s.replace(' ', replacement) + self.assertEquals(len(s), size) + self.assertEquals(s.count(replacement), size) + s = s.replace(replacement, ' ', size - 4) + self.assertEquals(len(s), size) + self.assertEquals(s.count(replacement), 4) + self.assertEquals(s[-10:], ' aaaa') + + @bigmemtest(minsize=_2G, memuse=2) + def test_rfind(self, size): + SUBSTR = ' abc def ghi' + sublen = len(SUBSTR) + s = ''.join([SUBSTR, '-' * size, SUBSTR]) + self.assertEquals(s.rfind(' '), sublen + size + SUBSTR.rfind(' ')) + self.assertEquals(s.rfind(SUBSTR), sublen + size) + self.assertEquals(s.rfind(' ', 0, size), SUBSTR.rfind(' ')) + self.assertEquals(s.rfind(SUBSTR, 0, sublen + size), 0) + self.assertEquals(s.rfind('i'), sublen + size + SUBSTR.rfind('i')) + self.assertEquals(s.rfind('i', 0, sublen), SUBSTR.rfind('i')) + self.assertEquals(s.rfind('i', 0, sublen + size), + SUBSTR.rfind('i')) + self.assertEquals(s.rfind('j'), -1) + + @bigmemtest(minsize=_2G, memuse=2) + def test_rindex(self, size): + SUBSTR = ' abc def ghi' + sublen = len(SUBSTR) + s = ''.join([SUBSTR, '-' * size, SUBSTR]) + self.assertEquals(s.rindex(' '), + sublen + size + SUBSTR.rindex(' ')) + self.assertEquals(s.rindex(SUBSTR), sublen + size) + self.assertEquals(s.rindex(' ', 0, sublen + size - 1), + SUBSTR.rindex(' ')) + self.assertEquals(s.rindex(SUBSTR, 0, sublen + size), 0) + self.assertEquals(s.rindex('i'), + sublen + size + SUBSTR.rindex('i')) + self.assertEquals(s.rindex('i', 0, sublen), SUBSTR.rindex('i')) + self.assertEquals(s.rindex('i', 0, sublen + size), + SUBSTR.rindex('i')) + self.assertRaises(ValueError, s.rindex, 'j') + + @bigmemtest(minsize=_2G + 10, memuse=1) + def test_rjust(self, size): + SUBSTR = ' abc def ghi' + s = SUBSTR.ljust(size) + self.failUnless(s.startswith(SUBSTR + ' ')) + self.assertEquals(len(s), size) + self.assertEquals(s.strip(), SUBSTR.strip()) + + @bigmemtest(minsize=_2G + 10, memuse=1) + def test_rstrip(self, size): + SUBSTR = ' abc def ghi' + s = SUBSTR.ljust(size) + self.assertEquals(len(s), size) + self.assertEquals(s.rstrip(), SUBSTR.rstrip()) + del s + s = SUBSTR.rjust(size) + self.assertEquals(len(s), size) + stripped = s.rstrip() + self.failUnless(stripped is s) + + # The test takes about size bytes to build a string, and then about + # sqrt(size) substrings of sqrt(size) in size and a list to + # hold sqrt(size) items. It's close but just over 2x size. + @bigmemtest(minsize=_2G, memuse=2.1) + def test_split_small(self, size): + # Crudely calculate an estimate so that the result of s.split won't + # take up an inordinate amount of memory + chunksize = int(size ** 0.5 + 2) + SUBSTR = 'a' + ' ' * chunksize + s = SUBSTR * chunksize + l = s.split() + self.assertEquals(len(l), chunksize) + self.assertEquals(set(l), set(['a'])) + del l + l = s.split('a') + self.assertEquals(len(l), chunksize + 1) + self.assertEquals(set(l), set(['', ' ' * chunksize])) + + # Allocates a string of twice size (and briefly two) and a list of + # size. Because of internal affairs, the s.split() call produces a + # list of size times the same one-character string, so we only + # suffer for the list size. (Otherwise, it'd cost another 48 times + # size in bytes!) Nevertheless, a list of size takes + # 8*size bytes. + @bigmemtest(minsize=_2G + 5, memuse=10) + def test_split_large(self, size): + s = ' a' * size + ' ' + l = s.split() + self.assertEquals(len(l), size) + self.assertEquals(set(l), set(['a'])) + del l + l = s.split('a') + self.assertEquals(len(l), size + 1) + self.assertEquals(set(l), set([' '])) + + @bigmemtest(minsize=_2G, memuse=2.1) + def test_splitlines(self, size): + # Crudely calculate an estimate so that the result of s.split won't + # take up an inordinate amount of memory + chunksize = int(size ** 0.5 + 2) // 2 + SUBSTR = ' ' * chunksize + '\n' + ' ' * chunksize + '\r\n' + s = SUBSTR * chunksize + l = s.splitlines() + self.assertEquals(len(l), chunksize * 2) + self.assertEquals(set(l), set([' ' * chunksize])) + + @bigmemtest(minsize=_2G, memuse=2) + def test_startswith(self, size): + SUBSTR = ' abc def ghi' + s = '-' * size + SUBSTR + self.failUnless(s.startswith(s)) + self.failUnless(s.startswith('-' * size)) + self.failIf(s.startswith(SUBSTR)) + + @bigmemtest(minsize=_2G, memuse=1) + def test_strip(self, size): + SUBSTR = ' abc def ghi ' + s = SUBSTR.rjust(size) + self.assertEquals(len(s), size) + self.assertEquals(s.strip(), SUBSTR.strip()) + del s + s = SUBSTR.ljust(size) + self.assertEquals(len(s), size) + self.assertEquals(s.strip(), SUBSTR.strip()) + + @bigmemtest(minsize=_2G, memuse=2) + def test_swapcase(self, size): + SUBSTR = "aBcDeFG12.'\xa9\x00" + sublen = len(SUBSTR) + repeats = size // sublen + 2 + s = SUBSTR * repeats + s = s.swapcase() + self.assertEquals(len(s), sublen * repeats) + self.assertEquals(s[:sublen * 3], SUBSTR.swapcase() * 3) + self.assertEquals(s[-sublen * 3:], SUBSTR.swapcase() * 3) + + @bigmemtest(minsize=_2G, memuse=2) + def test_title(self, size): + SUBSTR = 'SpaaHAaaAaham' + s = SUBSTR * (size // len(SUBSTR) + 2) + s = s.title() + self.failUnless(s.startswith((SUBSTR * 3).title())) + self.failUnless(s.endswith(SUBSTR.lower() * 3)) + + @bigmemtest(minsize=_2G, memuse=2) + def test_translate(self, size): + trans = string.maketrans('.aZ', '-!$') + SUBSTR = 'aZz.z.Aaz.' + sublen = len(SUBSTR) + repeats = size // sublen + 2 + s = SUBSTR * repeats + s = s.translate(trans) + self.assertEquals(len(s), repeats * sublen) + self.assertEquals(s[:sublen], SUBSTR.translate(trans)) + self.assertEquals(s[-sublen:], SUBSTR.translate(trans)) + self.assertEquals(s.count('.'), 0) + self.assertEquals(s.count('!'), repeats * 2) + self.assertEquals(s.count('z'), repeats * 3) + + @bigmemtest(minsize=_2G + 5, memuse=2) + def test_upper(self, size): + s = 'a' * size + s = s.upper() + self.assertEquals(len(s), size) + self.assertEquals(s.count('A'), size) + + @bigmemtest(minsize=_2G + 20, memuse=1) + def test_zfill(self, size): + SUBSTR = '-568324723598234' + s = SUBSTR.zfill(size) + self.failUnless(s.endswith('0' + SUBSTR[1:])) + self.failUnless(s.startswith('-0')) + self.assertEquals(len(s), size) + self.assertEquals(s.count('0'), size - len(SUBSTR)) + + @bigmemtest(minsize=_2G + 10, memuse=2) + def test_format(self, size): + s = '-' * size + sf = '%s' % (s,) + self.failUnless(s == sf) + del sf + sf = '..%s..' % (s,) + self.assertEquals(len(sf), len(s) + 4) + self.failUnless(sf.startswith('..-')) + self.failUnless(sf.endswith('-..')) + del s, sf + + size = (size // 2) + edge = '-' * size + s = ''.join([edge, '%s', edge]) + del edge + s = s % '...' + self.assertEquals(len(s), size * 2 + 3) + self.assertEquals(s.count('.'), 3) + self.assertEquals(s.count('-'), size * 2) + + @bigmemtest(minsize=_2G + 10, memuse=2) + def test_repr_small(self, size): + s = '-' * size + s = repr(s) + self.assertEquals(len(s), size + 2) + self.assertEquals(s[0], "'") + self.assertEquals(s[-1], "'") + self.assertEquals(s.count('-'), size) + del s + # repr() will create a string four times as large as this 'binary + # string', but we don't want to allocate much more than twice + # size in total. (We do extra testing in test_repr_large()) + size = size // 5 * 2 + s = '\x00' * size + s = repr(s) + self.assertEquals(len(s), size * 4 + 2) + self.assertEquals(s[0], "'") + self.assertEquals(s[-1], "'") + self.assertEquals(s.count('\\'), size) + self.assertEquals(s.count('0'), size * 2) + + @bigmemtest(minsize=_2G + 10, memuse=5) + def test_repr_large(self, size): + s = '\x00' * size + s = repr(s) + self.assertEquals(len(s), size * 4 + 2) + self.assertEquals(s[0], "'") + self.assertEquals(s[-1], "'") + self.assertEquals(s.count('\\'), size) + self.assertEquals(s.count('0'), size * 2) + + # This test is meaningful even with size < 2G, as long as the + # doubled string is > 2G (but it tests more if both are > 2G :) + @bigmemtest(minsize=_1G + 2, memuse=3) + def test_concat(self, size): + s = '.' * size + self.assertEquals(len(s), size) + s = s + s + self.assertEquals(len(s), size * 2) + self.assertEquals(s.count('.'), size * 2) + + # This test is meaningful even with size < 2G, as long as the + # repeated string is > 2G (but it tests more if both are > 2G :) + @bigmemtest(minsize=_1G + 2, memuse=3) + def test_repeat(self, size): + s = '.' * size + self.assertEquals(len(s), size) + s = s * 2 + self.assertEquals(len(s), size * 2) + self.assertEquals(s.count('.'), size * 2) + + @bigmemtest(minsize=_2G + 20, memuse=1) + def test_slice_and_getitem(self, size): + SUBSTR = '0123456789' + sublen = len(SUBSTR) + s = SUBSTR * (size // sublen) + stepsize = len(s) // 100 + stepsize = stepsize - (stepsize % sublen) + for i in range(0, len(s) - stepsize, stepsize): + self.assertEquals(s[i], SUBSTR[0]) + self.assertEquals(s[i:i + sublen], SUBSTR) + self.assertEquals(s[i:i + sublen:2], SUBSTR[::2]) + if i > 0: + self.assertEquals(s[i + sublen - 1:i - 1:-3], + SUBSTR[sublen::-3]) + # Make sure we do some slicing and indexing near the end of the + # string, too. + self.assertEquals(s[len(s) - 1], SUBSTR[-1]) + self.assertEquals(s[-1], SUBSTR[-1]) + self.assertEquals(s[len(s) - 10], SUBSTR[0]) + self.assertEquals(s[-sublen], SUBSTR[0]) + self.assertEquals(s[len(s):], '') + self.assertEquals(s[len(s) - 1:], SUBSTR[-1]) + self.assertEquals(s[-1:], SUBSTR[-1]) + self.assertEquals(s[len(s) - sublen:], SUBSTR) + self.assertEquals(s[-sublen:], SUBSTR) + self.assertEquals(len(s[:]), len(s)) + self.assertEquals(len(s[:len(s) - 5]), len(s) - 5) + self.assertEquals(len(s[5:-5]), len(s) - 10) + + self.assertRaises(IndexError, operator.getitem, s, len(s)) + self.assertRaises(IndexError, operator.getitem, s, len(s) + 1) + self.assertRaises(IndexError, operator.getitem, s, len(s) + 1<<31) + + @bigmemtest(minsize=_2G, memuse=2) + def test_contains(self, size): + SUBSTR = '0123456789' + edge = '-' * (size // 2) + s = ''.join([edge, SUBSTR, edge]) + del edge + self.failUnless(SUBSTR in s) + self.failIf(SUBSTR * 2 in s) + self.failUnless('-' in s) + self.failIf('a' in s) + s += 'a' + self.failUnless('a' in s) + + @bigmemtest(minsize=_2G + 10, memuse=2) + def test_compare(self, size): + s1 = '-' * size + s2 = '-' * size + self.failUnless(s1 == s2) + del s2 + s2 = s1 + 'a' + self.failIf(s1 == s2) + del s2 + s2 = '.' * size + self.failIf(s1 == s2) + + @bigmemtest(minsize=_2G + 10, memuse=1) + def test_hash(self, size): + # Not sure if we can do any meaningful tests here... Even if we + # start relying on the exact algorithm used, the result will be + # different depending on the size of the C 'long int'. Even this + # test is dodgy (there's no *guarantee* that the two things should + # have a different hash, even if they, in the current + # implementation, almost always do.) + s = '\x00' * size + h1 = hash(s) + del s + s = '\x00' * (size + 1) + self.failIf(h1 == hash(s)) + +class TupleTest(unittest.TestCase): + + # Tuples have a small, fixed-sized head and an array of pointers to + # data. Since we're testing 64-bit addressing, we can assume that the + # pointers are 8 bytes, and that thus that the tuples take up 8 bytes + # per size. + + # As a side-effect of testing long tuples, these tests happen to test + # having more than 2<<31 references to any given object. Hence the + # use of different types of objects as contents in different tests. + + @bigmemtest(minsize=_2G + 2, memuse=16) + def test_compare(self, size): + t1 = (u'',) * size + t2 = (u'',) * size + self.failUnless(t1 == t2) + del t2 + t2 = (u'',) * (size + 1) + self.failIf(t1 == t2) + del t2 + t2 = (1,) * size + self.failIf(t1 == t2) + + # Test concatenating into a single tuple of more than 2G in length, + # and concatenating a tuple of more than 2G in length separately, so + # the smaller test still gets run even if there isn't memory for the + # larger test (but we still let the tester know the larger test is + # skipped, in verbose mode.) + def basic_concat_test(self, size): + t = ((),) * size + self.assertEquals(len(t), size) + t = t + t + self.assertEquals(len(t), size * 2) + + @bigmemtest(minsize=_2G // 2 + 2, memuse=24) + def test_concat_small(self, size): + return self.basic_concat_test(size) + + @bigmemtest(minsize=_2G + 2, memuse=24) + def test_concat_large(self, size): + return self.basic_concat_test(size) + + @bigmemtest(minsize=_2G // 5 + 10, memuse=8*5) + def test_contains(self, size): + t = (1, 2, 3, 4, 5) * size + self.assertEquals(len(t), size * 5) + self.failUnless(5 in t) + self.failIf((1, 2, 3, 4, 5) in t) + self.failIf(0 in t) + + @bigmemtest(minsize=_2G + 10, memuse=8) + def test_hash(self, size): + t1 = (0,) * size + h1 = hash(t1) + del t1 + t2 = (0,) * (size + 1) + self.failIf(h1 == hash(t2)) + + @bigmemtest(minsize=_2G + 10, memuse=8) + def test_index_and_slice(self, size): + t = (None,) * size + self.assertEquals(len(t), size) + self.assertEquals(t[-1], None) + self.assertEquals(t[5], None) + self.assertEquals(t[size - 1], None) + self.assertRaises(IndexError, operator.getitem, t, size) + self.assertEquals(t[:5], (None,) * 5) + self.assertEquals(t[-5:], (None,) * 5) + self.assertEquals(t[20:25], (None,) * 5) + self.assertEquals(t[-25:-20], (None,) * 5) + self.assertEquals(t[size - 5:], (None,) * 5) + self.assertEquals(t[size - 5:size], (None,) * 5) + self.assertEquals(t[size - 6:size - 2], (None,) * 4) + self.assertEquals(t[size:size], ()) + self.assertEquals(t[size:size+5], ()) + + # Like test_concat, split in two. + def basic_test_repeat(self, size): + t = ('',) * size + self.assertEquals(len(t), size) + t = t * 2 + self.assertEquals(len(t), size * 2) + + @bigmemtest(minsize=_2G // 2 + 2, memuse=24) + def test_repeat_small(self, size): + return self.basic_test_repeat(size) + + @bigmemtest(minsize=_2G + 2, memuse=24) + def test_repeat_large(self, size): + return self.basic_test_repeat(size) + + # Like test_concat, split in two. + def basic_test_repr(self, size): + t = (0,) * size + s = repr(t) + # The repr of a tuple of 0's is exactly three times the tuple length. + self.assertEquals(len(s), size * 3) + self.assertEquals(s[:5], '(0, 0') + self.assertEquals(s[-5:], '0, 0)') + self.assertEquals(s.count('0'), size) + + @bigmemtest(minsize=_2G // 3 + 2, memuse=8+3) + def test_repr_small(self, size): + return self.basic_test_repr(size) + + @bigmemtest(minsize=_2G + 2, memuse=8+3) + def test_repr_large(self, size): + return self.basic_test_repr(size) + +class ListTest(unittest.TestCase): + + # Like tuples, lists have a small, fixed-sized head and an array of + # pointers to data, so 8 bytes per size. Also like tuples, we make the + # lists hold references to various objects to test their refcount + # limits. + + @bigmemtest(minsize=_2G + 2, memuse=16) + def test_compare(self, size): + l1 = [u''] * size + l2 = [u''] * size + self.failUnless(l1 == l2) + del l2 + l2 = [u''] * (size + 1) + self.failIf(l1 == l2) + del l2 + l2 = [2] * size + self.failIf(l1 == l2) + + # Test concatenating into a single list of more than 2G in length, + # and concatenating a list of more than 2G in length separately, so + # the smaller test still gets run even if there isn't memory for the + # larger test (but we still let the tester know the larger test is + # skipped, in verbose mode.) + def basic_test_concat(self, size): + l = [[]] * size + self.assertEquals(len(l), size) + l = l + l + self.assertEquals(len(l), size * 2) + + @bigmemtest(minsize=_2G // 2 + 2, memuse=24) + def test_concat_small(self, size): + return self.basic_test_concat(size) + + @bigmemtest(minsize=_2G + 2, memuse=24) + def test_concat_large(self, size): + return self.basic_test_concat(size) + + @bigmemtest(minsize=_2G // 5 + 10, memuse=8*5) + def test_contains(self, size): + l = [1, 2, 3, 4, 5] * size + self.assertEquals(len(l), size * 5) + self.failUnless(5 in l) + self.failIf([1, 2, 3, 4, 5] in l) + self.failIf(0 in l) + + @bigmemtest(minsize=_2G + 10, memuse=8) + def test_hash(self, size): + l = [0] * size + self.failUnlessRaises(TypeError, hash, l) + + @bigmemtest(minsize=_2G + 10, memuse=8) + def test_index_and_slice(self, size): + l = [None] * size + self.assertEquals(len(l), size) + self.assertEquals(l[-1], None) + self.assertEquals(l[5], None) + self.assertEquals(l[size - 1], None) + self.assertRaises(IndexError, operator.getitem, l, size) + self.assertEquals(l[:5], [None] * 5) + self.assertEquals(l[-5:], [None] * 5) + self.assertEquals(l[20:25], [None] * 5) + self.assertEquals(l[-25:-20], [None] * 5) + self.assertEquals(l[size - 5:], [None] * 5) + self.assertEquals(l[size - 5:size], [None] * 5) + self.assertEquals(l[size - 6:size - 2], [None] * 4) + self.assertEquals(l[size:size], []) + self.assertEquals(l[size:size+5], []) + + l[size - 2] = 5 + self.assertEquals(len(l), size) + self.assertEquals(l[-3:], [None, 5, None]) + self.assertEquals(l.count(5), 1) + self.assertRaises(IndexError, operator.setitem, l, size, 6) + self.assertEquals(len(l), size) + + l[size - 7:] = [1, 2, 3, 4, 5] + size -= 2 + self.assertEquals(len(l), size) + self.assertEquals(l[-7:], [None, None, 1, 2, 3, 4, 5]) + + l[:7] = [1, 2, 3, 4, 5] + size -= 2 + self.assertEquals(len(l), size) + self.assertEquals(l[:7], [1, 2, 3, 4, 5, None, None]) + + del l[size - 1] + size -= 1 + self.assertEquals(len(l), size) + self.assertEquals(l[-1], 4) + + del l[-2:] + size -= 2 + self.assertEquals(len(l), size) + self.assertEquals(l[-1], 2) + + del l[0] + size -= 1 + self.assertEquals(len(l), size) + self.assertEquals(l[0], 2) + + del l[:2] + size -= 2 + self.assertEquals(len(l), size) + self.assertEquals(l[0], 4) + + # Like test_concat, split in two. + def basic_test_repeat(self, size): + l = [] * size + self.failIf(l) + l = [''] * size + self.assertEquals(len(l), size) + l = l * 2 + self.assertEquals(len(l), size * 2) + + @bigmemtest(minsize=_2G // 2 + 2, memuse=24) + def test_repeat_small(self, size): + return self.basic_test_repeat(size) + + @bigmemtest(minsize=_2G + 2, memuse=24) + def test_repeat_large(self, size): + return self.basic_test_repeat(size) + + # Test repr-result of >2G + def basic_test_repr(self, size): + l = [0] * size + s = repr(l) + # The repr of a list of 0's is exactly three times the list length. + self.assertEquals(len(s), size * 3) + self.assertEquals(s[:5], '[0, 0') + self.assertEquals(s[-5:], '0, 0]') + self.assertEquals(s.count('0'), size) + + @bigmemtest(minsize=_2G // 3 + 2, memuse=8 + 3) + def test_repr_small(self, size): + return self.basic_test_repr(size) + + @bigmemtest(minsize=_2G + 2, memuse=8 + 3) + def test_repr_large(self, size): + return self.basic_test_repr(size) + + + @bigmemtest(minsize=_2G, memuse=8) + def test_append(self, size): + l = [object()] * size + l.append(object()) + self.assertEquals(len(l), size+1) + self.failUnless(l[-3] is l[-2]) + self.failIf(l[-2] is l[-1]) + + @bigmemtest(minsize=_2G // 5 + 2, memuse=8 * 5) + def test_count(self, size): + l = [1, 2, 3, 4, 5] * size + self.assertEquals(l.count(1), size) + self.assertEquals(l.count("1"), 0) + + def basic_test_extend(self, size): + l = [file] * size + l.extend(l) + self.assertEquals(len(l), size * 2) + self.failUnless(l[0] is l[-1]) + self.failUnless(l[size - 1] is l[size + 1]) + + @bigmemtest(minsize=_2G // 2 + 2, memuse=8) + def test_extend_small(self, size): + return self.basic_test_extend(size) + + @bigmemtest(minsize=_2G + 2, memuse=8) + def test_extend_large(self, size): + return self.basic_test_extend(size) + + @bigmemtest(minsize=_2G + 10, memuse=8) + def test_index(self, size): + l = [1L, 2L, 3L, 4L, 5L] * (size // 5) + self.assertEquals(l.index(1), 0) + self.assertEquals(l.index(5, size - 5), size - 1) + self.assertEquals(l.index(5, size - 5, size), size - 1) + self.assertRaises(ValueError, l.index, 1, size - 4, size) + self.assertRaises(ValueError, l.index, 6L) + + @bigmemtest(minsize=_2G + 10, memuse=8) + def test_insert(self, size): + l = [1.0] * size + l.insert(size - 1, "A") + size += 1 + self.assertEquals(len(l), size) + self.assertEquals(l[-3:], [1.0, "A", 1.0]) + + l.insert(size + 1, "B") + size += 1 + self.assertEquals(len(l), size) + self.assertEquals(l[-3:], ["A", 1.0, "B"]) + + l.insert(1, "C") + size += 1 + self.assertEquals(len(l), size) + self.assertEquals(l[:3], [1.0, "C", 1.0]) + self.assertEquals(l[size - 3:], ["A", 1.0, "B"]) + + @bigmemtest(minsize=_2G + 20, memuse=8) + def test_pop(self, size): + l = [u"a", u"b", u"c", u"d", u"e"] * (size // 5) + self.assertEquals(len(l), size) + + item = l.pop() + size -= 1 + self.assertEquals(len(l), size) + self.assertEquals(item, u"e") + + item = l.pop(0) + size -= 1 + self.assertEquals(len(l), size) + self.assertEquals(item, u"a") + + item = l.pop(size - 2) + size -= 1 + self.assertEquals(len(l), size) + self.assertEquals(item, u"c") + + @bigmemtest(minsize=_2G + 10, memuse=8) + def test_remove(self, size): + l = [10] * size + self.assertEquals(len(l), size) + + l.remove(10) + size -= 1 + self.assertEquals(len(l), size) + + l.append(5) + size += 1 + self.assertEquals(len(l), size) + self.assertEquals(l[-2:], [10, 5]) + l.remove(5) + size -= 1 + self.assertEquals(len(l), size) + self.assertEquals(l[-2:], [10, 10]) + + @bigmemtest(minsize=_2G + 10, memuse=8) + def test_reverse(self, size): + l = [1, 2, 3, 4, 5] * (size // 5) + l.reverse() + self.assertEquals(len(l), size) + self.assertEquals(l[-5:], [5, 4, 3, 2, 1]) + self.assertEquals(l[:5], [5, 4, 3, 2, 1]) + + @bigmemtest(minsize=_2G + 10, memuse=8) + def test_sort(self, size): + l = [1, 2, 3, 4, 5] * (size // 5) + l.sort() + self.assertEquals(len(l), size) + self.assertEquals(l.count(1), size // 5) + self.assertEquals(l[:10], [1] * 10) + self.assertEquals(l[-10:], [5] * 10) + +def test_main(): + test_support.run_unittest(StrTest, TupleTest, ListTest) + +if __name__ == '__main__': + if len(sys.argv) > 1: + test_support.set_memlimit(sys.argv[1]) + test_main() Modified: python/trunk/Lib/test/test_support.py ============================================================================== --- python/trunk/Lib/test/test_support.py (original) +++ python/trunk/Lib/test/test_support.py Wed Apr 26 19:11:16 2006 @@ -285,7 +285,7 @@ test-interpreted units.) 'memuse' is the number of 'bytes per size' for the test, or a good estimate of it. 'overhead' specifies fixed overhead, independant of the testsize, and defaults to 5Mb. - + The decorator tries to guess a good value for 'size' and passes it to the decorated test function. If minsize * memuse is more than the allowed memory use (as defined by max_memuse), the test is skipped. From python-checkins at python.org Wed Apr 26 19:19:45 2006 From: python-checkins at python.org (martin.v.loewis) Date: Wed, 26 Apr 2006 19:19:45 +0200 (CEST) Subject: [Python-checkins] r45732 - python/trunk/PCbuild/_bsddb.vcproj python/trunk/PCbuild/_elementtree.vcproj python/trunk/PCbuild/_msi.vcproj python/trunk/PCbuild/_socket.vcproj python/trunk/PCbuild/_testcapi.vcproj python/trunk/PCbuild/_tkinter.vcproj python/trunk/PCbuild/bz2.vcproj python/trunk/PCbuild/pythoncore.vcproj python/trunk/PCbuild/unicodedata.vcproj python/trunk/PCbuild/winsound.vcproj Message-ID: <20060426171945.8F7DB1E400D@bag.python.org> Author: martin.v.loewis Date: Wed Apr 26 19:19:44 2006 New Revision: 45732 Modified: python/trunk/PCbuild/_bsddb.vcproj python/trunk/PCbuild/_elementtree.vcproj python/trunk/PCbuild/_msi.vcproj python/trunk/PCbuild/_socket.vcproj python/trunk/PCbuild/_testcapi.vcproj python/trunk/PCbuild/_tkinter.vcproj python/trunk/PCbuild/bz2.vcproj python/trunk/PCbuild/pythoncore.vcproj python/trunk/PCbuild/unicodedata.vcproj python/trunk/PCbuild/winsound.vcproj Log: Use GS- and bufferoverlowU.lib where appropriate, for AMD64. Modified: python/trunk/PCbuild/_bsddb.vcproj ============================================================================== --- python/trunk/PCbuild/_bsddb.vcproj (original) +++ python/trunk/PCbuild/_bsddb.vcproj Wed Apr 26 19:19:44 2006 @@ -192,7 +192,7 @@ ATLMinimizesCRunTimeLibraryUsage="FALSE"> Author: thomas.wouters Date: Wed Apr 26 20:46:01 2006 New Revision: 45733 Modified: python/trunk/Lib/test/test_bigmem.py Log: Add tests for += and *= on strings, and fix the memory-use estimate for the list.extend tests (they were estimating half the actual use.) Modified: python/trunk/Lib/test/test_bigmem.py ============================================================================== --- python/trunk/Lib/test/test_bigmem.py (original) +++ python/trunk/Lib/test/test_bigmem.py Wed Apr 26 20:46:01 2006 @@ -696,6 +696,21 @@ def test_concat_large(self, size): return self.basic_test_concat(size) + def basic_test_inplace_concat(self, size): + l = [sys.stdout] * size + l += l + self.assertEquals(len(l), size * 2) + self.failUnless(l[0] is l[-1]) + self.failUnless(l[size - 1] is l[size + 1]) + + @bigmemtest(minsize=_2G // 2 + 2, memuse=8) + def test_inplace_concat_small(self, size): + return self.basic_test_inplace_concat(size) + + @bigmemtest(minsize=_2G + 2, memuse=8) + def test_inplace_concat_large(self, size): + return self.basic_test_inplace_concat(size) + @bigmemtest(minsize=_2G // 5 + 10, memuse=8*5) def test_contains(self, size): l = [1, 2, 3, 4, 5] * size @@ -781,7 +796,26 @@ def test_repeat_large(self, size): return self.basic_test_repeat(size) - # Test repr-result of >2G + def basic_test_inplace_repeat(self, size): + l = [''] + l *= size + self.assertEquals(len(l), size) + self.failUnless(l[0] is l[-1]) + del l + + l = [''] * size + l *= 2 + self.assertEquals(len(l), size * 2) + self.failUnless(l[size - 1] is l[-1]) + + @bigmemtest(minsize=_2G // 2 + 2, memuse=16) + def test_inplace_repeat_small(self, size): + return self.basic_test_inplace_repeat(size) + + @bigmemtest(minsize=_2G + 2, memuse=16) + def test_inplace_repeat_large(self, size): + return self.basic_test_inplace_repeat(size) + def basic_test_repr(self, size): l = [0] * size s = repr(l) @@ -821,11 +855,11 @@ self.failUnless(l[0] is l[-1]) self.failUnless(l[size - 1] is l[size + 1]) - @bigmemtest(minsize=_2G // 2 + 2, memuse=8) + @bigmemtest(minsize=_2G // 2 + 2, memuse=16) def test_extend_small(self, size): return self.basic_test_extend(size) - @bigmemtest(minsize=_2G + 2, memuse=8) + @bigmemtest(minsize=_2G + 2, memuse=16) def test_extend_large(self, size): return self.basic_test_extend(size) From buildbot at python.org Wed Apr 26 21:09:00 2006 From: buildbot at python.org (buildbot at python.org) Date: Wed, 26 Apr 2006 19:09:00 +0000 Subject: [Python-checkins] buildbot warnings in x86 XP trunk Message-ID: <20060426190900.531501E400B@bag.python.org> The Buildbot has detected a new failure of x86 XP trunk. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520XP%2520trunk/builds/585 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: thomas.wouters Build Had Warnings: warnings failed slave lost sincerely, -The Buildbot From python-checkins at python.org Wed Apr 26 21:14:47 2006 From: python-checkins at python.org (thomas.wouters) Date: Wed, 26 Apr 2006 21:14:47 +0200 (CEST) Subject: [Python-checkins] r45734 - python/trunk/Lib/test/test_bigmem.py Message-ID: <20060426191447.9CD1A1E400B@bag.python.org> Author: thomas.wouters Date: Wed Apr 26 21:14:46 2006 New Revision: 45734 Modified: python/trunk/Lib/test/test_bigmem.py Log: Some more test-size-estimate fixes: test_append and test_insert trigger a list resize, which overallocates. Modified: python/trunk/Lib/test/test_bigmem.py ============================================================================== --- python/trunk/Lib/test/test_bigmem.py (original) +++ python/trunk/Lib/test/test_bigmem.py Wed Apr 26 21:14:46 2006 @@ -833,8 +833,9 @@ def test_repr_large(self, size): return self.basic_test_repr(size) - - @bigmemtest(minsize=_2G, memuse=8) + # list overallocates ~1/8th of the total size (on first expansion) so + # the single list.append call puts memuse at 9 bytes per size. + @bigmemtest(minsize=_2G, memuse=9) def test_append(self, size): l = [object()] * size l.append(object()) @@ -872,7 +873,8 @@ self.assertRaises(ValueError, l.index, 1, size - 4, size) self.assertRaises(ValueError, l.index, 6L) - @bigmemtest(minsize=_2G + 10, memuse=8) + # This tests suffers from overallocation, just like test_append. + @bigmemtest(minsize=_2G + 10, memuse=9) def test_insert(self, size): l = [1.0] * size l.insert(size - 1, "A") @@ -920,6 +922,8 @@ size -= 1 self.assertEquals(len(l), size) + # Because of the earlier l.remove(), this append doesn't trigger + # a resize. l.append(5) size += 1 self.assertEquals(len(l), size) From python-checkins at python.org Wed Apr 26 21:20:27 2006 From: python-checkins at python.org (hyeshik.chang) Date: Wed, 26 Apr 2006 21:20:27 +0200 (CEST) Subject: [Python-checkins] r45735 - in python/trunk: Misc/NEWS Modules/_ctypes/libffi/configure Modules/_ctypes/libffi/configure.ac Modules/_ctypes/libffi/fficonfig.py.in Message-ID: <20060426192027.AE95D1E400B@bag.python.org> Author: hyeshik.chang Date: Wed Apr 26 21:20:26 2006 New Revision: 45735 Modified: python/trunk/Misc/NEWS python/trunk/Modules/_ctypes/libffi/configure python/trunk/Modules/_ctypes/libffi/configure.ac python/trunk/Modules/_ctypes/libffi/fficonfig.py.in Log: Fix build on MIPS for libffi. I haven't tested this yet because I don't have an access on MIPS machines. Will be tested by buildbot. :) Modified: python/trunk/Misc/NEWS ============================================================================== --- python/trunk/Misc/NEWS (original) +++ python/trunk/Misc/NEWS Wed Apr 26 21:20:26 2006 @@ -156,6 +156,8 @@ - Patch #1429775: Link extension modules with the shared libpython. +- Fixed a libffi build problem on MIPS systems. + C API ----- Modified: python/trunk/Modules/_ctypes/libffi/configure ============================================================================== --- python/trunk/Modules/_ctypes/libffi/configure (original) +++ python/trunk/Modules/_ctypes/libffi/configure Wed Apr 26 21:20:26 2006 @@ -310,7 +310,7 @@ # include #endif" -ac_subst_vars='SHELL PATH_SEPARATOR PACKAGE_NAME PACKAGE_TARNAME PACKAGE_VERSION PACKAGE_STRING PACKAGE_BUGREPORT exec_prefix prefix program_transform_name bindir sbindir libexecdir datadir sysconfdir sharedstatedir localstatedir libdir includedir oldincludedir infodir mandir build_alias host_alias target_alias DEFS ECHO_C ECHO_N ECHO_T LIBS build build_cpu build_vendor build_os host host_cpu host_vendor host_os target target_cpu target_vendor target_os CC ac_ct_CC EXEEXT OBJEXT CFLAGS CPP CPPFLAGS EGREP ALLOCA HAVE_LONG_DOUBLE TARGET TARGETDIR LIBOBJS LTLIBOBJS' +ac_subst_vars='SHELL PATH_SEPARATOR PACKAGE_NAME PACKAGE_TARNAME PACKAGE_VERSION PACKAGE_STRING PACKAGE_BUGREPORT exec_prefix prefix program_transform_name bindir sbindir libexecdir datadir sysconfdir sharedstatedir localstatedir libdir includedir oldincludedir infodir mandir build_alias host_alias target_alias DEFS ECHO_C ECHO_N ECHO_T LIBS build build_cpu build_vendor build_os host host_cpu host_vendor host_os target target_cpu target_vendor target_os CC ac_ct_CC EXEEXT OBJEXT CFLAGS CPP CPPFLAGS EGREP ALLOCA HAVE_LONG_DOUBLE TARGET TARGETDIR MKTARGET LIBOBJS LTLIBOBJS' ac_subst_files='' # Initialize some variables set by options. @@ -3534,6 +3534,8 @@ { (exit 1); exit 1; }; } fi +MKTARGET=$TARGET + case x$TARGET in xMIPS*) TARGET=MIPS ;; *) ;; @@ -5457,6 +5459,7 @@ + cat >>confdefs.h <<\_ACEOF #define FFI_NO_RAW_API 1 _ACEOF @@ -6129,6 +6132,7 @@ s, at HAVE_LONG_DOUBLE@,$HAVE_LONG_DOUBLE,;t t s, at TARGET@,$TARGET,;t t s, at TARGETDIR@,$TARGETDIR,;t t +s, at MKTARGET@,$MKTARGET,;t t s, at LIBOBJS@,$LIBOBJS,;t t s, at LTLIBOBJS@,$LTLIBOBJS,;t t CEOF Modified: python/trunk/Modules/_ctypes/libffi/configure.ac ============================================================================== --- python/trunk/Modules/_ctypes/libffi/configure.ac (original) +++ python/trunk/Modules/_ctypes/libffi/configure.ac Wed Apr 26 21:20:26 2006 @@ -70,6 +70,12 @@ AC_MSG_ERROR(["libffi has not been ported to $host."]) fi +dnl libffi changes TARGET for MIPS to define a such macro in the header +dnl while MIPS_IRIX or MIPS_LINUX is separatedly used to decide which +dnl files will be compiled. So, we need to keep the original decision +dnl of TARGET to use in fficonfig.py.in. +MKTARGET=$TARGET + case x$TARGET in xMIPS*) TARGET=MIPS ;; *) ;; @@ -183,6 +189,7 @@ AC_SUBST(TARGET) AC_SUBST(TARGETDIR) +AC_SUBST(MKTARGET) AC_SUBST(SHELL) Modified: python/trunk/Modules/_ctypes/libffi/fficonfig.py.in ============================================================================== --- python/trunk/Modules/_ctypes/libffi/fficonfig.py.in (original) +++ python/trunk/Modules/_ctypes/libffi/fficonfig.py.in Wed Apr 26 21:20:26 2006 @@ -27,7 +27,7 @@ } ffi_srcdir = '@srcdir@' -ffi_sources += ffi_platforms['@TARGET@'] +ffi_sources += ffi_platforms['@MKTARGET@'] ffi_sources = [os.path.join('@srcdir@', f) for f in ffi_sources] ffi_cflags = '@CFLAGS@' From buildbot at python.org Wed Apr 26 21:32:20 2006 From: buildbot at python.org (buildbot at python.org) Date: Wed, 26 Apr 2006 19:32:20 +0000 Subject: [Python-checkins] buildbot warnings in hppa Ubuntu dapper trunk Message-ID: <20060426193221.10AB21E4013@bag.python.org> The Buildbot has detected a new failure of hppa Ubuntu dapper trunk. Full details are available at: http://www.python.org/dev/buildbot/all/hppa%2520Ubuntu%2520dapper%2520trunk/builds/303 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: martin.v.loewis Build Had Warnings: warnings test sincerely, -The Buildbot From python-checkins at python.org Wed Apr 26 22:33:26 2006 From: python-checkins at python.org (david.goodger) Date: Wed, 26 Apr 2006 22:33:26 +0200 (CEST) Subject: [Python-checkins] r45736 - peps/trunk/pep-0000.txt peps/trunk/pep-3101.txt peps/trunk/pep-3102.txt Message-ID: <20060426203326.C3E371E4005@bag.python.org> Author: david.goodger Date: Wed Apr 26 22:33:25 2006 New Revision: 45736 Added: peps/trunk/pep-3101.txt (contents, props changed) peps/trunk/pep-3102.txt (contents, props changed) Modified: peps/trunk/pep-0000.txt Log: added two PEPs by Talin: 3101, Advanced String Formatting; and 3102, Keyword-Only Arguments Modified: peps/trunk/pep-0000.txt ============================================================================== --- peps/trunk/pep-0000.txt (original) +++ peps/trunk/pep-0000.txt Wed Apr 26 22:33:25 2006 @@ -104,6 +104,8 @@ S 358 The "bytes" Object Schemenauer S 359 The "make" Statement Bethard S 754 IEEE 754 Floating Point Special Values Warnes + S 3101 Advanced String Formatting Talin + S 3102 Keyword-Only Arguments Talin Finished PEPs (done, implemented in Subversion) @@ -425,7 +427,8 @@ P 3002 Procedure for Backwards-Incompatible Changes Bethard I 3099 Things that will Not Change in Python 3000 Brandl I 3100 Python 3.0 Plans Kuchling, Cannon - + S 3101 Advanced String Formatting Talin + S 3102 Keyword-Only Arguments Talin Key @@ -522,6 +525,7 @@ Smith, Kevin D. Kevin.Smith at theMorgue.org Stein, Greg gstein at lyra.org Suzi, Roman rnd at onego.ru + Talin talin at acm.org Taschuk, Steven staschuk at telusplanet.net Tirosh, Oren oren at hishome.net Warnes, Gregory R. warnes at users.sourceforge.net Added: peps/trunk/pep-3101.txt ============================================================================== --- (empty file) +++ peps/trunk/pep-3101.txt Wed Apr 26 22:33:25 2006 @@ -0,0 +1,346 @@ +PEP: 3101 +Title: Advanced String Formatting +Version: $Revision$ +Last-Modified: $Date$ +Author: Talin +Status: Draft +Type: Standards +Content-Type: text/plain +Created: 16-Apr-2006 +Python-Version: 3.0 +Post-History: + + +Abstract + + This PEP proposes a new system for built-in string formatting + operations, intended as a replacement for the existing '%' string + formatting operator. + + +Rationale + + Python currently provides two methods of string interpolation: + + - The '%' operator for strings. + + - The string.Template module. + + The scope of this PEP will be restricted to proposals for built-in + string formatting operations (in other words, methods of the + built-in string type). This does not obviate the need for more + sophisticated string-manipulation modules in the standard library + such as string.Template. In any case, string.Template will not be + discussed here, except to say that the this proposal will most + likely have some overlapping functionality with that module. + + The '%' operator is primarily limited by the fact that it is a + binary operator, and therefore can take at most two arguments. + One of those arguments is already dedicated to the format string, + leaving all other variables to be squeezed into the remaining + argument. The current practice is to use either a dictionary or a + tuple as the second argument, but as many people have commented + [1], this lacks flexibility. The "all or nothing" approach + (meaning that one must choose between only positional arguments, + or only named arguments) is felt to be overly constraining. + + +Specification + + The specification will consist of 4 parts: + + - Specification of a set of methods to be added to the built-in + string class. + + - Specification of a new syntax for format strings. + + - Specification of a new set of class methods to control the + formatting and conversion of objects. + + - Specification of an API for user-defined formatting classes. + + +String Methods + + The build-in string class will gain two new methods. The first + method is 'format', and takes an arbitrary number of positional + and keyword arguments: + + "The story of {0}, {1}, and {c}".format(a, b, c=d) + + Within a format string, each positional argument is identified + with a number, starting from zero, so in the above example, 'a' is + argument 0 and 'b' is argument 1. Each keyword argument is + identified by its keyword name, so in the above example, 'c' is + used to refer to the third argument. + + The result of the format call is an object of the same type + (string or unicode) as the format string. + + +Format Strings + + Brace characters ('curly braces') are used to indicate a + replacement field within the string: + + "My name is {0}".format('Fred') + + The result of this is the string: + + "My name is Fred" + + Braces can be escaped using a backslash: + + "My name is {0} :-\{\}".format('Fred') + + Which would produce: + + "My name is Fred :-{}" + + The element within the braces is called a 'field'. Fields consist + of a name, which can either be simple or compound, and an optional + 'conversion specifier'. + + Simple names are either names or numbers. If numbers, they must + be valid decimal numbers; if names, they must be valid Python + identifiers. A number is used to identify a positional argument, + while a name is used to identify a keyword argument. + + Compound names are a sequence of simple names seperated by + periods: + + "My name is {0.name} :-\{\}".format(dict(name='Fred')) + + Compound names can be used to access specific dictionary entries, + array elements, or object attributes. In the above example, the + '{0.name}' field refers to the dictionary entry 'name' within + positional argument 0. + + Each field can also specify an optional set of 'conversion + specifiers'. Conversion specifiers follow the field name, with a + colon (':') character separating the two: + + "My name is {0:8}".format('Fred') + + The meaning and syntax of the conversion specifiers depends on the + type of object that is being formatted, however many of the + built-in types will recognize a standard set of conversion + specifiers. + + The conversion specifier consists of a sequence of zero or more + characters, each of which can consist of any printable character + except for a non-escaped '}'. The format() method does not + attempt to intepret the conversion specifiers in any way; it + merely passes all of the characters between the first colon ':' + and the matching right brace ('}') to the various underlying + formatters (described later.) + + When using the 'fformat' variant, it is possible to omit the field + name entirely, and simply include the conversion specifiers: + + "My name is {:pad(23)}" + + This syntax is used to send special instructions to the custom + formatter object (such as instructing it to insert padding + characters up to a given column.) The interpretation of this + 'empty' field is entirely up to the custom formatter; no + standard interpretation will be defined in this PEP. + + If a custom formatter is not being used, then it is an error to + omit the field name. + + +Standard Conversion Specifiers + + For most built-in types, the conversion specifiers will be the + same or similar to the existing conversion specifiers used with + the '%' operator. Thus, instead of '%02.2x", you will say + '{0:2.2x}'. + + There are a few differences however: + + - The trailing letter is optional - you don't need to say '2.2d', + you can instead just say '2.2'. If the letter is omitted, the + value will be converted into its 'natural' form (that is, the + form that it take if str() or unicode() were called on it) + subject to the field length and precision specifiers (if + supplied). + + - Variable field width specifiers use a nested version of the {} + syntax, allowing the width specifier to be either a positional + or keyword argument: + + "{0:{1}.{2}d}".format(a, b, c) + + (Note: It might be easier to parse if these used a different + type of delimiter, such as parens - avoiding the need to create + a regex that handles the recursive case.) + + - The support for length modifiers (which are ignored by Python + anyway) is dropped. + + For non-built-in types, the conversion specifiers will be specific + to that type. An example is the 'datetime' class, whose + conversion specifiers are identical to the arguments to the + strftime() function: + + "Today is: {0:%x}".format(datetime.now()) + + +Controlling Formatting + + A class that wishes to implement a custom interpretation of its + conversion specifiers can implement a __format__ method: + + class AST: + def __format__(self, specifiers): + ... + + The 'specifiers' argument will be either a string object or a + unicode object, depending on the type of the original format + string. The __format__ method should test the type of the + specifiers parameter to determine whether to return a string or + unicode object. It is the responsibility of the __format__ method + to return an object of the proper type. + + string.format() will format each field using the following steps: + + 1) See if the value to be formatted has a __format__ method. If + it does, then call it. + + 2) Otherwise, check the internal formatter within string.format + that contains knowledge of certain builtin types. + + 3) Otherwise, call str() or unicode() as appropriate. + + +User-Defined Formatting Classes + + The code that interprets format strings can be called explicitly + from user code. This allows the creation of custom formatter + classes that can override the normal formatting rules. + + The string and unicode classes will have a class method called + 'cformat' that does all the actual work of formatting; The + format() method is just a wrapper that calls cformat. + + The parameters to the cformat function are: + + -- The format string (or unicode; the same function handles + both.) + -- A field format hook (see below) + -- A tuple containing the positional arguments + -- A dict containing the keyword arguments + + The cformat function will parse all of the fields in the format + string, and return a new string (or unicode) with all of the + fields replaced with their formatted values. + + For each field, the cformat function will attempt to call the + field format hook with the following arguments: + + field_hook(value, conversion, buffer) + + The 'value' field corresponds to the value being formatted, which + was retrieved from the arguments using the field name. (The + field_hook has no control over the selection of values, only + how they are formatted.) + + The 'conversion' argument is the conversion spec part of the + field, which will be either a string or unicode object, depending + on the type of the original format string. + + The 'buffer' argument is a Python array object, either a byte + array or unicode character array. The buffer object will contain + the partially constructed string; the field hook is free to modify + the contents of this buffer if needed. + + The field_hook will be called once per field. The field_hook may + take one of two actions: + + 1) Return False, indicating that the field_hook will not + process this field and the default formatting should be + used. This decision should be based on the type of the + value object, and the contents of the conversion string. + + 2) Append the formatted field to the buffer, and return True. + + +Alternate Syntax + + Naturally, one of the most contentious issues is the syntax of the + format strings, and in particular the markup conventions used to + indicate fields. + + Rather than attempting to exhaustively list all of the various + proposals, I will cover the ones that are most widely used + already. + + - Shell variable syntax: $name and $(name) (or in some variants, + ${name}). This is probably the oldest convention out there, and + is used by Perl and many others. When used without the braces, + the length of the variable is determined by lexically scanning + until an invalid character is found. + + This scheme is generally used in cases where interpolation is + implicit - that is, in environments where any string can contain + interpolation variables, and no special subsitution function + need be invoked. In such cases, it is important to prevent the + interpolation behavior from occuring accidentally, so the '$' + (which is otherwise a relatively uncommonly-used character) is + used to signal when the behavior should occur. + + It is the author's opinion, however, that in cases where the + formatting is explicitly invoked, that less care needs to be + taken to prevent accidental interpolation, in which case a + lighter and less unwieldy syntax can be used. + + - Printf and its cousins ('%'), including variations that add a + field index, so that fields can be interpolated out of order. + + - Other bracket-only variations. Various MUDs (Multi-User + Dungeons) such as MUSH have used brackets (e.g. [name]) to do + string interpolation. The Microsoft .Net libraries uses braces + ({}), and a syntax which is very similar to the one in this + proposal, although the syntax for conversion specifiers is quite + different. [2] + + - Backquoting. This method has the benefit of minimal syntactical + clutter, however it lacks many of the benefits of a function + call syntax (such as complex expression arguments, custom + formatters, etc.). + + - Other variations include Ruby's #{}, PHP's {$name}, and so + on. + + +Backwards Compatibility + + Backwards compatibility can be maintained by leaving the existing + mechanisms in place. The new system does not collide with any of + the method names of the existing string formatting techniques, so + both systems can co-exist until it comes time to deprecate the + older system. + + +References + + [1] [Python-3000] String formating operations in python 3k + http://mail.python.org/pipermail/python-3000/2006-April/000285.html + + [2] Composite Formatting - [.Net Framework Developer's Guide] + http://msdn.microsoft.com/library/en-us/cpguide/html/cpconcompositeformatting.asp?frame=true + + +Copyright + + This document has been placed in the public domain. + + +Local Variables: +mode: indented-text +indent-tabs-mode: nil +sentence-end-double-space: t +fill-column: 70 +coding: utf-8 +End: Added: peps/trunk/pep-3102.txt ============================================================================== --- (empty file) +++ peps/trunk/pep-3102.txt Wed Apr 26 22:33:25 2006 @@ -0,0 +1,184 @@ +PEP: 3102 +Title: Keyword-Only Arguments +Version: $Revision$ +Last-Modified: $Date$ +Author: Talin +Status: Draft +Type: Standards +Content-Type: text/plain +Created: 22-Apr-2006 +Python-Version: 3.0 +Post-History: + + +Abstract + + This PEP proposes a change to the way that function arguments are + assigned to named parameter slots. In particular, it enables the + declaration of "keyword-only" arguments: arguments that can only + be supplied by keyword and which will never be automatically + filled in by a positional argument. + + +Rationale + + The current Python function-calling paradigm allows arguments to + be specified either by position or by keyword. An argument can be + filled in either explicitly by name, or implicitly by position. + + There are often cases where it is desirable for a function to take + a variable number of arguments. The Python language supports this + using the 'varargs' syntax ('*name'), which specifies that any + 'left over' arguments be passed into the varargs parameter as a + tuple. + + One limitation on this is that currently, all of the regular + argument slots must be filled before the vararg slot can be. + + This is not always desirable. One can easily envision a function + which takes a variable number of arguments, but also takes one + or more 'options' in the form of keyword arguments. Currently, + the only way to do this is to define both a varargs argument, + and a 'keywords' argument (**kwargs), and then manually extract + the desired keywords from the dictionary. + + +Specification + + Syntactically, the proposed changes are fairly simple. The first + change is to allow regular arguments to appear after a varargs + argument: + + def sortwords(*wordlist, case_sensitive=False): + ... + + This function accepts any number of positional arguments, and it + also accepts a keyword option called 'case_sensitive'. This + option will never be filled in by a positional argument, but + must be explicitly specified by name. + + Keyword-only arguments are not required to have a default value. + Since Python requires that all arguments be bound to a value, + and since the only way to bind a value to a keyword-only argument + is via keyword, such arguments are therefore 'required keyword' + arguments. Such arguments must be supplied by the caller, and + they must be supplied via keyword. + + The second syntactical change is to allow the argument name to + be omitted for a varargs argument: + + def compare(a, b, *, key=None): + ... + + The reasoning behind this change is as follows. Imagine for a + moment a function which takes several positional arguments, as + well as a keyword argument: + + def compare(a, b, key=None): + ... + + Now, suppose you wanted to have 'key' be a keyword-only argument. + Under the above syntax, you could accomplish this by adding a + varargs argument immediately before the keyword argument: + + def compare(a, b, *ignore, key=None): + ... + + Unfortunately, the 'ignore' argument will also suck up any + erroneous positional arguments that may have been supplied by the + caller. Given that we'd prefer any unwanted arguments to raise an + error, we could do this: + + def compare(a, b, *ignore, key=None): + if ignore: # If ignore is not empty + raise TypeError + + As a convenient shortcut, we can simply omit the 'ignore' name, + meaning 'don't allow any positional arguments beyond this point'. + + +Function Calling Behavior + + The previous section describes the difference between the old + behavior and the new. However, it is also useful to have a + description of the new behavior that stands by itself, without + reference to the previous model. So this next section will + attempt to provide such a description. + + When a function is called, the input arguments are assigned to + formal parameters as follows: + + - For each formal parameter, there is a slot which will be used + to contain the value of the argument assigned to that + parameter. + + - Slots which have had values assigned to them are marked as + 'filled'. Slots which have no value assigned to them yet are + considered 'empty'. + + - Initially, all slots are marked as empty. + + - Positional arguments are assigned first, followed by keyword + arguments. + + - For each positional argument: + + o Attempt to bind the argument to the first unfilled + parameter slot. If the slot is not a vararg slot, then + mark the slot as 'filled'. + + o If the next unfilled slot is a vararg slot, and it does + not have a name, then it is an error. + + o Otherwise, if the next unfilled slot is a vararg slot then + all remaining non-keyword arguments are placed into the + vararg slot. + + - For each keyword argument: + + o If there is a parameter with the same name as the keyword, + then the argument value is assigned to that parameter slot. + However, if the parameter slot is already filled, then that + is an error. + + o Otherwise, if there is a 'keyword dictionary' argument, + the argument is added to the dictionary using the keyword + name as the dictionary key, unless there is already an + entry with that key, in which case it is an error. + + o Otherwise, if there is no keyword dictionary, and no + matching named parameter, then it is an error. + + - Finally: + + o If the vararg slot is not yet filled, assign an empty tuple + as its value. + + o For each remaining empty slot: if there is a default value + for that slot, then fill the slot with the default value. + If there is no default value, then it is an error. + + In accordance with the current Python implementation, any errors + encountered will be signaled by raising TypeError. (If you want + something different, that's a subject for a different PEP.) + + +Backwards Compatibility + + The function calling behavior specified in this PEP is a superset + of the existing behavior - that is, it is expected that any + existing programs will continue to work. + + +Copyright + + This document has been placed in the public domain. + + +Local Variables: +mode: indented-text +indent-tabs-mode: nil +sentence-end-double-space: t +fill-column: 70 +coding: utf-8 +End: From python-checkins at python.org Thu Apr 27 01:40:33 2006 From: python-checkins at python.org (fred.drake) Date: Thu, 27 Apr 2006 01:40:33 +0200 (CEST) Subject: [Python-checkins] r45737 - python/trunk/Doc/inst/inst.tex Message-ID: <20060426234033.3EC951E4010@bag.python.org> Author: fred.drake Date: Thu Apr 27 01:40:32 2006 New Revision: 45737 Modified: python/trunk/Doc/inst/inst.tex Log: one more place to use the current Python version Modified: python/trunk/Doc/inst/inst.tex ============================================================================== --- python/trunk/Doc/inst/inst.tex (original) +++ python/trunk/Doc/inst/inst.tex Thu Apr 27 01:40:32 2006 @@ -726,8 +726,8 @@ \envvar{PYTHONHOME} sets an alternate value for the prefix of the Python installation. For example, if \envvar{PYTHONHOME} is set to \samp{/www/python}, the search path will be set to \code{['', -'/www/python/lib/python2.2/', '/www/python/lib/python2.3/plat-linux2', -...]}. +'/www/python/lib/python\shortversion/', +'/www/python/lib/python\shortversion/plat-linux2', ...]}. The \envvar{PYTHONPATH} variable can be set to a list of paths that will be added to the beginning of \code{sys.path}. For example, if From python-checkins at python.org Thu Apr 27 02:02:26 2006 From: python-checkins at python.org (fred.drake) Date: Thu, 27 Apr 2006 02:02:26 +0200 (CEST) Subject: [Python-checkins] r45738 - python/trunk/Doc/inst/inst.tex Message-ID: <20060427000226.525361E4003@bag.python.org> Author: fred.drake Date: Thu Apr 27 02:02:24 2006 New Revision: 45738 Modified: python/trunk/Doc/inst/inst.tex Log: - update version numbers in file names again, until we have a better way - elaborate instructions for Cygwin support (closes SF #839709) Modified: python/trunk/Doc/inst/inst.tex ============================================================================== --- python/trunk/Doc/inst/inst.tex (original) +++ python/trunk/Doc/inst/inst.tex Thu Apr 27 02:02:24 2006 @@ -981,15 +981,15 @@ from the Python or ActiveState Web site. (Python is built with Microsoft Visual \Cpp, which uses COFF as the object file format.) For this reason you have to convert Python's library -\file{python24.lib} into the Borland format. You can do this as +\file{python25.lib} into the Borland format. You can do this as follows: \begin{verbatim} -coff2omf python24.lib python24_bcpp.lib +coff2omf python25.lib python25_bcpp.lib \end{verbatim} The \file{coff2omf} program comes with the Borland compiler. The file -\file{python24.lib} is in the \file{Libs} directory of your Python +\file{python25.lib} is in the \file{Libs} directory of your Python installation. If your extension uses other libraries (zlib,...) you have to convert them too. @@ -1053,17 +1053,23 @@ PExports 0.42h there.) \begin{verbatim} -pexports python24.dll >python24.def +pexports python25.dll >python25.def \end{verbatim} +The location of an installed \file{python25.dll} will depend on the +installation options and the version and language of Windows. In a +``just for me'' installation, it will appear in the root of the +installation directory. In a shared installation, it will be located +in the system directory. + Then you can create from these information an import library for gcc. \begin{verbatim} -dlltool --dllname python24.dll --def python24.def --output-lib libpython24.a +/cygwin/bin/dlltool --dllname python25.dll --def python25.def --output-lib libpython25.a \end{verbatim} The resulting library has to be placed in the same directory as -\file{python24.lib}. (Should be the \file{libs} directory under your +\file{python25.lib}. (Should be the \file{libs} directory under your Python installation directory.) If your extension uses other libraries (zlib,...) you might From python-checkins at python.org Thu Apr 27 02:20:15 2006 From: python-checkins at python.org (fred.drake) Date: Thu, 27 Apr 2006 02:20:15 +0200 (CEST) Subject: [Python-checkins] r45739 - python/trunk/Doc/lib/libstdtypes.tex Message-ID: <20060427002015.CD32B1E400B@bag.python.org> Author: fred.drake Date: Thu Apr 27 02:20:14 2006 New Revision: 45739 Modified: python/trunk/Doc/lib/libstdtypes.tex Log: add missing word Modified: python/trunk/Doc/lib/libstdtypes.tex ============================================================================== --- python/trunk/Doc/lib/libstdtypes.tex (original) +++ python/trunk/Doc/lib/libstdtypes.tex Thu Apr 27 02:20:14 2006 @@ -19,7 +19,7 @@ \function{str()} function). The latter function is implicitly used when an object is written by the \keyword{print}\stindex{print} statement. -(Information on \ulink{\keyword{print} statement}{../ref/print.html} +(Information on the \ulink{\keyword{print} statement}{../ref/print.html} and other language statements can be found in the \citetitle[../ref/ref.html]{Python Reference Manual} and the \citetitle[../tut/tut.html]{Python Tutorial}.) From buildbot at python.org Thu Apr 27 03:49:19 2006 From: buildbot at python.org (buildbot at python.org) Date: Thu, 27 Apr 2006 01:49:19 +0000 Subject: [Python-checkins] buildbot warnings in sparc Ubuntu dapper trunk Message-ID: <20060427014919.724051E4003@bag.python.org> The Buildbot has detected a new failure of sparc Ubuntu dapper trunk. Full details are available at: http://www.python.org/dev/buildbot/all/sparc%2520Ubuntu%2520dapper%2520trunk/builds/169 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: fred.drake Build Had Warnings: warnings test sincerely, -The Buildbot From buildbot at python.org Thu Apr 27 03:58:22 2006 From: buildbot at python.org (buildbot at python.org) Date: Thu, 27 Apr 2006 01:58:22 +0000 Subject: [Python-checkins] buildbot warnings in alpha Debian trunk Message-ID: <20060427015823.1C7BB1E4003@bag.python.org> The Buildbot has detected a new failure of alpha Debian trunk. Full details are available at: http://www.python.org/dev/buildbot/all/alpha%2520Debian%2520trunk/builds/29 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: fred.drake Build Had Warnings: warnings test sincerely, -The Buildbot From python-checkins at python.org Thu Apr 27 04:11:26 2006 From: python-checkins at python.org (anthony.baxter) Date: Thu, 27 Apr 2006 04:11:26 +0200 (CEST) Subject: [Python-checkins] r45740 - in python/trunk: Doc/commontex/boilerplate.tex Include/patchlevel.h Lib/idlelib/NEWS.txt Lib/idlelib/idlever.py Misc/NEWS Message-ID: <20060427021126.6C2821E4003@bag.python.org> Author: anthony.baxter Date: Thu Apr 27 04:11:24 2006 New Revision: 45740 Modified: python/trunk/Doc/commontex/boilerplate.tex python/trunk/Include/patchlevel.h python/trunk/Lib/idlelib/NEWS.txt python/trunk/Lib/idlelib/idlever.py python/trunk/Misc/NEWS Log: 2.5a2 Modified: python/trunk/Doc/commontex/boilerplate.tex ============================================================================== --- python/trunk/Doc/commontex/boilerplate.tex (original) +++ python/trunk/Doc/commontex/boilerplate.tex Thu Apr 27 04:11:24 2006 @@ -5,5 +5,5 @@ Email: \email{docs at python.org} } -\date{5th April 2006} % XXX update before final release! +\date{27th April 2006} % XXX update before final release! \input{patchlevel} % include Python version information Modified: python/trunk/Include/patchlevel.h ============================================================================== --- python/trunk/Include/patchlevel.h (original) +++ python/trunk/Include/patchlevel.h Thu Apr 27 04:11:24 2006 @@ -23,10 +23,10 @@ #define PY_MINOR_VERSION 5 #define PY_MICRO_VERSION 0 #define PY_RELEASE_LEVEL PY_RELEASE_LEVEL_ALPHA -#define PY_RELEASE_SERIAL 1 +#define PY_RELEASE_SERIAL 2 /* Version as a string */ -#define PY_VERSION "2.5a1" +#define PY_VERSION "2.5a2" /* Subversion Revision number of this file (not of the repository) */ #define PY_PATCHLEVEL_REVISION "$Revision$" Modified: python/trunk/Lib/idlelib/NEWS.txt ============================================================================== --- python/trunk/Lib/idlelib/NEWS.txt (original) +++ python/trunk/Lib/idlelib/NEWS.txt Thu Apr 27 04:11:24 2006 @@ -1,3 +1,8 @@ +What's New in IDLE 1.2a2? +========================= + +*Release date: 27-APR-2006* + What's New in IDLE 1.2a1? ========================= Modified: python/trunk/Lib/idlelib/idlever.py ============================================================================== --- python/trunk/Lib/idlelib/idlever.py (original) +++ python/trunk/Lib/idlelib/idlever.py Thu Apr 27 04:11:24 2006 @@ -1 +1 @@ -IDLE_VERSION = "1.2a1" +IDLE_VERSION = "1.2a2" Modified: python/trunk/Misc/NEWS ============================================================================== --- python/trunk/Misc/NEWS (original) +++ python/trunk/Misc/NEWS Thu Apr 27 04:11:24 2006 @@ -7,7 +7,7 @@ What's New in Python 2.5 alpha 2? ================================= -*Release date: XX-XXX-2006* +*Release date: 27-APR-2006* Core and builtins ----------------- From python-checkins at python.org Thu Apr 27 04:13:14 2006 From: python-checkins at python.org (anthony.baxter) Date: Thu, 27 Apr 2006 04:13:14 +0200 (CEST) Subject: [Python-checkins] r45741 - python/trunk/Misc/RPM/python-2.5.spec Message-ID: <20060427021314.997E71E4003@bag.python.org> Author: anthony.baxter Date: Thu Apr 27 04:13:13 2006 New Revision: 45741 Modified: python/trunk/Misc/RPM/python-2.5.spec Log: 2.5a2 Modified: python/trunk/Misc/RPM/python-2.5.spec ============================================================================== --- python/trunk/Misc/RPM/python-2.5.spec (original) +++ python/trunk/Misc/RPM/python-2.5.spec Thu Apr 27 04:13:13 2006 @@ -33,7 +33,7 @@ ################################# %define name python -%define version 2.5a1 +%define version 2.5a2 %define libvers 2.5 %define release 1pydotorg %define __prefix /usr From python-checkins at python.org Thu Apr 27 04:13:47 2006 From: python-checkins at python.org (anthony.baxter) Date: Thu, 27 Apr 2006 04:13:47 +0200 (CEST) Subject: [Python-checkins] r45742 - python/tags/r25a2 Message-ID: <20060427021347.A47911E4003@bag.python.org> Author: anthony.baxter Date: Thu Apr 27 04:13:47 2006 New Revision: 45742 Added: python/tags/r25a2/ - copied from r45741, python/trunk/ Log: Tagging for release of Python 2.5a2 From buildbot at python.org Thu Apr 27 04:53:20 2006 From: buildbot at python.org (buildbot at python.org) Date: Thu, 27 Apr 2006 02:53:20 +0000 Subject: [Python-checkins] buildbot warnings in hppa Ubuntu dapper trunk Message-ID: <20060427025320.79C7B1E4003@bag.python.org> The Buildbot has detected a new failure of hppa Ubuntu dapper trunk. Full details are available at: http://www.python.org/dev/buildbot/all/hppa%2520Ubuntu%2520dapper%2520trunk/builds/307 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: anthony.baxter Build Had Warnings: warnings test sincerely, -The Buildbot From python-checkins at python.org Thu Apr 27 10:43:41 2006 From: python-checkins at python.org (neal.norwitz) Date: Thu, 27 Apr 2006 10:43:41 +0200 (CEST) Subject: [Python-checkins] r45743 - peps/trunk/pep-0000.txt Message-ID: <20060427084341.388221E4019@bag.python.org> Author: neal.norwitz Date: Thu Apr 27 10:43:40 2006 New Revision: 45743 Modified: peps/trunk/pep-0000.txt Log: Add withdrawn to the key/legend and use W to denote those PEPs already withdrawn. Modified: peps/trunk/pep-0000.txt ============================================================================== --- peps/trunk/pep-0000.txt (original) +++ peps/trunk/pep-0000.txt Thu Apr 27 10:43:40 2006 @@ -179,7 +179,7 @@ Deferred, Abandoned, Withdrawn, and Rejected PEPs SR 204 Range Literals Wouters - IR 206 Python Advanced Library Kuchling + IW 206 Python Advanced Library Kuchling SD 211 Adding A New Outer Product Operator Wilson SD 212 Loop Counter Iteration Schneider-Kamp SD 213 Attribute Access Handlers Prescod @@ -204,11 +204,11 @@ SD 269 Pgen Module for Python Riehl SR 270 uniq method for list objects Petrone SR 271 Prefixing sys.path by command line option Giacometti - SR 274 Dict Comprehensions Warsaw + SW 274 Dict Comprehensions Warsaw SR 276 Simple Iterator for ints Althoff SR 281 Loop Counter Iteration with range and xrange Hetland SR 284 Integer for-loops Eppstein, Ewing - SR 288 Generators Attributes and Exceptions Hettinger + SW 288 Generators Attributes and Exceptions Hettinger SR 294 Type Names in the types Module Tirosh SR 295 Interpretation of multiline string constants Koltsov SR 296 Adding a bytes Object Type Gilbert @@ -221,7 +221,7 @@ SD 316 Programming by Contract for Python Way SR 317 Eliminate Implicit Exception Instantiation Taschuk SR 319 Python Synchronize/Asynchronize Block Pelletier - SR 321 Date/Time Parsing and Formatting Kuchling + SW 321 Date/Time Parsing and Formatting Kuchling SR 325 Resource-Release Support for Generators Pedroni SR 326 A Case for Top and Bottom Values Carlson, Reedy SR 329 Treating Builtins as Constants in the Standard Library Hettinger @@ -229,7 +229,7 @@ SR 332 Byte vectors and String/Unicode Unification Montanaro SR 336 Make None Callable McClelland SR 340 Anonymous Block Statements GvR - SR 346 User Defined ("with") Statements Coghlan + SW 346 User Defined ("with") Statements Coghlan SR 348 Exception Reorganization for Python 3.0 Cannon SD 349 Allow str() to return unicode strings Schemenauer SR 351 The freeze protocol Warsaw @@ -267,7 +267,7 @@ SF 203 Augmented Assignments Wouters SR 204 Range Literals Wouters S 205 Weak References Drake - IR 206 Python Advanced Library Kuchling + IW 206 Python Advanced Library Kuchling SF 207 Rich Comparisons GvR, Ascher SF 208 Reworking the Coercion Model Schemenauer, Lemburg S 209 Adding Multidimensional Arrays Barrett, Oliphant @@ -335,7 +335,7 @@ SR 271 Prefixing sys.path by command line option Giacometti IF 272 API for Block Encryption Algorithms v1.0 Kuchling SF 273 Import Modules from Zip Archives Ahlstrom - SR 274 Dict Comprehensions Warsaw + SW 274 Dict Comprehensions Warsaw S 275 Switching on Multiple Values Lemburg SR 276 Simple Iterator for ints Althoff SF 277 Unicode file name support for Windows NT Hodgson @@ -349,7 +349,7 @@ SF 285 Adding a bool type GvR S 286 Enhanced Argument Tuples von Loewis I 287 reStructuredText Docstring Format Goodger - SR 288 Generators Attributes and Exceptions Hettinger + SW 288 Generators Attributes and Exceptions Hettinger SF 289 Generator Expressions Hettinger I 290 Code Migration and Modernization Hettinger I 291 Backward Compatibility for Standard Library Norwitz @@ -381,7 +381,7 @@ SF 318 Decorators for Functions and Methods Smith, et al SR 319 Python Synchronize/Asynchronize Block Pelletier IF 320 Python 2.4 Release Schedule Warsaw, et al - SR 321 Date/Time Parsing and Formatting Kuchling + SW 321 Date/Time Parsing and Formatting Kuchling SF 322 Reverse Iteration Hettinger S 323 Copyable Iterators Martelli SF 324 subprocess - New POSIX process module Astrand @@ -406,7 +406,7 @@ SF 343 Anonymous Block Redux and Generator Enhancements GvR S 344 Exception Chaining and Embedded Tracebacks Yee S 345 Metadata for Python Software Packages 1.2 Jones - SR 346 User Defined ("with") Statements Coghlan + SW 346 User Defined ("with") Statements Coghlan P 347 Migrating the Python CVS to Subversion von L?wis SR 348 Exception Reorganization for Python 3.0 Cannon SD 349 Allow str() to return unicode strings Schemenauer @@ -438,6 +438,7 @@ A - Accepted proposal R - Rejected proposal + W - Withdrawn proposal D - Deferred proposal F - Final proposal From python-checkins at python.org Thu Apr 27 10:47:56 2006 From: python-checkins at python.org (neal.norwitz) Date: Thu, 27 Apr 2006 10:47:56 +0200 (CEST) Subject: [Python-checkins] r45744 - peps/trunk/pep-0000.txt peps/trunk/pep-0266.txt peps/trunk/pep-0298.txt peps/trunk/pep-0304.txt Message-ID: <20060427084756.641E91E4003@bag.python.org> Author: neal.norwitz Date: Thu Apr 27 10:47:55 2006 New Revision: 45744 Modified: peps/trunk/pep-0000.txt peps/trunk/pep-0266.txt peps/trunk/pep-0298.txt peps/trunk/pep-0304.txt Log: Authors withdrew some PEPs by mail on python-dev (Apr 26, 2006). Modified: peps/trunk/pep-0000.txt ============================================================================== --- peps/trunk/pep-0000.txt (original) +++ peps/trunk/pep-0000.txt Thu Apr 27 10:47:55 2006 @@ -79,7 +79,6 @@ S 243 Module Repository Upload Mechanism Reifschneider S 256 Docstring Processing System Framework Goodger S 258 Docutils Design Specification Goodger - S 266 Optimizing Global Variable/Attribute Access Montanaro S 267 Optimized Access to Module Namespaces Hylton S 268 Extended HTTP functionality and WebDAV Stein S 275 Switching on Multiple Values Lemburg @@ -87,9 +86,7 @@ S 286 Enhanced Argument Tuples von Loewis I 287 reStructuredText Docstring Format Goodger S 297 Support for System Upgrades Lemburg - S 298 The Locked Buffer Interface Heller S 302 New Import Hooks JvR, Moore - S 304 Controlling Generation of Bytecode Files Montanaro S 314 Metadata for Python Software Packages v1.1 Kuchling, Jones S 323 Copyable Iterators Martelli S 331 Locale-Independent Float/String Conversions Reis @@ -152,6 +149,7 @@ SF 289 Generator Expressions Hettinger SF 292 Simpler String Substitutions Warsaw SF 293 Codec Error Handling Callbacks D?rwald + SW 298 The Locked Buffer Interface Heller SF 301 Package Index and Metadata for Distutils Jones SF 305 CSV File API Montanaro, et al SF 307 Extensions to the pickle protocol GvR, Peters @@ -201,6 +199,7 @@ SR 259 Omit printing newline after newline GvR SD 262 Database of Installed Python Packages Kuchling SR 265 Sorting Dictionaries by Value Griffin + SW 266 Optimizing Global Variable/Attribute Access Montanaro SD 269 Pgen Module for Python Riehl SR 270 uniq method for list objects Petrone SR 271 Prefixing sys.path by command line option Giacometti @@ -214,6 +213,7 @@ SR 296 Adding a bytes Object Type Gilbert SR 299 Special __main__() function in modules Epler SR 303 Extend divmod() for Multiple Divisors Bellman + SW 304 Controlling Generation of Bytecode Files Montanaro SR 310 Reliable Acquisition/Release Pairs Hudson, Moore SD 312 Simple Implicit Lambda Suzi, Martelli SR 313 Adding Roman Numeral Literals to Python Meyer @@ -327,7 +327,7 @@ SF 263 Defining Python Source Code Encodings Lemburg SF 264 Future statements in simulated shells Hudson SR 265 Sorting Dictionaries by Value Griffin - S 266 Optimizing Global Variable/Attribute Access Montanaro + SW 266 Optimizing Global Variable/Attribute Access Montanaro S 267 Optimized Access to Module Namespaces Hylton S 268 Extended HTTP functionality and WebDAV Stein S 269 Pgen Module for Python Riehl @@ -359,12 +359,12 @@ SR 295 Interpretation of multiline string constants Koltsov SR 296 Adding a bytes Object Type Gilbert S 297 Support for System Upgrades Lemburg - S 298 The Locked Buffer Interface Heller + SW 298 The Locked Buffer Interface Heller SR 299 Special __main__() function in modules Epler SF 301 Package Index and Metadata for Distutils Jones S 302 New Import Hooks JvR, Moore SR 303 Extend divmod() for Multiple Divisors Bellman - S 304 Controlling Generation of Bytecode Files Montanaro + SW 304 Controlling Generation of Bytecode Files Montanaro SF 305 CSV File API Montanaro, et al I 306 How to Change Python's Grammar Hudson SF 307 Extensions to the pickle protocol GvR, Peters Modified: peps/trunk/pep-0266.txt ============================================================================== --- peps/trunk/pep-0266.txt (original) +++ peps/trunk/pep-0266.txt Thu Apr 27 10:47:55 2006 @@ -3,7 +3,7 @@ Version: $Revision$ Last-Modified: $Date$ Author: skip at pobox.com (Skip Montanaro) -Status: Draft +Status: Withdrawn Type: Standards Track Python-Version: 2.3 Created: 13-Aug-2001 Modified: peps/trunk/pep-0298.txt ============================================================================== --- peps/trunk/pep-0298.txt (original) +++ peps/trunk/pep-0298.txt Thu Apr 27 10:47:55 2006 @@ -3,7 +3,7 @@ Version: $Revision$ Last-Modified: $Date$ Author: Thomas Heller -Status: Draft +Status: Withdrawn Type: Standards Track Created: 26-Jul-2002 Python-Version: 2.3 Modified: peps/trunk/pep-0304.txt ============================================================================== --- peps/trunk/pep-0304.txt (original) +++ peps/trunk/pep-0304.txt Thu Apr 27 10:47:55 2006 @@ -3,7 +3,7 @@ Version: $Revision$ Last-Modified: $Date$ Author: Skip Montanaro -Status: Draft +Status: Withdrawn Type: Standards Track Content-Type: text/x-rst Created: 22-Jan-2003 From neal at metaslash.com Thu Apr 27 11:20:54 2006 From: neal at metaslash.com (Neal Norwitz) Date: Thu, 27 Apr 2006 05:20:54 -0400 Subject: [Python-checkins] Python Regression Test Failures doc (1) Message-ID: <20060427092054.GA26468@python.psfb.org> TEXINPUTS=/home/neal/python/trunk/Doc/commontex: python /home/neal/python/trunk/Doc/tools/mkhowto --html --about html/stdabout.dat --iconserver ../icons --favicon ../icons/pyfav.png --address "See About this document... for information on suggesting changes." --up-link ../index.html --up-title "Python Documentation Index" --global-module-index "../modindex.html" --dvips-safe --dir html/api api/api.tex *** Session transcript and error messages are in /home/neal/python/trunk/Doc/html/api/api.how. *** Exited with status 1. The relevant lines from the transcript are: ------------------------------------------------------------------------ +++ latex api This is TeX, Version 3.14159 (Web2C 7.4.5) (/home/neal/python/trunk/Doc/api/api.tex LaTeX2e <2001/06/01> Babel and hyphenation patterns for american, french, german, ngerman, n ohyphenation, loaded. (/home/neal/python/trunk/Doc/texinputs/manual.cls Document Class: manual 1998/03/03 Document class (Python manual) (/home/neal/python/trunk/Doc/texinputs/pypaper.sty (/usr/share/texmf/tex/latex/psnfss/times.sty) Using Times instead of Computer Modern. ) (/usr/share/texmf/tex/latex/misc/fancybox.sty Style option: `fancybox' v1.3 <2000/09/19> (tvz) ) (/usr/share/texmf/tex/latex/base/report.cls Document Class: report 2001/04/21 v1.4e Standard LaTeX document class (/usr/share/texmf/tex/latex/base/size10.clo)) (/home/neal/python/trunk/Doc/texinputs/fancyhdr.sty) Using fancier footers than usual. (/home/neal/python/trunk/Doc/texinputs/fncychap.sty) Using fancy chapter headings. (/home/neal/python/trunk/Doc/texinputs/python.sty (/usr/share/texmf/tex/latex/tools/longtable.sty) (/home/neal/python/trunk/Doc/texinputs/underscore.sty) (/usr/share/texmf/tex/latex/tools/verbatim.sty) (/usr/share/texmf/tex/latex/base/alltt.sty))) (/home/neal/python/trunk/Doc/commontex/boilerplate.tex ! LaTeX Error: Missing \begin{document}. See the LaTeX manual or LaTeX Companion for explanation. Type H for immediate help. ... l.8 < <<<<<< .mine ? ! Emergency stop. ... l.8 < <<<<<< .mine No pages of output. Transcript written on api.log. *** Session transcript and error messages are in /home/neal/python/trunk/Doc/html/api/api.how. *** Exited with status 1. +++ TEXINPUTS=/home/neal/python/trunk/Doc/api:/home/neal/python/trunk/Doc/commontex:/home/neal/python/trunk/Doc/paper-letter:/home/neal/python/trunk/Doc/texinputs: +++ latex api make: *** [html/api/api.html] Error 1 From python-checkins at python.org Thu Apr 27 11:47:04 2006 From: python-checkins at python.org (georg.brandl) Date: Thu, 27 Apr 2006 11:47:04 +0200 (CEST) Subject: [Python-checkins] r45745 - peps/trunk/pep-3099.txt Message-ID: <20060427094704.4AAD31E4003@bag.python.org> Author: georg.brandl Date: Thu Apr 27 11:47:03 2006 New Revision: 45745 Modified: peps/trunk/pep-3099.txt Log: Strings will stay iterable. Modified: peps/trunk/pep-3099.txt ============================================================================== --- peps/trunk/pep-3099.txt (original) +++ peps/trunk/pep-3099.txt Thu Apr 27 11:47:03 2006 @@ -31,6 +31,11 @@ Thread: "Parallel iteration syntax", http://mail.python.org/pipermail/python-3000/2006-March/000210.html +* Strings will stay iterable. + + Thread: "Making strings non-iterable", + http://mail.python.org/pipermail/python-3000/2006-April/000759.html + Builtins ======== From python-checkins at python.org Thu Apr 27 11:55:13 2006 From: python-checkins at python.org (georg.brandl) Date: Thu, 27 Apr 2006 11:55:13 +0200 (CEST) Subject: [Python-checkins] r45746 - peps/trunk/pep-3100.txt Message-ID: <20060427095513.65DD21E4003@bag.python.org> Author: georg.brandl Date: Thu Apr 27 11:55:12 2006 New Revision: 45746 Modified: peps/trunk/pep-3100.txt Log: Add another idea from python-3000 list. Modified: peps/trunk/pep-3100.txt ============================================================================== --- peps/trunk/pep-3100.txt (original) +++ peps/trunk/pep-3100.txt Thu Apr 27 11:55:12 2006 @@ -133,6 +133,7 @@ (e.g. ``dict.keys()``, ``dict.values()``, ``dict.items()``, etc.); iter* methods will be removed OR make keys(), etc. return views ala Java collections??? +* Make ``string.join()`` stringify its arguments? [26]_ To be removed: @@ -303,6 +304,9 @@ .. [25] python-3000 email ("Pronouncement on parameter lists") http://mail.python.org/pipermail/python-3000/2006-April/001175.html +.. [26] python-3000 email ("More wishful thinking") + http://mail.python.org/pipermail/python-3000/2006-April/000810.html + .. [#pep238] PEP 238 (Changing the Division Operator) http://www.python.org/dev/peps/pep-0238 From python-checkins at python.org Thu Apr 27 12:13:56 2006 From: python-checkins at python.org (georg.brandl) Date: Thu, 27 Apr 2006 12:13:56 +0200 (CEST) Subject: [Python-checkins] r45747 - peps/trunk/pep-3099.txt Message-ID: <20060427101356.081881E4003@bag.python.org> Author: georg.brandl Date: Thu Apr 27 12:13:55 2006 New Revision: 45747 Modified: peps/trunk/pep-3099.txt Log: Add "sorted genexp" pronouncement. Modified: peps/trunk/pep-3099.txt ============================================================================== --- peps/trunk/pep-3099.txt (original) +++ peps/trunk/pep-3099.txt Thu Apr 27 12:13:55 2006 @@ -36,6 +36,12 @@ Thread: "Making strings non-iterable", http://mail.python.org/pipermail/python-3000/2006-April/000759.html +* There will be no syntax to sort the result of a generator expression + or list comprehension. ``sorted()`` covers all use cases. + + Thread: "Adding sorting to generator comprehension", + http://mail.python.org/pipermail/python-3000/2006-April/001295.html + Builtins ======== From python-checkins at python.org Thu Apr 27 12:51:35 2006 From: python-checkins at python.org (georg.brandl) Date: Thu, 27 Apr 2006 12:51:35 +0200 (CEST) Subject: [Python-checkins] r45748 - peps/trunk/pep-3099.txt Message-ID: <20060427105135.5BB571E4003@bag.python.org> Author: georg.brandl Date: Thu Apr 27 12:51:34 2006 New Revision: 45748 Modified: peps/trunk/pep-3099.txt Log: Add "lambda" decision at a promiment location. Modified: peps/trunk/pep-3099.txt ============================================================================== --- peps/trunk/pep-3099.txt (original) +++ peps/trunk/pep-3099.txt Thu Apr 27 12:51:34 2006 @@ -21,6 +21,11 @@ Core language ============= +* ``lambda`` will not be renamed. + + Thread: "genexp syntax / lambda", + http://mail.python.org/pipermail/python-3000/2006-April/001042.html + * Python will not have programmable syntax. Thread: "It's a statement! It's a function! It's BOTH!", From python-checkins at python.org Thu Apr 27 14:22:38 2006 From: python-checkins at python.org (andrew.kuchling) Date: Thu, 27 Apr 2006 14:22:38 +0200 (CEST) Subject: [Python-checkins] r45749 - python/trunk/Doc/commontex/boilerplate.tex Message-ID: <20060427122238.D0C601E400B@bag.python.org> Author: andrew.kuchling Date: Thu Apr 27 14:22:37 2006 New Revision: 45749 Modified: python/trunk/Doc/commontex/boilerplate.tex Log: Now that 2.5a2 is out, revert to the current date Modified: python/trunk/Doc/commontex/boilerplate.tex ============================================================================== --- python/trunk/Doc/commontex/boilerplate.tex (original) +++ python/trunk/Doc/commontex/boilerplate.tex Thu Apr 27 14:22:37 2006 @@ -5,5 +5,5 @@ Email: \email{docs at python.org} } -\date{27th April 2006} % XXX update before final release! +\date{\today} % XXX update before final release! \input{patchlevel} % include Python version information From python-checkins at python.org Thu Apr 27 14:23:08 2006 From: python-checkins at python.org (andrew.kuchling) Date: Thu, 27 Apr 2006 14:23:08 +0200 (CEST) Subject: [Python-checkins] r45750 - python/trunk/Doc/whatsnew/whatsnew25.tex Message-ID: <20060427122308.2BECA1E4005@bag.python.org> Author: andrew.kuchling Date: Thu Apr 27 14:23:07 2006 New Revision: 45750 Modified: python/trunk/Doc/whatsnew/whatsnew25.tex Log: Bump document version Modified: python/trunk/Doc/whatsnew/whatsnew25.tex ============================================================================== --- python/trunk/Doc/whatsnew/whatsnew25.tex (original) +++ python/trunk/Doc/whatsnew/whatsnew25.tex Thu Apr 27 14:23:07 2006 @@ -7,7 +7,7 @@ % Count up the patches and bugs \title{What's New in Python 2.5} -\release{0.1} +\release{0.2} \author{A.M. Kuchling} \authoraddress{\email{amk at amk.ca}} From python-checkins at python.org Thu Apr 27 14:34:41 2006 From: python-checkins at python.org (andrew.kuchling) Date: Thu, 27 Apr 2006 14:34:41 +0200 (CEST) Subject: [Python-checkins] r45751 - python/trunk/Doc/lib/libxmlrpclib.tex Message-ID: <20060427123441.0ED051E4003@bag.python.org> Author: andrew.kuchling Date: Thu Apr 27 14:34:39 2006 New Revision: 45751 Modified: python/trunk/Doc/lib/libxmlrpclib.tex Log: [Bug #1477102] Add necessary import to example This may be a useful style question for the docs -- should examples show the necessary imports, or should it be assumed that the reader will figure it out? In the What's New, I'm not consistent but usually opt for omitting the imports. Modified: python/trunk/Doc/lib/libxmlrpclib.tex ============================================================================== --- python/trunk/Doc/lib/libxmlrpclib.tex (original) +++ python/trunk/Doc/lib/libxmlrpclib.tex Thu Apr 27 14:34:39 2006 @@ -340,6 +340,7 @@ \begin{verbatim} # simple test program (from the XML-RPC specification) +from xmlrpclib import ServerProxy # server = ServerProxy("http://localhost:8000") # local server server = ServerProxy("http://betty.userland.com") From python-checkins at python.org Thu Apr 27 14:35:35 2006 From: python-checkins at python.org (andrew.kuchling) Date: Thu, 27 Apr 2006 14:35:35 +0200 (CEST) Subject: [Python-checkins] r45752 - python/branches/release24-maint/Doc/lib/libxmlrpclib.tex Message-ID: <20060427123535.0A8421E4003@bag.python.org> Author: andrew.kuchling Date: Thu Apr 27 14:35:34 2006 New Revision: 45752 Modified: python/branches/release24-maint/Doc/lib/libxmlrpclib.tex Log: [Bug #1477102] Add necessary import to example Modified: python/branches/release24-maint/Doc/lib/libxmlrpclib.tex ============================================================================== --- python/branches/release24-maint/Doc/lib/libxmlrpclib.tex (original) +++ python/branches/release24-maint/Doc/lib/libxmlrpclib.tex Thu Apr 27 14:35:34 2006 @@ -318,6 +318,7 @@ \begin{verbatim} # simple test program (from the XML-RPC specification) +from xmlrpclib import ServerProxy # server = ServerProxy("http://localhost:8000") # local server server = ServerProxy("http://betty.userland.com") From python-checkins at python.org Thu Apr 27 14:38:36 2006 From: python-checkins at python.org (andrew.kuchling) Date: Thu, 27 Apr 2006 14:38:36 +0200 (CEST) Subject: [Python-checkins] r45753 - python/trunk/Doc/lib/libxmlrpclib.tex Message-ID: <20060427123836.0D0911E4003@bag.python.org> Author: andrew.kuchling Date: Thu Apr 27 14:38:35 2006 New Revision: 45753 Modified: python/trunk/Doc/lib/libxmlrpclib.tex Log: [Bug #1477140] Import Error base class Modified: python/trunk/Doc/lib/libxmlrpclib.tex ============================================================================== --- python/trunk/Doc/lib/libxmlrpclib.tex (original) +++ python/trunk/Doc/lib/libxmlrpclib.tex Thu Apr 27 14:38:35 2006 @@ -340,7 +340,7 @@ \begin{verbatim} # simple test program (from the XML-RPC specification) -from xmlrpclib import ServerProxy +from xmlrpclib import ServerProxy, Error # server = ServerProxy("http://localhost:8000") # local server server = ServerProxy("http://betty.userland.com") From python-checkins at python.org Thu Apr 27 14:42:54 2006 From: python-checkins at python.org (andrew.kuchling) Date: Thu, 27 Apr 2006 14:42:54 +0200 (CEST) Subject: [Python-checkins] r45754 - python/trunk/Doc/lib/libxmlrpclib.tex Message-ID: <20060427124254.DE8501E4003@bag.python.org> Author: andrew.kuchling Date: Thu Apr 27 14:42:54 2006 New Revision: 45754 Modified: python/trunk/Doc/lib/libxmlrpclib.tex Log: Mention the xmlrpclib.Error base class, which is used in one of the examples Modified: python/trunk/Doc/lib/libxmlrpclib.tex ============================================================================== --- python/trunk/Doc/lib/libxmlrpclib.tex (original) +++ python/trunk/Doc/lib/libxmlrpclib.tex Thu Apr 27 14:42:54 2006 @@ -81,9 +81,11 @@ This is the full set of data types supported by XML-RPC. Method calls may also raise a special \exception{Fault} instance, used to signal XML-RPC server errors, or \exception{ProtocolError} used to signal an -error in the HTTP/HTTPS transport layer. Note that even though starting -with Python 2.2 you can subclass builtin types, the xmlrpclib module -currently does not marshal instances of such subclasses. +error in the HTTP/HTTPS transport layer. Both \exception{Fault} and +\exception{ProtocolError} derive from a base class called +\exception{Error}. Note that even though starting with Python 2.2 you +can subclass builtin types, the xmlrpclib module currently does not +marshal instances of such subclasses. When passing strings, characters special to XML such as \samp{<}, \samp{>}, and \samp{\&} will be automatically escaped. However, it's From python-checkins at python.org Thu Apr 27 14:48:56 2006 From: python-checkins at python.org (andrew.kuchling) Date: Thu, 27 Apr 2006 14:48:56 +0200 (CEST) Subject: [Python-checkins] r45755 - python/branches/release24-maint/Doc/lib/libxmlrpclib.tex Message-ID: <20060427124856.D54A91E4013@bag.python.org> Author: andrew.kuchling Date: Thu Apr 27 14:48:56 2006 New Revision: 45755 Modified: python/branches/release24-maint/Doc/lib/libxmlrpclib.tex Log: Mention xmlrpclib.Error base class; import Error in example Modified: python/branches/release24-maint/Doc/lib/libxmlrpclib.tex ============================================================================== --- python/branches/release24-maint/Doc/lib/libxmlrpclib.tex (original) +++ python/branches/release24-maint/Doc/lib/libxmlrpclib.tex Thu Apr 27 14:48:56 2006 @@ -71,9 +71,11 @@ This is the full set of data types supported by XML-RPC. Method calls may also raise a special \exception{Fault} instance, used to signal XML-RPC server errors, or \exception{ProtocolError} used to signal an -error in the HTTP/HTTPS transport layer. Note that even though starting -with Python 2.2 you can subclass builtin types, the xmlrpclib module -currently does not marshal instances of such subclasses. +error in the HTTP/HTTPS transport layer. Both \exception{Fault} and +\exception{ProtocolError} derive from a base class called +\exception{Error}. Note that even though starting with Python 2.2 you +can subclass builtin types, the xmlrpclib module currently does not +marshal instances of such subclasses. When passing strings, characters special to XML such as \samp{<}, \samp{>}, and \samp{\&} will be automatically escaped. However, it's @@ -318,7 +320,7 @@ \begin{verbatim} # simple test program (from the XML-RPC specification) -from xmlrpclib import ServerProxy +from xmlrpclib import ServerProxy, Error # server = ServerProxy("http://localhost:8000") # local server server = ServerProxy("http://betty.userland.com") From buildbot at python.org Thu Apr 27 15:12:07 2006 From: buildbot at python.org (buildbot at python.org) Date: Thu, 27 Apr 2006 13:12:07 +0000 Subject: [Python-checkins] buildbot warnings in x86 OpenBSD 2.4 Message-ID: <20060427131207.59FBE1E4003@bag.python.org> The Buildbot has detected a new failure of x86 OpenBSD 2.4. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520OpenBSD%25202.4/builds/77 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch branches/release24-maint] HEAD Blamelist: andrew.kuchling Build Had Warnings: warnings test sincerely, -The Buildbot From anthony at interlink.com.au Thu Apr 27 15:21:05 2006 From: anthony at interlink.com.au (Anthony Baxter) Date: Thu, 27 Apr 2006 23:21:05 +1000 Subject: [Python-checkins] trunk is UNFROZEN Message-ID: <200604272321.07144.anthony@interlink.com.au> The release is done. The trunk is now unfrozen. Thanks, Anthony -- Anthony Baxter It's never too late to have a happy childhood. From python-checkins at python.org Thu Apr 27 15:41:08 2006 From: python-checkins at python.org (george.yoshida) Date: Thu, 27 Apr 2006 15:41:08 +0200 (CEST) Subject: [Python-checkins] r45756 - python/trunk/Doc/dist/dist.tex Message-ID: <20060427134108.0909F1E4003@bag.python.org> Author: george.yoshida Date: Thu Apr 27 15:41:07 2006 New Revision: 45756 Modified: python/trunk/Doc/dist/dist.tex Log: markup fix Modified: python/trunk/Doc/dist/dist.tex ============================================================================== --- python/trunk/Doc/dist/dist.tex (original) +++ python/trunk/Doc/dist/dist.tex Thu Apr 27 15:41:07 2006 @@ -1760,16 +1760,16 @@ URL from the \file{\$HOME/.pypirc} file (see section~\ref{pypirc} for more on this file). -You can use the \programopt{--sign} option to tell \command{upload} to +You can use the \longprogramopt{sign} option to tell \command{upload} to sign each uploaded file using GPG (GNU Privacy Guard). The \program{gpg} program must be available for execution on the system \envvar{PATH}. You can also specify which key to use for signing -using the \programopt{--identity=\var{name}} option. +using the \longprogramopt{identity=\var{name}} option. Other \command{upload} options include -\programopt{--repository=\var{url}} (which lets you override the +\longprogramopt{repository=\var{url}} (which lets you override the repository setting from \file{\$HOME/.pypirc}), and -\programopt{--show-response} (which displays the full response text +\longprogramopt{show-response} (which displays the full response text from the PyPI server for help in debugging upload problems). \chapter{Examples} From python-checkins at python.org Thu Apr 27 15:46:59 2006 From: python-checkins at python.org (thomas.wouters) Date: Thu, 27 Apr 2006 15:46:59 +0200 (CEST) Subject: [Python-checkins] r45757 - python/trunk/Lib/test/test_bigmem.py Message-ID: <20060427134659.D88CB1E4003@bag.python.org> Author: thomas.wouters Date: Thu Apr 27 15:46:59 2006 New Revision: 45757 Modified: python/trunk/Lib/test/test_bigmem.py Log: Some more size-estimate fixes, for large-list-tests. Modified: python/trunk/Lib/test/test_bigmem.py ============================================================================== --- python/trunk/Lib/test/test_bigmem.py (original) +++ python/trunk/Lib/test/test_bigmem.py Thu Apr 27 15:46:59 2006 @@ -703,11 +703,11 @@ self.failUnless(l[0] is l[-1]) self.failUnless(l[size - 1] is l[size + 1]) - @bigmemtest(minsize=_2G // 2 + 2, memuse=8) + @bigmemtest(minsize=_2G // 2 + 2, memuse=24) def test_inplace_concat_small(self, size): return self.basic_test_inplace_concat(size) - @bigmemtest(minsize=_2G + 2, memuse=8) + @bigmemtest(minsize=_2G + 2, memuse=24) def test_inplace_concat_large(self, size): return self.basic_test_inplace_concat(size) From buildbot at python.org Thu Apr 27 15:57:16 2006 From: buildbot at python.org (buildbot at python.org) Date: Thu, 27 Apr 2006 13:57:16 +0000 Subject: [Python-checkins] buildbot warnings in hppa Ubuntu dapper 2.4 Message-ID: <20060427135717.2314E1E4003@bag.python.org> The Buildbot has detected a new failure of hppa Ubuntu dapper 2.4. Full details are available at: http://www.python.org/dev/buildbot/all/hppa%2520Ubuntu%2520dapper%25202.4/builds/41 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch branches/release24-maint] HEAD Blamelist: andrew.kuchling Build Had Warnings: warnings test sincerely, -The Buildbot From buildbot at python.org Thu Apr 27 16:54:25 2006 From: buildbot at python.org (buildbot at python.org) Date: Thu, 27 Apr 2006 14:54:25 +0000 Subject: [Python-checkins] buildbot warnings in hppa Ubuntu dapper trunk Message-ID: <20060427145425.B5BFE1E4003@bag.python.org> The Buildbot has detected a new failure of hppa Ubuntu dapper trunk. Full details are available at: http://www.python.org/dev/buildbot/all/hppa%2520Ubuntu%2520dapper%2520trunk/builds/309 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: andrew.kuchling Build Had Warnings: warnings failed slave lost sincerely, -The Buildbot From buildbot at python.org Thu Apr 27 17:08:44 2006 From: buildbot at python.org (buildbot at python.org) Date: Thu, 27 Apr 2006 15:08:44 +0000 Subject: [Python-checkins] buildbot warnings in alpha Debian 2.4 Message-ID: <20060427150845.274C31E4005@bag.python.org> The Buildbot has detected a new failure of alpha Debian 2.4. Full details are available at: http://www.python.org/dev/buildbot/all/alpha%2520Debian%25202.4/builds/4 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch branches/release24-maint] HEAD Blamelist: andrew.kuchling Build Had Warnings: warnings test sincerely, -The Buildbot From python-checkins at python.org Thu Apr 27 17:50:45 2006 From: python-checkins at python.org (thomas.heller) Date: Thu, 27 Apr 2006 17:50:45 +0200 (CEST) Subject: [Python-checkins] r45758 - python/trunk/setup.py Message-ID: <20060427155045.779951E4014@bag.python.org> Author: thomas.heller Date: Thu Apr 27 17:50:42 2006 New Revision: 45758 Modified: python/trunk/setup.py Log: Rerun the libffi configuration if any of the files used for that are newer then fficonfig.py. Modified: python/trunk/setup.py ============================================================================== --- python/trunk/setup.py (original) +++ python/trunk/setup.py Thu Apr 27 17:50:42 2006 @@ -1267,7 +1267,12 @@ '_ctypes', 'libffi')) ffi_configfile = os.path.join(ffi_builddir, 'fficonfig.py') - if self.force or not os.path.exists(ffi_configfile): + from distutils.dep_util import newer_group + + config_sources = [os.path.join(ffi_srcdir, fname) + for fname in os.listdir(ffi_srcdir)] + if self.force or newer_group(config_sources, + ffi_configfile): from distutils.dir_util import mkpath mkpath(ffi_builddir) config_args = [] From buildbot at python.org Thu Apr 27 18:19:43 2006 From: buildbot at python.org (buildbot at python.org) Date: Thu, 27 Apr 2006 16:19:43 +0000 Subject: [Python-checkins] buildbot warnings in alpha Debian trunk Message-ID: <20060427161943.5ADAA1E4003@bag.python.org> The Buildbot has detected a new failure of alpha Debian trunk. Full details are available at: http://www.python.org/dev/buildbot/all/alpha%2520Debian%2520trunk/builds/32 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: andrew.kuchling Build Had Warnings: warnings test sincerely, -The Buildbot From buildbot at python.org Thu Apr 27 18:24:29 2006 From: buildbot at python.org (buildbot at python.org) Date: Thu, 27 Apr 2006 16:24:29 +0000 Subject: [Python-checkins] buildbot warnings in hppa Ubuntu dapper trunk Message-ID: <20060427162430.0F8781E4003@bag.python.org> The Buildbot has detected a new failure of hppa Ubuntu dapper trunk. Full details are available at: http://www.python.org/dev/buildbot/all/hppa%2520Ubuntu%2520dapper%2520trunk/builds/311 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: thomas.heller Build Had Warnings: warnings failed slave lost sincerely, -The Buildbot From buildbot at python.org Thu Apr 27 18:29:13 2006 From: buildbot at python.org (buildbot at python.org) Date: Thu, 27 Apr 2006 16:29:13 +0000 Subject: [Python-checkins] buildbot warnings in sparc Ubuntu dapper 2.4 Message-ID: <20060427162913.33B041E4006@bag.python.org> The Buildbot has detected a new failure of sparc Ubuntu dapper 2.4. Full details are available at: http://www.python.org/dev/buildbot/all/sparc%2520Ubuntu%2520dapper%25202.4/builds/39 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch branches/release24-maint] HEAD Blamelist: andrew.kuchling Build Had Warnings: warnings test sincerely, -The Buildbot From python-checkins at python.org Thu Apr 27 18:53:54 2006 From: python-checkins at python.org (talin) Date: Thu, 27 Apr 2006 18:53:54 +0200 (CEST) Subject: [Python-checkins] r45759 - peps/trunk/pep-3101.txt Message-ID: <20060427165354.B46511E4003@bag.python.org> Author: talin Date: Thu Apr 27 18:53:54 2006 New Revision: 45759 Modified: peps/trunk/pep-3101.txt Log: Removed references to previous 'fformat' proposal. Added clarification about relationship with string.Template Modified: peps/trunk/pep-3101.txt ============================================================================== --- peps/trunk/pep-3101.txt (original) +++ peps/trunk/pep-3101.txt Thu Apr 27 18:53:54 2006 @@ -22,28 +22,29 @@ Python currently provides two methods of string interpolation: - - The '%' operator for strings. + - The '%' operator for strings. [1] - - The string.Template module. + - The string.Template module. [2] The scope of this PEP will be restricted to proposals for built-in string formatting operations (in other words, methods of the - built-in string type). This does not obviate the need for more - sophisticated string-manipulation modules in the standard library - such as string.Template. In any case, string.Template will not be - discussed here, except to say that the this proposal will most - likely have some overlapping functionality with that module. - + built-in string type). + The '%' operator is primarily limited by the fact that it is a binary operator, and therefore can take at most two arguments. One of those arguments is already dedicated to the format string, leaving all other variables to be squeezed into the remaining argument. The current practice is to use either a dictionary or a tuple as the second argument, but as many people have commented - [1], this lacks flexibility. The "all or nothing" approach + [3], this lacks flexibility. The "all or nothing" approach (meaning that one must choose between only positional arguments, or only named arguments) is felt to be overly constraining. + While there is some overlap between this proposal and + string.Template, it is felt that each serves a distinct need, + and that one does not obviate the other. In any case, + string.Template will not be discussed here. + Specification @@ -62,9 +63,9 @@ String Methods - The build-in string class will gain two new methods. The first - method is 'format', and takes an arbitrary number of positional - and keyword arguments: + The build-in string class will gain a new method, 'format', + which takes takes an arbitrary number of positional and keyword + arguments: "The story of {0}, {1}, and {c}".format(a, b, c=d) @@ -135,20 +136,6 @@ and the matching right brace ('}') to the various underlying formatters (described later.) - When using the 'fformat' variant, it is possible to omit the field - name entirely, and simply include the conversion specifiers: - - "My name is {:pad(23)}" - - This syntax is used to send special instructions to the custom - formatter object (such as instructing it to insert padding - characters up to a given column.) The interpretation of this - 'empty' field is entirely up to the custom formatter; no - standard interpretation will be defined in this PEP. - - If a custom formatter is not being used, then it is an error to - omit the field name. - Standard Conversion Specifiers @@ -303,7 +290,7 @@ string interpolation. The Microsoft .Net libraries uses braces ({}), and a syntax which is very similar to the one in this proposal, although the syntax for conversion specifiers is quite - different. [2] + different. [4] - Backquoting. This method has the benefit of minimal syntactical clutter, however it lacks many of the benefits of a function @@ -314,6 +301,13 @@ on. +Sample Implementation + + A rought prototype of the underlying 'cformat' function has been + coded in Python, however it needs much refinement before being + submitted. + + Backwards Compatibility Backwards compatibility can be maintained by leaving the existing @@ -325,10 +319,16 @@ References - [1] [Python-3000] String formating operations in python 3k + [1] Python Library Reference - String formating operations + http://docs.python.org/lib/typesseq-strings.html + + [2] Python Library References - Template strings + http://docs.python.org/lib/node109.html + + [3] [Python-3000] String formating operations in python 3k http://mail.python.org/pipermail/python-3000/2006-April/000285.html - [2] Composite Formatting - [.Net Framework Developer's Guide] + [4] Composite Formatting - [.Net Framework Developer's Guide] http://msdn.microsoft.com/library/en-us/cpguide/html/cpconcompositeformatting.asp?frame=true From python-checkins at python.org Thu Apr 27 19:33:43 2006 From: python-checkins at python.org (andrew.kuchling) Date: Thu, 27 Apr 2006 19:33:43 +0200 (CEST) Subject: [Python-checkins] r45760 - peps/trunk/pep-0356.txt Message-ID: <20060427173343.88A861E4003@bag.python.org> Author: andrew.kuchling Date: Thu Apr 27 19:33:42 2006 New Revision: 45760 Modified: peps/trunk/pep-0356.txt Log: Move mailbox change into 'completed' Modified: peps/trunk/pep-0356.txt ============================================================================== --- peps/trunk/pep-0356.txt (original) +++ peps/trunk/pep-0356.txt Thu Apr 27 19:33:42 2006 @@ -98,6 +98,9 @@ - Added PEP 302 zipfile/__loader__ support to the following modules: warnings, linecache, inspect, traceback, site, and doctest + - Add write support for mailboxes from the code in sandbox/mailbox. + (Owner: A.M. Kuchling. It would still be good if another person + would take a look at the new code.) Possible features for 2.5 @@ -194,12 +197,6 @@ "timing" (listed as obsolete), "cl" (listed as possibly not up-to-date), and "sv" (listed as obsolete hardware specific). - - Add write support for mailboxes from the code in sandbox/mailbox. - AMK will take one last look at the code for backward-compatibility - problems and then apply the changes before 2.5a2. - (Owner: A.M. Kuchling. It would still be good if another person - would take a look at the new code.) - Copyright This document has been placed in the public domain. From buildbot at python.org Thu Apr 27 19:34:01 2006 From: buildbot at python.org (buildbot at python.org) Date: Thu, 27 Apr 2006 17:34:01 +0000 Subject: [Python-checkins] buildbot warnings in sparc Ubuntu dapper trunk Message-ID: <20060427173402.0186E1E4003@bag.python.org> The Buildbot has detected a new failure of sparc Ubuntu dapper trunk. Full details are available at: http://www.python.org/dev/buildbot/all/sparc%2520Ubuntu%2520dapper%2520trunk/builds/173 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: andrew.kuchling,george.yoshida,thomas.wouters Build Had Warnings: warnings test sincerely, -The Buildbot From python-checkins at python.org Thu Apr 27 19:34:13 2006 From: python-checkins at python.org (andrew.kuchling) Date: Thu, 27 Apr 2006 19:34:13 +0200 (CEST) Subject: [Python-checkins] r45761 - peps/trunk/pep-0356.txt Message-ID: <20060427173413.5555C1E4003@bag.python.org> Author: andrew.kuchling Date: Thu Apr 27 19:34:13 2006 New Revision: 45761 Modified: peps/trunk/pep-0356.txt Log: Record alpha2 release Modified: peps/trunk/pep-0356.txt ============================================================================== --- peps/trunk/pep-0356.txt (original) +++ peps/trunk/pep-0356.txt Thu Apr 27 19:34:13 2006 @@ -39,7 +39,7 @@ unfolds. alpha 1: April 5, 2006 [completed] - alpha 2: April 25, 2006 [planned] + alpha 2: April 27, 2006 [completed] alpha 3: May 27, 2006 [planned] beta 1: June 24, 2006 [planned] beta 2: July 15, 2006 [planned] From python-checkins at python.org Thu Apr 27 19:37:21 2006 From: python-checkins at python.org (andrew.kuchling) Date: Thu, 27 Apr 2006 19:37:21 +0200 (CEST) Subject: [Python-checkins] r45762 - peps/trunk/pep-0356.txt Message-ID: <20060427173721.20EDD1E4005@bag.python.org> Author: andrew.kuchling Date: Thu Apr 27 19:37:19 2006 New Revision: 45762 Modified: peps/trunk/pep-0356.txt Log: Mention MacOS icons; add links to Windows and MacOS info (can't find the Unix icons) Modified: peps/trunk/pep-0356.txt ============================================================================== --- peps/trunk/pep-0356.txt (original) +++ peps/trunk/pep-0356.txt Thu Apr 27 19:37:19 2006 @@ -136,7 +136,9 @@ - Support for building "fat" Mac binaries (Intel and PPC) (Owner: Ronald Oussoren) - - Add new icons for Windows and Unix with the new Python logo? + - Add new icons for Windows, MacOS and Unix with the new Python logo? + Windows: http://mail.python.org/pipermail/python-dev/2006-April/063738.html + MacOS: http://hcs.harvard.edu/~jrus/python/prettified-py-icons.png - Check the various bits of code in Demo/ all still work, update or remove the ones that don't. From python-checkins at python.org Thu Apr 27 19:48:03 2006 From: python-checkins at python.org (guido.van.rossum) Date: Thu, 27 Apr 2006 19:48:03 +0200 (CEST) Subject: [Python-checkins] r45763 - peps/trunk/pep-3100.txt Message-ID: <20060427174803.E17341E4005@bag.python.org> Author: guido.van.rossum Date: Thu Apr 27 19:48:03 2006 New Revision: 45763 Modified: peps/trunk/pep-3100.txt Log: Add set literals. Clarify that the new string type will be 'str'. Modified: peps/trunk/pep-3100.txt ============================================================================== --- peps/trunk/pep-3100.txt (original) +++ peps/trunk/pep-3100.txt Thu Apr 27 19:48:03 2006 @@ -104,6 +104,12 @@ and semantics is evil. * Attributes on functions of the form ``func_whatever`` will be renamed ``__whatever__`` [25]_ +* Set literals and comprehensions [27]_ + {/} means set(); {x} means set([x]); {x, y} means set([x, y]). + {F(x) for x in S if P(x)} means set(F(x) for x in S if P(x)). + NB. {range(x)} means set([range(x)]), NOT set(range(x)). + There's no frozenset literal; they are too rarely needed. + The {/} part is still controversial. To be removed: @@ -129,6 +135,7 @@ * Remove distinction between int and long types [1]_ (int may become an abstract base type, with short and long subtypes.) * Make all strings be Unicode, and have a separate bytes() type [1]_ + The new string type will be called 'str'. * Return iterators instead of lists where appropriate for atomic type methods (e.g. ``dict.keys()``, ``dict.values()``, ``dict.items()``, etc.); iter* methods will be removed @@ -307,6 +314,9 @@ .. [26] python-3000 email ("More wishful thinking") http://mail.python.org/pipermail/python-3000/2006-April/000810.html + [27] python-3000 email ("sets in P3K?") + http://mail.python.org/pipermail/python-3000/2006-April/001286.html + .. [#pep238] PEP 238 (Changing the Division Operator) http://www.python.org/dev/peps/pep-0238 From python-checkins at python.org Thu Apr 27 20:02:45 2006 From: python-checkins at python.org (phillip.eby) Date: Thu, 27 Apr 2006 20:02:45 +0200 (CEST) Subject: [Python-checkins] r45764 - in sandbox/trunk/setuptools: pkg_resources.py setuptools/tests/test_resources.py Message-ID: <20060427180245.BEE5C1E4006@bag.python.org> Author: phillip.eby Date: Thu Apr 27 20:02:44 2006 New Revision: 45764 Modified: sandbox/trunk/setuptools/pkg_resources.py sandbox/trunk/setuptools/setuptools/tests/test_resources.py Log: Fix entry point parsing when a standalone module name has whitespace between it and the extras. Modified: sandbox/trunk/setuptools/pkg_resources.py ============================================================================== --- sandbox/trunk/setuptools/pkg_resources.py (original) +++ sandbox/trunk/setuptools/pkg_resources.py Thu Apr 27 20:02:44 2006 @@ -1791,7 +1791,7 @@ src ) else: - return cls(name.strip(), value.lstrip(), attrs, extras, dist) + return cls(name.strip(), value.strip(), attrs, extras, dist) parse = classmethod(parse) Modified: sandbox/trunk/setuptools/setuptools/tests/test_resources.py ============================================================================== --- sandbox/trunk/setuptools/setuptools/tests/test_resources.py (original) +++ sandbox/trunk/setuptools/setuptools/tests/test_resources.py Thu Apr 27 20:02:44 2006 @@ -254,18 +254,18 @@ def checkSubMap(self, m): self.assertEqual(str(m), - "{" - "'feature2': EntryPoint.parse(" + "{'feature2': EntryPoint.parse(" "'feature2 = another.module:SomeClass [extra1,extra2]'), " + "'feature3': EntryPoint.parse('feature3 = this.module [something]'), " "'feature1': EntryPoint.parse(" - "'feature1 = somemodule:somefunction')" - "}" + "'feature1 = somemodule:somefunction')}" ) submap_str = """ # define features for blah blah feature1 = somemodule:somefunction feature2 = another.module:SomeClass [extra1,extra2] + feature3 = this.module [something] """ def testParseList(self): From python-checkins at python.org Thu Apr 27 20:05:53 2006 From: python-checkins at python.org (phillip.eby) Date: Thu, 27 Apr 2006 20:05:53 +0200 (CEST) Subject: [Python-checkins] r45765 - in sandbox/branches/setuptools-0.6: pkg_resources.py setuptools/tests/test_resources.py Message-ID: <20060427180553.F3D501E400F@bag.python.org> Author: phillip.eby Date: Thu Apr 27 20:05:53 2006 New Revision: 45765 Modified: sandbox/branches/setuptools-0.6/pkg_resources.py sandbox/branches/setuptools-0.6/setuptools/tests/test_resources.py Log: Fix entry point parsing when a standalone module name has whitespace between it and the extras. Modified: sandbox/branches/setuptools-0.6/pkg_resources.py ============================================================================== --- sandbox/branches/setuptools-0.6/pkg_resources.py (original) +++ sandbox/branches/setuptools-0.6/pkg_resources.py Thu Apr 27 20:05:53 2006 @@ -1873,7 +1873,7 @@ src ) else: - return cls(name.strip(), value.lstrip(), attrs, extras, dist) + return cls(name.strip(), value.strip(), attrs, extras, dist) parse = classmethod(parse) Modified: sandbox/branches/setuptools-0.6/setuptools/tests/test_resources.py ============================================================================== --- sandbox/branches/setuptools-0.6/setuptools/tests/test_resources.py (original) +++ sandbox/branches/setuptools-0.6/setuptools/tests/test_resources.py Thu Apr 27 20:05:53 2006 @@ -254,18 +254,18 @@ def checkSubMap(self, m): self.assertEqual(str(m), - "{" - "'feature2': EntryPoint.parse(" + "{'feature2': EntryPoint.parse(" "'feature2 = another.module:SomeClass [extra1,extra2]'), " + "'feature3': EntryPoint.parse('feature3 = this.module [something]'), " "'feature1': EntryPoint.parse(" - "'feature1 = somemodule:somefunction')" - "}" + "'feature1 = somemodule:somefunction')}" ) submap_str = """ # define features for blah blah feature1 = somemodule:somefunction feature2 = another.module:SomeClass [extra1,extra2] + feature3 = this.module [something] """ def testParseList(self): From buildbot at python.org Thu Apr 27 21:29:20 2006 From: buildbot at python.org (buildbot at python.org) Date: Thu, 27 Apr 2006 19:29:20 +0000 Subject: [Python-checkins] buildbot warnings in MIPS Debian trunk Message-ID: <20060427192920.8C6D81E4003@bag.python.org> The Buildbot has detected a new failure of MIPS Debian trunk. Full details are available at: http://www.python.org/dev/buildbot/all/MIPS%2520Debian%2520trunk/builds/39 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: thomas.heller Build Had Warnings: warnings test sincerely, -The Buildbot From neal at metaslash.com Thu Apr 27 23:19:48 2006 From: neal at metaslash.com (Neal Norwitz) Date: Thu, 27 Apr 2006 17:19:48 -0400 Subject: [Python-checkins] Python Regression Test Failures doc (1) Message-ID: <20060427211948.GA6886@python.psfb.org> TEXINPUTS=/home/neal/python/trunk/Doc/commontex: python /home/neal/python/trunk/Doc/tools/mkhowto --html --about html/stdabout.dat --iconserver ../icons --favicon ../icons/pyfav.png --address "See About this document... for information on suggesting changes." --up-link ../index.html --up-title "Python Documentation Index" --global-module-index "../modindex.html" --dvips-safe --dir html/api api/api.tex *** Session transcript and error messages are in /home/neal/python/trunk/Doc/html/api/api.how. *** Exited with status 1. The relevant lines from the transcript are: ------------------------------------------------------------------------ +++ latex api This is TeX, Version 3.14159 (Web2C 7.4.5) (/home/neal/python/trunk/Doc/api/api.tex LaTeX2e <2001/06/01> Babel and hyphenation patterns for american, french, german, ngerman, n ohyphenation, loaded. (/home/neal/python/trunk/Doc/texinputs/manual.cls Document Class: manual 1998/03/03 Document class (Python manual) (/home/neal/python/trunk/Doc/texinputs/pypaper.sty (/usr/share/texmf/tex/latex/psnfss/times.sty) Using Times instead of Computer Modern. ) (/usr/share/texmf/tex/latex/misc/fancybox.sty Style option: `fancybox' v1.3 <2000/09/19> (tvz) ) (/usr/share/texmf/tex/latex/base/report.cls Document Class: report 2001/04/21 v1.4e Standard LaTeX document class (/usr/share/texmf/tex/latex/base/size10.clo)) (/home/neal/python/trunk/Doc/texinputs/fancyhdr.sty) Using fancier footers than usual. (/home/neal/python/trunk/Doc/texinputs/fncychap.sty) Using fancy chapter headings. (/home/neal/python/trunk/Doc/texinputs/python.sty (/usr/share/texmf/tex/latex/tools/longtable.sty) (/home/neal/python/trunk/Doc/texinputs/underscore.sty) (/usr/share/texmf/tex/latex/tools/verbatim.sty) (/usr/share/texmf/tex/latex/base/alltt.sty))) (/home/neal/python/trunk/Doc/commontex/boilerplate.tex ! LaTeX Error: Missing \begin{document}. See the LaTeX manual or LaTeX Companion for explanation. Type H for immediate help. ... l.8 < <<<<<< .mine ? ! Emergency stop. ... l.8 < <<<<<< .mine No pages of output. Transcript written on api.log. *** Session transcript and error messages are in /home/neal/python/trunk/Doc/html/api/api.how. *** Exited with status 1. +++ TEXINPUTS=/home/neal/python/trunk/Doc/api:/home/neal/python/trunk/Doc/commontex:/home/neal/python/trunk/Doc/paper-letter:/home/neal/python/trunk/Doc/texinputs: +++ latex api make: *** [html/api/api.html] Error 1 From fdrake at acm.org Thu Apr 27 23:32:36 2006 From: fdrake at acm.org (Fred L. Drake, Jr.) Date: Thu, 27 Apr 2006 17:32:36 -0400 Subject: [Python-checkins] Python Regression Test Failures doc (1) In-Reply-To: <20060427211948.GA6886@python.psfb.org> References: <20060427211948.GA6886@python.psfb.org> Message-ID: <200604271732.36708.fdrake@acm.org> On Thursday 27 April 2006 17:19, Neal Norwitz wrote: > TEXINPUTS=/home/neal/python/trunk/Doc/commontex: python > /home/neal/python/trunk/Doc/tools/mkhowto --html --about html/stdabout.dat > --iconserver ../icons --favicon ../icons/pyfav.png --address "See href=\"about.html\">About this document... for information on > suggesting changes." --up-link ../index.html --up-title "Python > Documentation Index" --global-module-index "../modindex.html" --dvips-safe > --dir html/api api/api.tex *** Session transcript and error messages are > in /home/neal/python/trunk/Doc/html/api/api.how. *** Exited with status 1. > The relevant lines from the transcript are: ... > l.8 < > <<<<<< .mine > ? > ! Emergency stop. > ... > > l.8 < > <<<<<< .mine Haven't we seen these kinds of turds from this process before? -Fred -- Fred L. Drake, Jr. From buildbot at python.org Thu Apr 27 23:47:19 2006 From: buildbot at python.org (buildbot at python.org) Date: Thu, 27 Apr 2006 21:47:19 +0000 Subject: [Python-checkins] buildbot warnings in x86 cygwin trunk Message-ID: <20060427214719.586DC1E4006@bag.python.org> The Buildbot has detected a new failure of x86 cygwin trunk. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520cygwin%2520trunk/builds/308 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: thomas.heller Build Had Warnings: warnings test sincerely, -The Buildbot From python-checkins at python.org Fri Apr 28 00:37:51 2006 From: python-checkins at python.org (thomas.wouters) Date: Fri, 28 Apr 2006 00:37:51 +0200 (CEST) Subject: [Python-checkins] r45766 - python/trunk/Lib/test/test_bigmem.py Message-ID: <20060427223751.0F4D91E4006@bag.python.org> Author: thomas.wouters Date: Fri Apr 28 00:37:50 2006 New Revision: 45766 Modified: python/trunk/Lib/test/test_bigmem.py Log: Some style fixes and size-calculation fixes. Also do the small-memory run using a prime number, rather than a convenient power-of-2-and-multiple-of-5, so incorrect testing algorithms fail more easily. Modified: python/trunk/Lib/test/test_bigmem.py ============================================================================== --- python/trunk/Lib/test/test_bigmem.py (original) +++ python/trunk/Lib/test/test_bigmem.py Fri Apr 28 00:37:50 2006 @@ -417,7 +417,7 @@ self.failUnless(sf.endswith('-..')) del s, sf - size = (size // 2) + size //= 2 edge = '-' * size s = ''.join([edge, '%s', edge]) del edge @@ -591,7 +591,7 @@ def test_concat_large(self, size): return self.basic_concat_test(size) - @bigmemtest(minsize=_2G // 5 + 10, memuse=8*5) + @bigmemtest(minsize=_2G // 5 + 10, memuse=8 * 5) def test_contains(self, size): t = (1, 2, 3, 4, 5) * size self.assertEquals(len(t), size * 5) @@ -650,11 +650,11 @@ self.assertEquals(s[-5:], '0, 0)') self.assertEquals(s.count('0'), size) - @bigmemtest(minsize=_2G // 3 + 2, memuse=8+3) + @bigmemtest(minsize=_2G // 3 + 2, memuse=8 + 3) def test_repr_small(self, size): return self.basic_test_repr(size) - @bigmemtest(minsize=_2G + 2, memuse=8+3) + @bigmemtest(minsize=_2G + 2, memuse=8 + 3) def test_repr_large(self, size): return self.basic_test_repr(size) @@ -711,7 +711,7 @@ def test_inplace_concat_large(self, size): return self.basic_test_inplace_concat(size) - @bigmemtest(minsize=_2G // 5 + 10, memuse=8*5) + @bigmemtest(minsize=_2G // 5 + 10, memuse=8 * 5) def test_contains(self, size): l = [1, 2, 3, 4, 5] * size self.assertEquals(len(l), size * 5) @@ -864,9 +864,10 @@ def test_extend_large(self, size): return self.basic_test_extend(size) - @bigmemtest(minsize=_2G + 10, memuse=8) + @bigmemtest(minsize=_2G // 5 + 2, memuse=8 * 5) def test_index(self, size): - l = [1L, 2L, 3L, 4L, 5L] * (size // 5) + l = [1L, 2L, 3L, 4L, 5L] * size + size *= 5 self.assertEquals(l.index(1), 0) self.assertEquals(l.index(5, size - 5), size - 1) self.assertEquals(l.index(5, size - 5, size), size - 1) @@ -893,25 +894,29 @@ self.assertEquals(l[:3], [1.0, "C", 1.0]) self.assertEquals(l[size - 3:], ["A", 1.0, "B"]) - @bigmemtest(minsize=_2G + 20, memuse=8) + @bigmemtest(minsize=_2G // 5 + 4, memuse=8 * 5) def test_pop(self, size): - l = [u"a", u"b", u"c", u"d", u"e"] * (size // 5) + l = [u"a", u"b", u"c", u"d", u"e"] * size + size *= 5 self.assertEquals(len(l), size) item = l.pop() size -= 1 self.assertEquals(len(l), size) self.assertEquals(item, u"e") + self.assertEquals(l[-2:], [u"c", u"d"]) item = l.pop(0) size -= 1 self.assertEquals(len(l), size) self.assertEquals(item, u"a") + self.assertEquals(l[:2], [u"b", u"c"]) item = l.pop(size - 2) size -= 1 self.assertEquals(len(l), size) self.assertEquals(item, u"c") + self.assertEquals(l[-2:], [u"b", u"d"]) @bigmemtest(minsize=_2G + 10, memuse=8) def test_remove(self, size): @@ -933,20 +938,20 @@ self.assertEquals(len(l), size) self.assertEquals(l[-2:], [10, 10]) - @bigmemtest(minsize=_2G + 10, memuse=8) + @bigmemtest(minsize=_2G // 5 + 2, memuse=8 * 5) def test_reverse(self, size): - l = [1, 2, 3, 4, 5] * (size // 5) + l = [1, 2, 3, 4, 5] * size l.reverse() - self.assertEquals(len(l), size) + self.assertEquals(len(l), size * 5) self.assertEquals(l[-5:], [5, 4, 3, 2, 1]) self.assertEquals(l[:5], [5, 4, 3, 2, 1]) - @bigmemtest(minsize=_2G + 10, memuse=8) + @bigmemtest(minsize=_2G // 5 + 2, memuse=8 * 5) def test_sort(self, size): - l = [1, 2, 3, 4, 5] * (size // 5) + l = [1, 2, 3, 4, 5] * size l.sort() - self.assertEquals(len(l), size) - self.assertEquals(l.count(1), size // 5) + self.assertEquals(len(l), size * 5) + self.assertEquals(l.count(1), size) self.assertEquals(l[:10], [1] * 10) self.assertEquals(l[-10:], [5] * 10) From python-checkins at python.org Fri Apr 28 00:38:32 2006 From: python-checkins at python.org (thomas.wouters) Date: Fri, 28 Apr 2006 00:38:32 +0200 (CEST) Subject: [Python-checkins] r45767 - python/trunk/Lib/test/test_support.py Message-ID: <20060427223832.E1CCA1E4006@bag.python.org> Author: thomas.wouters Date: Fri Apr 28 00:38:32 2006 New Revision: 45767 Modified: python/trunk/Lib/test/test_support.py Log: Do the small-memory run of big-meormy tests using a prime number, rather than a convenient power-of-2-and-multiple-of-5, so incorrect testing algorithms fail more easily. Modified: python/trunk/Lib/test/test_support.py ============================================================================== --- python/trunk/Lib/test/test_support.py (original) +++ python/trunk/Lib/test/test_support.py Fri Apr 28 00:38:32 2006 @@ -298,7 +298,7 @@ # we still want to run the tests with size set to a few kb, # to make sure they work. We still want to avoid using # too much memory, though, but we do that noisily. - maxsize = 1024*5 + maxsize = 5147 self.failIf(maxsize * memuse + overhead > 20 * _1M) else: maxsize = int((max_memuse - overhead) / memuse) From nnorwitz at gmail.com Fri Apr 28 00:52:05 2006 From: nnorwitz at gmail.com (Neal Norwitz) Date: Thu, 27 Apr 2006 15:52:05 -0700 Subject: [Python-checkins] Python Regression Test Failures doc (1) In-Reply-To: <200604271732.36708.fdrake@acm.org> References: <20060427211948.GA6886@python.psfb.org> <200604271732.36708.fdrake@acm.org> Message-ID: Yup, each release. :-) I was keeping an outstanding change to set the date in boilerplate.tex to \today. If the date is always reset, I won't need outstanding changes and we shouldn't see these any more. n -- On 4/27/06, Fred L. Drake, Jr. wrote: > On Thursday 27 April 2006 17:19, Neal Norwitz wrote: > > TEXINPUTS=/home/neal/python/trunk/Doc/commontex: python > > /home/neal/python/trunk/Doc/tools/mkhowto --html --about html/stdabout.dat > > --iconserver ../icons --favicon ../icons/pyfav.png --address "See > href=\"about.html\">About this document... for information on > > suggesting changes." --up-link ../index.html --up-title "Python > > Documentation Index" --global-module-index "../modindex.html" --dvips-safe > > --dir html/api api/api.tex *** Session transcript and error messages are > > in /home/neal/python/trunk/Doc/html/api/api.how. *** Exited with status 1. > > The relevant lines from the transcript are: > ... > > l.8 < > > <<<<<< .mine > > ? > > ! Emergency stop. > > ... > > > > l.8 < > > <<<<<< .mine > > Haven't we seen these kinds of turds from this process before? > > > -Fred > > -- > Fred L. Drake, Jr. > > From python-checkins at python.org Fri Apr 28 00:53:07 2006 From: python-checkins at python.org (david.goodger) Date: Fri, 28 Apr 2006 00:53:07 +0200 (CEST) Subject: [Python-checkins] r45768 - python/trunk/Misc/developers.txt Message-ID: <20060427225307.9DC441E4006@bag.python.org> Author: david.goodger Date: Fri Apr 28 00:53:05 2006 New Revision: 45768 Modified: python/trunk/Misc/developers.txt Log: Added SVN access for Steven Bethard and Talin, for PEP updating. Modified: python/trunk/Misc/developers.txt ============================================================================== --- python/trunk/Misc/developers.txt (original) +++ python/trunk/Misc/developers.txt Fri Apr 28 00:53:05 2006 @@ -17,6 +17,12 @@ Permissions History ------------------- +- Steven Bethard was given SVN access on 27 Apr 2006 by DJG, for PEP + update access. + +- Talin was given SVN access on 27 Apr 2006 by DJG, for PEP update + access. + - George Yoshida (SF name "quiver") added to the SourceForge Python project 14 Apr 2006, by Tim Peters, as a tracker admin. See contemporaneous python-checkins thread with the unlikely Subject: @@ -102,3 +108,4 @@ NCN: Neal Norwitz RDH: Raymond Hettinger TGP: Tim Peters +DJG: David Goodger From python-checkins at python.org Fri Apr 28 01:13:21 2006 From: python-checkins at python.org (thomas.wouters) Date: Fri, 28 Apr 2006 01:13:21 +0200 (CEST) Subject: [Python-checkins] r45770 - in python/trunk: Include/pyerrors.h Lib/test/exception_hierarchy.txt Lib/test/regrtest.py Lib/test/test_import.py Python/exceptions.c Python/import.c Message-ID: <20060427231321.CB16A1E4006@bag.python.org> Author: thomas.wouters Date: Fri Apr 28 01:13:20 2006 New Revision: 45770 Modified: python/trunk/Include/pyerrors.h python/trunk/Lib/test/exception_hierarchy.txt python/trunk/Lib/test/regrtest.py python/trunk/Lib/test/test_import.py python/trunk/Python/exceptions.c python/trunk/Python/import.c Log: - Add new Warning class, ImportWarning - Warn-raise ImportWarning when importing would have picked up a directory as package, if only it'd had an __init__.py. This swaps two tests (for case-ness and __init__-ness), but case-test is not really more expensive, and it's not in a speed-critical section. - Test for the new warning by importing a common non-package directory on sys.path: site-packages - In regrtest.py, silence warnings generated by the build-environment because Modules/ (which is added to sys.path for Setup-created modules) has 'zlib' and '_ctypes' directories without __init__.py's. Modified: python/trunk/Include/pyerrors.h ============================================================================== --- python/trunk/Include/pyerrors.h (original) +++ python/trunk/Include/pyerrors.h Fri Apr 28 01:13:20 2006 @@ -108,6 +108,7 @@ PyAPI_DATA(PyObject *) PyExc_OverflowWarning; PyAPI_DATA(PyObject *) PyExc_RuntimeWarning; PyAPI_DATA(PyObject *) PyExc_FutureWarning; +PyAPI_DATA(PyObject *) PyExc_ImportWarning; /* Convenience functions */ Modified: python/trunk/Lib/test/exception_hierarchy.txt ============================================================================== --- python/trunk/Lib/test/exception_hierarchy.txt (original) +++ python/trunk/Lib/test/exception_hierarchy.txt Fri Apr 28 01:13:20 2006 @@ -44,3 +44,4 @@ +-- UserWarning +-- FutureWarning +-- OverflowWarning [not generated by the interpreter] + +-- ImportWarning Modified: python/trunk/Lib/test/regrtest.py ============================================================================== --- python/trunk/Lib/test/regrtest.py (original) +++ python/trunk/Lib/test/regrtest.py Fri Apr 28 01:13:20 2006 @@ -138,6 +138,12 @@ warnings.filterwarnings("ignore", "hex/oct constants", FutureWarning, "") +# Ignore ImportWarnings that only occur in the source tree, +# (because of modules with the same name as source-directories in Modules/) +for mod in ("ctypes", "gzip", "test.test_zipimport", "test.test_zlib"): + warnings.filterwarnings(module=".*%s$" % (mod,), + action="ignore", category=ImportWarning) + # MacOSX (a.k.a. Darwin) has a default stack size that is too small # for deeply recursive regular expressions. We see this as crashes in # the Python test suite when running test_re.py and test_sre.py. The Modified: python/trunk/Lib/test/test_import.py ============================================================================== --- python/trunk/Lib/test/test_import.py (original) +++ python/trunk/Lib/test/test_import.py Fri Apr 28 01:13:20 2006 @@ -205,3 +205,20 @@ assert y is test.test_support, y.__name__ test_import_name_binding() + +def test_import_initless_directory_warning(): + import warnings + oldfilters = warnings.filters[:] + warnings.simplefilter('error', ImportWarning); + try: + # Just a random non-package directory we always expect to be + # somewhere in sys.path... + __import__("site-packages") + except ImportWarning: + pass + else: + raise AssertionError + finally: + warnings.filters = oldfilters + +test_import_initless_directory_warning() Modified: python/trunk/Python/exceptions.c ============================================================================== --- python/trunk/Python/exceptions.c (original) +++ python/trunk/Python/exceptions.c Fri Apr 28 01:13:20 2006 @@ -1647,6 +1647,8 @@ "Base class for warnings about constructs that will change semantically " "in the future."); +PyDoc_STRVAR(ImportWarning__doc__, +"Base class for warnings about probable mistakes in module imports"); /* module global functions */ @@ -1719,6 +1721,7 @@ PyObject *PyExc_OverflowWarning; PyObject *PyExc_RuntimeWarning; PyObject *PyExc_FutureWarning; +PyObject *PyExc_ImportWarning; @@ -1818,6 +1821,8 @@ RuntimeWarning__doc__}, {"FutureWarning", &PyExc_FutureWarning, &PyExc_Warning, FutureWarning__doc__}, + {"ImportWarning", &PyExc_ImportWarning, &PyExc_Warning, + ImportWarning__doc__}, /* Sentinel */ {NULL} }; Modified: python/trunk/Python/import.c ============================================================================== --- python/trunk/Python/import.c (original) +++ python/trunk/Python/import.c Fri Apr 28 01:13:20 2006 @@ -1271,19 +1271,42 @@ #ifdef HAVE_STAT if (stat(buf, &statbuf) == 0 && /* it exists */ S_ISDIR(statbuf.st_mode) && /* it's a directory */ - find_init_module(buf) && /* it has __init__.py */ - case_ok(buf, len, namelen, name)) { /* and case matches */ - Py_XDECREF(copy); - return &fd_package; + case_ok(buf, len, namelen, name)) { /* case matches */ + if (find_init_module(buf)) { /* and has __init__.py */ + Py_XDECREF(copy); + return &fd_package; + } + else { + char warnstr[MAXPATHLEN+80]; + sprintf(warnstr, "Not importing directory " + "'%.*s': missing __init__.py", + MAXPATHLEN, buf); + if (PyErr_Warn(PyExc_ImportWarning, + warnstr)) { + Py_XDECREF(copy); + return NULL; + } + } } #else /* XXX How are you going to test for directories? */ #ifdef RISCOS if (isdir(buf) && - find_init_module(buf) && case_ok(buf, len, namelen, name)) { - Py_XDECREF(copy); - return &fd_package; + if (find_init_module(buf)) { + Py_XDECREF(copy); + return &fd_package; + } + else { + char warnstr[MAXPATHLEN+80]; + sprintf(warnstr, "Not importing directory " + "'%.*s': missing __init__.py", + MAXPATHLEN, buf); + if (PyErr_Warn(PyExc_ImportWarning, + warnstr)) { + Py_XDECREF(copy); + return NULL; + } } #endif #endif From buildbot at python.org Fri Apr 28 01:34:50 2006 From: buildbot at python.org (buildbot at python.org) Date: Thu, 27 Apr 2006 23:34:50 +0000 Subject: [Python-checkins] buildbot warnings in alpha Debian trunk Message-ID: <20060427233450.455221E4006@bag.python.org> The Buildbot has detected a new failure of alpha Debian trunk. Full details are available at: http://www.python.org/dev/buildbot/all/alpha%2520Debian%2520trunk/builds/34 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: thomas.wouters Build Had Warnings: warnings test sincerely, -The Buildbot From python-checkins at python.org Fri Apr 28 01:41:28 2006 From: python-checkins at python.org (thomas.wouters) Date: Fri, 28 Apr 2006 01:41:28 +0200 (CEST) Subject: [Python-checkins] r45771 - python/trunk/Lib/test/regrtest.py Message-ID: <20060427234128.640751E4006@bag.python.org> Author: thomas.wouters Date: Fri Apr 28 01:41:27 2006 New Revision: 45771 Modified: python/trunk/Lib/test/regrtest.py Log: Add more ignores of ImportWarnings; these are all just potential triggers (since they won't trigger if zlib is already sucessfully imported); they were found by grepping .py files, instead of looking at warning output :) Modified: python/trunk/Lib/test/regrtest.py ============================================================================== --- python/trunk/Lib/test/regrtest.py (original) +++ python/trunk/Lib/test/regrtest.py Fri Apr 28 01:41:27 2006 @@ -140,7 +140,9 @@ # Ignore ImportWarnings that only occur in the source tree, # (because of modules with the same name as source-directories in Modules/) -for mod in ("ctypes", "gzip", "test.test_zipimport", "test.test_zlib"): +for mod in ("ctypes", "gzip", "zipfile", "tarfile", "encodings.zlib_codec", + "test.test_zipimport", "test.test_zlib", "test.test_zipfile", + "test.test_codecs", "test.string_tests"): warnings.filterwarnings(module=".*%s$" % (mod,), action="ignore", category=ImportWarning) From buildbot at python.org Fri Apr 28 01:53:45 2006 From: buildbot at python.org (buildbot at python.org) Date: Thu, 27 Apr 2006 23:53:45 +0000 Subject: [Python-checkins] buildbot warnings in x86 Ubuntu dapper (icc) trunk Message-ID: <20060427235345.EA99D1E4006@bag.python.org> The Buildbot has detected a new failure of x86 Ubuntu dapper (icc) trunk. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520Ubuntu%2520dapper%2520%2528icc%2529%2520trunk/builds/272 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: thomas.wouters Build Had Warnings: warnings test sincerely, -The Buildbot From buildbot at python.org Fri Apr 28 02:23:28 2006 From: buildbot at python.org (buildbot at python.org) Date: Fri, 28 Apr 2006 00:23:28 +0000 Subject: [Python-checkins] buildbot failure in x86 cygwin trunk Message-ID: <20060428002328.615131E4010@bag.python.org> The Buildbot has detected a new failure of x86 cygwin trunk. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520cygwin%2520trunk/builds/311 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: thomas.wouters BUILD FAILED: failed failed slave lost sincerely, -The Buildbot From buildbot at python.org Fri Apr 28 02:23:33 2006 From: buildbot at python.org (buildbot at python.org) Date: Fri, 28 Apr 2006 00:23:33 +0000 Subject: [Python-checkins] buildbot failure in x86 Ubuntu dapper (icc) trunk Message-ID: <20060428002333.9179A1E400F@bag.python.org> The Buildbot has detected a new failure of x86 Ubuntu dapper (icc) trunk. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520Ubuntu%2520dapper%2520%2528icc%2529%2520trunk/builds/273 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: david.goodger,thomas.wouters BUILD FAILED: failed failed slave lost sincerely, -The Buildbot From buildbot at python.org Fri Apr 28 03:57:01 2006 From: buildbot at python.org (buildbot at python.org) Date: Fri, 28 Apr 2006 01:57:01 +0000 Subject: [Python-checkins] buildbot warnings in alpha Debian trunk Message-ID: <20060428015701.A8D771E4006@bag.python.org> The Buildbot has detected a new failure of alpha Debian trunk. Full details are available at: http://www.python.org/dev/buildbot/all/alpha%2520Debian%2520trunk/builds/36 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: thomas.wouters Build Had Warnings: warnings test sincerely, -The Buildbot From buildbot at python.org Fri Apr 28 05:10:17 2006 From: buildbot at python.org (buildbot at python.org) Date: Fri, 28 Apr 2006 03:10:17 +0000 Subject: [Python-checkins] buildbot warnings in sparc Ubuntu dapper trunk Message-ID: <20060428031017.BED901E4006@bag.python.org> The Buildbot has detected a new failure of sparc Ubuntu dapper trunk. Full details are available at: http://www.python.org/dev/buildbot/all/sparc%2520Ubuntu%2520dapper%2520trunk/builds/176 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: david.goodger,thomas.wouters Build Had Warnings: warnings test sincerely, -The Buildbot From python-checkins at python.org Fri Apr 28 05:39:58 2006 From: python-checkins at python.org (neal.norwitz) Date: Fri, 28 Apr 2006 05:39:58 +0200 (CEST) Subject: [Python-checkins] r45772 - peps/trunk/pep-0000.txt peps/trunk/pep-0209.txt Message-ID: <20060428033958.A577F1E4006@bag.python.org> Author: neal.norwitz Date: Fri Apr 28 05:39:57 2006 New Revision: 45772 Modified: peps/trunk/pep-0000.txt peps/trunk/pep-0209.txt Log: Travis sent me mail asking me to withdraw this PEP. Modified: peps/trunk/pep-0000.txt ============================================================================== --- peps/trunk/pep-0000.txt (original) +++ peps/trunk/pep-0000.txt Fri Apr 28 05:39:57 2006 @@ -73,7 +73,6 @@ Open PEPs (under consideration) - S 209 Adding Multidimensional Arrays Barrett, Oliphant S 228 Reworking Python's Numeric Model Zadka, GvR S 237 Unifying Long Integers and Integers Zadka, GvR S 243 Module Repository Upload Mechanism Reifschneider @@ -178,6 +177,7 @@ SR 204 Range Literals Wouters IW 206 Python Advanced Library Kuchling + SW 209 Adding Multidimensional Arrays Barrett, Oliphant SD 211 Adding A New Outer Product Operator Wilson SD 212 Loop Counter Iteration Schneider-Kamp SD 213 Attribute Access Handlers Prescod @@ -270,7 +270,7 @@ IW 206 Python Advanced Library Kuchling SF 207 Rich Comparisons GvR, Ascher SF 208 Reworking the Coercion Model Schemenauer, Lemburg - S 209 Adding Multidimensional Arrays Barrett, Oliphant + SW 209 Adding Multidimensional Arrays Barrett, Oliphant SD 210 Decoupling the Interpreter Loop Ascher SD 211 Adding A New Outer Product Operator Wilson SD 212 Loop Counter Iteration Schneider-Kamp Modified: peps/trunk/pep-0209.txt ============================================================================== --- peps/trunk/pep-0209.txt (original) +++ peps/trunk/pep-0209.txt Fri Apr 28 05:39:57 2006 @@ -4,7 +4,7 @@ Last-Modified: $Date$ Author: barrett at stsci.edu (Paul Barrett), oliphant at ee.byu.edu (Travis Oliphant) Python-Version: 2.2 -Status: Draft +Status: Withdrawn Type: Standards Track Created: 03-Jan-2001 Post-History: From python-checkins at python.org Fri Apr 28 06:32:21 2006 From: python-checkins at python.org (neal.norwitz) Date: Fri, 28 Apr 2006 06:32:21 +0200 (CEST) Subject: [Python-checkins] r45773 - python/trunk/Lib/pkgutil.py Message-ID: <20060428043221.45DE61E400B@bag.python.org> Author: neal.norwitz Date: Fri Apr 28 06:32:20 2006 New Revision: 45773 Modified: python/trunk/Lib/pkgutil.py Log: Add some whitespace to be more consistent. Modified: python/trunk/Lib/pkgutil.py ============================================================================== --- python/trunk/Lib/pkgutil.py (original) +++ python/trunk/Lib/pkgutil.py Fri Apr 28 06:32:20 2006 @@ -31,7 +31,7 @@ def simplegeneric(func): """Make a trivial single-dispatch generic function""" registry = {} - def wrapper(*args,**kw): + def wrapper(*args, **kw): ob = args[0] try: cls = ob.__class__ @@ -41,18 +41,19 @@ mro = cls.__mro__ except AttributeError: try: - class cls(cls,object): pass + class cls(cls, object): + pass mro = cls.__mro__[1:] except TypeError: mro = object, # must be an ExtensionClass or some such :( for t in mro: if t in registry: - return registry[t](*args,**kw) + return registry[t](*args, **kw) else: - return func(*args,**kw) + return func(*args, **kw) try: wrapper.__name__ = func.__name__ - except (TypeError,AttributeError): + except (TypeError, AttributeError): pass # Python 2.3 doesn't allow functions to be renamed def register(typ, func=None): @@ -70,8 +71,9 @@ def walk_packages(path=None, prefix='', onerror=None): """Yield submodule names+loaders recursively, for path or sys.path""" - def seen(p,m={}): - if p in m: return True + def seen(p, m={}): + if p in m: + return True m[p] = True for importer, name, ispkg in iter_modules(path, prefix): @@ -110,7 +112,7 @@ #@simplegeneric def iter_importer_modules(importer, prefix=''): - if not hasattr(importer,'iter_modules'): + if not hasattr(importer, 'iter_modules'): return [] return importer.iter_modules(prefix) @@ -336,7 +338,7 @@ pass else: importer = None - sys.path_importer_cache.setdefault(path_item,importer) + sys.path_importer_cache.setdefault(path_item, importer) if importer is None: try: @@ -377,7 +379,7 @@ pkg = '.'.join(fullname.split('.')[:-1]) if pkg not in sys.modules: __import__(pkg) - path = getattr(sys.modules[pkg],'__path__',None) or [] + path = getattr(sys.modules[pkg], '__path__', None) or [] else: for importer in sys.meta_path: yield importer @@ -404,7 +406,7 @@ module_or_name = sys.modules[module_or_name] if isinstance(module_or_name, ModuleType): module = module_or_name - loader = getattr(module,'__loader__',None) + loader = getattr(module, '__loader__', None) if loader is not None: return loader fullname = module.__name__ From python-checkins at python.org Fri Apr 28 06:34:44 2006 From: python-checkins at python.org (neal.norwitz) Date: Fri, 28 Apr 2006 06:34:44 +0200 (CEST) Subject: [Python-checkins] r45774 - python/trunk/Lib/test/test_compiler.py Message-ID: <20060428043444.35BA41E4006@bag.python.org> Author: neal.norwitz Date: Fri Apr 28 06:34:43 2006 New Revision: 45774 Modified: python/trunk/Lib/test/test_compiler.py Log: Try to really fix the slow buildbots this time. Printing to stdout, doesn't mean the data was actually written. It depends on the buffering, so we need to flush. This will hopefully really fix the buildbots getting killed due to no output on the slow bots. Modified: python/trunk/Lib/test/test_compiler.py ============================================================================== --- python/trunk/Lib/test/test_compiler.py (original) +++ python/trunk/Lib/test/test_compiler.py Fri Apr 28 06:34:43 2006 @@ -26,6 +26,7 @@ next_time = time.time() + _PRINT_WORKING_MSG_INTERVAL print >>sys.__stdout__, \ ' testCompileLibrary still working, be patient...' + sys.__stdout__.flush() if not basename.endswith(".py"): continue From nnorwitz at gmail.com Fri Apr 28 06:49:14 2006 From: nnorwitz at gmail.com (Neal Norwitz) Date: Thu, 27 Apr 2006 21:49:14 -0700 Subject: [Python-checkins] buildbot warnings in sparc Ubuntu dapper trunk In-Reply-To: <444E1F54.3060102@gmail.com> References: <20060425123140.6AAFE1E4020@bag.python.org> <444E1F54.3060102@gmail.com> Message-ID: On 4/25/06, Nick Coghlan wrote: > > The sparc Ubuntu and alpha Debian buildbot slaves both died again with the 30 > minute timeout in test_compiler, even after Neal's fix the other day. I think I found the problem. I wasn't flushing stdout. So it's possible that nothing was received by the buildbot. Let's see if we see any more spurious failures. n From buildbot at python.org Fri Apr 28 07:18:28 2006 From: buildbot at python.org (buildbot at python.org) Date: Fri, 28 Apr 2006 05:18:28 +0000 Subject: [Python-checkins] buildbot warnings in hppa Ubuntu dapper trunk Message-ID: <20060428051828.ED3541E4006@bag.python.org> The Buildbot has detected a new failure of hppa Ubuntu dapper trunk. Full details are available at: http://www.python.org/dev/buildbot/all/hppa%2520Ubuntu%2520dapper%2520trunk/builds/314 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: neal.norwitz Build Had Warnings: warnings test sincerely, -The Buildbot From buildbot at python.org Fri Apr 28 07:21:02 2006 From: buildbot at python.org (buildbot at python.org) Date: Fri, 28 Apr 2006 05:21:02 +0000 Subject: [Python-checkins] buildbot warnings in x86 cygwin trunk Message-ID: <20060428052102.A26AE1E4006@bag.python.org> The Buildbot has detected a new failure of x86 cygwin trunk. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520cygwin%2520trunk/builds/312 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: neal.norwitz Build Had Warnings: warnings test sincerely, -The Buildbot From python-checkins at python.org Fri Apr 28 07:28:07 2006 From: python-checkins at python.org (neal.norwitz) Date: Fri, 28 Apr 2006 07:28:07 +0200 (CEST) Subject: [Python-checkins] r45775 - in python/trunk: Mac/Modules/file/_Filemodule.c Mac/Modules/mlte/_Mltemodule.c Modules/_sqlite/cursor.c Message-ID: <20060428052807.1D3F61E4006@bag.python.org> Author: neal.norwitz Date: Fri Apr 28 07:28:05 2006 New Revision: 45775 Modified: python/trunk/Mac/Modules/file/_Filemodule.c python/trunk/Mac/Modules/mlte/_Mltemodule.c python/trunk/Modules/_sqlite/cursor.c Log: Fix some warnings on Mac OS X 10.4 Modified: python/trunk/Mac/Modules/file/_Filemodule.c ============================================================================== --- python/trunk/Mac/Modules/file/_Filemodule.c (original) +++ python/trunk/Mac/Modules/file/_Filemodule.c Fri Apr 28 07:28:05 2006 @@ -105,13 +105,14 @@ FSSpec fss2; int tocopy; - err = FSMakeFSSpec(fss->vRefNum, fss->parID, "", &fss2); + err = FSMakeFSSpec(fss->vRefNum, fss->parID, + (unsigned char*)"", &fss2); if (err) return err; err = FSpMakeFSRef(&fss2, &fsr); if (err) return err; - err = (OSErr)FSRefMakePath(&fsr, path, len-1); + err = (OSErr)FSRefMakePath(&fsr, (unsigned char*)path, len-1); if (err) return err; /* This part is not 100% safe: we append the filename part, but @@ -123,12 +124,12 @@ if ((strlen(path) + tocopy) >= len) tocopy = len - strlen(path) - 1; if (tocopy > 0) - strncat(path, fss->name+1, tocopy); + strncat(path, (char*)fss->name+1, tocopy); } else { if (err) return err; - err = (OSErr)FSRefMakePath(&fsr, path, len); + err = (OSErr)FSRefMakePath(&fsr, (unsigned char*)path, len); if (err) return err; } Modified: python/trunk/Mac/Modules/mlte/_Mltemodule.c ============================================================================== --- python/trunk/Mac/Modules/mlte/_Mltemodule.c (original) +++ python/trunk/Mac/Modules/mlte/_Mltemodule.c Fri Apr 28 07:28:05 2006 @@ -49,23 +49,6 @@ } /* -** Parse an optional rect -*/ -static int -OptRectPtr_Convert(PyObject *v, Rect **p_itself) -{ - static Rect r; - - if (v == Py_None) - { - *p_itself = NULL; - return 1; - } - *p_itself = &r; - return PyMac_GetRect(v, *p_itself); -} - -/* ** Parse an optional GWorld */ static int Modified: python/trunk/Modules/_sqlite/cursor.c ============================================================================== --- python/trunk/Modules/_sqlite/cursor.c (original) +++ python/trunk/Modules/_sqlite/cursor.c Fri Apr 28 07:28:05 2006 @@ -27,8 +27,12 @@ #include "sqlitecompat.h" /* used to decide wether to call PyInt_FromLong or PyLong_FromLongLong */ +#ifndef INT32_MIN #define INT32_MIN (-2147483647 - 1) +#endif +#ifndef INT32_MAX #define INT32_MAX 2147483647 +#endif PyObject* cursor_iternext(Cursor *self); From python-checkins at python.org Fri Apr 28 07:28:30 2006 From: python-checkins at python.org (neal.norwitz) Date: Fri, 28 Apr 2006 07:28:30 +0200 (CEST) Subject: [Python-checkins] r45776 - python/trunk/Python/codecs.c Message-ID: <20060428052830.D8E2D1E4006@bag.python.org> Author: neal.norwitz Date: Fri Apr 28 07:28:30 2006 New Revision: 45776 Modified: python/trunk/Python/codecs.c Log: Fix a warning on alpha Modified: python/trunk/Python/codecs.c ============================================================================== --- python/trunk/Python/codecs.c (original) +++ python/trunk/Python/codecs.c Fri Apr 28 07:28:30 2006 @@ -70,7 +70,7 @@ if (ch == ' ') ch = '-'; else - ch = tolower(ch); + ch = tolower(Py_CHARMASK(ch)); p[i] = ch; } return v; From python-checkins at python.org Fri Apr 28 07:28:54 2006 From: python-checkins at python.org (neal.norwitz) Date: Fri, 28 Apr 2006 07:28:54 +0200 (CEST) Subject: [Python-checkins] r45777 - python/trunk/Python/structmember.c Message-ID: <20060428052854.9D1751E4006@bag.python.org> Author: neal.norwitz Date: Fri Apr 28 07:28:54 2006 New Revision: 45777 Modified: python/trunk/Python/structmember.c Log: Fix a warning on ppc (debian) Modified: python/trunk/Python/structmember.c ============================================================================== --- python/trunk/Python/structmember.c (original) +++ python/trunk/Python/structmember.c Fri Apr 28 07:28:54 2006 @@ -260,8 +260,9 @@ PyErr_BadArgument(); return -1; } else { - *(PY_LONG_LONG*)addr = PyLong_AsLongLong(v); - if ((*addr == -1) && PyErr_Occurred()) { + PY_LONG_LONG value; + *(PY_LONG_LONG*)addr = value = PyLong_AsLongLong(v); + if ((value == -1) && PyErr_Occurred()) { return -1; } } @@ -271,8 +272,10 @@ PyErr_BadArgument(); return -1; } else { - *(unsigned PY_LONG_LONG*)addr = PyLong_AsUnsignedLongLong(v); - if ((*addr == -1) && PyErr_Occurred()) { + unsigned PY_LONG_LONG value; + *(unsigned PY_LONG_LONG*)addr = value = PyLong_AsUnsignedLongLong(v); + if ((value == (unsigned PY_LONG_LONG)-1) && + PyErr_Occurred()) { return -1; } } From python-checkins at python.org Fri Apr 28 18:09:47 2006 From: python-checkins at python.org (george.yoshida) Date: Fri, 28 Apr 2006 18:09:47 +0200 (CEST) Subject: [Python-checkins] r45778 - python/trunk/Doc/api/concrete.tex Message-ID: <20060428160947.34FF61E4006@bag.python.org> Author: george.yoshida Date: Fri Apr 28 18:09:45 2006 New Revision: 45778 Modified: python/trunk/Doc/api/concrete.tex Log: fix markup glitch Modified: python/trunk/Doc/api/concrete.tex ============================================================================== --- python/trunk/Doc/api/concrete.tex (original) +++ python/trunk/Doc/api/concrete.tex Fri Apr 28 18:09:45 2006 @@ -1393,7 +1393,7 @@ The \var{mapping} table must map Unicode ordinal integers to Unicode ordinal integers or None (causing deletion of the character). - Mapping tables need only provide the method{__getitem__()} + Mapping tables need only provide the \method{__getitem__()} interface; dictionaries and sequences work well. Unmapped character ordinals (ones which cause a \exception{LookupError}) are left untouched and are copied as-is. From python-checkins at python.org Fri Apr 28 18:11:51 2006 From: python-checkins at python.org (george.yoshida) Date: Fri, 28 Apr 2006 18:11:51 +0200 (CEST) Subject: [Python-checkins] r45779 - python/branches/release24-maint/Doc/api/concrete.tex Message-ID: <20060428161151.BEA231E4014@bag.python.org> Author: george.yoshida Date: Fri Apr 28 18:11:51 2006 New Revision: 45779 Modified: python/branches/release24-maint/Doc/api/concrete.tex Log: fix markup glitch(backport from rev.45778) Modified: python/branches/release24-maint/Doc/api/concrete.tex ============================================================================== --- python/branches/release24-maint/Doc/api/concrete.tex (original) +++ python/branches/release24-maint/Doc/api/concrete.tex Fri Apr 28 18:11:51 2006 @@ -1366,7 +1366,7 @@ The \var{mapping} table must map Unicode ordinal integers to Unicode ordinal integers or None (causing deletion of the character). - Mapping tables need only provide the method{__getitem__()} + Mapping tables need only provide the \method{__getitem__()} interface; dictionaries and sequences work well. Unmapped character ordinals (ones which cause a \exception{LookupError}) are left untouched and are copied as-is. From buildbot at python.org Fri Apr 28 18:17:02 2006 From: buildbot at python.org (buildbot at python.org) Date: Fri, 28 Apr 2006 16:17:02 +0000 Subject: [Python-checkins] buildbot failure in x86 cygwin trunk Message-ID: <20060428161702.416F41E401F@bag.python.org> The Buildbot has detected a new failure of x86 cygwin trunk. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520cygwin%2520trunk/builds/314 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: george.yoshida BUILD FAILED: failed failed slave lost sincerely, -The Buildbot From buildbot at python.org Fri Apr 28 18:17:02 2006 From: buildbot at python.org (buildbot at python.org) Date: Fri, 28 Apr 2006 16:17:02 +0000 Subject: [Python-checkins] buildbot failure in x86 Ubuntu dapper (icc) trunk Message-ID: <20060428161702.437051E4023@bag.python.org> The Buildbot has detected a new failure of x86 Ubuntu dapper (icc) trunk. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520Ubuntu%2520dapper%2520%2528icc%2529%2520trunk/builds/277 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: george.yoshida BUILD FAILED: failed failed slave lost sincerely, -The Buildbot From python-checkins at python.org Fri Apr 28 18:31:18 2006 From: python-checkins at python.org (georg.brandl) Date: Fri, 28 Apr 2006 18:31:18 +0200 (CEST) Subject: [Python-checkins] r45780 - python/trunk/Lib/webbrowser.py Message-ID: <20060428163118.36A511E4006@bag.python.org> Author: georg.brandl Date: Fri Apr 28 18:31:17 2006 New Revision: 45780 Modified: python/trunk/Lib/webbrowser.py Log: Add SeaMonkey to the list of Mozilla browsers. Modified: python/trunk/Lib/webbrowser.py ============================================================================== --- python/trunk/Lib/webbrowser.py (original) +++ python/trunk/Lib/webbrowser.py Fri Apr 28 18:31:17 2006 @@ -447,7 +447,7 @@ # First, the Mozilla/Netscape browsers for browser in ("mozilla-firefox", "firefox", "mozilla-firebird", "firebird", - "mozilla", "netscape"): + "seamonkey", "mozilla", "netscape"): if _iscommand(browser): register(browser, None, Mozilla(browser)) From python-checkins at python.org Fri Apr 28 18:36:55 2006 From: python-checkins at python.org (georg.brandl) Date: Fri, 28 Apr 2006 18:36:55 +0200 (CEST) Subject: [Python-checkins] r45781 - python/trunk/Doc/lib/libposixpath.tex Message-ID: <20060428163655.8C3511E400C@bag.python.org> Author: georg.brandl Date: Fri Apr 28 18:36:55 2006 New Revision: 45781 Modified: python/trunk/Doc/lib/libposixpath.tex Log: Bug #1475009: clarify ntpath.join behavior with absolute components Modified: python/trunk/Doc/lib/libposixpath.tex ============================================================================== --- python/trunk/Doc/lib/libposixpath.tex (original) +++ python/trunk/Doc/lib/libposixpath.tex Fri Apr 28 18:36:55 2006 @@ -146,8 +146,9 @@ \end{funcdesc} \begin{funcdesc}{join}{path1\optional{, path2\optional{, ...}}} -Joins one or more path components intelligently. If any component is -an absolute path, all previous components are thrown away, and joining +Join one or more path components intelligently. If any component is +an absolute path, all previous components (on Windows, including the +previous drive letter, if there was one) are thrown away, and joining continues. The return value is the concatenation of \var{path1}, and optionally \var{path2}, etc., with exactly one directory separator (\code{os.sep}) inserted between components, unless \var{path2} is From python-checkins at python.org Fri Apr 28 18:36:58 2006 From: python-checkins at python.org (georg.brandl) Date: Fri, 28 Apr 2006 18:36:58 +0200 (CEST) Subject: [Python-checkins] r45782 - python/branches/release24-maint/Doc/lib/libposixpath.tex Message-ID: <20060428163658.900671E400C@bag.python.org> Author: georg.brandl Date: Fri Apr 28 18:36:58 2006 New Revision: 45782 Modified: python/branches/release24-maint/Doc/lib/libposixpath.tex Log: Bug #1475009: clarify ntpath.join behavior with absolute components (backport from rev. 45781) Modified: python/branches/release24-maint/Doc/lib/libposixpath.tex ============================================================================== --- python/branches/release24-maint/Doc/lib/libposixpath.tex (original) +++ python/branches/release24-maint/Doc/lib/libposixpath.tex Fri Apr 28 18:36:58 2006 @@ -146,8 +146,9 @@ \end{funcdesc} \begin{funcdesc}{join}{path1\optional{, path2\optional{, ...}}} -Joins one or more path components intelligently. If any component is -an absolute path, all previous components are thrown away, and joining +Join one or more path components intelligently. If any component is +an absolute path, all previous components (on Windows, including the +previous drive letter, if there was one) are thrown away, and joining continues. The return value is the concatenation of \var{path1}, and optionally \var{path2}, etc., with exactly one directory separator (\code{os.sep}) inserted between components, unless \var{path2} is From python-checkins at python.org Fri Apr 28 18:40:14 2006 From: python-checkins at python.org (george.yoshida) Date: Fri, 28 Apr 2006 18:40:14 +0200 (CEST) Subject: [Python-checkins] r45783 - python/trunk/Doc/lib/libtarfile.tex Message-ID: <20060428164014.E09A71E4006@bag.python.org> Author: george.yoshida Date: Fri Apr 28 18:40:14 2006 New Revision: 45783 Modified: python/trunk/Doc/lib/libtarfile.tex Log: correct a dead link Modified: python/trunk/Doc/lib/libtarfile.tex ============================================================================== --- python/trunk/Doc/lib/libtarfile.tex (original) +++ python/trunk/Doc/lib/libtarfile.tex Fri Apr 28 18:40:14 2006 @@ -128,7 +128,7 @@ \seemodule{zipfile}{Documentation of the \refmodule{zipfile} standard module.} - \seetitle[http://www.gnu.org/software/tar/manual/html_chapter/tar_8.html\#SEC134] + \seetitle[http://www.gnu.org/software/tar/manual/html_node/tar_134.html\#SEC134] {GNU tar manual, Basic Tar Format}{Documentation for tar archive files, including GNU tar extensions.} \end{seealso} From python-checkins at python.org Fri Apr 28 18:41:24 2006 From: python-checkins at python.org (george.yoshida) Date: Fri, 28 Apr 2006 18:41:24 +0200 (CEST) Subject: [Python-checkins] r45784 - python/branches/release24-maint/Doc/lib/libtarfile.tex Message-ID: <20060428164124.9AA501E4006@bag.python.org> Author: george.yoshida Date: Fri Apr 28 18:41:24 2006 New Revision: 45784 Modified: python/branches/release24-maint/Doc/lib/libtarfile.tex Log: correct a dead link(backport from rev.45783) Modified: python/branches/release24-maint/Doc/lib/libtarfile.tex ============================================================================== --- python/branches/release24-maint/Doc/lib/libtarfile.tex (original) +++ python/branches/release24-maint/Doc/lib/libtarfile.tex Fri Apr 28 18:41:24 2006 @@ -127,7 +127,7 @@ \seemodule{zipfile}{Documentation of the \refmodule{zipfile} standard module.} - \seetitle[http://www.gnu.org/software/tar/manual/html_chapter/tar_8.html\#SEC134] + \seetitle[http://www.gnu.org/software/tar/manual/html_node/tar_134.html\#SEC134] {GNU tar manual, Basic Tar Format}{Documentation for tar archive files, including GNU tar extensions.} \end{seealso} From buildbot at python.org Fri Apr 28 18:48:43 2006 From: buildbot at python.org (buildbot at python.org) Date: Fri, 28 Apr 2006 16:48:43 +0000 Subject: [Python-checkins] buildbot warnings in x86 OpenBSD 2.4 Message-ID: <20060428164843.F18951E4006@bag.python.org> The Buildbot has detected a new failure of x86 OpenBSD 2.4. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520OpenBSD%25202.4/builds/79 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch branches/release24-maint] HEAD Blamelist: george.yoshida Build Had Warnings: warnings test sincerely, -The Buildbot From python-checkins at python.org Fri Apr 28 18:54:25 2006 From: python-checkins at python.org (georg.brandl) Date: Fri, 28 Apr 2006 18:54:25 +0200 (CEST) Subject: [Python-checkins] r45785 - python/trunk/Lib/shutil.py Message-ID: <20060428165425.64F471E4015@bag.python.org> Author: georg.brandl Date: Fri Apr 28 18:54:25 2006 New Revision: 45785 Modified: python/trunk/Lib/shutil.py Log: Bug #1472949: stringify IOErrors in shutil.copytree when appending them to the Error errors list. Modified: python/trunk/Lib/shutil.py ============================================================================== --- python/trunk/Lib/shutil.py (original) +++ python/trunk/Lib/shutil.py Fri Apr 28 18:54:25 2006 @@ -122,7 +122,7 @@ copy2(srcname, dstname) # XXX What about devices, sockets etc.? except (IOError, os.error), why: - errors.append((srcname, dstname, why)) + errors.append((srcname, dstname, str(why))) # catch the Error from the recursive copytree so that we can # continue with other files except Error, err: From buildbot at python.org Fri Apr 28 18:56:28 2006 From: buildbot at python.org (buildbot at python.org) Date: Fri, 28 Apr 2006 16:56:28 +0000 Subject: [Python-checkins] buildbot warnings in hppa Ubuntu dapper trunk Message-ID: <20060428165628.653DD1E400C@bag.python.org> The Buildbot has detected a new failure of hppa Ubuntu dapper trunk. Full details are available at: http://www.python.org/dev/buildbot/all/hppa%2520Ubuntu%2520dapper%2520trunk/builds/316 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: george.yoshida Build Had Warnings: warnings test sincerely, -The Buildbot From python-checkins at python.org Fri Apr 28 18:58:53 2006 From: python-checkins at python.org (georg.brandl) Date: Fri, 28 Apr 2006 18:58:53 +0200 (CEST) Subject: [Python-checkins] r45786 - python/trunk/Lib/distutils/util.py Message-ID: <20060428165853.5CBD51E4006@bag.python.org> Author: georg.brandl Date: Fri Apr 28 18:58:52 2006 New Revision: 45786 Modified: python/trunk/Lib/distutils/util.py Log: Bug #1478326: don't allow '/' in distutils.util.get_platform machine names since this value is used to name the build directory. Modified: python/trunk/Lib/distutils/util.py ============================================================================== --- python/trunk/Lib/distutils/util.py (original) +++ python/trunk/Lib/distutils/util.py Fri Apr 28 18:58:52 2006 @@ -45,6 +45,7 @@ osname = string.lower(osname) osname = string.replace(osname, '/', '') machine = string.replace(machine, ' ', '_') + machine = string.replace(machine, '/', '-') if osname[:5] == "linux": # At least on Linux/Intel, 'machine' is the processor -- From python-checkins at python.org Fri Apr 28 18:58:56 2006 From: python-checkins at python.org (georg.brandl) Date: Fri, 28 Apr 2006 18:58:56 +0200 (CEST) Subject: [Python-checkins] r45787 - python/branches/release24-maint/Lib/distutils/util.py Message-ID: <20060428165856.9CF461E4006@bag.python.org> Author: georg.brandl Date: Fri Apr 28 18:58:56 2006 New Revision: 45787 Modified: python/branches/release24-maint/Lib/distutils/util.py Log: Bug #1478326: don't allow '/' in distutils.util.get_platform machine names since this value is used to name the build directory. (backport from rev. 45786) Modified: python/branches/release24-maint/Lib/distutils/util.py ============================================================================== --- python/branches/release24-maint/Lib/distutils/util.py (original) +++ python/branches/release24-maint/Lib/distutils/util.py Fri Apr 28 18:58:56 2006 @@ -45,6 +45,7 @@ osname = string.lower(osname) osname = string.replace(osname, '/', '') machine = string.replace(machine, ' ', '_') + machine = string.replace(machine, '/', '-') if osname[:5] == "linux": # At least on Linux/Intel, 'machine' is the processor -- From python-checkins at python.org Fri Apr 28 19:02:19 2006 From: python-checkins at python.org (thomas.heller) Date: Fri, 28 Apr 2006 19:02:19 +0200 (CEST) Subject: [Python-checkins] r45788 - python/trunk/Lib/ctypes/test/test_pointers.py Message-ID: <20060428170219.DE5411E4006@bag.python.org> Author: thomas.heller Date: Fri Apr 28 19:02:18 2006 New Revision: 45788 Modified: python/trunk/Lib/ctypes/test/test_pointers.py Log: Remove a duplicated test (the same test is in test_incomplete.py). Modified: python/trunk/Lib/ctypes/test/test_pointers.py ============================================================================== --- python/trunk/Lib/ctypes/test/test_pointers.py (original) +++ python/trunk/Lib/ctypes/test/test_pointers.py Fri Apr 28 19:02:18 2006 @@ -133,27 +133,6 @@ self.failUnlessEqual(p[0], 42) self.failUnlessEqual(p.contents.value, 42) - def test_incomplete(self): - lpcell = POINTER("cell") - class cell(Structure): - _fields_ = [("value", c_int), - ("next", lpcell)] - SetPointerType(lpcell, cell) - - # Make a structure containing a pointer to itself: - c = cell() - c.value = 42 - c.next = pointer(c) - - result = [] - for i in range(8): - result.append(c.value) - c = c.next[0] - self.failUnlessEqual(result, [42] * 8) - - from ctypes import _pointer_type_cache - del _pointer_type_cache[cell] - def test_charpp( self ): """Test that a character pointer-to-pointer is correctly passed""" dll = CDLL(_ctypes_test.__file__) From buildbot at python.org Fri Apr 28 19:15:58 2006 From: buildbot at python.org (buildbot at python.org) Date: Fri, 28 Apr 2006 17:15:58 +0000 Subject: [Python-checkins] buildbot warnings in x86 Ubuntu dapper (icc) 2.4 Message-ID: <20060428171558.910021E4006@bag.python.org> The Buildbot has detected a new failure of x86 Ubuntu dapper (icc) 2.4. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520Ubuntu%2520dapper%2520%2528icc%2529%25202.4/builds/47 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch branches/release24-maint] HEAD Blamelist: george.yoshida Build Had Warnings: warnings test sincerely, -The Buildbot From buildbot at python.org Fri Apr 28 19:32:16 2006 From: buildbot at python.org (buildbot at python.org) Date: Fri, 28 Apr 2006 17:32:16 +0000 Subject: [Python-checkins] buildbot warnings in x86 XP-2 2.4 Message-ID: <20060428173216.8AFA31E4006@bag.python.org> The Buildbot has detected a new failure of x86 XP-2 2.4. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520XP-2%25202.4/builds/92 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch branches/release24-maint] HEAD Blamelist: georg.brandl,george.yoshida Build Had Warnings: warnings test sincerely, -The Buildbot From buildbot at python.org Fri Apr 28 19:43:15 2006 From: buildbot at python.org (buildbot at python.org) Date: Fri, 28 Apr 2006 17:43:15 +0000 Subject: [Python-checkins] buildbot warnings in x86 cygwin trunk Message-ID: <20060428174315.46A281E400C@bag.python.org> The Buildbot has detected a new failure of x86 cygwin trunk. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520cygwin%2520trunk/builds/315 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: georg.brandl Build Had Warnings: warnings test sincerely, -The Buildbot From python-checkins at python.org Fri Apr 28 20:04:08 2006 From: python-checkins at python.org (phillip.eby) Date: Fri, 28 Apr 2006 20:04:08 +0200 (CEST) Subject: [Python-checkins] r45789 - sandbox/trunk/setuptools/EasyInstall.txt Message-ID: <20060428180408.456081E4018@bag.python.org> Author: phillip.eby Date: Fri Apr 28 20:04:07 2006 New Revision: 45789 Modified: sandbox/trunk/setuptools/EasyInstall.txt Log: Remove out-of-date info about zipfile analysis Modified: sandbox/trunk/setuptools/EasyInstall.txt ============================================================================== --- sandbox/trunk/setuptools/EasyInstall.txt (original) +++ sandbox/trunk/setuptools/EasyInstall.txt Fri Apr 28 20:04:07 2006 @@ -549,10 +549,6 @@ * Possible use of ``inspect`` functions that expect to manipulate source files (e.g. ``inspect.getsource()``) - * Any data files or C extensions (this restriction will be removed in a future - release, once the ``pkg_resources`` runtime has been hardened for multi-user - environments) - If any of the above are found in the package being installed, EasyInstall will assume that the package cannot be safely run from a zipfile, and unzip it to a directory instead. You can override this analysis with the ``-zip-ok`` flag, @@ -786,7 +782,7 @@ EasyInstall will not actually build or install the requested projects or their dependencies; it will just find and extract them for you. See `Editing and Viewing Source Packages`_ above for more details. - + ``--build-directory=DIR, -b DIR`` (UPDATED in 0.6a1) Set the directory used to build source packages. If a package is built from a source distribution or checkout, it will be extracted to a @@ -1085,7 +1081,7 @@ [install] install_lib = ~/py-lib install_scripts = ~/bin - + Be sure to do this *before* you try to run the ``ez_setup.py`` installation script. Then, follow the standard `installation instructions`_, but make sure that ``~/py-lib`` is listed in your ``PYTHONPATH`` environment variable. From python-checkins at python.org Fri Apr 28 20:12:21 2006 From: python-checkins at python.org (phillip.eby) Date: Fri, 28 Apr 2006 20:12:21 +0200 (CEST) Subject: [Python-checkins] r45790 - in sandbox/trunk/setuptools: EasyInstall.txt setuptools/command/bdist_egg.py Message-ID: <20060428181221.9D95C1E4006@bag.python.org> Author: phillip.eby Date: Fri Apr 28 20:12:18 2006 New Revision: 45790 Modified: sandbox/trunk/setuptools/EasyInstall.txt sandbox/trunk/setuptools/setuptools/command/bdist_egg.py Log: Python 2.5 supports -m w/zipped modules, and Python 2.3 has no -m, so there's no need to treat script modules as zip-unsafe unless we're running 2.4 exactly. Modified: sandbox/trunk/setuptools/EasyInstall.txt ============================================================================== --- sandbox/trunk/setuptools/EasyInstall.txt (original) +++ sandbox/trunk/setuptools/EasyInstall.txt Fri Apr 28 20:12:18 2006 @@ -549,6 +549,8 @@ * Possible use of ``inspect`` functions that expect to manipulate source files (e.g. ``inspect.getsource()``) + * Top-level modules that might be scripts used with ``python -m`` (Python 2.4) + If any of the above are found in the package being installed, EasyInstall will assume that the package cannot be safely run from a zipfile, and unzip it to a directory instead. You can override this analysis with the ``-zip-ok`` flag, Modified: sandbox/trunk/setuptools/setuptools/command/bdist_egg.py ============================================================================== --- sandbox/trunk/setuptools/setuptools/command/bdist_egg.py (original) +++ sandbox/trunk/setuptools/setuptools/command/bdist_egg.py Fri Apr 28 20:12:18 2006 @@ -393,7 +393,7 @@ log.warn("%s: module MAY be using inspect.%s", module, bad) safe = False if '__name__' in symbols and '__main__' in symbols and '.' not in module: - if get_python_version()>="2.4": + if sys.version[:3]=="2.4": # -m works w/zipfiles in 2.5 log.warn("%s: top-level module may be 'python -m' script", module) safe = False return safe From python-checkins at python.org Fri Apr 28 20:13:46 2006 From: python-checkins at python.org (phillip.eby) Date: Fri, 28 Apr 2006 20:13:46 +0200 (CEST) Subject: [Python-checkins] r45791 - in sandbox/branches/setuptools-0.6: EasyInstall.txt setuptools/command/bdist_egg.py Message-ID: <20060428181346.7E22D1E4006@bag.python.org> Author: phillip.eby Date: Fri Apr 28 20:13:46 2006 New Revision: 45791 Modified: sandbox/branches/setuptools-0.6/EasyInstall.txt sandbox/branches/setuptools-0.6/setuptools/command/bdist_egg.py Log: Backport doc fixes and 2.4-only -m safety check to 0.6 branch Modified: sandbox/branches/setuptools-0.6/EasyInstall.txt ============================================================================== --- sandbox/branches/setuptools-0.6/EasyInstall.txt (original) +++ sandbox/branches/setuptools-0.6/EasyInstall.txt Fri Apr 28 20:13:46 2006 @@ -545,9 +545,7 @@ * Possible use of ``inspect`` functions that expect to manipulate source files (e.g. ``inspect.getsource()``) - * Any data files or C extensions (this restriction will be removed in a future - release, once the ``pkg_resources`` runtime has been hardened for multi-user - environments) + * Top-level modules that might be scripts used with ``python -m`` (Python 2.4) If any of the above are found in the package being installed, EasyInstall will assume that the package cannot be safely run from a zipfile, and unzip it to @@ -782,7 +780,7 @@ EasyInstall will not actually build or install the requested projects or their dependencies; it will just find and extract them for you. See `Editing and Viewing Source Packages`_ above for more details. - + ``--build-directory=DIR, -b DIR`` (UPDATED in 0.6a1) Set the directory used to build source packages. If a package is built from a source distribution or checkout, it will be extracted to a @@ -1081,7 +1079,7 @@ [install] install_lib = ~/py-lib install_scripts = ~/bin - + Be sure to do this *before* you try to run the ``ez_setup.py`` installation script. Then, follow the standard `installation instructions`_, but make sure that ``~/py-lib`` is listed in your ``PYTHONPATH`` environment variable. Modified: sandbox/branches/setuptools-0.6/setuptools/command/bdist_egg.py ============================================================================== --- sandbox/branches/setuptools-0.6/setuptools/command/bdist_egg.py (original) +++ sandbox/branches/setuptools-0.6/setuptools/command/bdist_egg.py Fri Apr 28 20:13:46 2006 @@ -393,7 +393,7 @@ log.warn("%s: module MAY be using inspect.%s", module, bad) safe = False if '__name__' in symbols and '__main__' in symbols and '.' not in module: - if get_python_version()>="2.4": + if sys.version[:3]=="2.4": # -m works w/zipfiles in 2.5 log.warn("%s: top-level module may be 'python -m' script", module) safe = False return safe From buildbot at python.org Fri Apr 28 20:25:46 2006 From: buildbot at python.org (buildbot at python.org) Date: Fri, 28 Apr 2006 18:25:46 +0000 Subject: [Python-checkins] buildbot warnings in x86 Ubuntu dapper (icc) trunk Message-ID: <20060428182546.80A281E4006@bag.python.org> The Buildbot has detected a new failure of x86 Ubuntu dapper (icc) trunk. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520Ubuntu%2520dapper%2520%2528icc%2529%2520trunk/builds/278 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: georg.brandl Build Had Warnings: warnings test sincerely, -The Buildbot From buildbot at python.org Fri Apr 28 20:31:19 2006 From: buildbot at python.org (buildbot at python.org) Date: Fri, 28 Apr 2006 18:31:19 +0000 Subject: [Python-checkins] buildbot warnings in x86 OpenBSD trunk Message-ID: <20060428183119.E39161E4019@bag.python.org> The Buildbot has detected a new failure of x86 OpenBSD trunk. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520OpenBSD%2520trunk/builds/492 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: georg.brandl,thomas.heller Build Had Warnings: warnings test sincerely, -The Buildbot From buildbot at python.org Fri Apr 28 20:35:14 2006 From: buildbot at python.org (buildbot at python.org) Date: Fri, 28 Apr 2006 18:35:14 +0000 Subject: [Python-checkins] buildbot warnings in x86 XP trunk Message-ID: <20060428183514.EE4191E401E@bag.python.org> The Buildbot has detected a new failure of x86 XP trunk. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520XP%2520trunk/builds/604 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: georg.brandl,george.yoshida,thomas.heller Build Had Warnings: warnings test sincerely, -The Buildbot From buildbot at python.org Fri Apr 28 20:40:26 2006 From: buildbot at python.org (buildbot at python.org) Date: Fri, 28 Apr 2006 18:40:26 +0000 Subject: [Python-checkins] buildbot warnings in ppc Debian unstable 2.4 Message-ID: <20060428184026.55B131E4006@bag.python.org> The Buildbot has detected a new failure of ppc Debian unstable 2.4. Full details are available at: http://www.python.org/dev/buildbot/all/ppc%2520Debian%2520unstable%25202.4/builds/45 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch branches/release24-maint] HEAD Blamelist: georg.brandl Build Had Warnings: warnings test sincerely, -The Buildbot From buildbot at python.org Fri Apr 28 20:44:55 2006 From: buildbot at python.org (buildbot at python.org) Date: Fri, 28 Apr 2006 18:44:55 +0000 Subject: [Python-checkins] buildbot warnings in alpha Debian 2.4 Message-ID: <20060428184455.2621B1E4006@bag.python.org> The Buildbot has detected a new failure of alpha Debian 2.4. Full details are available at: http://www.python.org/dev/buildbot/all/alpha%2520Debian%25202.4/builds/6 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch branches/release24-maint] HEAD Blamelist: george.yoshida Build Had Warnings: warnings test sincerely, -The Buildbot From buildbot at python.org Fri Apr 28 20:44:55 2006 From: buildbot at python.org (buildbot at python.org) Date: Fri, 28 Apr 2006 18:44:55 +0000 Subject: [Python-checkins] buildbot warnings in ia64 Debian unstable 2.4 Message-ID: <20060428184456.085DC1E4006@bag.python.org> The Buildbot has detected a new failure of ia64 Debian unstable 2.4. Full details are available at: http://www.python.org/dev/buildbot/all/ia64%2520Debian%2520unstable%25202.4/builds/44 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch branches/release24-maint] HEAD Blamelist: georg.brandl,george.yoshida Build Had Warnings: warnings test sincerely, -The Buildbot From buildbot at python.org Fri Apr 28 20:49:28 2006 From: buildbot at python.org (buildbot at python.org) Date: Fri, 28 Apr 2006 18:49:28 +0000 Subject: [Python-checkins] buildbot warnings in g4 osx.4 trunk Message-ID: <20060428184928.D86A91E4006@bag.python.org> The Buildbot has detected a new failure of g4 osx.4 trunk. Full details are available at: http://www.python.org/dev/buildbot/all/g4%2520osx.4%2520trunk/builds/580 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: georg.brandl,thomas.heller Build Had Warnings: warnings test sincerely, -The Buildbot From python-checkins at python.org Fri Apr 28 21:09:26 2006 From: python-checkins at python.org (georg.brandl) Date: Fri, 28 Apr 2006 21:09:26 +0200 (CEST) Subject: [Python-checkins] r45792 - in python/trunk: Lib/test/test_datetime.py Modules/datetimemodule.c Message-ID: <20060428190926.0A8921E4006@bag.python.org> Author: georg.brandl Date: Fri Apr 28 21:09:24 2006 New Revision: 45792 Modified: python/trunk/Lib/test/test_datetime.py python/trunk/Modules/datetimemodule.c Log: Bug #1478429: make datetime.datetime.fromtimestamp accept every float, possibly "rounding up" to the next whole second. Modified: python/trunk/Lib/test/test_datetime.py ============================================================================== --- python/trunk/Lib/test/test_datetime.py (original) +++ python/trunk/Lib/test/test_datetime.py Fri Apr 28 21:09:24 2006 @@ -1400,6 +1400,12 @@ got = self.theclass.utcfromtimestamp(ts) self.verify_field_equality(expected, got) + def test_microsecond_rounding(self): + # Test whether fromtimestamp "rounds up" floats that are less + # than one microsecond smaller than an integer. + self.assertEquals(self.theclass.fromtimestamp(0.9999999), + self.theclass.fromtimestamp(1)) + def test_insane_fromtimestamp(self): # It's possible that some platform maps time_t to double, # and that this test will fail there. This test should Modified: python/trunk/Modules/datetimemodule.c ============================================================================== --- python/trunk/Modules/datetimemodule.c (original) +++ python/trunk/Modules/datetimemodule.c Fri Apr 28 21:09:24 2006 @@ -3683,6 +3683,13 @@ return NULL; fraction = timestamp - (double)timet; us = (int)round_to_long(fraction * 1e6); + /* If timestamp is less than one microsecond smaller than a + * full second, round up. Otherwise, ValueErrors are raised + * for some floats. */ + if (us == 1000000) { + timet += 1; + us = 0; + } return datetime_from_timet_and_us(cls, f, timet, us, tzinfo); } From python-checkins at python.org Fri Apr 28 21:09:29 2006 From: python-checkins at python.org (georg.brandl) Date: Fri, 28 Apr 2006 21:09:29 +0200 (CEST) Subject: [Python-checkins] r45793 - in python/branches/release24-maint: Lib/test/test_datetime.py Modules/datetimemodule.c Message-ID: <20060428190929.EFB941E4006@bag.python.org> Author: georg.brandl Date: Fri Apr 28 21:09:29 2006 New Revision: 45793 Modified: python/branches/release24-maint/Lib/test/test_datetime.py python/branches/release24-maint/Modules/datetimemodule.c Log: Bug #1478429: make datetime.datetime.fromtimestamp accept every float, possibly "rounding up" to the next whole second. (backport from rev. 45792) Modified: python/branches/release24-maint/Lib/test/test_datetime.py ============================================================================== --- python/branches/release24-maint/Lib/test/test_datetime.py (original) +++ python/branches/release24-maint/Lib/test/test_datetime.py Fri Apr 28 21:09:29 2006 @@ -1400,6 +1400,12 @@ got = self.theclass.utcfromtimestamp(ts) self.verify_field_equality(expected, got) + def test_microsecond_rounding(self): + # Test whether fromtimestamp "rounds up" floats that are less + # than one microsecond smaller than an integer. + self.assertEquals(self.theclass.fromtimestamp(0.9999999), + self.theclass.fromtimestamp(1)) + def test_insane_fromtimestamp(self): # It's possible that some platform maps time_t to double, # and that this test will fail there. This test should Modified: python/branches/release24-maint/Modules/datetimemodule.c ============================================================================== --- python/branches/release24-maint/Modules/datetimemodule.c (original) +++ python/branches/release24-maint/Modules/datetimemodule.c Fri Apr 28 21:09:29 2006 @@ -3682,6 +3682,13 @@ return NULL; fraction = timestamp - (double)timet; us = (int)round_to_long(fraction * 1e6); + /* If timestamp is less than one microsecond smaller than a + * full second, round up. Otherwise, ValueErrors are raised + * for some floats. */ + if (us == 1000000) { + timet += 1; + us = 0; + } return datetime_from_timet_and_us(cls, f, timet, us, tzinfo); } From buildbot at python.org Fri Apr 28 21:14:36 2006 From: buildbot at python.org (buildbot at python.org) Date: Fri, 28 Apr 2006 19:14:36 +0000 Subject: [Python-checkins] buildbot failure in x86 cygwin 2.4 Message-ID: <20060428191437.0617E1E4006@bag.python.org> The Buildbot has detected a new failure of x86 cygwin 2.4. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520cygwin%25202.4/builds/53 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch branches/release24-maint] HEAD Blamelist: georg.brandl BUILD FAILED: failed failed slave lost sincerely, -The Buildbot From buildbot at python.org Fri Apr 28 21:51:30 2006 From: buildbot at python.org (buildbot at python.org) Date: Fri, 28 Apr 2006 19:51:30 +0000 Subject: [Python-checkins] buildbot warnings in ia64 Debian unstable trunk Message-ID: <20060428195130.712801E4007@bag.python.org> The Buildbot has detected a new failure of ia64 Debian unstable trunk. Full details are available at: http://www.python.org/dev/buildbot/all/ia64%2520Debian%2520unstable%2520trunk/builds/304 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: georg.brandl,thomas.heller Build Had Warnings: warnings test sincerely, -The Buildbot From buildbot at python.org Fri Apr 28 21:54:17 2006 From: buildbot at python.org (buildbot at python.org) Date: Fri, 28 Apr 2006 19:54:17 +0000 Subject: [Python-checkins] buildbot warnings in x86 OpenBSD 2.4 Message-ID: <20060428195418.032101E4007@bag.python.org> The Buildbot has detected a new failure of x86 OpenBSD 2.4. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520OpenBSD%25202.4/builds/82 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch branches/release24-maint] HEAD Blamelist: georg.brandl Build Had Warnings: warnings test sincerely, -The Buildbot From python-checkins at python.org Fri Apr 28 22:06:00 2006 From: python-checkins at python.org (guido.van.rossum) Date: Fri, 28 Apr 2006 22:06:00 +0200 (CEST) Subject: [Python-checkins] r45794 - peps/trunk/pep-3099.txt Message-ID: <20060428200600.27E9E1E400F@bag.python.org> Author: guido.van.rossum Date: Fri Apr 28 22:05:59 2006 New Revision: 45794 Modified: peps/trunk/pep-3099.txt Log: No Unicode for operators or names. Modified: peps/trunk/pep-3099.txt ============================================================================== --- peps/trunk/pep-3099.txt (original) +++ peps/trunk/pep-3099.txt Fri Apr 28 22:05:59 2006 @@ -47,6 +47,12 @@ Thread: "Adding sorting to generator comprehension", http://mail.python.org/pipermail/python-3000/2006-April/001295.html +* Python won't use Unicode characters for anything except string + literals or comments. + + Thread: sets in P3K? + http://mail.python.org/pipermail/python-3000/2006-April/01474.html + Builtins ======== From buildbot at python.org Fri Apr 28 22:12:48 2006 From: buildbot at python.org (buildbot at python.org) Date: Fri, 28 Apr 2006 20:12:48 +0000 Subject: [Python-checkins] buildbot warnings in sparc Ubuntu dapper 2.4 Message-ID: <20060428201248.603BA1E4007@bag.python.org> The Buildbot has detected a new failure of sparc Ubuntu dapper 2.4. Full details are available at: http://www.python.org/dev/buildbot/all/sparc%2520Ubuntu%2520dapper%25202.4/builds/41 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch branches/release24-maint] HEAD Blamelist: george.yoshida Build Had Warnings: warnings test sincerely, -The Buildbot From buildbot at python.org Fri Apr 28 23:30:43 2006 From: buildbot at python.org (buildbot at python.org) Date: Fri, 28 Apr 2006 21:30:43 +0000 Subject: [Python-checkins] buildbot warnings in MIPS Debian 2.4 Message-ID: <20060428213043.E6D281E4015@bag.python.org> The Buildbot has detected a new failure of MIPS Debian 2.4. Full details are available at: http://www.python.org/dev/buildbot/all/MIPS%2520Debian%25202.4/builds/6 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch branches/release24-maint] HEAD Blamelist: george.yoshida Build Had Warnings: warnings test sincerely, -The Buildbot From buildbot at python.org Sat Apr 29 00:45:43 2006 From: buildbot at python.org (buildbot at python.org) Date: Fri, 28 Apr 2006 22:45:43 +0000 Subject: [Python-checkins] buildbot warnings in alpha Debian trunk Message-ID: <20060428224544.0340A1E401B@bag.python.org> The Buildbot has detected a new failure of alpha Debian trunk. Full details are available at: http://www.python.org/dev/buildbot/all/alpha%2520Debian%2520trunk/builds/41 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: georg.brandl Build Had Warnings: warnings test sincerely, -The Buildbot From buildbot at python.org Sat Apr 29 01:00:01 2006 From: buildbot at python.org (buildbot at python.org) Date: Fri, 28 Apr 2006 23:00:01 +0000 Subject: [Python-checkins] buildbot warnings in sparc Ubuntu dapper trunk Message-ID: <20060428230002.079A81E4007@bag.python.org> The Buildbot has detected a new failure of sparc Ubuntu dapper trunk. Full details are available at: http://www.python.org/dev/buildbot/all/sparc%2520Ubuntu%2520dapper%2520trunk/builds/180 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: georg.brandl,george.yoshida,thomas.heller Build Had Warnings: warnings test sincerely, -The Buildbot From buildbot at python.org Sat Apr 29 01:44:10 2006 From: buildbot at python.org (buildbot at python.org) Date: Fri, 28 Apr 2006 23:44:10 +0000 Subject: [Python-checkins] buildbot warnings in x86 cygwin 2.4 Message-ID: <20060428234410.855521E4008@bag.python.org> The Buildbot has detected a new failure of x86 cygwin 2.4. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520cygwin%25202.4/builds/54 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch branches/release24-maint] HEAD Blamelist: georg.brandl Build Had Warnings: warnings test sincerely, -The Buildbot From buildbot at python.org Sat Apr 29 02:14:03 2006 From: buildbot at python.org (buildbot at python.org) Date: Sat, 29 Apr 2006 00:14:03 +0000 Subject: [Python-checkins] buildbot warnings in x86 cygwin trunk Message-ID: <20060429001403.30A7B1E400F@bag.python.org> The Buildbot has detected a new failure of x86 cygwin trunk. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520cygwin%2520trunk/builds/317 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: georg.brandl Build Had Warnings: warnings test sincerely, -The Buildbot From buildbot at python.org Sat Apr 29 02:15:49 2006 From: buildbot at python.org (buildbot at python.org) Date: Sat, 29 Apr 2006 00:15:49 +0000 Subject: [Python-checkins] buildbot warnings in alpha Debian 2.4 Message-ID: <20060429001549.BA2221E4007@bag.python.org> The Buildbot has detected a new failure of alpha Debian 2.4. Full details are available at: http://www.python.org/dev/buildbot/all/alpha%2520Debian%25202.4/builds/8 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch branches/release24-maint] HEAD Blamelist: georg.brandl Build Had Warnings: warnings test sincerely, -The Buildbot From python-checkins at python.org Sat Apr 29 02:34:37 2006 From: python-checkins at python.org (andrew.kuchling) Date: Sat, 29 Apr 2006 02:34:37 +0200 (CEST) Subject: [Python-checkins] r45795 - sandbox/trunk/wsgiref-docs sandbox/trunk/wsgiref-docs/wsgiref.tex Message-ID: <20060429003437.7FFB81E4007@bag.python.org> Author: andrew.kuchling Date: Sat Apr 29 02:34:36 2006 New Revision: 45795 Added: sandbox/trunk/wsgiref-docs/ sandbox/trunk/wsgiref-docs/wsgiref.tex Log: Begin rough outline of wsgiref docs Added: sandbox/trunk/wsgiref-docs/wsgiref.tex ============================================================================== --- (empty file) +++ sandbox/trunk/wsgiref-docs/wsgiref.tex Sat Apr 29 02:34:36 2006 @@ -0,0 +1,51 @@ + +\section{wsgiref} + +XXX write introduction + +\subsection{Web Server Gateway Interface} + +Brief overview of application interface + +app(environ, start_response): + call start_response(status, header) + return iterable w/ text context + +Reference to PEP 333 + +\subsection{wsgiref.handlers} + +BaseHandler + +SimpleHandler + +BaseCGIHandler + +CGIHandler + + +\subsection{wsgiref.util} + +FileWrapper + +% XXX document guess_scheme, application_uri, request_uri, shift_path_info, +% setup_testing_defaults? + +\subsection{wsgiref.simple_server} + +WSGIServer + +WSGIRequestHandler + +demo_app + +\subsection{Examples} + +Simple application: trivial 'hello world' + +Set up a server to run one application + +Other ideas? + + + From python-checkins at python.org Sat Apr 29 04:43:31 2006 From: python-checkins at python.org (george.yoshida) Date: Sat, 29 Apr 2006 04:43:31 +0200 (CEST) Subject: [Python-checkins] r45796 - python/trunk/Doc/lib/librunpy.tex Message-ID: <20060429024331.DD8341E4008@bag.python.org> Author: george.yoshida Date: Sat Apr 29 04:43:30 2006 New Revision: 45796 Modified: python/trunk/Doc/lib/librunpy.tex Log: grammar fix Modified: python/trunk/Doc/lib/librunpy.tex ============================================================================== --- python/trunk/Doc/lib/librunpy.tex (original) +++ python/trunk/Doc/lib/librunpy.tex Sat Apr 29 04:43:30 2006 @@ -10,7 +10,7 @@ \versionadded{2.5} The \module{runpy} module is used to locate and run Python modules -without importing them first. It's main use is to implement the +without importing them first. Its main use is to implement the \programopt{-m} command line switch that allows scripts to be located using the Python module namespace rather than the filesystem. From python-checkins at python.org Sat Apr 29 04:55:12 2006 From: python-checkins at python.org (neal.norwitz) Date: Sat, 29 Apr 2006 04:55:12 +0200 (CEST) Subject: [Python-checkins] r45797 - peps/trunk/pep-0356.txt Message-ID: <20060429025512.40FC01E4007@bag.python.org> Author: neal.norwitz Date: Sat Apr 29 04:55:11 2006 New Revision: 45797 Modified: peps/trunk/pep-0356.txt Log: * Ronald is basically done with the fat Mac builds. * No proposed keywords will have warnings per Guido. He also doesn't see a benefit to a @deprecated decorator. * No owner for bdist_deb means it's not going in. * Guido postponed the python pgen Add owners for missing doc. Modified: peps/trunk/pep-0356.txt ============================================================================== --- peps/trunk/pep-0356.txt (original) +++ peps/trunk/pep-0356.txt Sat Apr 29 04:55:11 2006 @@ -102,41 +102,24 @@ (Owner: A.M. Kuchling. It would still be good if another person would take a look at the new code.) + - Support for building "fat" Mac binaries (Intel and PPC) + Possible features for 2.5 Each feature below should implemented prior to beta1 or will require BDFL approval for inclusion in 2.5. - - SyntaxWarnings for the following proposed keywords: - - - 'do'? (PEP 315) - - 'super' - - 'interface' and 'implements'? (PEP 245) - - 'switch' and 'case'? (PEP 275) - - 'True' and 'False'? - - - Add builtin @deprecated decorator? - - Add @decorator decorator to functional, rename to functools? - Remove the fpectl module? - Modules under consideration for inclusion: - - bdist_deb in distutils package - (Owner: ???) - http://mail.python.org/pipermail/python-dev/2006-February/060926.html - - wsgiref to the standard library (Owner: Phillip Eby) - - pure python pgen module - (Owner: Guido) - - - Support for building "fat" Mac binaries (Intel and PPC) - (Owner: Ronald Oussoren) - - Add new icons for Windows, MacOS and Unix with the new Python logo? + (Owner: ???) Windows: http://mail.python.org/pipermail/python-dev/2006-April/063738.html MacOS: http://hcs.harvard.edu/~jrus/python/prettified-py-icons.png @@ -160,6 +143,9 @@ Deferred until 2.6: + - bdist_deb in distutils package + http://mail.python.org/pipermail/python-dev/2006-February/060926.html + - bdist_egg in distutils package (though bsdist_egg is supported in setuptools) @@ -167,10 +153,10 @@ Open issues - Missing documentation - * ctypes - * ElementTree/cElementTree - * msilib - * pysqlite + * ctypes (Thomas Heller) + * ElementTree/cElementTree (Fredrik Lundh) + * msilib (Martin von Loewis) + * pysqlite (Gerhard Haering) * setuptools (written, needs conversion to proper format) - AST compiler problems From python-checkins at python.org Sat Apr 29 04:55:52 2006 From: python-checkins at python.org (neal.norwitz) Date: Sat, 29 Apr 2006 04:55:52 +0200 (CEST) Subject: [Python-checkins] r45798 - peps/trunk/pep-0356.txt Message-ID: <20060429025552.1ADF01E4007@bag.python.org> Author: neal.norwitz Date: Sat Apr 29 04:55:51 2006 New Revision: 45798 Modified: peps/trunk/pep-0356.txt Log: pgen can go into 2.6 Modified: peps/trunk/pep-0356.txt ============================================================================== --- peps/trunk/pep-0356.txt (original) +++ peps/trunk/pep-0356.txt Sat Apr 29 04:55:51 2006 @@ -149,6 +149,9 @@ - bdist_egg in distutils package (though bsdist_egg is supported in setuptools) + - pure python pgen module + (Owner: Guido) + Open issues From python-checkins at python.org Sat Apr 29 05:22:46 2006 From: python-checkins at python.org (neal.norwitz) Date: Sat, 29 Apr 2006 05:22:46 +0200 (CEST) Subject: [Python-checkins] r45799 - python/branches/release24-maint/Lib/test/test_compiler.py Message-ID: <20060429032246.D626D1E4007@bag.python.org> Author: neal.norwitz Date: Sat Apr 29 05:22:45 2006 New Revision: 45799 Modified: python/branches/release24-maint/Lib/test/test_compiler.py Log: Backport r45774: Try to really fix the slow buildbots this time. Printing to stdout, doesn't mean the data was actually written. It depends on the buffering, so we need to flush. This will hopefully really fix the buildbots getting killed due to no output on the slow bots. Modified: python/branches/release24-maint/Lib/test/test_compiler.py ============================================================================== --- python/branches/release24-maint/Lib/test/test_compiler.py (original) +++ python/branches/release24-maint/Lib/test/test_compiler.py Sat Apr 29 05:22:45 2006 @@ -25,6 +25,7 @@ next_time = time.time() + _PRINT_WORKING_MSG_INTERVAL print >>sys.__stdout__, \ ' testCompileLibrary still working, be patient...' + sys.__stdout__.flush() if not basename.endswith(".py"): continue From buildbot at python.org Sat Apr 29 06:51:03 2006 From: buildbot at python.org (buildbot at python.org) Date: Sat, 29 Apr 2006 04:51:03 +0000 Subject: [Python-checkins] buildbot warnings in hppa Ubuntu dapper trunk Message-ID: <20060429045103.63E7D1E4008@bag.python.org> The Buildbot has detected a new failure of hppa Ubuntu dapper trunk. Full details are available at: http://www.python.org/dev/buildbot/all/hppa%2520Ubuntu%2520dapper%2520trunk/builds/320 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: george.yoshida Build Had Warnings: warnings test sincerely, -The Buildbot From buildbot at python.org Sat Apr 29 10:47:51 2006 From: buildbot at python.org (buildbot at python.org) Date: Sat, 29 Apr 2006 08:47:51 +0000 Subject: [Python-checkins] buildbot warnings in sparc Ubuntu dapper trunk Message-ID: <20060429084751.BD88C1E4008@bag.python.org> The Buildbot has detected a new failure of sparc Ubuntu dapper trunk. Full details are available at: http://www.python.org/dev/buildbot/all/sparc%2520Ubuntu%2520dapper%2520trunk/builds/182 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: george.yoshida Build Had Warnings: warnings test sincerely, -The Buildbot From python-checkins at python.org Sat Apr 29 13:31:37 2006 From: python-checkins at python.org (ronald.oussoren) Date: Sat, 29 Apr 2006 13:31:37 +0200 (CEST) Subject: [Python-checkins] r45800 - in python/trunk: Lib/distutils/sysconfig.py Makefile.pre.in README configure configure.in pyconfig.h.in setup.py Message-ID: <20060429113137.A3DE61E4002@bag.python.org> Author: ronald.oussoren Date: Sat Apr 29 13:31:35 2006 New Revision: 45800 Modified: python/trunk/Lib/distutils/sysconfig.py python/trunk/Makefile.pre.in python/trunk/README python/trunk/configure python/trunk/configure.in python/trunk/pyconfig.h.in python/trunk/setup.py Log: Patch 1471883: --enable-universalsdk on Mac OS X Modified: python/trunk/Lib/distutils/sysconfig.py ============================================================================== --- python/trunk/Lib/distutils/sysconfig.py (original) +++ python/trunk/Lib/distutils/sysconfig.py Sat Apr 29 13:31:35 2006 @@ -366,8 +366,8 @@ # MACOSX_DEPLOYMENT_TARGET: configure bases some choices on it so # it needs to be compatible. # If it isn't set we set it to the configure-time value - if sys.platform == 'darwin' and g.has_key('CONFIGURE_MACOSX_DEPLOYMENT_TARGET'): - cfg_target = g['CONFIGURE_MACOSX_DEPLOYMENT_TARGET'] + if sys.platform == 'darwin' and g.has_key('MACOSX_DEPLOYMENT_TARGET'): + cfg_target = g['MACOSX_DEPLOYMENT_TARGET'] cur_target = os.getenv('MACOSX_DEPLOYMENT_TARGET', '') if cur_target == '': cur_target = cfg_target Modified: python/trunk/Makefile.pre.in ============================================================================== --- python/trunk/Makefile.pre.in (original) +++ python/trunk/Makefile.pre.in Sat Apr 29 13:31:35 2006 @@ -108,13 +108,17 @@ BUILDEXE= @BUILDEXEEXT@ # Short name and location for Mac OS X Python framework +UNIVERSALSDK=@UNIVERSALSDK@ PYTHONFRAMEWORK= @PYTHONFRAMEWORK@ PYTHONFRAMEWORKDIR= @PYTHONFRAMEWORKDIR@ PYTHONFRAMEWORKPREFIX= @PYTHONFRAMEWORKPREFIX@ PYTHONFRAMEWORKINSTALLDIR= @PYTHONFRAMEWORKINSTALLDIR@ # Deployment target selected during configure, to be checked -# by distutils -CONFIGURE_MACOSX_DEPLOYMENT_TARGET=@CONFIGURE_MACOSX_DEPLOYMENT_TARGET@ +# by distutils. The export statement is needed to ensure that the +# deployment target is active during build. +MACOSX_DEPLOYMENT_TARGET=@CONFIGURE_MACOSX_DEPLOYMENT_TARGET@ + at EXPORT_MACOSX_DEPLOYMENT_TARGET@export MACOSX_DEPLOYMENT_TARGET + # Options to enable prebinding (for fast startup prior to Mac OS X 10.3) OTHER_LIBTOOL_OPT=@OTHER_LIBTOOL_OPT@ @@ -377,8 +381,17 @@ $(RESSRCDIR)/version.plist \ $(RESSRCDIR)/English.lproj/InfoPlist.strings $(INSTALL) -d -m $(DIRMODE) $(PYTHONFRAMEWORKDIR)/Versions/$(VERSION) - libtool -o $(LDLIBRARY) -dynamic $(OTHER_LIBTOOL_OPT) $(LIBRARY) \ - @LIBTOOL_CRUFT@ + if test "${UNIVERSALSDK}"; then \ + $(CC) -o $(LDLIBRARY) -arch i386 -arch ppc -dynamiclib \ + -isysroot "${UNIVERSALSDK}" \ + -all_load $(LIBRARY) -Wl,-single_module \ + -install_name $(DESTDIR)$(PYTHONFRAMEWORKINSTALLDIR)/Versions/$(VERSION)/Python \ + -compatibility_version $(VERSION) \ + -current_version $(VERSION); \ + else \ + libtool -o $(LDLIBRARY) -dynamic $(OTHER_LIBTOOL_OPT) $(LIBRARY) \ + @LIBTOOL_CRUFT@ ;\ + fi $(INSTALL) -d -m $(DIRMODE) \ $(PYTHONFRAMEWORKDIR)/Versions/$(VERSION)/Resources/English.lproj $(INSTALL_DATA) $(RESSRCDIR)/Info.plist \ @@ -568,6 +581,19 @@ -$(TESTPYTHON) $(TESTPROG) $(TESTOPTS) -uall $(TESTPYTHON) $(TESTPROG) $(TESTOPTS) -uall +# Run the unitests for both architectures in a Universal build on OSX +# Must be run on an Intel box. +testuniversal: all platform + if [ `arch` != 'i386' ];then \ + echo "This can only be used on OSX/i386" ;\ + exit 1 ;\ + fi + -find $(srcdir)/Lib -name '*.py[co]' -print | xargs rm -f + -$(TESTPYTHON) $(TESTPROG) $(TESTOPTS) -uall + $(TESTPYTHON) $(TESTPROG) $(TESTOPTS) -uall + $(RUNSHARED) /usr/libexec/oah/translate ./$(BUILDPYTHON) -E -tt $(TESTPROG) $(TESTOPTS) -uall + + # Like testall, but with a single pass only buildbottest: all platform $(TESTPYTHON) $(TESTPROG) $(TESTOPTS) -uall -rw Modified: python/trunk/README ============================================================================== --- python/trunk/README (original) +++ python/trunk/README Sat Apr 29 13:31:35 2006 @@ -579,7 +579,12 @@ want to use any Aqua-based GUI toolkit (whether Tkinter, wxPython, Carbon, Cocoa or anything else). - See Mac/OSX/README for more information on framework builds. + You may also want to try the configure option "--enable-universalsdk" + which builds Python as a universal binary with support for the + i386 and PPC architetures. This requires Xcode 2.1 or later to build. + + See Mac/OSX/README for more information on framework and + universal builds. Cygwin: With recent (relative to the time of writing, 2001-12-19) Cygwin installations, there are problems with the interaction Modified: python/trunk/configure ============================================================================== --- python/trunk/configure (original) +++ python/trunk/configure Sat Apr 29 13:31:35 2006 @@ -1,5 +1,5 @@ #! /bin/sh -# From configure.in Revision: 45387 . +# From configure.in Revision: 45392 . # Guess values for system-dependent variables and create Makefiles. # Generated by GNU Autoconf 2.59 for python 2.5. # @@ -312,7 +312,7 @@ # include #endif" -ac_subst_vars='SHELL PATH_SEPARATOR PACKAGE_NAME PACKAGE_TARNAME PACKAGE_VERSION PACKAGE_STRING PACKAGE_BUGREPORT exec_prefix prefix program_transform_name bindir sbindir libexecdir datadir sysconfdir sharedstatedir localstatedir libdir includedir oldincludedir infodir mandir build_alias host_alias target_alias DEFS ECHO_C ECHO_N ECHO_T LIBS VERSION SOVERSION CONFIG_ARGS PYTHONFRAMEWORK PYTHONFRAMEWORKDIR PYTHONFRAMEWORKPREFIX PYTHONFRAMEWORKINSTALLDIR MACHDEP SGI_ABI EXTRAPLATDIR EXTRAMACHDEPPATH CONFIGURE_MACOSX_DEPLOYMENT_TARGET CC CFLAGS LDFLAGS CPPFLAGS ac_ct_CC EXEEXT OBJEXT CXX MAINCC CPP EGREP BUILDEXEEXT LIBRARY LDLIBRARY DLLLIBRARY BLDLIBRARY LDLIBRARYDIR INSTSONAME RUNSHARED LINKCC RANLIB ac_ct_RANLIB AR SVNVERSION INSTALL_PROGRAM INSTALL_SCRIPT INSTALL_DATA LN OPT BASECFLAGS OTHER_LIBTOOL_OPT LIBTOOL_CRUFT SO LDSHARED BLDSHARED CCSHARED LINKFORSHARED CFLAGSFORSHARED SHLIBS USE_SIGNAL_MODULE SIGNAL_OBJS USE_THREAD_MODULE LDLAST THREADOBJ DLINCLDIR DYNLOADFILE MACHDEP_OBJS TRUE LIBOBJS HAVE_GETHOSTBYNAME_R_6_ARG HAVE_GETHOSTBYNAME_R_5_ARG HAVE_GETHOSTBYNAME_R_3_ARG HAVE_GETHOSTBYNAME_R HAVE_GETHOSTBYNAME LIBM LIBC UNICODE_OBJS THREADHEADERS SRCDIRS LTLIBOBJS' +ac_subst_vars='SHELL PATH_SEPARATOR PACKAGE_NAME PACKAGE_TARNAME PACKAGE_VERSION PACKAGE_STRING PACKAGE_BUGREPORT exec_prefix prefix program_transform_name bindir sbindir libexecdir datadir sysconfdir sharedstatedir localstatedir libdir includedir oldincludedir infodir mandir build_alias host_alias target_alias DEFS ECHO_C ECHO_N ECHO_T LIBS VERSION SOVERSION CONFIG_ARGS UNIVERSALSDK PYTHONFRAMEWORK PYTHONFRAMEWORKDIR PYTHONFRAMEWORKPREFIX PYTHONFRAMEWORKINSTALLDIR MACHDEP SGI_ABI EXTRAPLATDIR EXTRAMACHDEPPATH CONFIGURE_MACOSX_DEPLOYMENT_TARGET EXPORT_MACOSX_DEPLOYMENT_TARGET CC CFLAGS LDFLAGS CPPFLAGS ac_ct_CC EXEEXT OBJEXT CXX MAINCC CPP EGREP BUILDEXEEXT LIBRARY LDLIBRARY DLLLIBRARY BLDLIBRARY LDLIBRARYDIR INSTSONAME RUNSHARED LINKCC RANLIB ac_ct_RANLIB AR SVNVERSION INSTALL_PROGRAM INSTALL_SCRIPT INSTALL_DATA LN OPT BASECFLAGS OTHER_LIBTOOL_OPT LIBTOOL_CRUFT SO LDSHARED BLDSHARED CCSHARED LINKFORSHARED CFLAGSFORSHARED SHLIBS USE_SIGNAL_MODULE SIGNAL_OBJS USE_THREAD_MODULE LDLAST THREADOBJ DLINCLDIR DYNLOADFILE MACHDEP_OBJS TRUE LIBOBJS HAVE_GETHOSTBYNAME_R_6_ARG HAVE_GETHOSTBYNAME_R_5_ARG HAVE_GETHOSTBYNAME_R_3_ARG HAVE_GETHOSTBYNAME_R HAVE_GETHOSTBYNAME LIBM LIBC UNICODE_OBJS THREADHEADERS SRCDIRS LTLIBOBJS' ac_subst_files='' # Initialize some variables set by options. @@ -845,6 +845,8 @@ Optional Features: --disable-FEATURE do not include FEATURE (same as --enable-FEATURE=no) --enable-FEATURE[=ARG] include FEATURE [ARG=yes] + --enable-universalsdk[SDKDIR] + Build agains Mac OS X 10.4u SDK (ppc/i386) --enable-framework[=INSTALLDIR] Build (MacOSX|Darwin) framework --enable-shared disable/enable building shared python library @@ -1400,6 +1402,33 @@ CONFIG_ARGS="$ac_configure_args" +# Check whether --enable-universalsdk or --disable-universalsdk was given. +if test "${enable_universalsdk+set}" = set; then + enableval="$enable_universalsdk" + + case $enableval in + yes) + enableval=/Developer/SDKs/MacOSX10.4u.sdk + ;; + esac + case $enableval in + no) + UNIVERSALSDK= + enable_universalsdk= + ;; + *) + UNIVERSALSDK=$enableval + ;; + esac + +else + + UNIVERSALSDK= + enable_universalsdk= + +fi; + + # Check whether --enable-framework or --disable-framework was given. if test "${enable_framework+set}" = set; then enableval="$enable_framework" @@ -1617,7 +1646,9 @@ # it may influence the way we can build extensions, so distutils # needs to check it + CONFIGURE_MACOSX_DEPLOYMENT_TARGET= +EXPORT_MACOSX_DEPLOYMENT_TARGET='#' # checks for alternative programs @@ -3807,6 +3838,10 @@ # is there any other compiler on Darwin besides gcc? Darwin*) BASECFLAGS="$BASECFLAGS -Wno-long-double -no-cpp-precomp -mno-fused-madd" + if test "${enable_universalsdk}"; then + BASECFLAGS="-arch ppc -arch i386 -isysroot ${UNIVERSALSDK} ${BASECFLAGS}" + fi + ;; OSF*) BASECFLAGS="$BASECFLAGS -mieee" @@ -10753,7 +10788,12 @@ case $ac_sys_system/$ac_sys_release in Darwin/[01567]\..*) - LIBTOOL_CRUFT="-framework System -lcc_dynamic -arch_only `arch`" + LIBTOOL_CRUFT="-framework System -lcc_dynamic" + if test "${enable_universalsdk}"; then + : + else + LIBTOOL_CRUFT="${LIBTOOL_CRUFT} -arch_only `arch`" + fi LIBTOOL_CRUFT=$LIBTOOL_CRUFT' -install_name $(PYTHONFRAMEWORKINSTALLDIR)/Versions/$(VERSION)/$(PYTHONFRAMEWORK)' LIBTOOL_CRUFT=$LIBTOOL_CRUFT' -compatibility_version $(VERSION) -current_version $(VERSION)';; Darwin/*) @@ -10888,9 +10928,16 @@ # Use -undefined dynamic_lookup whenever possible (10.3 and later). # This allows an extension to be used in any Python cur_target=`sw_vers -productVersion | sed 's/\(10\.[0-9]*\).*/\1/'` + if test ${cur_target} '>' 10.2; then + cur_target=10.3 + fi CONFIGURE_MACOSX_DEPLOYMENT_TARGET=${MACOSX_DEPLOYMENT_TARGET-${cur_target}} + EXPORT_MACOSX_DEPLOYMENT_TARGET='' if test ${MACOSX_DEPLOYMENT_TARGET-${cur_target}} '>' 10.2 then + if test "${enable_universalsdk}"; then + LDFLAGS="-arch i386 -arch ppc -isysroot ${UNIVERSALSDK} ${LDFLAGS}" + fi LDSHARED='$(CC) $(LDFLAGS) -bundle -undefined dynamic_lookup' BLDSHARED="$LDSHARED" else @@ -20313,6 +20360,8 @@ esac + + # Check whether right shifting a negative integer extends the sign bit # or fills with zeros (like the Cray J90, according to Tim Peters). echo "$as_me:$LINENO: checking whether right shift extends the sign bit" >&5 @@ -22395,6 +22444,7 @@ s, at VERSION@,$VERSION,;t t s, at SOVERSION@,$SOVERSION,;t t s, at CONFIG_ARGS@,$CONFIG_ARGS,;t t +s, at UNIVERSALSDK@,$UNIVERSALSDK,;t t s, at PYTHONFRAMEWORK@,$PYTHONFRAMEWORK,;t t s, at PYTHONFRAMEWORKDIR@,$PYTHONFRAMEWORKDIR,;t t s, at PYTHONFRAMEWORKPREFIX@,$PYTHONFRAMEWORKPREFIX,;t t @@ -22404,6 +22454,7 @@ s, at EXTRAPLATDIR@,$EXTRAPLATDIR,;t t s, at EXTRAMACHDEPPATH@,$EXTRAMACHDEPPATH,;t t s, at CONFIGURE_MACOSX_DEPLOYMENT_TARGET@,$CONFIGURE_MACOSX_DEPLOYMENT_TARGET,;t t +s, at EXPORT_MACOSX_DEPLOYMENT_TARGET@,$EXPORT_MACOSX_DEPLOYMENT_TARGET,;t t s, at CC@,$CC,;t t s, at CFLAGS@,$CFLAGS,;t t s, at LDFLAGS@,$LDFLAGS,;t t Modified: python/trunk/configure.in ============================================================================== --- python/trunk/configure.in (original) +++ python/trunk/configure.in Sat Apr 29 13:31:35 2006 @@ -60,6 +60,29 @@ AC_SUBST(CONFIG_ARGS) CONFIG_ARGS="$ac_configure_args" +AC_ARG_ENABLE(universalsdk, + AC_HELP_STRING(--enable-universalsdk@<:@SDKDIR@:>@, Build agains Mac OS X 10.4u SDK (ppc/i386)), +[ + case $enableval in + yes) + enableval=/Developer/SDKs/MacOSX10.4u.sdk + ;; + esac + case $enableval in + no) + UNIVERSALSDK= + enable_universalsdk= + ;; + *) + UNIVERSALSDK=$enableval + ;; + esac +],[ + UNIVERSALSDK= + enable_universalsdk= +]) +AC_SUBST(UNIVERSALSDK) + dnl quadrigraphs "@<:@" and "@:>@" produce "[" and "]" in the output AC_ARG_ENABLE(framework, AC_HELP_STRING(--enable-framework@<:@=INSTALLDIR@:>@, Build (MacOSX|Darwin) framework), @@ -258,7 +281,9 @@ # it may influence the way we can build extensions, so distutils # needs to check it AC_SUBST(CONFIGURE_MACOSX_DEPLOYMENT_TARGET) +AC_SUBST(EXPORT_MACOSX_DEPLOYMENT_TARGET) CONFIGURE_MACOSX_DEPLOYMENT_TARGET= +EXPORT_MACOSX_DEPLOYMENT_TARGET='#' # checks for alternative programs @@ -740,6 +765,10 @@ # is there any other compiler on Darwin besides gcc? Darwin*) BASECFLAGS="$BASECFLAGS -Wno-long-double -no-cpp-precomp -mno-fused-madd" + if test "${enable_universalsdk}"; then + BASECFLAGS="-arch ppc -arch i386 -isysroot ${UNIVERSALSDK} ${BASECFLAGS}" + fi + ;; OSF*) BASECFLAGS="$BASECFLAGS -mieee" @@ -1263,7 +1292,12 @@ AC_SUBST(LIBTOOL_CRUFT) case $ac_sys_system/$ac_sys_release in Darwin/@<:@01567@:>@\..*) - LIBTOOL_CRUFT="-framework System -lcc_dynamic -arch_only `arch`" + LIBTOOL_CRUFT="-framework System -lcc_dynamic" + if test "${enable_universalsdk}"; then + : + else + LIBTOOL_CRUFT="${LIBTOOL_CRUFT} -arch_only `arch`" + fi LIBTOOL_CRUFT=$LIBTOOL_CRUFT' -install_name $(PYTHONFRAMEWORKINSTALLDIR)/Versions/$(VERSION)/$(PYTHONFRAMEWORK)' LIBTOOL_CRUFT=$LIBTOOL_CRUFT' -compatibility_version $(VERSION) -current_version $(VERSION)';; Darwin/*) @@ -1386,9 +1420,16 @@ # Use -undefined dynamic_lookup whenever possible (10.3 and later). # This allows an extension to be used in any Python cur_target=`sw_vers -productVersion | sed 's/\(10\.[[0-9]]*\).*/\1/'` + if test ${cur_target} '>' 10.2; then + cur_target=10.3 + fi CONFIGURE_MACOSX_DEPLOYMENT_TARGET=${MACOSX_DEPLOYMENT_TARGET-${cur_target}} + EXPORT_MACOSX_DEPLOYMENT_TARGET='' if test ${MACOSX_DEPLOYMENT_TARGET-${cur_target}} '>' 10.2 then + if test "${enable_universalsdk}"; then + LDFLAGS="-arch i386 -arch ppc -isysroot ${UNIVERSALSDK} ${LDFLAGS}" + fi LDSHARED='$(CC) $(LDFLAGS) -bundle -undefined dynamic_lookup' BLDSHARED="$LDSHARED" else @@ -2873,6 +2914,23 @@ # check for endianness AC_C_BIGENDIAN +AH_VERBATIM([WORDS_BIGENDIAN], +[ + /* Define to 1 if your processor stores words with the most significant byte + first (like Motorola and SPARC, unlike Intel and VAX). + + The block below does compile-time checking for endianness on platforms + that use GCC and therefore allows compiling fat binaries on OSX by using + '-arch ppc -arch i386' as the compile flags. The phrasing was choosen + such that the configure-result is used on systems that don't use GCC. + */ +#ifdef __BIG_ENDIAN__ +#define WORDS_BIGENDIAN 1 +#else +#ifndef __LITTLE_ENDIAN__ +#undef WORDS_BIGENDIAN +#endif +#endif]) # Check whether right shifting a negative integer extends the sign bit # or fills with zeros (like the Cray J90, according to Tim Peters). Modified: python/trunk/pyconfig.h.in ============================================================================== --- python/trunk/pyconfig.h.in (original) +++ python/trunk/pyconfig.h.in Sat Apr 29 13:31:35 2006 @@ -846,9 +846,22 @@ /* Define to profile with the Pentium timestamp counter */ #undef WITH_TSC -/* Define to 1 if your processor stores words with the most significant byte - first (like Motorola and SPARC, unlike Intel and VAX). */ + + /* Define to 1 if your processor stores words with the most significant byte + first (like Motorola and SPARC, unlike Intel and VAX). + + The block below does compile-time checking for endianness on platforms + that use GCC and therefore allows compiling fat binaries on OSX by using + '-arch ppc -arch i386' as the compile flags. The phrasing was choosen + such that the configure-result is used on systems that don't use GCC. + */ +#ifdef __BIG_ENDIAN__ +#define WORDS_BIGENDIAN 1 +#else +#ifndef __LITTLE_ENDIAN__ #undef WORDS_BIGENDIAN +#endif +#endif /* Define to 1 if on AIX 3. System headers sometimes define this. Modified: python/trunk/setup.py ============================================================================== --- python/trunk/setup.py (original) +++ python/trunk/setup.py Sat Apr 29 13:31:35 2006 @@ -913,20 +913,11 @@ # # More information on Expat can be found at www.libexpat.org. # - if sys.byteorder == "little": - xmlbo = "1234" - else: - xmlbo = "4321" expatinc = os.path.join(os.getcwd(), srcdir, 'Modules', 'expat') define_macros = [ - ('XML_NS', '1'), - ('XML_DTD', '1'), - ('BYTEORDER', xmlbo), - ('XML_CONTEXT_BYTES','1024'), - ] - for feature_macro in ['HAVE_MEMMOVE', 'HAVE_BCOPY']: - if config_h_vars.has_key(feature_macro): - define_macros.append((feature_macro, '1')) + ('HAVE_EXPAT_CONFIG_H', '1'), + ] + exts.append(Extension('pyexpat', define_macros = define_macros, include_dirs = [expatinc], From python-checkins at python.org Sat Apr 29 13:53:15 2006 From: python-checkins at python.org (andrew.kuchling) Date: Sat, 29 Apr 2006 13:53:15 +0200 (CEST) Subject: [Python-checkins] r45801 - python/trunk/Doc/whatsnew/whatsnew25.tex Message-ID: <20060429115315.D94691E4002@bag.python.org> Author: andrew.kuchling Date: Sat Apr 29 13:53:15 2006 New Revision: 45801 Modified: python/trunk/Doc/whatsnew/whatsnew25.tex Log: Add item Modified: python/trunk/Doc/whatsnew/whatsnew25.tex ============================================================================== --- python/trunk/Doc/whatsnew/whatsnew25.tex (original) +++ python/trunk/Doc/whatsnew/whatsnew25.tex Sat Apr 29 13:53:15 2006 @@ -1962,6 +1962,11 @@ now uses the \cfunction{dlopen()} function instead of MacOS-specific functions. +\item MacOS X: a \longprogramopt{enable-universalsdk} switch was added +to the \program{configure} script that compiles the interpreter as a +universal binary able to run on both PowerPC and Intel processors. +(Contributed by Ronald Oussoren.) + \item Windows: \file{.dll} is no longer supported as a filename extension for extension modules. \file{.pyd} is now the only filename extension that will be searched for. From buildbot at python.org Sat Apr 29 13:53:50 2006 From: buildbot at python.org (buildbot at python.org) Date: Sat, 29 Apr 2006 11:53:50 +0000 Subject: [Python-checkins] buildbot warnings in x86 gentoo trunk Message-ID: <20060429115350.D36B91E4002@bag.python.org> The Buildbot has detected a new failure of x86 gentoo trunk. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520gentoo%2520trunk/builds/659 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: ronald.oussoren Build Had Warnings: warnings test sincerely, -The Buildbot From buildbot at python.org Sat Apr 29 13:53:53 2006 From: buildbot at python.org (buildbot at python.org) Date: Sat, 29 Apr 2006 11:53:53 +0000 Subject: [Python-checkins] buildbot warnings in amd64 gentoo trunk Message-ID: <20060429115354.011AE1E4002@bag.python.org> The Buildbot has detected a new failure of amd64 gentoo trunk. Full details are available at: http://www.python.org/dev/buildbot/all/amd64%2520gentoo%2520trunk/builds/679 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: ronald.oussoren Build Had Warnings: warnings test sincerely, -The Buildbot From buildbot at python.org Sat Apr 29 13:57:09 2006 From: buildbot at python.org (buildbot at python.org) Date: Sat, 29 Apr 2006 11:57:09 +0000 Subject: [Python-checkins] buildbot warnings in x86 OpenBSD trunk Message-ID: <20060429115709.5D0961E4002@bag.python.org> The Buildbot has detected a new failure of x86 OpenBSD trunk. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520OpenBSD%2520trunk/builds/495 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: ronald.oussoren Build Had Warnings: warnings test sincerely, -The Buildbot From python-checkins at python.org Sat Apr 29 14:10:30 2006 From: python-checkins at python.org (andrew.kuchling) Date: Sat, 29 Apr 2006 14:10:30 +0200 (CEST) Subject: [Python-checkins] r45802 - python/trunk/Doc/whatsnew/whatsnew23.tex Message-ID: <20060429121030.2F03B1E4002@bag.python.org> Author: andrew.kuchling Date: Sat Apr 29 14:10:28 2006 New Revision: 45802 Modified: python/trunk/Doc/whatsnew/whatsnew23.tex Log: Make case of 'ZIP' consistent Modified: python/trunk/Doc/whatsnew/whatsnew23.tex ============================================================================== --- python/trunk/Doc/whatsnew/whatsnew23.tex (original) +++ python/trunk/Doc/whatsnew/whatsnew23.tex Sat Apr 29 14:10:28 2006 @@ -318,7 +318,7 @@ %====================================================================== -\section{PEP 273: Importing Modules from Zip Archives} +\section{PEP 273: Importing Modules from ZIP Archives} The new \module{zipimport} module adds support for importing modules from a ZIP-format archive. You don't need to import the From python-checkins at python.org Sat Apr 29 14:10:43 2006 From: python-checkins at python.org (andrew.kuchling) Date: Sat, 29 Apr 2006 14:10:43 +0200 (CEST) Subject: [Python-checkins] r45803 - python/trunk/Doc/whatsnew/whatsnew25.tex Message-ID: <20060429121043.D9EE61E4002@bag.python.org> Author: andrew.kuchling Date: Sat Apr 29 14:10:43 2006 New Revision: 45803 Modified: python/trunk/Doc/whatsnew/whatsnew25.tex Log: Add item Modified: python/trunk/Doc/whatsnew/whatsnew25.tex ============================================================================== --- python/trunk/Doc/whatsnew/whatsnew25.tex (original) +++ python/trunk/Doc/whatsnew/whatsnew25.tex Sat Apr 29 14:10:43 2006 @@ -2,7 +2,6 @@ \usepackage{distutils} % $Id$ -% Describe the pkgutil module % Fix XXX comments % Count up the patches and bugs @@ -1436,6 +1435,11 @@ arguments instead. The ability to return \code{None} was deprecated in Python 2.4, so this completes the removal of the feature. +\item The \module{pkgutil} module, containing various utility +functions for finding packages, was enhanced to support PEP 302's +import hooks and now also works for packages stored in ZIP-format archives. +(Contributed by Phillip J. Eby.) + \item The old \module{regex} and \module{regsub} modules, which have been deprecated ever since Python 2.0, have finally been deleted. Other deleted modules: \module{statcache}, \module{tzparse}, From buildbot at python.org Sat Apr 29 14:11:29 2006 From: buildbot at python.org (buildbot at python.org) Date: Sat, 29 Apr 2006 12:11:29 +0000 Subject: [Python-checkins] buildbot warnings in g4 osx.4 trunk Message-ID: <20060429121129.C03731E4002@bag.python.org> The Buildbot has detected a new failure of g4 osx.4 trunk. Full details are available at: http://www.python.org/dev/buildbot/all/g4%2520osx.4%2520trunk/builds/583 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: ronald.oussoren Build Had Warnings: warnings test sincerely, -The Buildbot From buildbot at python.org Sat Apr 29 14:13:26 2006 From: buildbot at python.org (buildbot at python.org) Date: Sat, 29 Apr 2006 12:13:26 +0000 Subject: [Python-checkins] buildbot warnings in sparc solaris10 gcc trunk Message-ID: <20060429121326.EDD931E4002@bag.python.org> The Buildbot has detected a new failure of sparc solaris10 gcc trunk. Full details are available at: http://www.python.org/dev/buildbot/all/sparc%2520solaris10%2520gcc%2520trunk/builds/588 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: ronald.oussoren Build Had Warnings: warnings test sincerely, -The Buildbot From python-checkins at python.org Sat Apr 29 14:20:31 2006 From: python-checkins at python.org (marc-andre.lemburg) Date: Sat, 29 Apr 2006 14:20:31 +0200 (CEST) Subject: [Python-checkins] r45804 - peps/trunk/pep-0356.txt Message-ID: <20060429122031.BF0101E4002@bag.python.org> Author: marc-andre.lemburg Date: Sat Apr 29 14:20:31 2006 New Revision: 45804 Modified: peps/trunk/pep-0356.txt Log: Add note about added pybench benchmark. Modified: peps/trunk/pep-0356.txt ============================================================================== --- peps/trunk/pep-0356.txt (original) +++ peps/trunk/pep-0356.txt Sat Apr 29 14:20:31 2006 @@ -98,6 +98,8 @@ - Added PEP 302 zipfile/__loader__ support to the following modules: warnings, linecache, inspect, traceback, site, and doctest + - Added pybench Python benchmark suite -- by Marc-Andre Lemburg + - Add write support for mailboxes from the code in sandbox/mailbox. (Owner: A.M. Kuchling. It would still be good if another person would take a look at the new code.) From python-checkins at python.org Sat Apr 29 14:21:14 2006 From: python-checkins at python.org (marc-andre.lemburg) Date: Sat, 29 Apr 2006 14:21:14 +0200 (CEST) Subject: [Python-checkins] r45805 - peps/trunk/pep-0291.txt Message-ID: <20060429122114.253661E4002@bag.python.org> Author: marc-andre.lemburg Date: Sat Apr 29 14:21:13 2006 New Revision: 45805 Modified: peps/trunk/pep-0291.txt Log: Add note regarding pybench benchmark. Modified: peps/trunk/pep-0291.txt ============================================================================== --- peps/trunk/pep-0291.txt (original) +++ peps/trunk/pep-0291.txt Sat Apr 29 14:21:13 2006 @@ -77,6 +77,8 @@ decorators, int/long unification, set/frozenset, reversed(), sorted(), "".rsplit() + 2.4 ??? + Backward Compatible Packages, Modules, and Tools @@ -93,6 +95,7 @@ Just van Rossum pkgutil Phillip Eby 2.3 platform Marc-Andre Lemburg 1.5.2 + pybench Marc-Andre Lemburg 1.5.2 [3] sre Fredrik Lundh 2.1 subprocess Peter Astrand 2.2 xml (PyXML) Martin v. Loewis 2.0 @@ -116,6 +119,12 @@ The decision will be revisited for Python 2.5 and not changed unless compelling advantages arise. + [3] pybench lives under the Tools/ directory. Compatibility with + older Python version is needed in order to be able to compare + performance between Python versions. New features may still + be used in new tests, which may then be configured to fail + gracefully on import by the tool in older Python versions. + Copyright From python-checkins at python.org Sat Apr 29 14:22:05 2006 From: python-checkins at python.org (marc-andre.lemburg) Date: Sat, 29 Apr 2006 14:22:05 +0200 (CEST) Subject: [Python-checkins] r45806 - peps/trunk/pep-0297.txt Message-ID: <20060429122205.0B1221E4002@bag.python.org> Author: marc-andre.lemburg Date: Sat Apr 29 14:22:04 2006 New Revision: 45806 Modified: peps/trunk/pep-0297.txt Log: Retarget PEP 297 for Python 2.6. Add comment about the possibility to use the strategy for hot-fixes. Modified: peps/trunk/pep-0297.txt ============================================================================== --- peps/trunk/pep-0297.txt (original) +++ peps/trunk/pep-0297.txt Sat Apr 29 14:22:04 2006 @@ -5,7 +5,7 @@ Author: mal at lemburg.com (Marc-Andr? Lemburg) Status: Draft Type: Standards Track -Python-Version: 2.3 +Python-Version: 2.6 Created: 19-Jul-2001 Post-History: @@ -32,6 +32,9 @@ packages without having to wait for a new patch level release of the Python interpreter to bring along the changes. + On some occasions, it may also be desirable to update modules of + the standard library without going through the whole Python release + cycle, e.g. in order to provide hot-fixes for security problems. Proposed Solutions @@ -79,7 +82,7 @@ Scope - Solution 1: Python 2.3 and up + Solution 1: Python 2.6 and up Solution 2: all Python versions supported by distutils From python-checkins at python.org Sat Apr 29 14:24:06 2006 From: python-checkins at python.org (marc-andre.lemburg) Date: Sat, 29 Apr 2006 14:24:06 +0200 (CEST) Subject: [Python-checkins] r45807 - peps/trunk/pep-0275.txt Message-ID: <20060429122406.1741C1E4008@bag.python.org> Author: marc-andre.lemburg Date: Sat Apr 29 14:24:05 2006 New Revision: 45807 Modified: peps/trunk/pep-0275.txt Log: Retarget PEP 275 for Python 2.6. Modified: peps/trunk/pep-0275.txt ============================================================================== --- peps/trunk/pep-0275.txt (original) +++ peps/trunk/pep-0275.txt Sat Apr 29 14:24:05 2006 @@ -5,7 +5,7 @@ Author: mal at lemburg.com (Marc-Andr? Lemburg) Status: Draft Type: Standards Track -Python-Version: 2.4 +Python-Version: 2.6 Created: 10-Nov-2001 Post-History: @@ -17,7 +17,7 @@ Problem - Up to Python 2.3, the typical way of writing multi-value switches + Up to Python 2.5, the typical way of writing multi-value switches has been to use long switch constructs of the following type: if x == 'first state': From python-checkins at python.org Sat Apr 29 14:37:26 2006 From: python-checkins at python.org (martin.v.loewis) Date: Sat, 29 Apr 2006 14:37:26 +0200 (CEST) Subject: [Python-checkins] r45808 - in python/trunk: Misc/NEWS Modules/expat/expat_config.h Message-ID: <20060429123726.4A7771E4013@bag.python.org> Author: martin.v.loewis Date: Sat Apr 29 14:37:25 2006 New Revision: 45808 Added: python/trunk/Modules/expat/expat_config.h (contents, props changed) Modified: python/trunk/Misc/NEWS Log: Further changes for #1471883: Edit Misc/NEWS, and add expat_config.h. Modified: python/trunk/Misc/NEWS ============================================================================== --- python/trunk/Misc/NEWS (original) +++ python/trunk/Misc/NEWS Sat Apr 29 14:37:25 2006 @@ -142,6 +142,8 @@ Build ----- +- Patch #1471883: Add --enable-universalsdk. + - The Makefile now has a reindent target, which runs reindent.py on the library. Added: python/trunk/Modules/expat/expat_config.h ============================================================================== --- (empty file) +++ python/trunk/Modules/expat/expat_config.h Sat Apr 29 14:37:25 2006 @@ -0,0 +1,19 @@ +/* + * Expat configuration for python. This file is not part of the expat + * distribution. + */ +#ifndef EXPAT_CONFIG_H +#define EXPAT_CONFIG_H + +#include +#ifdef WORDS_BIGENDIAN +#define BYTEORDER 4321 +#else +#define BYTEORDER 1234 +#endif + +#define XML_NS 1 +#define XML_DTD 1 +#define XML_CONTEXT_BYTES 1024 + +#endif /* EXPAT_CONFIG_H */ From python-checkins at python.org Sat Apr 29 23:29:50 2006 From: python-checkins at python.org (brett.cannon) Date: Sat, 29 Apr 2006 23:29:50 +0200 (CEST) Subject: [Python-checkins] r45809 - python/trunk/Lib/contextlib.py Message-ID: <20060429212950.A9CEB1E4008@bag.python.org> Author: brett.cannon Date: Sat Apr 29 23:29:50 2006 New Revision: 45809 Modified: python/trunk/Lib/contextlib.py Log: Fix docstring for contextfactory; mentioned old contextmanager name. Modified: python/trunk/Lib/contextlib.py ============================================================================== --- python/trunk/Lib/contextlib.py (original) +++ python/trunk/Lib/contextlib.py Sat Apr 29 23:29:50 2006 @@ -53,7 +53,7 @@ Typical usage: - @contextmanager + @contextfactory def some_generator(): try: From buildbot at python.org Sun Apr 30 00:13:13 2006 From: buildbot at python.org (buildbot at python.org) Date: Sat, 29 Apr 2006 22:13:13 +0000 Subject: [Python-checkins] buildbot warnings in ppc Debian unstable trunk Message-ID: <20060429221313.338DB1E4008@bag.python.org> The Buildbot has detected a new failure of ppc Debian unstable trunk. Full details are available at: http://www.python.org/dev/buildbot/all/ppc%2520Debian%2520unstable%2520trunk/builds/326 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: brett.cannon Build Had Warnings: warnings test sincerely, -The Buildbot From buildbot at python.org Sun Apr 30 00:20:10 2006 From: buildbot at python.org (buildbot at python.org) Date: Sat, 29 Apr 2006 22:20:10 +0000 Subject: [Python-checkins] buildbot warnings in alpha Tru64 5.1 trunk Message-ID: <20060429222010.E36DB1E4008@bag.python.org> The Buildbot has detected a new failure of alpha Tru64 5.1 trunk. Full details are available at: http://www.python.org/dev/buildbot/all/alpha%2520Tru64%25205.1%2520trunk/builds/382 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: brett.cannon Build Had Warnings: warnings test sincerely, -The Buildbot From python-checkins at python.org Sun Apr 30 01:12:42 2006 From: python-checkins at python.org (gerhard.haering) Date: Sun, 30 Apr 2006 01:12:42 +0200 (CEST) Subject: [Python-checkins] r45810 - in python/trunk/Doc: Makefile.deps lib/lib.tex lib/libsqlite3.tex Message-ID: <20060429231242.6CAEC1E400D@bag.python.org> Author: gerhard.haering Date: Sun Apr 30 01:12:41 2006 New Revision: 45810 Added: python/trunk/Doc/lib/libsqlite3.tex Modified: python/trunk/Doc/Makefile.deps python/trunk/Doc/lib/lib.tex Log: This is the start of documentation for the sqlite3 module. Please feel free to find a better place for the link to it than alongside bsddb & friends. Modified: python/trunk/Doc/Makefile.deps ============================================================================== --- python/trunk/Doc/Makefile.deps (original) +++ python/trunk/Doc/Makefile.deps Sun Apr 30 01:12:41 2006 @@ -350,7 +350,8 @@ lib/libturtle.tex \ lib/libtarfile.tex \ lib/libcsv.tex \ - lib/libcfgparser.tex + lib/libcfgparser.tex \ + lib/libsqlite3.tex # LaTeX source files for Macintosh Library Modules. MACFILES= $(HOWTOSTYLES) $(INDEXSTYLES) $(COMMONTEX) \ Modified: python/trunk/Doc/lib/lib.tex ============================================================================== --- python/trunk/Doc/lib/lib.tex (original) +++ python/trunk/Doc/lib/lib.tex Sun Apr 30 01:12:41 2006 @@ -224,6 +224,7 @@ \input{libdbhash} \input{libbsddb} \input{libdumbdbm} +\input{libsqlite3} % ============= Added: python/trunk/Doc/lib/libsqlite3.tex ============================================================================== --- (empty file) +++ python/trunk/Doc/lib/libsqlite3.tex Sun Apr 30 01:12:41 2006 @@ -0,0 +1,105 @@ +\section{\module{sqlite3} --- + DB-API 2.0 interface for SQLite databases} + +\declaremodule{builtin}{sqlite3} +\modulesynopsis{A DB-API 2.0 interface based on SQLite 3.x.} + + + +The module defines the following: + +\begin{datadesc}{PARSE_DECLTYPES} +This constant is meant to be used with the detect_types parameter of the connect function. + +Setting it makes the sqlite3 module parse the declared type for each column it +returns. It will parse out the first word of the declared type, i. e. for +"integer primary key", it will parse out "integer". Then for that column, it +will look into pysqlite's converters dictionary and use the converter function +registered for that type there. Converter names are case-sensitive! +\end{datadesc} + + +\begin{datadesc}{PARSE_COLNAMES} + +Setting this makes pysqlite parse the column name for each column it returns. +It will look for a string formed [mytype] in there, and then decide that +'mytype' is the type of the column. It will try to find an entry of 'mytype' in +the converters dictionary and then use the converter function found there to +return the value. The column name found in cursor.description is only the first +word of the column name, i. e. if you use something like 'as "x [datetime]"' +in your SQL, then pysqlite will parse out everything until the first blank for +the column name: the column name would simply be "x". +\end{datadesc} + +\begin{funcdesc}{connect}{database\optional{, timeout, isolation_level, detect_types, check_same_thread, factory}} +Opens a connection to the SQLite database file \var{database}. You can use +\code{":memory:"} to open a database connection to a database that resides in +RAM instead of on disk. + +When a database is accessed by multiple connections, and one of the processes +modifies the database, the SQLite database is locked until that transaction is +committed. The \var{timeout} parameter specifies how long the connection should +wait for the lock to go away until raising an exception. The default for the +timeout parameter is 5.0 (five seconds). + +For the \var{isolation_level} parameter, please see TODO: link property of +Connection objects. + +SQLite natively supports only the types TEXT, INTEGER, FLOAT, BLOB and NULL. If +you want to use other types, like you have to add support for them yourself. +The \var{detect_types} parameter and the using custom *converters* registered with +the module-level *register_converter* function allow you to easily do that. + +\var{detect_types} defaults to 0 (i. e. off, no type detection), you can set it +to any combination of *PARSE_DECLTYPES* and *PARSE_COLNAMES* to turn type +detection on. + +By default, the sqlite3 module uses its Connection class for the connect call. +You can, however, subclass the Connection class and make .connect() use your +class instead by providing your class for the \var{factory} parameter. + +Consult the section `4. SQLite and Python types`_ of this manual for details. + +The sqlite3 module internally uses a statement cache to avoid SQL parsing +overhead. If you want to explicitly set the number of statements that are +cached for the connection, you can set the \var{cached_statements} parameter. +The currently implemented default is to cache 100 statements. +\end{funcdesc} + +\begin{funcdesc}{register_converter}{typename, callable} + +Registers a callable to convert a bytestring from the database into a custom +Python type. The callable will be invoked for all database values that are of +the type \var{typename}. Confer the parameter **detect_types** of the +**connect** method for how the type detection works. Note that the case of +\var{typename} and the name of the type in your query must match! +\end{funcdesc} + +\begin{funcdesc}{register_adapter}{type, callable} +Registers a callable to convert the custom Python type \var{type} into one of +SQLite's supported types. The callable \var{callable} accepts as single +parameter the Python value, and must return a value of the following types: +int, long, float, str (UTF-8 encoded), unicode or buffer. +\end{funcdesc} + + + + + + +\subsection{Connection Objects \label{Connection-Objects}} + +A \class{Connection} instance has the following attributes and methods: + +\member{isolation_level} + Get or set the current isolation level. None for autocommit mode or one + of "DEFERRED", "IMMEDIATE" or "EXLUSIVE". See `5. Controlling + Transactions`_ for a more detailed explanation. + +\begin{methoddesc}{cursor}{\optional{cursorClass}} + The cursor method accepts a single optional parameter \var{cursorClass}. + This is a custom cursor class which must extend sqlite3.Cursor. +\end{methoddesc} + +TODO: execute* + From buildbot at python.org Sun Apr 30 01:41:07 2006 From: buildbot at python.org (buildbot at python.org) Date: Sat, 29 Apr 2006 23:41:07 +0000 Subject: [Python-checkins] buildbot warnings in x86 W2k trunk Message-ID: <20060429234107.812FA1E4014@bag.python.org> The Buildbot has detected a new failure of x86 W2k trunk. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520W2k%2520trunk/builds/601 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: gerhard.haering Build Had Warnings: warnings test sincerely, -The Buildbot From python-checkins at python.org Sun Apr 30 03:07:10 2006 From: python-checkins at python.org (andrew.kuchling) Date: Sun, 30 Apr 2006 03:07:10 +0200 (CEST) Subject: [Python-checkins] r45811 - python/trunk/Doc/whatsnew/whatsnew25.tex Message-ID: <20060430010710.8115B1E400C@bag.python.org> Author: andrew.kuchling Date: Sun Apr 30 03:07:09 2006 New Revision: 45811 Modified: python/trunk/Doc/whatsnew/whatsnew25.tex Log: Add two items Modified: python/trunk/Doc/whatsnew/whatsnew25.tex ============================================================================== --- python/trunk/Doc/whatsnew/whatsnew25.tex (original) +++ python/trunk/Doc/whatsnew/whatsnew25.tex Sun Apr 30 03:07:09 2006 @@ -1125,6 +1125,14 @@ # -*- coding: latin1 -*- \end{verbatim} +\item One error that Python programmers sometimes make is forgetting +to include an \file{__init__.py} module in a package directory. +Debugging this mistake can be confusing, and usually requires running +Python with the \programopt{-v} switch to log all the paths searched. +In Python 2.5, a new \exception{ImportWarning} warning is raised when +an import would have picked up a directory as a package but no +\file{__init__.py} was found. (Implemented by Thomas Wouters.) + \item The list of base classes in a class definition can now be empty. As an example, this is now legal: @@ -1440,6 +1448,15 @@ import hooks and now also works for packages stored in ZIP-format archives. (Contributed by Phillip J. Eby.) +\item The pybench benchmark suite by Marc-Andr\'e~Lemburg is now +included in the \file{Tools/pybench} directory. The pybench suite is +an improvement on the commonly used \file{pystone.py} program because +pybench provides a more detailed measurement of the interpreter's +performance. It times particular operations such as function calls, +tuple slicing, method lookups, and numeric operations, instead of +performing many different operations and reducing the result to a +single number as \file{pystone.py} does. + \item The old \module{regex} and \module{regsub} modules, which have been deprecated ever since Python 2.0, have finally been deleted. Other deleted modules: \module{statcache}, \module{tzparse}, From buildbot at python.org Sun Apr 30 04:27:29 2006 From: buildbot at python.org (buildbot at python.org) Date: Sun, 30 Apr 2006 02:27:29 +0000 Subject: [Python-checkins] buildbot warnings in alpha Debian trunk Message-ID: <20060430022729.2DBC71E400C@bag.python.org> The Buildbot has detected a new failure of alpha Debian trunk. Full details are available at: http://www.python.org/dev/buildbot/all/alpha%2520Debian%2520trunk/builds/47 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: andrew.kuchling Build Had Warnings: warnings test sincerely, -The Buildbot From python-checkins at python.org Sun Apr 30 05:49:57 2006 From: python-checkins at python.org (george.yoshida) Date: Sun, 30 Apr 2006 05:49:57 +0200 (CEST) Subject: [Python-checkins] r45814 - python/trunk/Doc/lib/libgetpass.tex Message-ID: <20060430034957.B75821E400C@bag.python.org> Author: george.yoshida Date: Sun Apr 30 05:49:56 2006 New Revision: 45814 Modified: python/trunk/Doc/lib/libgetpass.tex Log: Use \versionchanged instead of \versionadded for new parameter support. Modified: python/trunk/Doc/lib/libgetpass.tex ============================================================================== --- python/trunk/Doc/lib/libgetpass.tex (original) +++ python/trunk/Doc/lib/libgetpass.tex Sun Apr 30 05:49:56 2006 @@ -19,7 +19,7 @@ \code{sys.stdout} (this argument is ignored on Windows). Availability: Macintosh, \UNIX, Windows. - \versionadded[The \var{stream} parameter]{2.5} + \versionchanged[The \var{stream} parameter was added]{2.5} \end{funcdesc} From g.brandl at gmx.net Sun Apr 30 07:00:12 2006 From: g.brandl at gmx.net (Georg Brandl) Date: Sun, 30 Apr 2006 07:00:12 +0200 Subject: [Python-checkins] r45810 - in python/trunk/Doc: Makefile.deps lib/lib.tex lib/libsqlite3.tex In-Reply-To: <20060429231242.6CAEC1E400D@bag.python.org> References: <20060429231242.6CAEC1E400D@bag.python.org> Message-ID: gerhard.haering wrote: > Added: python/trunk/Doc/lib/libsqlite3.tex > ============================================================================== > --- (empty file) > +++ python/trunk/Doc/lib/libsqlite3.tex Sun Apr 30 01:12:41 2006 > @@ -0,0 +1,105 @@ > +\section{\module{sqlite3} --- > + DB-API 2.0 interface for SQLite databases} > + > +\declaremodule{builtin}{sqlite3} > +\modulesynopsis{A DB-API 2.0 interface based on SQLite 3.x.} I find that confusing -- isn't it a DB-API interface to SQLite? Georg From python-checkins at python.org Sun Apr 30 09:06:12 2006 From: python-checkins at python.org (georg.brandl) Date: Sun, 30 Apr 2006 09:06:12 +0200 (CEST) Subject: [Python-checkins] r45815 - in python/trunk: Doc/lib/liburllib2.tex Lib/test/test_urllib2.py Lib/test/test_urllib2net.py Lib/urllib2.py Misc/NEWS Message-ID: <20060430070612.AD4931E400A@bag.python.org> Author: georg.brandl Date: Sun Apr 30 09:06:11 2006 New Revision: 45815 Modified: python/trunk/Doc/lib/liburllib2.tex python/trunk/Lib/test/test_urllib2.py python/trunk/Lib/test/test_urllib2net.py python/trunk/Lib/urllib2.py python/trunk/Misc/NEWS Log: Patch #1470846: fix urllib2 ProxyBasicAuthHandler. Modified: python/trunk/Doc/lib/liburllib2.tex ============================================================================== --- python/trunk/Doc/lib/liburllib2.tex (original) +++ python/trunk/Doc/lib/liburllib2.tex Sun Apr 30 09:06:11 2006 @@ -621,14 +621,20 @@ \subsection{AbstractBasicAuthHandler Objects \label{abstract-basic-auth-handler}} -\begin{methoddesc}[AbstractBasicAuthHandler]{handle_authentication_request} +\begin{methoddesc}[AbstractBasicAuthHandler]{http_error_auth_reqed} {authreq, host, req, headers} Handle an authentication request by getting a user/password pair, and re-trying the request. \var{authreq} should be the name of the header where the information about the realm is included in the request, -\var{host} is the host to authenticate to, \var{req} should be the -(failed) \class{Request} object, and \var{headers} should be the error -headers. +\var{host} specifies the URL and path to authenticate for, \var{req} +should be the (failed) \class{Request} object, and \var{headers} +should be the error headers. + +\var{host} is either an authority (e.g. \code{"python.org"}) or a URL +containing an authority component (e.g. \code{"http://python.org/"}). +In either case, the authority must not contain a userinfo component +(so, \code{"python.org"} and \code{"python.org:80"} are fine, +\code{"joe:password at python.org"} is not). \end{methoddesc} @@ -653,7 +659,7 @@ \subsection{AbstractDigestAuthHandler Objects \label{abstract-digest-auth-handler}} -\begin{methoddesc}[AbstractDigestAuthHandler]{handle_authentication_request} +\begin{methoddesc}[AbstractDigestAuthHandler]{http_error_auth_reqed} {authreq, host, req, headers} \var{authreq} should be the name of the header where the information about the realm is included in the request, \var{host} should be the host to Modified: python/trunk/Lib/test/test_urllib2.py ============================================================================== --- python/trunk/Lib/test/test_urllib2.py (original) +++ python/trunk/Lib/test/test_urllib2.py Sun Apr 30 09:06:11 2006 @@ -10,10 +10,7 @@ # XXX # Request # CacheFTPHandler (hard to write) -# parse_keqv_list, parse_http_list (I'm leaving this for Anthony Baxter -# and Greg Stein, since they're doing Digest Authentication) -# Authentication stuff (ditto) -# CustomProxy, CustomProxyHandler +# parse_keqv_list, parse_http_list, HTTPDigestAuthHandler class TrivialTests(unittest.TestCase): def test_trivial(self): @@ -49,6 +46,70 @@ self.assertEquals(urllib2.parse_http_list(string), list) +def test_password_manager(self): + """ + >>> mgr = urllib2.HTTPPasswordMgr() + >>> add = mgr.add_password + >>> add("Some Realm", "http://example.com/", "joe", "password") + >>> add("Some Realm", "http://example.com/ni", "ni", "ni") + >>> add("c", "http://example.com/foo", "foo", "ni") + >>> add("c", "http://example.com/bar", "bar", "nini") + >>> add("b", "http://example.com/", "first", "blah") + >>> add("b", "http://example.com/", "second", "spam") + >>> add("a", "http://example.com", "1", "a") + >>> add("Some Realm", "http://c.example.com:3128", "3", "c") + >>> add("Some Realm", "d.example.com", "4", "d") + >>> add("Some Realm", "e.example.com:3128", "5", "e") + + >>> mgr.find_user_password("Some Realm", "example.com") + ('joe', 'password') + >>> mgr.find_user_password("Some Realm", "http://example.com") + ('joe', 'password') + >>> mgr.find_user_password("Some Realm", "http://example.com/") + ('joe', 'password') + >>> mgr.find_user_password("Some Realm", "http://example.com/spam") + ('joe', 'password') + >>> mgr.find_user_password("Some Realm", "http://example.com/spam/spam") + ('joe', 'password') + >>> mgr.find_user_password("c", "http://example.com/foo") + ('foo', 'ni') + >>> mgr.find_user_password("c", "http://example.com/bar") + ('bar', 'nini') + + Currently, we use the highest-level path where more than one match: + + >>> mgr.find_user_password("Some Realm", "http://example.com/ni") + ('joe', 'password') + + Use latest add_password() in case of conflict: + + >>> mgr.find_user_password("b", "http://example.com/") + ('second', 'spam') + + No special relationship between a.example.com and example.com: + + >>> mgr.find_user_password("a", "http://example.com/") + ('1', 'a') + >>> mgr.find_user_password("a", "http://a.example.com/") + (None, None) + + Ports: + + >>> mgr.find_user_password("Some Realm", "c.example.com") + (None, None) + >>> mgr.find_user_password("Some Realm", "c.example.com:3128") + ('3', 'c') + >>> mgr.find_user_password("Some Realm", "http://c.example.com:3128") + ('3', 'c') + >>> mgr.find_user_password("Some Realm", "d.example.com") + ('4', 'd') + >>> mgr.find_user_password("Some Realm", "e.example.com:3128") + ('5', 'e') + + """ + pass + + class MockOpener: addheaders = [] def open(self, req, data=None): @@ -89,6 +150,8 @@ return self.handle(self.meth_name, self.action, *args) class MockHandler: + # useful for testing handler machinery + # see add_ordered_mock_handlers() docstring handler_order = 500 def __init__(self, methods): self._define_methods(methods) @@ -161,6 +224,50 @@ opener.add_handler(h) return handlers +def build_test_opener(*handler_instances): + opener = OpenerDirector() + for h in handler_instances: + opener.add_handler(h) + return opener + +class MockHTTPHandler(urllib2.BaseHandler): + # useful for testing redirections and auth + # sends supplied headers and code as first response + # sends 200 OK as second response + def __init__(self, code, headers): + self.code = code + self.headers = headers + self.reset() + def reset(self): + self._count = 0 + self.requests = [] + def http_open(self, req): + import mimetools, httplib, copy + from StringIO import StringIO + self.requests.append(copy.deepcopy(req)) + if self._count == 0: + self._count = self._count + 1 + name = httplib.responses[self.code] + msg = mimetools.Message(StringIO(self.headers)) + return self.parent.error( + "http", req, MockFile(), self.code, name, msg) + else: + self.req = req + msg = mimetools.Message(StringIO("\r\n\r\n")) + return MockResponse(200, "OK", msg, "", req.get_full_url()) + +class MockPasswordManager: + def add_password(self, realm, uri, user, password): + self.realm = realm + self.url = uri + self.user = user + self.password = password + def find_user_password(self, realm, authuri): + self.target_realm = realm + self.target_url = authuri + return self.user, self.password + + class OpenerDirectorTests(unittest.TestCase): def test_handled(self): @@ -612,33 +719,18 @@ urllib2.HTTPRedirectHandler.max_redirections) def test_cookie_redirect(self): - class MockHTTPHandler(urllib2.HTTPHandler): - def __init__(self): self._count = 0 - def http_open(self, req): - import mimetools - from StringIO import StringIO - if self._count == 0: - self._count = self._count + 1 - msg = mimetools.Message( - StringIO("Location: http://www.cracker.com/\r\n\r\n")) - return self.parent.error( - "http", req, MockFile(), 302, "Found", msg) - else: - self.req = req - msg = mimetools.Message(StringIO("\r\n\r\n")) - return MockResponse(200, "OK", msg, "", req.get_full_url()) # cookies shouldn't leak into redirected requests from cookielib import CookieJar - from urllib2 import build_opener, HTTPHandler, HTTPError, \ - HTTPCookieProcessor from test.test_cookielib import interact_netscape cj = CookieJar() interact_netscape(cj, "http://www.example.com/", "spam=eggs") - hh = MockHTTPHandler() - cp = HTTPCookieProcessor(cj) - o = build_opener(hh, cp) + hh = MockHTTPHandler(302, "Location: http://www.cracker.com/\r\n\r\n") + hdeh = urllib2.HTTPDefaultErrorHandler() + hrh = urllib2.HTTPRedirectHandler() + cp = urllib2.HTTPCookieProcessor(cj) + o = build_test_opener(hh, hdeh, hrh, cp) o.open("http://www.example.com/") self.assert_(not hh.req.has_header("Cookie")) @@ -659,6 +751,71 @@ self.assertEqual([(handlers[0], "http_open")], [tup[0:2] for tup in o.calls]) + def test_basic_auth(self): + opener = OpenerDirector() + password_manager = MockPasswordManager() + auth_handler = urllib2.HTTPBasicAuthHandler(password_manager) + realm = "ACME Widget Store" + http_handler = MockHTTPHandler( + 401, 'WWW-Authenticate: Basic realm="%s"\r\n\r\n' % realm) + self._test_basic_auth(opener, auth_handler, "Authorization", + realm, http_handler, password_manager, + "http://acme.example.com/protected", + "http://acme.example.com/protected", + ) + + def test_proxy_basic_auth(self): + opener = OpenerDirector() + ph = urllib2.ProxyHandler(dict(http="proxy.example.com:3128")) + opener.add_handler(ph) + password_manager = MockPasswordManager() + auth_handler = urllib2.ProxyBasicAuthHandler(password_manager) + realm = "ACME Networks" + http_handler = MockHTTPHandler( + 407, 'Proxy-Authenticate: Basic realm="%s"\r\n\r\n' % realm) + self._test_basic_auth(opener, auth_handler, "Proxy-authorization", + realm, http_handler, password_manager, + "http://acme.example.com:3128/protected", + "proxy.example.com:3128", + ) + + def _test_basic_auth(self, opener, auth_handler, auth_header, + realm, http_handler, password_manager, + request_url, protected_url): + import base64, httplib + user, password = "wile", "coyote" + opener.add_handler(auth_handler) + opener.add_handler(http_handler) + + # .add_password() fed through to password manager + auth_handler.add_password(realm, request_url, user, password) + self.assertEqual(realm, password_manager.realm) + self.assertEqual(request_url, password_manager.url) + self.assertEqual(user, password_manager.user) + self.assertEqual(password, password_manager.password) + + r = opener.open(request_url) + + # should have asked the password manager for the username/password + self.assertEqual(password_manager.target_realm, realm) + self.assertEqual(password_manager.target_url, protected_url) + + # expect one request without authorization, then one with + self.assertEqual(len(http_handler.requests), 2) + self.assertFalse(http_handler.requests[0].has_header(auth_header)) + userpass = '%s:%s' % (user, password) + auth_hdr_value = 'Basic '+base64.encodestring(userpass).strip() + self.assertEqual(http_handler.requests[1].get_header(auth_header), + auth_hdr_value) + + # if the password manager can't find a password, the handler won't + # handle the HTTP auth error + password_manager.user = password_manager.password = None + http_handler.reset() + r = opener.open(request_url) + self.assertEqual(len(http_handler.requests), 1) + self.assertFalse(http_handler.requests[0].has_header(auth_header)) + class MiscTests(unittest.TestCase): @@ -830,20 +987,12 @@ cfh.setTimeout(1) handlers.append(cfh) -## # XXX try out some custom proxy objects too! -## def at_cnri(req): -## host = req.get_host() -## debug(host) -## if host[-18:] == '.cnri.reston.va.us': -## return True -## p = CustomProxy('http', at_cnri, 'proxy.cnri.reston.va.us') -## ph = CustomProxyHandler(p) -## handlers.append(ph) - return handlers def test_main(verbose=None): + from test import test_urllib2 + test_support.run_doctest(test_urllib2, verbose) test_support.run_doctest(urllib2, verbose) tests = (TrivialTests, OpenerDirectorTests, Modified: python/trunk/Lib/test/test_urllib2net.py ============================================================================== --- python/trunk/Lib/test/test_urllib2net.py (original) +++ python/trunk/Lib/test/test_urllib2net.py Sun Apr 30 09:06:11 2006 @@ -23,6 +23,46 @@ f = urllib2.urlopen("http://www.python.org/") x = f.read() + +class AuthTests(unittest.TestCase): + """Tests urllib2 authentication features.""" + +## Disabled at the moment since there is no page under python.org which +## could be used to HTTP authentication. +# +# def test_basic_auth(self): +# import httplib +# +# test_url = "http://www.python.org/test/test_urllib2/basic_auth" +# test_hostport = "www.python.org" +# test_realm = 'Test Realm' +# test_user = 'test.test_urllib2net' +# test_password = 'blah' +# +# # failure +# try: +# urllib2.urlopen(test_url) +# except urllib2.HTTPError, exc: +# self.assertEqual(exc.code, 401) +# else: +# self.fail("urlopen() should have failed with 401") +# +# # success +# auth_handler = urllib2.HTTPBasicAuthHandler() +# auth_handler.add_password(test_realm, test_hostport, +# test_user, test_password) +# opener = urllib2.build_opener(auth_handler) +# f = opener.open('http://localhost/') +# response = urllib2.urlopen("http://www.python.org/") +# +# # The 'userinfo' URL component is deprecated by RFC 3986 for security +# # reasons, let's not implement it! (it's already implemented for proxy +# # specification strings (that is, URLs or authorities specifying a +# # proxy), so we must keep that) +# self.assertRaises(httplib.InvalidURL, +# urllib2.urlopen, "http://evil:thing at example.com") + + class urlopenNetworkTests(unittest.TestCase): """Tests urllib2.urlopen using the network. @@ -86,7 +126,8 @@ def test_main(): test_support.requires("network") - test_support.run_unittest(URLTimeoutTest, urlopenNetworkTests) + test_support.run_unittest(URLTimeoutTest, urlopenNetworkTests, + AuthTests) if __name__ == "__main__": test_main() Modified: python/trunk/Lib/urllib2.py ============================================================================== --- python/trunk/Lib/urllib2.py (original) +++ python/trunk/Lib/urllib2.py Sun Apr 30 09:06:11 2006 @@ -612,7 +612,6 @@ ('http', 'joe', 'password', 'proxy.example.com') """ - from urlparse import _splitnetloc scheme, r_scheme = splittype(proxy) if not r_scheme.startswith("/"): # authority @@ -673,6 +672,7 @@ return self.parent.open(req) class HTTPPasswordMgr: + def __init__(self): self.passwd = {} @@ -696,10 +696,15 @@ def reduce_uri(self, uri): """Accept netloc or URI and extract only the netloc and path""" - parts = urlparse.urlparse(uri) + parts = urlparse.urlsplit(uri) if parts[1]: + # URI return parts[1], parts[2] or '/' + elif parts[0]: + # host:port + return uri, '/' else: + # host return parts[2], '/' def is_suburi(self, base, test): @@ -742,6 +747,8 @@ self.add_password = self.passwd.add_password def http_error_auth_reqed(self, authreq, host, req, headers): + # host may be an authority (without userinfo) or a URL with an + # authority # XXX could be multiple headers authreq = headers.get(authreq, None) if authreq: @@ -752,10 +759,7 @@ return self.retry_http_basic_auth(host, req, realm) def retry_http_basic_auth(self, host, req, realm): - # TODO(jhylton): Remove the host argument? It depends on whether - # retry_http_basic_auth() is consider part of the public API. - # It probably is. - user, pw = self.passwd.find_user_password(realm, req.get_full_url()) + user, pw = self.passwd.find_user_password(realm, host) if pw is not None: raw = "%s:%s" % (user, pw) auth = 'Basic %s' % base64.encodestring(raw).strip() @@ -766,14 +770,15 @@ else: return None + class HTTPBasicAuthHandler(AbstractBasicAuthHandler, BaseHandler): auth_header = 'Authorization' def http_error_401(self, req, fp, code, msg, headers): - host = urlparse.urlparse(req.get_full_url())[1] + url = req.get_full_url() return self.http_error_auth_reqed('www-authenticate', - host, req, headers) + url, req, headers) class ProxyBasicAuthHandler(AbstractBasicAuthHandler, BaseHandler): @@ -781,9 +786,13 @@ auth_header = 'Proxy-authorization' def http_error_407(self, req, fp, code, msg, headers): - host = req.get_host() + # http_error_auth_reqed requires that there is no userinfo component in + # authority. Assume there isn't one, since urllib2 does not (and + # should not, RFC 3986 s. 3.2.1) support requests for URLs containing + # userinfo. + authority = req.get_host() return self.http_error_auth_reqed('proxy-authenticate', - host, req, headers) + authority, req, headers) def randombytes(n): Modified: python/trunk/Misc/NEWS ============================================================================== --- python/trunk/Misc/NEWS (original) +++ python/trunk/Misc/NEWS Sun Apr 30 09:06:11 2006 @@ -86,6 +86,8 @@ Library ------- +- Patch #1470846: fix urllib2 ProxyBasicAuthHandler. + - Patch #1475231: ``doctest`` has a new ``SKIP`` option, which causes a doctest to be skipped (the code is not run, and the expected output or exception is ignored). From buildbot at python.org Sun Apr 30 09:53:38 2006 From: buildbot at python.org (buildbot at python.org) Date: Sun, 30 Apr 2006 07:53:38 +0000 Subject: [Python-checkins] buildbot warnings in alpha Tru64 5.1 trunk Message-ID: <20060430075338.7967F1E400C@bag.python.org> The Buildbot has detected a new failure of alpha Tru64 5.1 trunk. Full details are available at: http://www.python.org/dev/buildbot/all/alpha%2520Tru64%25205.1%2520trunk/builds/386 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: georg.brandl Build Had Warnings: warnings test sincerely, -The Buildbot From python-checkins at python.org Sun Apr 30 10:36:09 2006 From: python-checkins at python.org (georg.brandl) Date: Sun, 30 Apr 2006 10:36:09 +0200 (CEST) Subject: [Python-checkins] r45816 - peps/trunk/pep-0356.txt Message-ID: <20060430083609.0A2481E4036@bag.python.org> Author: georg.brandl Date: Sun Apr 30 10:36:08 2006 New Revision: 45816 Modified: peps/trunk/pep-0356.txt Log: Add expat 2.0 upgrade. Modified: peps/trunk/pep-0356.txt ============================================================================== --- peps/trunk/pep-0356.txt (original) +++ peps/trunk/pep-0356.txt Sun Apr 30 10:36:08 2006 @@ -120,6 +120,9 @@ - wsgiref to the standard library (Owner: Phillip Eby) + - Upgrade pyexpat to use expat 2.0? + (Patch by Trent Mick, #1462338) + - Add new icons for Windows, MacOS and Unix with the new Python logo? (Owner: ???) Windows: http://mail.python.org/pipermail/python-dev/2006-April/063738.html From buildbot at python.org Sun Apr 30 10:44:10 2006 From: buildbot at python.org (buildbot at python.org) Date: Sun, 30 Apr 2006 08:44:10 +0000 Subject: [Python-checkins] buildbot warnings in hppa Ubuntu dapper trunk Message-ID: <20060430084410.7B0D21E400C@bag.python.org> The Buildbot has detected a new failure of hppa Ubuntu dapper trunk. Full details are available at: http://www.python.org/dev/buildbot/all/hppa%2520Ubuntu%2520dapper%2520trunk/builds/329 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: georg.brandl Build Had Warnings: warnings test sincerely, -The Buildbot From python-checkins at python.org Sun Apr 30 10:57:36 2006 From: python-checkins at python.org (georg.brandl) Date: Sun, 30 Apr 2006 10:57:36 +0200 (CEST) Subject: [Python-checkins] r45817 - in python/trunk/Lib: distutils/command/upload.py plat-mac/pimp.py poplib.py test/test_stringprep.py test/test_unicodedata.py urllib2.py Message-ID: <20060430085736.A68F51E400C@bag.python.org> Author: georg.brandl Date: Sun Apr 30 10:57:35 2006 New Revision: 45817 Modified: python/trunk/Lib/distutils/command/upload.py python/trunk/Lib/plat-mac/pimp.py python/trunk/Lib/poplib.py python/trunk/Lib/test/test_stringprep.py python/trunk/Lib/test/test_unicodedata.py python/trunk/Lib/urllib2.py Log: In stdlib, use hashlib instead of deprecated md5 and sha modules. Modified: python/trunk/Lib/distutils/command/upload.py ============================================================================== --- python/trunk/Lib/distutils/command/upload.py (original) +++ python/trunk/Lib/distutils/command/upload.py Sun Apr 30 10:57:35 2006 @@ -6,7 +6,7 @@ from distutils.core import Command from distutils.spawn import spawn from distutils import log -from md5 import md5 +from hashlib import md5 import os import socket import platform Modified: python/trunk/Lib/plat-mac/pimp.py ============================================================================== --- python/trunk/Lib/plat-mac/pimp.py (original) +++ python/trunk/Lib/plat-mac/pimp.py Sun Apr 30 10:57:35 2006 @@ -21,7 +21,7 @@ import plistlib import distutils.util import distutils.sysconfig -import md5 +import hashlib import tarfile import tempfile import shutil @@ -693,7 +693,7 @@ sys.stderr.write("Warning: no MD5Sum for %s\n" % self.fullname()) return 1 data = open(self.archiveFilename, 'rb').read() - checksum = md5.new(data).hexdigest() + checksum = hashlib.md5(data).hexdigest() return checksum == self._dict['MD5Sum'] def unpackPackageOnly(self, output=None): Modified: python/trunk/Lib/poplib.py ============================================================================== --- python/trunk/Lib/poplib.py (original) +++ python/trunk/Lib/poplib.py Sun Apr 30 10:57:35 2006 @@ -295,8 +295,8 @@ m = self.timestamp.match(self.welcome) if not m: raise error_proto('-ERR APOP not supported by server') - import md5 - digest = md5.new(m.group(1)+secret).digest() + import hashlib + digest = hashlib.md5(m.group(1)+secret).digest() digest = ''.join(map(lambda x:'%02x'%ord(x), digest)) return self._shortcmd('APOP %s %s' % (user, digest)) Modified: python/trunk/Lib/test/test_stringprep.py ============================================================================== --- python/trunk/Lib/test/test_stringprep.py (original) +++ python/trunk/Lib/test/test_stringprep.py Sun Apr 30 10:57:35 2006 @@ -2,7 +2,6 @@ # Since we don't have them, this test checks only a few codepoints. from test.test_support import verify, vereq -import sha import stringprep from stringprep import * @@ -73,6 +72,7 @@ # unicode database. Instead, stringprep.py asserts the version of # the database. +# import hashlib # predicates = [k for k in dir(stringprep) if k.startswith("in_table")] # predicates.sort() # for p in predicates: @@ -83,6 +83,6 @@ # if f(unichr(i)): # data[i] = "1" # data = "".join(data) -# h = sha.sha() +# h = hashlib.sha1() # h.update(data) -# print p,h.hexdigest() +# print p, h.hexdigest() Modified: python/trunk/Lib/test/test_unicodedata.py ============================================================================== --- python/trunk/Lib/test/test_unicodedata.py (original) +++ python/trunk/Lib/test/test_unicodedata.py Sun Apr 30 10:57:35 2006 @@ -6,7 +6,7 @@ """#" import unittest, test.test_support -import sha +import hashlib encoding = 'utf-8' @@ -19,7 +19,7 @@ expectedchecksum = 'a6555cd209d960dcfa17bfdce0c96d91cfa9a9ba' def test_method_checksum(self): - h = sha.sha() + h = hashlib.sha1() for i in range(65536): char = unichr(i) data = [ @@ -79,7 +79,7 @@ def test_function_checksum(self): data = [] - h = sha.sha() + h = hashlib.sha1() for i in range(0x10000): char = unichr(i) Modified: python/trunk/Lib/urllib2.py ============================================================================== --- python/trunk/Lib/urllib2.py (original) +++ python/trunk/Lib/urllib2.py Sun Apr 30 10:57:35 2006 @@ -88,14 +88,13 @@ import ftplib import httplib import inspect -import md5 +import hashlib import mimetypes import mimetools import os import posixpath import random import re -import sha import socket import sys import time @@ -869,8 +868,8 @@ # and server to avoid chosen plaintext attacks, to provide mutual # authentication, and to provide some message integrity protection. # This isn't a fabulous effort, but it's probably Good Enough. - dig = sha.new("%s:%s:%s:%s" % (self.nonce_count, nonce, time.ctime(), - randombytes(8))).hexdigest() + dig = hashlib.sha1("%s:%s:%s:%s" % (self.nonce_count, nonce, time.ctime(), + randombytes(8))).hexdigest() return dig[:16] def get_authorization(self, req, chal): @@ -932,9 +931,9 @@ def get_algorithm_impls(self, algorithm): # lambdas assume digest modules are imported at the top level if algorithm == 'MD5': - H = lambda x: md5.new(x).hexdigest() + H = lambda x: hashlib.md5(x).hexdigest() elif algorithm == 'SHA': - H = lambda x: sha.new(x).hexdigest() + H = lambda x: hashlib.sha1(x).hexdigest() # XXX MD5-sess KD = lambda s, d: H("%s:%s" % (s, d)) return H, KD From python-checkins at python.org Sun Apr 30 10:57:57 2006 From: python-checkins at python.org (georg.brandl) Date: Sun, 30 Apr 2006 10:57:57 +0200 (CEST) Subject: [Python-checkins] r45818 - peps/trunk/pep-0356.txt Message-ID: <20060430085757.891E81E400C@bag.python.org> Author: georg.brandl Date: Sun Apr 30 10:57:57 2006 New Revision: 45818 Modified: peps/trunk/pep-0356.txt Log: Stdlib now uses hashlib instead of md5 und sha. Typos fix. Modified: peps/trunk/pep-0356.txt ============================================================================== --- peps/trunk/pep-0356.txt (original) +++ peps/trunk/pep-0356.txt Sun Apr 30 10:57:57 2006 @@ -83,7 +83,6 @@ (replaces old md5 and sha modules) - msilib -- for creating MSI files and bdist_msi in distutils. - There are no docs yet. - pysqlite @@ -106,6 +105,7 @@ - Support for building "fat" Mac binaries (Intel and PPC) + Possible features for 2.5 Each feature below should implemented prior to beta1 or @@ -142,9 +142,6 @@ - Python core should compile cleanly with g++ (Owner: Anthony) - - upgrade stdlib to use newer hashlib module instead of older - deprecated md5/sha modules. - (Owner: Anthony) Deferred until 2.6: @@ -152,7 +149,7 @@ http://mail.python.org/pipermail/python-dev/2006-February/060926.html - bdist_egg in distutils package - (though bsdist_egg is supported in setuptools) + (though bdist_egg is supported in setuptools) - pure python pgen module (Owner: Guido) From python-checkins at python.org Sun Apr 30 11:23:59 2006 From: python-checkins at python.org (georg.brandl) Date: Sun, 30 Apr 2006 11:23:59 +0200 (CEST) Subject: [Python-checkins] r45819 - python/trunk/Lib/urllib.py Message-ID: <20060430092359.CA1CF1E400C@bag.python.org> Author: georg.brandl Date: Sun Apr 30 11:23:59 2006 New Revision: 45819 Modified: python/trunk/Lib/urllib.py Log: Patch #1470976: don't NLST files when retrieving over FTP. Modified: python/trunk/Lib/urllib.py ============================================================================== --- python/trunk/Lib/urllib.py (original) +++ python/trunk/Lib/urllib.py Sun Apr 30 11:23:59 2006 @@ -856,13 +856,6 @@ self.ftp.voidcmd(cmd) conn = None if file and not isdir: - # Use nlst to see if the file exists at all - try: - self.ftp.nlst(file) - except ftplib.error_perm, reason: - raise IOError, ('ftp error', reason), sys.exc_info()[2] - # Restore the transfer mode! - self.ftp.voidcmd(cmd) # Try to retrieve as a file try: cmd = 'RETR ' + file From buildbot at python.org Sun Apr 30 12:19:39 2006 From: buildbot at python.org (buildbot at python.org) Date: Sun, 30 Apr 2006 10:19:39 +0000 Subject: [Python-checkins] buildbot warnings in x86 gentoo trunk Message-ID: <20060430101939.9C03C1E400C@bag.python.org> The Buildbot has detected a new failure of x86 gentoo trunk. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520gentoo%2520trunk/builds/669 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: georg.brandl Build Had Warnings: warnings test sincerely, -The Buildbot From python-checkins at python.org Sun Apr 30 13:12:04 2006 From: python-checkins at python.org (georg.brandl) Date: Sun, 30 Apr 2006 13:12:04 +0200 (CEST) Subject: [Python-checkins] r45820 - python/branches/release24-maint/Modules/cPickle.c Message-ID: <20060430111204.0A6941E400D@bag.python.org> Author: georg.brandl Date: Sun Apr 30 13:12:03 2006 New Revision: 45820 Modified: python/branches/release24-maint/Modules/cPickle.c Log: Bug #1473625: stop pickling in text mode make floats locale dependent. Modified: python/branches/release24-maint/Modules/cPickle.c ============================================================================== --- python/branches/release24-maint/Modules/cPickle.c (original) +++ python/branches/release24-maint/Modules/cPickle.c Sun Apr 30 13:12:03 2006 @@ -1141,7 +1141,9 @@ else { char c_str[250]; c_str[0] = FLOAT; - PyOS_snprintf(c_str + 1, sizeof(c_str) - 1, "%.17g\n", x); + PyOS_ascii_formatd(c_str + 1, sizeof(c_str) - 2, "%.17g", x); + /* Extend the formatted string with a newline character */ + strcat(c_str, "\n"); if (self->write_func(self, c_str, strlen(c_str)) < 0) return -1; From python-checkins at python.org Sun Apr 30 13:13:57 2006 From: python-checkins at python.org (georg.brandl) Date: Sun, 30 Apr 2006 13:13:57 +0200 (CEST) Subject: [Python-checkins] r45821 - in python/trunk: Lib/test/pickletester.py Lib/test/test_builtin.py Lib/test/test_logging.py Lib/test/test_support.py Lib/test/test_unicode.py Modules/cPickle.c Message-ID: <20060430111357.77C8F1E400C@bag.python.org> Author: georg.brandl Date: Sun Apr 30 13:13:56 2006 New Revision: 45821 Modified: python/trunk/Lib/test/pickletester.py python/trunk/Lib/test/test_builtin.py python/trunk/Lib/test/test_logging.py python/trunk/Lib/test/test_support.py python/trunk/Lib/test/test_unicode.py python/trunk/Modules/cPickle.c Log: Bug #1473625: stop cPickle making float dumps locale dependent in protocol 0. On the way, add a decorator to test_support to facilitate running single test functions in different locales with automatic cleanup. Modified: python/trunk/Lib/test/pickletester.py ============================================================================== --- python/trunk/Lib/test/pickletester.py (original) +++ python/trunk/Lib/test/pickletester.py Sun Apr 30 13:13:56 2006 @@ -4,7 +4,8 @@ import pickletools import copy_reg -from test.test_support import TestFailed, have_unicode, TESTFN +from test.test_support import TestFailed, have_unicode, TESTFN, \ + run_with_locale # Tests that try a number of pickle protocols should have a # for proto in protocols: @@ -527,6 +528,11 @@ got = self.loads(p) self.assertEqual(n, got) + @run_with_locale('LC_ALL', 'de_DE', 'fr_FR') + def test_float_format(self): + # make sure that floats are formatted locale independent + self.assertEqual(self.dumps(1.2)[0:3], 'F1.') + def test_reduce(self): pass Modified: python/trunk/Lib/test/test_builtin.py ============================================================================== --- python/trunk/Lib/test/test_builtin.py (original) +++ python/trunk/Lib/test/test_builtin.py Sun Apr 30 13:13:56 2006 @@ -1,7 +1,8 @@ # Python test set -- built-in functions import test.test_support, unittest -from test.test_support import fcmp, have_unicode, TESTFN, unlink, run_unittest +from test.test_support import fcmp, have_unicode, TESTFN, unlink, \ + run_unittest, run_with_locale from operator import neg import sys, warnings, cStringIO, random, UserDict @@ -554,33 +555,20 @@ # Implementation limitation in PyFloat_FromString() self.assertRaises(ValueError, float, unicode("1"*10000)) + @run_with_locale('LC_NUMERIC', 'fr_FR', 'de_DE') def test_float_with_comma(self): # set locale to something that doesn't use '.' for the decimal point - try: - import locale - orig_locale = locale.setlocale(locale.LC_NUMERIC) - locale.setlocale(locale.LC_NUMERIC, 'fr_FR') - except: - # if we can't set the locale, just ignore this test - return - - try: - self.assertEqual(locale.localeconv()['decimal_point'], ',') - except: - # this test is worthless, just skip it and reset the locale - locale.setlocale(locale.LC_NUMERIC, orig_locale) + import locale + if not locale.localeconv()['decimal_point'] == ',': return - try: - self.assertEqual(float(" 3,14 "), 3.14) - self.assertEqual(float(" +3,14 "), 3.14) - self.assertEqual(float(" -3,14 "), -3.14) - self.assertRaises(ValueError, float, " 0x3.1 ") - self.assertRaises(ValueError, float, " -0x3.p-1 ") - self.assertEqual(float(" 25.e-1 "), 2.5) - self.assertEqual(fcmp(float(" .25e-1 "), .025), 0) - finally: - locale.setlocale(locale.LC_NUMERIC, orig_locale) + self.assertEqual(float(" 3,14 "), 3.14) + self.assertEqual(float(" +3,14 "), 3.14) + self.assertEqual(float(" -3,14 "), -3.14) + self.assertRaises(ValueError, float, " 0x3.1 ") + self.assertRaises(ValueError, float, " -0x3.p-1 ") + self.assertEqual(float(" 25.e-1 "), 2.5) + self.assertEqual(fcmp(float(" .25e-1 "), .025), 0) def test_floatconversion(self): # Make sure that calls to __float__() work properly Modified: python/trunk/Lib/test/test_logging.py ============================================================================== --- python/trunk/Lib/test/test_logging.py (original) +++ python/trunk/Lib/test/test_logging.py Sun Apr 30 13:13:56 2006 @@ -28,6 +28,7 @@ import os, sys, string, struct, types, cPickle, cStringIO import socket, tempfile, threading, time import logging, logging.handlers, logging.config +from test.test_support import run_with_locale BANNER = "-- %-10s %-6s ---------------------------------------------------\n" @@ -657,19 +658,11 @@ pass rootLogger.removeHandler(hdlr) +# Set the locale to the platform-dependent default. I have no idea +# why the test does this, but in any case we save the current locale +# first and restore it at the end. + at run_with_locale('LC_ALL', '') def test_main(): - import locale - # Set the locale to the platform-dependent default. I have no idea - # why the test does this, but in any case we save the current locale - # first so we can restore it at the end. - try: - original_locale = locale.setlocale(locale.LC_ALL) - locale.setlocale(locale.LC_ALL, '') - except (ValueError, locale.Error): - # this happens on a Solaris box which only supports "C" locale - # or a Mac OS X box which supports very little locale stuff at all - original_locale = None - # Save and restore the original root logger level across the tests. # Otherwise, e.g., if any test using cookielib runs after test_logging, # cookielib's debug-level logger tries to log messages, leading to @@ -681,8 +674,6 @@ try: test_main_inner() finally: - if original_locale is not None: - locale.setlocale(locale.LC_ALL, original_locale) root_logger.setLevel(original_logging_level) if __name__ == "__main__": Modified: python/trunk/Lib/test/test_support.py ============================================================================== --- python/trunk/Lib/test/test_support.py (original) +++ python/trunk/Lib/test/test_support.py Sun Apr 30 13:13:56 2006 @@ -252,6 +252,42 @@ return open(fn) #======================================================================= +# Decorator for running a function in a different locale, correctly resetting +# it afterwards. + +def run_with_locale(catstr, *locales): + def decorator(func): + def inner(*args, **kwds): + try: + import locale + category = getattr(locale, catstr) + orig_locale = locale.setlocale(category) + except AttributeError: + # if the test author gives us an invalid category string + raise + except: + # cannot retrieve original locale, so do nothing + locale = orig_locale = None + else: + for loc in locales: + try: + locale.setlocale(category, loc) + break + except: + pass + + # now run the function, resetting the locale on exceptions + try: + return func(*args, **kwds) + finally: + if locale and orig_locale: + locale.setlocale(category, orig_locale) + inner.func_name = func.func_name + inner.__doc__ = func.__doc__ + return inner + return decorator + +#======================================================================= # Big-memory-test support. Separate from 'resources' because memory use should be configurable. # Some handy shorthands. Note that these are used for byte-limits as well Modified: python/trunk/Lib/test/test_unicode.py ============================================================================== --- python/trunk/Lib/test/test_unicode.py (original) +++ python/trunk/Lib/test/test_unicode.py Sun Apr 30 13:13:56 2006 @@ -410,20 +410,11 @@ def __str__(self): return u'\u1234' self.assertEqual('%s' % Wrapper(), u'\u1234') - + + @test_support.run_with_locale('LC_ALL', 'de_DE', 'fr_FR') def test_format_float(self): - try: - import locale - orig_locale = locale.setlocale(locale.LC_ALL) - locale.setlocale(locale.LC_ALL, 'de_DE') - except (ImportError, locale.Error): - return # skip if we can't set locale - - try: - # should not format with a comma, but always with C locale - self.assertEqual(u'1.0', u'%.1f' % 1.0) - finally: - locale.setlocale(locale.LC_ALL, orig_locale) + # should not format with a comma, but always with C locale + self.assertEqual(u'1.0', u'%.1f' % 1.0) def test_constructor(self): # unicode(obj) tests (this maps to PyObject_Unicode() at C level) Modified: python/trunk/Modules/cPickle.c ============================================================================== --- python/trunk/Modules/cPickle.c (original) +++ python/trunk/Modules/cPickle.c Sun Apr 30 13:13:56 2006 @@ -1151,7 +1151,9 @@ else { char c_str[250]; c_str[0] = FLOAT; - PyOS_snprintf(c_str + 1, sizeof(c_str) - 1, "%.17g\n", x); + PyOS_ascii_formatd(c_str + 1, sizeof(c_str) - 2, "%.17g", x); + /* Extend the formatted string with a newline character */ + strcat(c_str, "\n"); if (self->write_func(self, c_str, strlen(c_str)) < 0) return -1; From buildbot at python.org Sun Apr 30 15:00:00 2006 From: buildbot at python.org (buildbot at python.org) Date: Sun, 30 Apr 2006 13:00:00 +0000 Subject: [Python-checkins] buildbot warnings in x86 Ubuntu dapper (icc) trunk Message-ID: <20060430130000.6B4E61E400C@bag.python.org> The Buildbot has detected a new failure of x86 Ubuntu dapper (icc) trunk. Full details are available at: http://www.python.org/dev/buildbot/all/x86%2520Ubuntu%2520dapper%2520%2528icc%2529%2520trunk/builds/290 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: georg.brandl Build Had Warnings: warnings test sincerely, -The Buildbot From python-checkins at python.org Sun Apr 30 17:59:26 2006 From: python-checkins at python.org (phillip.eby) Date: Sun, 30 Apr 2006 17:59:26 +0200 (CEST) Subject: [Python-checkins] r45822 - python/trunk/Lib/inspect.py Message-ID: <20060430155926.ADE981E400D@bag.python.org> Author: phillip.eby Date: Sun Apr 30 17:59:26 2006 New Revision: 45822 Modified: python/trunk/Lib/inspect.py Log: Fix infinite regress when inspecting or frames. Modified: python/trunk/Lib/inspect.py ============================================================================== --- python/trunk/Lib/inspect.py (original) +++ python/trunk/Lib/inspect.py Sun Apr 30 17:59:26 2006 @@ -353,7 +353,13 @@ if 'b' in mode and string.lower(filename[-len(suffix):]) == suffix: # Looks like a binary file. We want to only return a text file. return None - if os.path.exists(filename) or hasattr(getmodule(object), '__loader__'): + if os.path.exists(filename): + return filename + # Ugly but necessary - '' and '' mean that getmodule() + # would infinitely recurse, because they're not real files nor loadable + # Note that this means that writing a PEP 302 loader that uses '<' + # at the start of a filename is now not a good idea. :( + if filename[:1]!='<' and hasattr(getmodule(object), '__loader__'): return filename def getabsfile(object): From python-checkins at python.org Sun Apr 30 18:01:39 2006 From: python-checkins at python.org (guido.van.rossum) Date: Sun, 30 Apr 2006 18:01:39 +0200 (CEST) Subject: [Python-checkins] r45823 - sandbox/trunk/sio/io.py Message-ID: <20060430160139.1F6211E400D@bag.python.org> Author: guido.van.rossum Date: Sun Apr 30 18:01:38 2006 New Revision: 45823 Added: sandbox/trunk/sio/io.py (contents, props changed) Log: Check in a very early draft for a Py3K I/O library. Don't look yet. Added: sandbox/trunk/sio/io.py ============================================================================== --- (empty file) +++ sandbox/trunk/sio/io.py Sun Apr 30 18:01:38 2006 @@ -0,0 +1,245 @@ +"""New I/O library for Python 3.0. + +This is inspired by sandbox/sio/sio.py but uses bytes for low-level +I/O and unicode strings for high-level I/O. + +""" + +# XXX What about thread-safety? + +__all__ = ["open"] + +import os + +DEFAULT_BUFSIZE = 8*1024 # International standard buffer size + +class IBinaryFile: + + """Abstract base class documenting the API for a binary file. + + All operations may raise IOError, OSError or some other low-level + I/O-related exception (e.g. socket.error). + + """ + + def readable(self): + """Returns True if the file supports the read() operation.""" + + def writable(self): + """Returns True if the file supports the write() operation.""" + + def seekable(self): + """Returns True if the file supports seek() and tell().""" + + def truncatable(self): + """Returns True if the file supports truncate().""" + + def read(self, n): + """Reads up to n bytes from a readable file. + + Returns a bytes object whose length is <= n. + + The returned object is empty only if EOF is reached. + + A short read is nothing unusal and doesn't mean that the next + read will report EOF. + + """ + + def write(self, b): + """Writes b, which must be a bytes object. + + Not all bytes may be written. At least one byte is written + unless b is empty. + + The return value is the number of bytes written; it is only + zero when b is empty. (When no bytes are written, an + exception is raised.) + + """ + + def tell(self): + """Returns the current byte offset from the start of the file.""" + + def seek(self, offset, whence=0): + """Positions the file at the given byte offset. + + If whence is 0, the offset is measured from the start of the + file. If it is 1, the offset is measured from the current + position. If it is 2, the offset is measured from the end of + the file. + + A negative offset may be specified if whence is 1 or 2 but the + effect is undefined if the resulting position is negative or + beyond EOF. + + Returns the new byte offset. + + """ + + def truncate(self, offset): + """Truncates the file at the given byte offset. + + Sets the file position to the new EOF position. + + The effect is undefined if offset is negative or beyond EOF. + + Returns the new file size, in bytes. + + """ + + +class OSBinaryFile: + + """A binary file using I/O on Unix file descriptors.""" + + def __init__(self, fd): + self._fd = fd + + def readable(self): + return True # May be a lie -- but how to tell the mode? + + def writable(self): + return True # May be a lie -- but how to tell the mode? + + def seekable(self): + return True # May be a lie -- but how to tell the mode? + + def truncatable(self): + return True # May be a lie -- but how to tell the mode? + + def read(self, n): + b = os.read(self._fd, n) + if not isinstance(b, bytes): + b = bytes(b) + return b + + def write(self, b): + assert isinstance(b, bytes) + return os.write(self._fd, b) + + def seek(self, pos, whence=0): + return os.lseek(self._fd, pos, whence) + + def tell(self): + return os.lseek(self._fd, 0, 1) + + def truncate(self, offset): + os.ftruncate(self._fd, offset) + return os.lseek(self._fd, 0, 2) + + # XXX ioctl, fcntl? What else? + + +class BufferingBinaryFileWrapper: + + """A buffering wrapper for a binary file. + + This provides exactly the same API but uses internal buffering to + speed up small reads and writes. An additional flush() method + forces data out to the underlying binary file or throws away + unused read-ahead data. + + WHen this is used for reading *and* writing, it assumes the + underlying file is seekable. For a socket, you need to open two + separate buffering wrappers: one for reading, and a separate one + for writing. + + """ + + def __init__(self, file, bufsize=DEFAULT_BUFSIZE): + bufsize = bufsize.__index__() + if bufsize <= 0: + raise ValueError("bufsize must be > 0, not %d" % bufsize) + self._file = file + self._bufsize = bufsize + self._buffer = bytes() + self._bufptr = 0 + self._writing = False + # Invariants: + # 0 <= self._bufptr <= len(self._buffer) <= self._bufsize + # Meaning of self._bufptr: + # if self._writing: + # self._buffer[ : self._bufptr] = data to be flushed + # else: + # self._buffer[self._bufptr : ] = data to be read + + def flush(self): + if self._writing: + start = 0 + while start < self._bufptr: + start += self._file.write(self._buffer[start : self._bufptr]) + self._bufptr = 0 + elif self._bufptr < len(self._buffer): + self._file.seek(self._bufptr - len(self._buffer), 1) + self._buffer = bytes() + self._bufptr = 0 + + def read(self, n): + n = n.__index__() + if not self._file.readable(): + raise IOError("file is not open for reading") + if self._writing: + self.flush() + self._writing = False + result = None + while n > 0: + if self._bufptr < len(self._buffer): + data = self._buffer[self._bufptr : self._bufptr + n] + self._bufptr += len(data) + n -= len(data) + if result is None: + result = data + else: + result += data + else: + self._buffer = self._file.read(max(n, self._bufsize)) + self._bufptr = 0 + if len(self._buffer) == 0: + break + if result is None: + result = bytes() + return result + + def write(self, b): + if not isinstance(b, bytes): + raise TypeError("write() requires bytes, not %r" % type(b)) + if not self._file.writable(): + raise IOError("file is not open for writing") + if not self._writing: + self.flush() + self._writing = True + if ((self._bufptr == 0 and len(b) >= self._bufsize) + or + (self._bufptr + len(b) >= 2*self._bufsize)): + self.flush() + start = 0 + while start < len(b): + if start == 0: + start += self._file.write(b) + else: + start += self._file.write(b[start : ]) + return len(b) + if self._bufptr == 0 and not self._buffer: + self._buffer = bytes(self._bufsize) + if self._bufptr + len(b) < len(self._buffer): + self._buffer[self._bufptr : self._bufptr + len(b)] = b + elif self._bufptr + len(b) == len(self._buffer): + self._buffer[self._bufptr : ] = b + self.flush() + else: + n = len(self._buffer) - self._bufptr + self._buffer[self._bufptr : ] = b[ : n] + self.flush() + self._bufptr[ : len(b) - n] = b[n : ] + self._bufptr = len(b) - n + + +def main(): + import sys + fd = sys.stdout.fileno() + f = OSBinaryFile(fd) + f.write(bytes(u"hello world\n", "ascii")) + +if __name__ == "__main__": + main() From buildbot at python.org Sun Apr 30 19:40:29 2006 From: buildbot at python.org (buildbot at python.org) Date: Sun, 30 Apr 2006 17:40:29 +0000 Subject: [Python-checkins] buildbot warnings in hppa Ubuntu dapper trunk Message-ID: <20060430174029.EB1881E4017@bag.python.org> The Buildbot has detected a new failure of hppa Ubuntu dapper trunk. Full details are available at: http://www.python.org/dev/buildbot/all/hppa%2520Ubuntu%2520dapper%2520trunk/builds/333 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: phillip.eby Build Had Warnings: warnings test sincerely, -The Buildbot From python-checkins at python.org Sun Apr 30 19:42:27 2006 From: python-checkins at python.org (georg.brandl) Date: Sun, 30 Apr 2006 19:42:27 +0200 (CEST) Subject: [Python-checkins] r45824 - python/trunk/Lib/inspect.py Message-ID: <20060430174227.455D91E400D@bag.python.org> Author: georg.brandl Date: Sun Apr 30 19:42:26 2006 New Revision: 45824 Modified: python/trunk/Lib/inspect.py Log: Fix another problem in inspect: if the module for an object cannot be found, don't try to give its __dict__ to linecache. Modified: python/trunk/Lib/inspect.py ============================================================================== --- python/trunk/Lib/inspect.py (original) +++ python/trunk/Lib/inspect.py Sun Apr 30 19:42:26 2006 @@ -412,7 +412,11 @@ in the file and the line number indexes a line in that list. An IOError is raised if the source code cannot be retrieved.""" file = getsourcefile(object) or getfile(object) - lines = linecache.getlines(file, getmodule(object).__dict__) + module = getmodule(object) + if module: + lines = linecache.getlines(file, module.__dict__) + else: + lines = linecache.getlines(file) if not lines: raise IOError('could not get source code') From python-checkins at python.org Sun Apr 30 20:14:55 2006 From: python-checkins at python.org (georg.brandl) Date: Sun, 30 Apr 2006 20:14:55 +0200 (CEST) Subject: [Python-checkins] r45825 - in python/trunk: Doc/lib/librlcompleter.tex Lib/rlcompleter.py Lib/test/test_sundry.py Misc/NEWS Message-ID: <20060430181455.484411E400D@bag.python.org> Author: georg.brandl Date: Sun Apr 30 20:14:54 2006 New Revision: 45825 Modified: python/trunk/Doc/lib/librlcompleter.tex python/trunk/Lib/rlcompleter.py python/trunk/Lib/test/test_sundry.py python/trunk/Misc/NEWS Log: Patch #1472854: make the rlcompleter.Completer class usable on non- UNIX platforms. Modified: python/trunk/Doc/lib/librlcompleter.tex ============================================================================== --- python/trunk/Doc/lib/librlcompleter.tex (original) +++ python/trunk/Doc/lib/librlcompleter.tex Sun Apr 30 20:14:54 2006 @@ -2,18 +2,17 @@ Completion function for GNU readline} \declaremodule{standard}{rlcompleter} - \platform{Unix} \sectionauthor{Moshe Zadka}{moshez at zadka.site.co.il} -\modulesynopsis{Python identifier completion for the GNU readline library.} +\modulesynopsis{Python identifier completion, suitable for the GNU readline library.} -The \module{rlcompleter} module defines a completion function for +The \module{rlcompleter} module defines a completion function suitable for the \refmodule{readline} module by completing valid Python identifiers and keywords. -This module is \UNIX-specific due to its dependence on the -\refmodule{readline} module. - -The \module{rlcompleter} module defines the \class{Completer} class. +When this module is imported on a \UNIX\ platform with the \module{readline} +module available, an instance of the \class{Completer} class is automatically +created and its \method{complete} method is set as the \module{readline} +completer. Example: @@ -44,6 +43,9 @@ \end{verbatim} +On platforms without \module{readline}, the \class{Completer} class defined +by this module can still be used for custom purposes. + \subsection{Completer Objects \label{completer-objects}} Completer objects have the following method: Modified: python/trunk/Lib/rlcompleter.py ============================================================================== --- python/trunk/Lib/rlcompleter.py (original) +++ python/trunk/Lib/rlcompleter.py Sun Apr 30 20:14:54 2006 @@ -39,7 +39,6 @@ """ -import readline import __builtin__ import __main__ @@ -147,4 +146,9 @@ ret = ret + get_class_members(base) return ret -readline.set_completer(Completer().complete) +try: + import readline +except ImportError: + pass +else: + readline.set_completer(Completer().complete) Modified: python/trunk/Lib/test/test_sundry.py ============================================================================== --- python/trunk/Lib/test/test_sundry.py (original) +++ python/trunk/Lib/test/test_sundry.py Sun Apr 30 20:14:54 2006 @@ -50,11 +50,7 @@ import py_compile import pydoc import rexec -try: - import rlcompleter # not available on Windows -except ImportError: - if verbose: - print "skipping rlcompleter" +import rlcompleter import sched import smtplib import sndhdr Modified: python/trunk/Misc/NEWS ============================================================================== --- python/trunk/Misc/NEWS (original) +++ python/trunk/Misc/NEWS Sun Apr 30 20:14:54 2006 @@ -86,6 +86,9 @@ Library ------- +- Patch #1472854: make the rlcompleter.Completer class usable on non- + UNIX platforms. + - Patch #1470846: fix urllib2 ProxyBasicAuthHandler. - Patch #1475231: ``doctest`` has a new ``SKIP`` option, which causes From buildbot at python.org Sun Apr 30 20:36:01 2006 From: buildbot at python.org (buildbot at python.org) Date: Sun, 30 Apr 2006 18:36:01 +0000 Subject: [Python-checkins] buildbot warnings in amd64 gentoo trunk Message-ID: <20060430183601.4AC381E400D@bag.python.org> The Buildbot has detected a new failure of amd64 gentoo trunk. Full details are available at: http://www.python.org/dev/buildbot/all/amd64%2520gentoo%2520trunk/builds/693 Buildbot URL: http://www.python.org/dev/buildbot/all/ Build Reason: Build Source Stamp: [branch trunk] HEAD Blamelist: georg.brandl Build Had Warnings: warnings test sincerely, -The Buildbot From python-checkins at python.org Sun Apr 30 21:34:20 2006 From: python-checkins at python.org (georg.brandl) Date: Sun, 30 Apr 2006 21:34:20 +0200 (CEST) Subject: [Python-checkins] r45826 - python/trunk/Doc/ref/ref3.tex python/trunk/Doc/ref/ref7.tex Message-ID: <20060430193420.63E011E400D@bag.python.org> Author: georg.brandl Date: Sun Apr 30 21:34:19 2006 New Revision: 45826 Modified: python/trunk/Doc/ref/ref3.tex python/trunk/Doc/ref/ref7.tex Log: Patch #1479438: add \keyword markup for "with". Modified: python/trunk/Doc/ref/ref3.tex ============================================================================== --- python/trunk/Doc/ref/ref3.tex (original) +++ python/trunk/Doc/ref/ref3.tex Sun Apr 30 21:34:19 2006 @@ -2149,9 +2149,9 @@ than writing individual \method{__enter__()} and \method{__exit__()} methods on a separate object when the state to be managed is complex. -With statement context objects also need to implement this method; they -are required to return themselves (that is, this method will simply -return \var{self}). +\keyword{with} statement context objects also need to implement this +method; they are required to return themselves (that is, this method +will simply return \var{self}). \end{methoddesc} \begin{methoddesc}[with statement context]{__enter__}{self} Modified: python/trunk/Doc/ref/ref7.tex ============================================================================== --- python/trunk/Doc/ref/ref7.tex (original) +++ python/trunk/Doc/ref/ref7.tex Sun Apr 30 21:34:19 2006 @@ -315,7 +315,7 @@ \versionadded{2.5} The \keyword{with} statement is used to wrap the execution of a block -with methods defined by a context manager or with statement context +with methods defined by a context manager or \keyword{with} statement context object (see section~\ref{context-managers}). This allows common \keyword{try}...\keyword{except}...\keyword{finally} usage patterns to be encapsulated for convenient reuse. @@ -332,7 +332,7 @@ \item The context expression is evaluated, to obtain a context manager. \item The context manger's \method{__context__()} method is -invoked to obtain a with statement context object. +invoked to obtain a \keyword{with} statement context object. \item The context object's \method{__enter__()} method is invoked. From python-checkins at python.org Sun Apr 30 23:19:32 2006 From: python-checkins at python.org (andrew.kuchling) Date: Sun, 30 Apr 2006 23:19:32 +0200 (CEST) Subject: [Python-checkins] r45827 - python/trunk/Doc/howto/urllib2.rst Message-ID: <20060430211932.A017B1E400D@bag.python.org> Author: andrew.kuchling Date: Sun Apr 30 23:19:31 2006 New Revision: 45827 Added: python/trunk/Doc/howto/urllib2.rst Log: Add urllib2 HOWTO from Michael Foord Added: python/trunk/Doc/howto/urllib2.rst ============================================================================== --- (empty file) +++ python/trunk/Doc/howto/urllib2.rst Sun Apr 30 23:19:31 2006 @@ -0,0 +1,410 @@ +============================================== + HOWTO Fetch Internet Resources Using urllib2 +============================================== +------------------------------------------ + Fetching URLs With Python +------------------------------------------ + + +.. note:: + + There is an French translation of this HOWTO, available at `urllib2 - Le Manuel manquant `_. + +.. contents:: urllib2 Tutorial + + +Introduction +============ + +.. sidebar:: Related Articles + + You may also find useful the following articles on fetching web resources with Python : + + * `Basic Authentication `_ + + A tutorial on *Basic Authentication*, with exampels in Python. + + * `cookielib and ClientCookie `_ + + How to handle cookies, when fetching web pages with Python. + + This HOWTO is written by `Michael Foord `_. + +**urllib2** is a Python_ module for fetching URLs (Uniform Resource Locators). It offers a very simple interface, in the form of the *urlopen* function. This is capable of fetching URLs using a variety of different protocols. It also offers a slightly more complex interface for handling common situations - like basic authentication, cookies, proxies, and so on. These are provided by objects called handlers and openers. + +For straightforward situations *urlopen* is very easy to use. But as soon as you encounter errors, or non-trivial cases, you will need some understanding of the HyperText Transfer Protocol. The most comprehensive reference to HTTP is :RFC:`2616`. This is a technical document and not intended to be easy to read. This HOWTO aims to illustrate using *urllib2*, with enough detail about HTTP to help you through. It is not intended to replace the `urllib2 docs`_ [#]_, but is supplementary to them. + + +Fetching URLs +============= + +HTTP is based on requests and responses - the client makes requests and servers send responses. Python mirrors this by having you form a ``Request`` object which represents the request you are making. In it's simplest form you create a Request object that specifies the URL you want to fetch [#]_. Calling ``urlopen`` with this Request object returns a handle on the page requested. This handle is a file like object : :: + + import urllib2 + + the_url = 'http://www.voidspace.org.uk' + req = urllib2.Request(the_url) + handle = urllib2.urlopen(req) + the_page = handle.read() + +There are two extra things that Request objects allow you to do. Sometimes you want to **POST** data to a CGI (Common Gateway Interface) [#]_ or other web application. This is what your browser does when you fill in a FORM on the web. You may be mimicking a FORM submission, or transmitting data to your own application. In either case the data needs to be encoded for safe transmission over HTTP, and then passed to the Request object as the ``data`` argument. The encoding is done using a function from the ``urllib`` library *not* from ``urllib2``. :: + + import urllib + import urllib2 + + the_url = 'http://www.someserver.com/cgi-bin/register.cgi' + values = {'name' : 'Michael Foord', + 'location' : 'Northampton', + 'language' : 'Python' } + + data = urllib.urlencode(values) + req = urllib2.Request(the_url, data) + handle = urllib2.urlopen(req) + the_page = handle.read() + +Some websites [#]_ dislike being browsed by programs, or send different versions to different browsers [#]_ . By default urllib2 identifies itself as ``Python-urllib/2.4``, which may confuse the site, or just plain not work. The way a browser identifies itself is through the ``User-Agent`` header [#]_. When you create a Request object you can pass a dictionary of headers in. The following example makes the same request as above, but identifies itself as a version of Internet Explorer [#]_. :: + + import urllib + import urllib2 + + the_url = 'http://www.someserver.com/cgi-bin/register.cgi' + user_agent = 'Mozilla/4.0 (compatible; MSIE 5.5; Windows NT)' + values = {'name' : 'Michael Foord', + 'location' : 'Northampton', + 'language' : 'Python' } + headers = { 'User-Agent' : user_agent } + + data = urllib.urlencode(values) + req = urllib2.Request(the_url, data, headers) + handle = urllib2.urlopen(req) + the_page = handle.read() + +The handle also has two useful methods. See the section on `info and geturl`_ which comes after we have a look at what happens when things go wrong. + + +Coping With Errors +================== + +*urlopen* raises ``URLError`` or ``HTTPError`` in the event of an error. ``HTTPError`` is a subclass of ``URLError``, which is a subclass of ``IOError``. This means you can trap for ``IOError`` if you want. :: + + req = urllib2.Request(some_url) + try: + handle = urllib2.urlopen(req) + except IOError: + print 'Something went wrong' + else: + print handle.read() + +URLError +-------- + +If the request fails to reach a server then urlopen will raise a ``URLError``. This will usually be because there is no network connection (no route to the specified server), or the specified server doesn't exist. + +In this case, the exception raised will have a 'reason' attribute, which is a tuple containing an error code and a text error message. + +e.g. :: + + >>> req = urllib2.Request('http://www.pretend_server.org') + >>> try: urllib2.urlopen(req) + >>> except IOError, e: + >>> print e.reason + >>> + (4, 'getaddrinfo failed') + + +HTTPError +--------- + +If the request reaches a server, but the server is unable to fulfil the request, it returns an error code. The default handlers will hande some of these errors for you. For those it can't handle, urlopen will raise an ``HTTPError``. Typical errors include '404' (page not found), '403' (request forbidden), and '401' (authentication required). + +See http://www.w3.org/Protocols/HTTP/HTRESP.html for a reference on all the http error codes. + +The ``HTTPError`` instance raised will have an integer 'code' attribute, which corresponds to the error sent by the server. + +There is a useful dictionary of response codes in ``HTTPBaseServer``, that shows all the defined response codes. Because the default handlers handle redirects (codes in the 300 range), and codes in the 100-299 range indicate success, you will usually only see error codes in the 400-599 range. + +Error Codes +~~~~~~~~~~~ + +.. note:: + + As of Python 2.5 a dictionary like this one has become part of ``urllib2``. + +:: + + # Table mapping response codes to messages; entries have the + # form {code: (shortmessage, longmessage)}. + httpresponses = { + 100: ('Continue', 'Request received, please continue'), + 101: ('Switching Protocols', + 'Switching to new protocol; obey Upgrade header'), + + 200: ('OK', 'Request fulfilled, document follows'), + 201: ('Created', 'Document created, URL follows'), + 202: ('Accepted', + 'Request accepted, processing continues off-line'), + 203: ('Non-Authoritative Information', + 'Request fulfilled from cache'), + 204: ('No response', 'Request fulfilled, nothing follows'), + 205: ('Reset Content', 'Clear input form for further input.'), + 206: ('Partial Content', 'Partial content follows.'), + + 300: ('Multiple Choices', + 'Object has several resources -- see URI list'), + 301: ('Moved Permanently', + 'Object moved permanently -- see URI list'), + 302: ('Found', 'Object moved temporarily -- see URI list'), + 303: ('See Other', 'Object moved -- see Method and URL list'), + 304: ('Not modified', + 'Document has not changed since given time'), + 305: ('Use Proxy', + 'You must use proxy specified in Location' + ' to access this resource.'), + 307: ('Temporary Redirect', + 'Object moved temporarily -- see URI list'), + + 400: ('Bad request', + 'Bad request syntax or unsupported method'), + 401: ('Unauthorized', + 'No permission -- see authorization schemes'), + 402: ('Payment required', + 'No payment -- see charging schemes'), + 403: ('Forbidden', + 'Request forbidden -- authorization will not help'), + 404: ('Not Found', 'Nothing matches the given URI'), + 405: ('Method Not Allowed', + 'Specified method is invalid for this server.'), + 406: ('Not Acceptable', + 'URI not available in preferred format.'), + 407: ('Proxy Authentication Required', + 'You must authenticate with ' + 'this proxy before proceeding.'), + 408: ('Request Time-out', + 'Request timed out; try again later.'), + 409: ('Conflict', 'Request conflict.'), + 410: ('Gone', + 'URI no longer exists and has been permanently removed.'), + 411: ('Length Required', 'Client must specify Content-Length.'), + 412: ('Precondition Failed', + 'Precondition in headers is false.'), + 413: ('Request Entity Too Large', 'Entity is too large.'), + 414: ('Request-URI Too Long', 'URI is too long.'), + 415: ('Unsupported Media Type', + 'Entity body in unsupported format.'), + 416: ('Requested Range Not Satisfiable', + 'Cannot satisfy request range.'), + 417: ('Expectation Failed', + 'Expect condition could not be satisfied.'), + + 500: ('Internal error', 'Server got itself in trouble'), + 501: ('Not Implemented', + 'Server does not support this operation'), + 502: ('Bad Gateway', + 'Invalid responses from another server/proxy.'), + 503: ('Service temporarily overloaded', + 'The server cannot ' + 'process the request due to a high load'), + 504: ('Gateway timeout', + 'The gateway server did not receive a timely response'), + 505: ('HTTP Version not supported', 'Cannot fulfill request.'), + } + +When an error is raised the server responds by returning an http error code *and* an error page. You can use the ``HTTPError`` instance as a handle on the page returned. This means that as well as the code attribute, it also has read, geturl, and info, methods. :: + + >>> req = urllib2.Request('http://www.python.org/fish.html') + >>> try: + >>> urllib2.urlopen(req) + >>> except IOError, e: + >>> print e.code + >>> print e.read() + >>> + 404 + + + Error 404: File Not Found + ...... etc... + +Wrapping it Up +-------------- + +So if you want to be prepared for ``HTTPError`` *or* ``URLError`` there are two +basic approaches. I prefer the second approach. + +Number 1 +~~~~~~~~ + +:: + + + from urllib2 import Request, urlopen, URLError, HTTPError + req = Request(someurl) + try: + handle = urlopen(req) + except HTTPError, e: + print 'The server couldn\'t fulfill the request.' + print 'Error code: ', e.code + except URLError, e: + print 'We failed to reach a server.' + print 'Reason: ', e.reason + else: + # everything is fine + + +.. note:: + + The ``except HTTPError`` *must* come first, otherwise ``except URLError`` will *also* catch an ``HTTPError``. + +Number 2 +~~~~~~~~ + +:: + + from urllib2 import Request, urlopen + req = Request(someurl) + try: + handle = urlopen(req) + except IOError, e: + if hasattr(e, 'reason'): + print 'We failed to reach a server.' + print 'Reason: ', e.reason + elif hasattr(e, 'code'): + print 'The server couldn\'t fulfill the request.' + print 'Error code: ', e.code + else: + # everything is fine + + +info and geturl +=============== + +The handle returned by urlopen (or the ``HTTPError`` instance) has two useful methods ``info`` and ``geturl``. + +**geturl** - this returns the real url of the page fetched. This is useful because ``urlopen`` (or the openener object used) may have followed a redirect. The url of the page fetched may not be the same as the url requested. + +**info** - this returns a dictionary like object that describes the page fetched, particularly the headers sent by the server. It is actually an ``httplib.HTTPMessage`` instance. In versions of Python prior to 2.3.4 it wasn't safe to iterate over the object directly, so you should iterate over the list returned by ``msg.keys()`` instead. + +Typical headers include 'content-length', 'content-type', and so on. See the `Quick Reference to HTTP Headers`_ for a useful reference on the different sort of headers. + + +Openers and Handlers +==================== + +Openers and handlers are slightly esoteric parts of **urllib2**. When you fetch a URL you use an opener. Normally we have been using the default opener - via ``urlopen`` - but you can create custom openers. Openers use handlers. + +``build_opener`` is used to create ``opener`` objects - for fetching URLs with specific handlers installed. Handlers can handle cookies, authentication, and other common but slightly specialised situations. Opener objects have an ``open`` method, which can be called directly to fetch urls in the same way as the ``urlopen`` function. + +``install_opener`` can be used to make an ``opener`` object the default opener. This means that calls to ``urlopen`` will use the opener you have installed. + + +Basic Authentication +==================== + +To illustrate creating and installing a handler we will use the ``HTTPBasicAuthHandler``. For a more detailed discussion of this subject - including an explanation of how Basic Authentication works - see the `Basic Authentication Tutorial`_. + +When authentication is required, the server sends a header (as well as the 401 error code) requesting authentication. This specifies the authentication scheme and a 'realm'. The header looks like : ``www-authenticate: SCHEME realm="REALM"``. + +e.g. :: + + www-authenticate: Basic realm="cPanel" + + +The client should then retry the request with the appropriate name and password for the realm included as a header in the request. This is 'basic authentication'. In order to simplify this process we can create an instance of ``HTTPBasicAuthHandler`` and an opener to use this handler. + +The ``HTTPBasicAuthHandler`` uses an object called a password manager to handle the mapping of URIs and realms to passwords and usernames. If you know what the realm is (from the authentication header sent by the server), then you can use a ``HTTPPasswordMgr``. Generally there is only one realm per URI, so it is possible to use ``HTTPPasswordMgrWithDefaultRealm``. This allows you to specify a default username and password for a URI. This will be supplied in the absence of yoou providing an alternative combination for a specific realm. We signify this by providing ``None`` as the realm argument to the ``add_password`` method. + +The toplevelurl is the first url that requires authentication. This is usually a 'super-url' of any others in the same realm. :: + + password_mgr = urllib2.HTTPPasswordMgrWithDefaultRealm() + # create a password manager + + password_mgr.add_password(None, + top_level_url, username, password) + # add the username and password + # if we knew the realm, we could + # use it instead of ``None`` + + handler = urllib2.HTTPBasicAuthHandler(password_mgr) + # create the handler + + opener = urllib2.build_opener(handler) + # from handler to opener + + opener.open(a_url) + # use the opener to fetch a URL + + urllib2.install_opener(opener) + # install the opener + # now all calls to urllib2.urlopen use our opener + +.. note:: + + In the above example we only supplied our ``HHTPBasicAuthHandler`` to ``build_opener``. By default openers have the handlers for normal situations - ``ProxyHandler``, ``UnknownHandler``, ``HTTPHandler``, ``HTTPDefaultErrorHandler``, ``HTTPRedirectHandler``, ``FTPHandler``, ``FileHandler``, ``HTTPErrorProcessor``. The only reason to explicitly supply these to ``build_opener`` (which chains handlers provided as a list), would be to change the order they appear in the chain. + +One thing not to get bitten by is that the ``top_level_url`` in the code above *must not* contain the protocol - the ``http://`` part. So if the URL we are trying to access is ``http://www.someserver.com/path/page.html``, then we set : :: + + top_level_url = "www.someserver.com/path/page.html" + # *no* http:// !! + +It took me a long time to track that down the first time I tried to use handlers. + + +Proxies +======= + +**urllib2** will auto-detect your proxy settings and use those. This is through the ``ProxyHandler`` which is part of the normal handler chain. Normally that's a good thing, but there are occasions when it may not be helpful [#]_. In order to do this we need to setup our own ``ProxyHandler``, with no proxies defined. This is done using similar steps to setting up a `Basic Authentication`_ handler : :: + + >>> proxy_support = urllib2.ProxyHandler({}) + >>> opener = urllib2.build_opener(proxy_support) + >>> urllib2.install_opener(opener) + +.. caution:: + + Currently ``urllib2`` *does not* support fetching of ``https`` locations through + a proxy. This can be a problem. + +Sockets and Layers +================== + +The Python support for fetching resources from the web is layered. urllib2 uses the httplib library, which in turn uses the socket library. + +As of Python 2.3 you can specify how long a socket should wait for a response before timing out. This can be useful in applications which have to fetch web pages. By default the socket module has *no timeout* and can hang. To set the timeout use : :: + + import socket + import urllib2 + + timeout = 10 + # timeout in seconds + socket.setdefaulttimeout(timeout) + + req = urllib2.Request('http://www.voidspace.org.uk') + handle = urllib2.urlopen(req) + # this call to urllib2.urlopen + # now uses the default timeout + # we have set in the socket module + + +------- + + +Footnotes +=========== + +.. [#] Possibly some of this tutorial will make it into the standard library docs for versions of Python after 2.4.1. +.. [#] You *can* fetch URLs directly with urlopen, without using a request object. It's more explicit, and therefore more Pythonic, to use ``urllib2.Request`` though. It also makes it easier to add headers to your request. +.. [#] For an introduction to the CGI protocol see `Writing Web Applications in Python`_. +.. [#] Like Google for example. The *proper* way to use google from a program is to use PyGoogle_ of course. See `Voidspace Google`_ for some examples of using the Google API. +.. [#] Browser sniffing is a very bad practise for website design - building sites using web standards is much more sensible. Unfortunately a lot of sites still send different versions to different browsers. +.. [#] The user agent for MSIE 6 is *'Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)'* +.. [#] For details of more HTTP request headers, see `Quick Reference to HTTP Headers`_. + +.. [#] In my case I have to use a proxy to access the internet at work. If you attempt to fetch *localhost* URLs through this proxy it blocks them. IE is set to use the proxy, which urllib2 picks up on. In order to test scripts with a localhost server, I have to prevent urllib2 from using the proxy. + +.. _Python: http://www.python.org +.. _urllib2 docs: http://docs.python.org/lib/module-urllib2.html +.. _Quick Reference to HTTP Headers: http://www.cs.tut.fi/~jkorpela/http.html +.. _PyGoogle: http://pygoogle.sourceforge.net +.. _Voidspace Google: http://www.voidspace.org.uk/python/recipebook.shtml#google +.. _Writing Web Applications in Python: http://www.pyzine.com/Issue008/Section_Articles/article_CGIOne.html +.. _Basic Authentication Tutorial: http://www.voidspace.org.uk/python/articles/authentication.shtml \ No newline at end of file From python-checkins at python.org Sun Apr 30 23:19:50 2006 From: python-checkins at python.org (andrew.kuchling) Date: Sun, 30 Apr 2006 23:19:50 +0200 (CEST) Subject: [Python-checkins] r45828 - python/trunk/Doc/whatsnew/whatsnew25.tex Message-ID: <20060430211950.24E561E400D@bag.python.org> Author: andrew.kuchling Date: Sun Apr 30 23:19:49 2006 New Revision: 45828 Modified: python/trunk/Doc/whatsnew/whatsnew25.tex Log: Add item Modified: python/trunk/Doc/whatsnew/whatsnew25.tex ============================================================================== --- python/trunk/Doc/whatsnew/whatsnew25.tex (original) +++ python/trunk/Doc/whatsnew/whatsnew25.tex Sun Apr 30 23:19:49 2006 @@ -1468,6 +1468,12 @@ \code{sys.path}, so unless your programs explicitly added the directory to \code{sys.path}, this removal shouldn't affect your code. +\item The \module{rlcompleter} module is no longer +dependent on importing the \module{readline} module and +therefore now works on non-{\UNIX} platforms. +(Patch from Robert Kiendl.) +% Patch #1472854 + \item The \module{socket} module now supports \constant{AF_NETLINK} sockets on Linux, thanks to a patch from Philippe Biondi. Netlink sockets are a Linux-specific mechanism for communications From python-checkins at python.org Sun Apr 30 23:26:42 2006 From: python-checkins at python.org (barry.warsaw) Date: Sun, 30 Apr 2006 23:26:42 +0200 (CEST) Subject: [Python-checkins] r45829 - in python/branches/release24-maint/Lib: email/_parseaddr.py email/test/test_email.py rfc822.py test/test_rfc822.py Message-ID: <20060430212642.A72671E400E@bag.python.org> Author: barry.warsaw Date: Sun Apr 30 23:26:41 2006 New Revision: 45829 Modified: python/branches/release24-maint/Lib/email/_parseaddr.py python/branches/release24-maint/Lib/email/test/test_email.py python/branches/release24-maint/Lib/rfc822.py python/branches/release24-maint/Lib/test/test_rfc822.py Log: Patch #1464708 from William McVey: fixed handling of nested comments in mail addresses. E.g. "Foo ((Foo Bar)) " Fixes for both rfc822.py and email package. This patch needs to be back ported to Python 2.3 for email 2.5 and forward ported to Python 2.5 for email 4.0. (I'll do both soon) Modified: python/branches/release24-maint/Lib/email/_parseaddr.py ============================================================================== --- python/branches/release24-maint/Lib/email/_parseaddr.py (original) +++ python/branches/release24-maint/Lib/email/_parseaddr.py Sun Apr 30 23:26:41 2006 @@ -360,6 +360,7 @@ break elif allowcomments and self.field[self.pos] == '(': slist.append(self.getcomment()) + continue # have already advanced pos from getcomment elif self.field[self.pos] == '\\': quote = True else: Modified: python/branches/release24-maint/Lib/email/test/test_email.py ============================================================================== --- python/branches/release24-maint/Lib/email/test/test_email.py (original) +++ python/branches/release24-maint/Lib/email/test/test_email.py Sun Apr 30 23:26:41 2006 @@ -2212,6 +2212,12 @@ ['foo: ;', '"Jason R. Mastaler" ']), [('', ''), ('Jason R. Mastaler', 'jason at dom.ain')]) + def test_getaddresses_embedded_comment(self): + """Test proper handling of a nested comment""" + eq = self.assertEqual + addrs = Utils.getaddresses(['User ((nested comment)) ']) + eq(addrs[0][1], 'foo at bar.com') + def test_utils_quote_unquote(self): eq = self.assertEqual msg = Message() Modified: python/branches/release24-maint/Lib/rfc822.py ============================================================================== --- python/branches/release24-maint/Lib/rfc822.py (original) +++ python/branches/release24-maint/Lib/rfc822.py Sun Apr 30 23:26:41 2006 @@ -711,6 +711,7 @@ break elif allowcomments and self.field[self.pos] == '(': slist.append(self.getcomment()) + continue # have already advanced pos from getcomment elif self.field[self.pos] == '\\': quote = 1 else: Modified: python/branches/release24-maint/Lib/test/test_rfc822.py ============================================================================== --- python/branches/release24-maint/Lib/test/test_rfc822.py (original) +++ python/branches/release24-maint/Lib/test/test_rfc822.py Sun Apr 30 23:26:41 2006 @@ -45,6 +45,10 @@ print 'extra parsed address:', repr(n), repr(a) continue i = i + 1 + self.assertEqual(mn, n, + "Un-expected name: %s != %s" % (`mn`, `n`)) + self.assertEqual(ma, a, + "Un-expected address: %s != %s" % (`ma`, `a`)) if mn == n and ma == a: pass else: @@ -129,6 +133,12 @@ 'To: person at dom.ain (User J. Person)\n\n', [('User J. Person', 'person at dom.ain')]) + def test_doublecomment(self): + # The RFC allows comments within comments in an email addr + self.check( + 'To: person at dom.ain ((User J. Person)), John Doe \n\n', + [('User J. Person', 'person at dom.ain'), ('John Doe', 'foo at bar.com')]) + def test_twisted(self): # This one is just twisted. I don't know what the proper # result should be, but it shouldn't be to infloop, which is