Mailman 3 February 2003 - Python-Dev

Import lock knowledge required!
by Mark Hammond 17 Feb '03

17 Feb '03

I've found a bug related to the import lock causing a deadlock in 2.3. The bug can be found at http://www.python.org/sf/683658 The bug is fairly self-explanatory, so I won't repeat it here. The proposed patch exposes the import lock to the Python C API, allowing optionally a non-blocking attempt to acquire the lock. I should note that the only reason I propose this is because I can see no other option! I would appreciate it if people knowledgeable about the import lock and its implications could have a quick look at this bug, and comment on my proposed patch. Thanks, Mark.

5 5

incorrect regression tests
by Neal Norwitz 17 Feb '03

17 Feb '03

I found 7 regression tests which weren't being executed by regrtest. I checked in fixes for 6 of them. test_zipimport was the other test, but there was a problem. More on that below. To be run, tests must do one of following: 1) have a def test_main() which runs the tests 2) run the tests as a side-effect of import I'm wondering if regrtest.py should grow an option to print a list of all the tests which don't define test_main? When adding a new test, it will be easy to verify the test is, in fact, being run. Perhaps also a list of known tests which don't have a test_main and are expected to run on import? (I think there's over 100.) When test_zipimport is run by regrtest, there is memory corruption: [neal@epoch 2.3]$ ./python -E -tt ./Lib/test/regrtest.py test_zipimport test_zipimport Debug memory block at address p=0x40280508: 74 bytes originally requested The 4 pad bytes at p-4 are FORBIDDENBYTE, as expected. The 4 pad bytes at tail=0x40280552 are not all FORBIDDENBYTE (0xfb): at tail+0: 0x00 *** OUCH at tail+1: 0xfb at tail+2: 0xfb at tail+3: 0xfb The block was made by call #51959 to debug malloc/realloc. Data at p: 64 65 66 20 67 65 74 5f ... 69 6c 65 5f 5f 0a 00 0a Fatal Python error: bad trailing pad byte Aborted (core dumped) I didn't check in the test or investigate the cause. Below is a patch to the test so it will run under regrtest. Neal -- Index: test_zipimport.py =================================================================== RCS file: /cvsroot/python/python/dist/src/Lib/test/test_zipimport.py,v retrieving revision 1.5 diff -w -u -r1.5 test_zipimport.py --- test_zipimport.py 9 Jan 2003 22:27:10 -0000 1.5 +++ test_zipimport.py 17 Feb 2003 14:58:45 -0000 @@ -186,6 +186,9 @@ compression = ZIP_DEFLATED -if __name__ == "__main__": +def test_main(): test_support.run_unittest(UncompressedZipImportTestCase) test_support.run_unittest(CompressedZipImportTestCase) + +if __name__ == "__main__": + test_main()

2 1

Re: [Python-Dev] non-binary operators
by Gerald S. Williams 17 Feb '03

17 Feb '03

Gary Herron wrote: > Good point. I've never actually used (a<b<c), but I'm pleasently > surprised when reminded of it's existence. There is however a > difference. A whole string of '<' operators (or even of string of > mixed '<' and '>' operators) can be understood by considering each > operation in isolation as a binary operation. Trying hard to avoid PEP 308 discussions: :-) Yes and no. You cannot collapse the expression, so these binary "operations" are really only partial operations. Granted, with th*n/el*e the left-most operand participates in both partial operations, but I'll leave that discussion to c.l.py. I was just pointing out that there's precedent for (a OP b OP c) != ((a OP b) OP c) != (a OP (b OP c)). -Jerry

1 0

CALL_ATTR, A Method Proposal
by Glyph Lefkowitz 17 Feb '03

17 Feb '03

I have an idea that may result in significant optimization of python. However, I have an incomplete understanding of what's involved, and I haven't had enough time to puzzle out enough of Python's internals to write up a full PEP describing this. Method calls appear are a full 20% slower (simple benchmark included) than function calls, and Python function calls are already pretty slow. By my understanding, one of the reasons for the difference is that if you have a method call like this: a = A() a.b() what's really happening is something along the lines of: temp = new.instancemethod(getattr(a.__class__, "b"), a, A) temp() free(temp) This causes an unnecessary memory allocation: since the instancemethod object is immediately being created, then called, then garbage collected. Looking at the output of dis.dis, I can see there are also 3 bytecodes being evaluated rather than 1. My proposal is to treat method calls as syntactically different from function calls. Rather than x.y() being completely synonymous with getattr(x, "y")(), it could be analogous to 'x.y = z' or 'del x.y'. For symmetry with these statement types, the new bytecode could be called CALL_ATTR. I think this is an important thing to consider as systems like Zope and Twisted move towards using component models and Interfaces in Python. The fact that direct function calls are so much faster puts efficiency directly at odds with structured flexibility. With a method call primitive comparable to function calls, most python code, especially in systems that make heavy use of inter-object communication patterns, would immediately get as much as a 15% speed boost. CALL_ATTR should be implementable with no impact on existing python code, except bytecode hacks. It should be possible to retain a fully backwards-compatible __getattr__ method, for places where method objects are used (including the C API). Likewise, the default __callattr__ could be set up to first check if __getattr__ is defined, then the instance's dictionary or __slots__. For additional speed gains, new-object-model classes could set '__fast_methods__ = True' and gain a semantic distinction between __getattr__ and __callattr__. Better still, I think that Jython could use the subtle semantic change to make Java reflection less expensive. (Java's `new' is more expensive than C's `malloc', after all.) I have a sneaking suspicion that this would also be good for security purposes. I haven't yet come up with a specific case where this is a big deal, but I think capability-style data-hiding would be simplified if filtering method-calls were different from filtering attributes. I hope this idea is useful to some of you, -- | <`'> | Glyph Lefkowitz: Traveling Sorcerer | | < _/ > | Lead Developer, the Twisted project | | < ___/ > | http://www.twistedmatrix.com | import time class A: def b(self): pass def b(self): pass a = A() def wallclock(f): then = time.time() f() now = time.time() elapsed = now - then return elapsed NCALLS = 1000000 def methods(): for x in xrange(NCALLS): a.b() def functions(): for x in xrange(NCALLS): b(a) print wallclock(methods) print wallclock(functions) % python2.0 methods.py 1.85362696648 1.47611200809 % python2.1 methods.py 1.21098303795 0.972733020782 % python2.2 methods.py 1.15857589245 0.914402961731 % jython methods.py 63.396000027656555 51.51300001144409

7 12

Changes to logging
by Vinay Sajip 17 Feb '03

17 Feb '03

I've got a set of patches which I want to make to logging: 1. Added the record being processed as a parameter to handleError. 2. Handler.handle returns the result of applying the filter to the record. 3. Added a seek(0, 2) in RotatingFileHandler before the tell() call. This is because under Windows, tell() returns 0 until the first actual write. 4. Renamed warn and WARN to warning and WARNING. This may break existing code, but the standard Python module will use warning/WARNING rather than warn/WARN. The fatal and FATAL synonyms for critical and CRITICAL have also been removed. 5. Added a defaultEncoding attribute and some simple support for encoding Unicode messages. 6. Added process ID to the list of LogRecord attributes. 7. Modified Logger.removeHandler so that it does not close the handler on removal. As far as point 4 is concerned, I feel that the last discussion on python-dev was inconclusive. The consensus seemed to say WARNING and CRITICAL were OK, but then there was some doubt about WARNING due to the number of changes Zope would need. Though Zope is an important application, I'm not sure Zope changes should be the yardstick for this sort of change;

3 5

FWD: Python execvpe symlink race condition.
by Aahz 16 Feb '03

16 Feb '03

Not sure what to do about this, but I don't have the time to check into it. ----- Forwarded message from Access-=-Denied support <info(a)ad2u.gr> ----- > Forwarded-By: aahz(a)pythoncraft.com > Reply-To: info(a)ad2u.gr > To: webmaster(a)python.org > Subject: Python execvpe symlink race condition. > From: Access-=-Denied support <info(a)ad2u.gr> > Organization: Access-=-Denied > Date: Sun, 16 Feb 2003 03:33:38 -0000 > > > Dear webmaster, > > > The proof of concept code will be published in a few days. That will leave > you plenty of time > to deal with the bug. > > > > Looking forward to hearing from you, > > Access-=-Denied support > > > > START OF ADVISORY > ----------------- > > AD2U Security Advisory -- 15/02/2003 > > Python 2.2.x execvpe Symlink Race Condition > > Summary > -------- > > A Symlink race condition exists in all version of Python programming > language. Probably any posix system running python is vulnerable. > > #### IMPORTANT #### > This vulnerability is not closely related to the recent execvpe > vulnerability, but the idea is derived from there. > > > > Description > ---------------- > > The vulnerability can be observed in two library files that come with > python. > > tempfile.py at def mktemp(suffix=""): > ************************************* > while 1: > i = _counter.get_next() > file = os.path.join(dir, pre + str(i) + suffix) > if not os.path.exists(file): > return file > > > os.py at def _execvpe(file, args, env=None): > ******************************************** > import tempfile > t = tempfile.mktemp() > # Exec a file that is guaranteed not to exist > try: execv(t, ('blah',)) > except error, _notfound: pass > > It is possible, to create a link of the file to be executed (t) in the > limited time window between the statements > os.path.exists(file) in tempfile.py and execv(t, ('blah',)) in os.py are > executed. > > > IMPACT > ------ > > Python is not running suid on most platforms, so to gain root privileges is > a little hard. However, you can attack scripts that use execvpe function > and you will gain the privileges of the > user running the script. > > > PATCH > ----- > > The vulnerability of the tempfile.mktemp() > function is known, and python have "declared" mktemp() deprecated. > Af far it concerns the execvpe function, because it uses mktemp() function > it is vulnerable to this attack. > > Vendor has been notified. > > > Exploit > ------- > > There is exploit code available for this bug and will be published soon. > > A simple run of the epxloit is demonstrated below: > > root@prezaki:~# ./python.sh > Python 2.2.x Symlink Race Condition exploit > Access-=-Denied Networks (c) mzozd(a)ad2u.gr, 2003 > This is a proof of concept code!!! For educational purposes only > Creating suidshell script > Building python file... > Be patient, it will take a few moments > ...................................................................... > ...................................................................... > ...................................................................... > ...................................................................... > ...................................................................... > ...................................................................... > ...................................................................... > You got your suidshell... > -rwsr-xr-x 1 root root 532960 Feb 15 02:42 /tmp/.sh > Cleaning environment > > root@prezaki:~# > > > DISCLAIMER > ---------- > > The author(s) does(do) not have any responsibility for any malicious > use of this advisory or proof of concept code. The code and the > information provided here are for educational purposes only. > The author(s) will NOT be held responsible for any direct or indirect > damages caused by the information or the code > provided here. This advisory is OPEN for public distribution > EXCEPT for Symantec Corporation, Security Focus, Bugtraq or > any other company affiliated with Symantec. Articles that are > based on the information posted here SHOULD include a link > to this advisory or clearly refer the SOURCE. > This disclaimer is not to be modified by any means and must > be included 'as-is' in other documents. The material provided > here, in any form, is copyright property of Access-=-Denied Networks. > > > Acknowledgements > ---------------- > > Discovery and proof of concept code by MzOzD > Email at mzozd(a)ad2u.gr > > > REFERENCES > ---------- > > RHSA-2002:202-33.txt ADVISORY > http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=156556 > http://cve.mitre.org/cgi-bin/cvename.cgi?name=CAN-2002-1119 ----- End forwarded message ----- -- Aahz (aahz(a)pythoncraft.com) <*> http://www.pythoncraft.com/ Register for PyCon now! http://www.python.org/pycon/reg.html

5 11

Re: Python execvpe symlink race condition
by Guido van Rossum 16 Feb '03

16 Feb '03

Dear submitter, Thanks for your vulnerability report. The vulnerability you reported has already been fixed in Python 2.2.2, released on October 14, 2002. As of that version, os.execvpe() no longer calls mktemp(). --Guido van Rossum (home page: http://www.python.org/~guido/)

1 0

Re: execve vulnerability
by Aahz 16 Feb '03

16 Feb '03

On Sun, Feb 16, 2003, Bill Bumgarner wrote: > > Can you please let me know if the fix is in the maintenance branch of > 2.2 and which version it is present in? > > I need to make a recommendation to Apple as to how to deal with this > issue -- knowing the exact tree to grab would be very helpful. This seems like a good reason to issue 2.2.3. Per PEP 6, we need a Patch Czar to champion it. Anyone want to take it up? -- Aahz (aahz(a)pythoncraft.com) <*> http://www.pythoncraft.com/ Register for PyCon now! http://www.python.org/pycon/reg.html

2 2

Weekly Python Bug/Patch Summary
by Skip Montanaro 16 Feb '03

16 Feb '03

Bug/Patch Summary ----------------- 336 open / 3332 total bugs (no change) 127 open / 1982 total patches (+6) New Bugs -------- Whats new: Obligatory source code encodings if not ASCII (2003-02-09) http://python.org/sf/683416 Tutorial: info on Source Code Encoding is missing (2003-02-09) http://python.org/sf/683486 PyErr_Warn may cause import deadlock (2003-02-09) http://python.org/sf/683658 HTMLParser attribute parsing bug (2003-02-10) http://python.org/sf/683938 Freeze can not be made to exclude implicits (2003-02-11) http://python.org/sf/684679 socket module on solaris (2003-02-11) http://python.org/sf/684903 IDE asks for attention when quitting (2003-02-11) http://python.org/sf/684975 bundlebuilder and spaces in filenames (2003-02-12) http://python.org/sf/685373 python-mode loops on if/else (2003-02-12) http://python.org/sf/685465 2 (more) bugs in turtle (2003-02-12) http://python.org/sf/685773 turtle circle() documentation error (2003-02-12) http://python.org/sf/685775 raw_input defers alarm signal (2003-02-13) http://python.org/sf/685846 os.spawnv(P_WAIT, ...) on Linux doesn't handle EINTR (2003-02-14) http://python.org/sf/686667 Profilier hooked into SystemExit (2003-02-15) http://python.org/sf/687297 New Patches ----------- unicode support for os.listdir() (2003-02-09) http://python.org/sf/683592 Add download_url to setup() (2003-02-10) http://python.org/sf/683939 AutoThreadState implementation (2003-02-10) http://python.org/sf/684256 Fix for verbosity (2003-02-10) http://python.org/sf/684398 extending readline functionality (2003-02-11) http://python.org/sf/684500 Allow freeze to exclude implicits (2003-02-11) http://python.org/sf/684677 extend readline functionality in pdb (2003-02-11) http://python.org/sf/684944 fix for bug 501716 (2003-02-11) http://python.org/sf/684981 fix for 680789: reprs in arraymodule (2003-02-11) http://python.org/sf/685051 imputil must respect __path__ for package submodules (2003-02-12) http://python.org/sf/685268 logging.handlers.SMTPHandler and single 'to' address (2003-02-13) http://python.org/sf/686379 add os.sep and friends to os.path (2003-02-13) http://python.org/sf/686397 Add addition and moving messages to Maildir (2003-02-14) http://python.org/sf/686545 Remove unncessary casts for PyEval_GetFrame() (2003-02-14) http://python.org/sf/686601 Closed Bugs ----------- Missing trailing newline should not raise SyntaxError (2002-01-09) http://python.org/sf/501622 pty.fork problem (2002-02-07) http://python.org/sf/514345 isinstance() should respect __class__ (2002-03-20) http://python.org/sf/532767 import lock should be exposed (2002-07-13) http://python.org/sf/580952 sys.executable in applets (2002-10-19) http://python.org/sf/625728 Instantiation init-less class with param (2002-11-14) http://python.org/sf/638610 socket.inet_aton("255.255.255.255") (2002-11-24) http://python.org/sf/643005 -hex/oct const generates wrong code (2002-12-31) http://python.org/sf/660455 sets module review (2003-01-07) http://python.org/sf/663701 Another parsermodule validation error (2003-01-31) http://python.org/sf/678518 'compile' refuses BOM. (2003-02-03) http://python.org/sf/679880 __module__ broken for extension classes (2003-02-04) http://python.org/sf/680429 C subtypes do not inherit tp_as_buffer (2003-02-05) http://python.org/sf/681367 Source encoding rules are extreme. (2003-02-06) http://python.org/sf/681960 Closed Patches -------------- handle unary op of constant in transformer.py (2003-01-03) http://python.org/sf/661536 BZ2File leaking fd and memory (2003-01-03) http://python.org/sf/661796 Compile _randommodule.c and datetimemodule.c under cygwin (2003-01-20) http://python.org/sf/671176 extending readline functionality (2003-01-27) http://python.org/sf/675551 Cygwin array module patch (2003-01-29) http://python.org/sf/676837 Cygwin _iconv_codec module patch (2003-01-29) http://python.org/sf/676839 improved readline usage in pdb (2003-01-30) http://python.org/sf/677719 Fix for the bug #679880: 'compile' refuses BOM. (2003-02-04) http://python.org/sf/680474 lookbehind tests (2003-02-07) http://python.org/sf/682432 Patch for bug 580952: import lock should be exposed (2003-02-08) http://python.org/sf/683257

1 0

PEP 308: "then" "else" for deprecating "and" "or" side effects
by Christian Tismer 15 Feb '03

15 Feb '03

Dear community, there has been some favor of my silly/simple a then b else c proposal. Please let me elaborate a little on this, with respect to the existing "and" and "or operators. "and" and "or" are two-headed beasts: At first glance, they pretend to be logical operators. At second inspection, they do this logic by promoting one of their operands. This is not obviously bad, and it is consistant with the logic. Due to the presence of a real boolean type, we should ask ourselves if "and" and "or" shouldn't be deprecated in their current meaning, towards being "boolean only"? I think it would be nicer if "and" and "or" would only produce true or false. But this would break lots of code. Therefore, I propose to keep them as they are, but to deprecate using them to carry their operand values instead of a true truth value. This would be still allowed, but there are new, more explicit operators, which should be used instead for new Python code: The binary operators "then" and "else". The ternary operator "then else". For a clear definition, see the function definitions below. Example: Parameter cleanup. Parameters are often massaged like this: "or" case: arg = arg or "Default" The meaning of this phrase is: "If the user has provided no argument value that is true (non-empty or not 0 most of the time), then provide some default value". arg = arg then arg else "Default" would be the verbose ternary replacement of this. A short replacement would read better: arg = arg else "Default" which means "If arg is false, replace it by 'Default'". "and" case: arg = arg and 42 The meaning of this phrase is: "If the user provided any argument that is true, use some default value". arg = arg then 42 else arg would be the verbose replacement of this. A short replacement would read better: arg = arg then 42 Summary: -------- "and" and "or" are behaving like logical operators. They happen to do this by not providing a plain truth value, but by promoting one of their operands. "and" and "or" are valid operators, when used in a logical context, only, meaning only to use the truth value. They have the side effect of carrying an operand, which I regard as a design flaw, but this is bearable when used properly. Using "and" and "or", when they are meant as "then" or "else" should be deprectaed as bad style in the long term. Almost all instances of "and" and "or" used to carry a value can be replaced by "then" or "else" with ease, which makes the resulting code much more readable without prior knowledge. "and" and "or" do have a builtin perception of symmetry, which is not related to the truth, and hard to explain to newcomers. Therefore, I propose to deprecate their use to promote a value, and I propose to introduce "then" and "else" as dual operators, together with the "then else" pair as ternary operator, to be used in any context where the inquiry should promote the right operand without evaluating its truth value. Definition: ----------- Here is my "cure" to "and" and "or" with side-effects: a then b -------- is equivalent to def then_proc(a, b): if a: return b else: return a a else b -------- is equivalent to def else_proc(a, b): if a: return a else: return b a then b else c --------------- is equivalent to def then_else_proc(a, b, c): if a: return b else: return c "and" and "or" should be replaced by the above constructs, in all contexts where "and" and "or" are not pure logical, but are meant to propagate specific values, for all new code. kind regards -- chris p.s.: Please correct me positively, if I made a mistake at 4:15am -- Christian Tismer :^) <mailto:tismer@tismer.com> Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 pager +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/

6 8