Mailman 3 September 2007 - Python-Dev

Word size inconsistencies in C extension modules
by Luke Mewburn Sept. 10, 2007

Sept. 10, 2007

Hi folks. While working on an in-house application that uses the curses module, we noticed that it didn't work as expected on an AIX system (powerpc 64-bit big-endian LP64), using python 2.3.5. On a hunch, I took a look through the _cursesmodule.c code and noticed the use of PyArg_ParseTuple()'s "l" decoding mode to retrieve a "long" from python into a C type (attr_t) that on AIX is an int. On 64-bit LP64 platforms, sizeof(long) > sizeof(int), so this doesn't quite work, especially on big-endian systems. Further research into curses shows that different platforms use a different underlying C type for the attr_t type (int, unsigned int, long, unsigned long), so changing the PyArg_ParseTuple() to using the "i" decoding mode probably wasn't portable. I documented this problem and provided a patch that fixes it against the head of the svn trunk in http://bugs.python.org/issue1114 (because the problem appears to still exist in the latest code.) My workaround was to use a separate explicit C "long" to decode the value from python into, and then just assign that to the final value and hope that the type promotion does the right thing on the native platfomr. My questions are: (a) What's the "preferred" style in python extension modules of parsing a number from python into a C type, where the C type size may change on different platforms? Is my method of guessing what the largest common size will be (long, unsigned long, ...), reading into that, and assigning to the final type, acceptable? (b) Is there a desire to see the standard python C extension modules cleaned up to use the answer to (a), especially where said modules may be susceptable to the word size problems I mentioned? (64bit big-endian platforms such as powerpc and sparc64 are good for detecting word-size lossage) cheers, Luke.

2 3

tests expanded for SSL module -- other suggestions?
by Bill Janssen Sept. 10, 2007

Sept. 10, 2007

I'm looking for suggestions for other SSL module tests. Here's the result of running my (not yet checked-in) test_ssl.py module in verbose mode. I'm pretty happy with the codebase right now, and barring other tests, I'm ready to check it in and start on the 3.x patch (or perhaps the 2.3 package). In the client/server tests, a new server thread is created for each test. In the STARTTLS test, several messages are exchanged in the clear, then the client sends a STARTTLS message and after the server replies "OK", initiates the TLS handshake. It would be nice to have an external HTTPS server on python.org that could be used for an HTTPS connection test. Is there one? Bill % ./python.exe ./Lib/test/regrtest.py -u all -v test_ssl test_ssl testCrucialConstants (test.test_ssl.BasicTests) ... ok testParseCert (test.test_ssl.BasicTests) ... {'notAfter': 'Feb 16 16:54:50 2013 GMT', 'subject': ((('countryName', u'US'),), (('stateOrProvinceName', u'Delaware'),), (('localityName', u'Wilmington'),), (('organizationName', u'Python Software Foundation'),), (('organizationalUnitName', u'SSL'),), (('commonName', u'somemachine.python.org'),))} ok testRAND (test.test_ssl.BasicTests) ... RAND_status is 1 (sufficient randomness) ok testSSLconnect (test.test_ssl.BasicTests) ... ok testEcho (test.test_ssl.ConnectedTests) ... server: new connection from ('127.0.0.1', 51840) server: connection cipher is now ('AES256-SHA', 'TLSv1/SSLv3', 256) client: sending 'FOO\n'... server: read 'FOO\n', sending back 'foo\n'... client: read 'foo\n' client: closing connection. server: client closed connection ok testMalformedCert (test.test_ssl.ConnectedTests) ... ok testMalformedKey (test.test_ssl.ConnectedTests) ... ok testNULLcert (test.test_ssl.ConnectedTests) ... ok testReadCert (test.test_ssl.ConnectedTests) ... {'notAfter': 'Feb 16 16:54:50 2013 GMT', 'subject': ((('countryName', u'US'),), (('stateOrProvinceName', u'Delaware'),), (('localityName', u'Wilmington'),), (('organizationName', u'Python Software Foundation'),), (('organizationalUnitName', u'SSL'),), (('commonName', u'somemachine.python.org'),))} Connection cipher is ('AES256-SHA', 'TLSv1/SSLv3', 256). ok testRudeShutdown (test.test_ssl.ConnectedTests) ... ok testSSL2 (test.test_ssl.ConnectedTests) ... SSLv2->SSLv2 CERT_NONE SSLv2->SSLv2 CERT_OPTIONAL SSLv2->SSLv2 CERT_REQUIRED SSLv23->SSLv2 CERT_NONE {SSLv3->SSLv2} CERT_NONE {TLSv1->SSLv2} CERT_NONE ok testSSL23 (test.test_ssl.ConnectedTests) ... {SSLv2->SSLv23} CERT_NONE SSLv3->SSLv23 CERT_NONE SSLv23->SSLv23 CERT_NONE TLSv1->SSLv23 CERT_NONE {SSLv2->SSLv23} CERT_OPTIONAL SSLv3->SSLv23 CERT_OPTIONAL SSLv23->SSLv23 CERT_OPTIONAL TLSv1->SSLv23 CERT_OPTIONAL {SSLv2->SSLv23} CERT_REQUIRED SSLv3->SSLv23 CERT_REQUIRED SSLv23->SSLv23 CERT_REQUIRED TLSv1->SSLv23 CERT_REQUIRED ok testSSL3 (test.test_ssl.ConnectedTests) ... SSLv3->SSLv3 CERT_NONE SSLv3->SSLv3 CERT_OPTIONAL SSLv3->SSLv3 CERT_REQUIRED {SSLv2->SSLv3} CERT_NONE {SSLv23->SSLv3} CERT_NONE {TLSv1->SSLv3} CERT_NONE ok testSTARTTLS (test.test_ssl.ConnectedTests) ... client: sending 'msg 1'... server: new connection from ('127.0.0.1', 51870) server: read 'msg 1', sending back 'msg 1'... client: read 'msg 1' from server client: sending 'MSG 2'... server: read 'MSG 2', sending back 'msg 2'... client: read 'msg 2' from server client: sending 'STARTTLS'... server: read STARTTLS from client, sending OK... client: read 'OK\n' from server, starting TLS... server: connection cipher is now ('AES256-SHA', 'TLSv1/SSLv3', 256) client: sending 'MSG 3'... server: read 'MSG 3', sending back 'msg 3'... client: read 'msg 3' from server client: sending 'msg 4'... server: read 'msg 4', sending back 'msg 4'... client: read 'msg 4' from server client: closing connection. server: client closed connection ok testTLS1 (test.test_ssl.ConnectedTests) ... TLSv1->TLSv1 CERT_NONE TLSv1->TLSv1 CERT_OPTIONAL TLSv1->TLSv1 CERT_REQUIRED {SSLv2->TLSv1} CERT_NONE {SSLv3->TLSv1} CERT_NONE {SSLv23->TLSv1} CERT_NONE ok ---------------------------------------------------------------------- Ran 15 tests in 6.866s OK 1 test OK. CAUTION: stdout isn't compared in verbose mode: a test that passes in verbose mode may fail without it. [23679 refs]

1 0

can't run test_tcl remotely logged in on an OS X machine
by Bill Janssen Sept. 9, 2007

Sept. 9, 2007

"test_tcl" fails on me (OS X 10.4.10 on an Intel Mac, remotely logged in via SSH and X Windows): % test_tcl 2007-09-08 17:00:22.629 python.exe[4163] CFLog (0): CFMessagePort: bootstrap_register(): failed 1100 (0x44c), port = 0x3a03, name = 'Processes-0.58327041' See /usr/include/servers/bootstrap_defs.h for the error codes. 2007-09-08 17:00:22.630 python.exe[4163] CFLog (99): CFMessagePortCreateLocal(): failed to name Mach port (Processes-0.58327041) CFMessagePortCreateLocal failed (name = Processes-0.58327041 error = 0) Abort % This is on the trunk. Bill

2 2

what platforms require RAND_add() before using SSL?
by Bill Janssen Sept. 9, 2007

Sept. 9, 2007

There are some functions in _ssl.c for gathering randomness from a daemon, and adding that randomness to the pseudo-random number generator in SSL, before using SSL. There's a note there saying that "on some platform" this is necessary. Anyone know which platforms? Bill

2 2

Porting python
by Giszo Sept. 9, 2007

Sept. 9, 2007

Hi! I've tried to port Python (2.3.6 and 2.5.1) to my own OS. The compilation of the python library is done after a few hours of work. When i try to run the compiled executable i got an error shown on the following screenshot: http://giszo.lame.hu/jshot/screens/screen31.png After a little while of debugging i know that it fails bootstrapping the exceptions because the initializer function failes to get the "__builtin__" module. Adding debug printfs to the bltinmodule.c init code it looks like the builtin module is initialized properly. I'd like to ask some help where i should start checking the code to fix the error. Thanks! ________________________________________________ Message sent using UebiMiau 2.7.9

2 1

Re: [Python-Dev] [Python-3000] 3.0 crypto
by Bill Janssen Sept. 8, 2007

Sept. 8, 2007

> We're already linking against the OpenSSL EVP libraries for hashlib > (and against the OpenSSL SSL libraries for the SSL support). It > wouldn't be hard to expose the EVP functions a bit more, essentially > as hash functions that return long (and reversible) hashes: > > encryptor = opensslevp.encryptor("AES-256-CBC", ...maybe some options...) > encryptor.update(...some plaintext...) Almost certainly this signature should be encryptor = opensslevp.encryptor("AES-256-CBC", KEY, ...options...) and correspondingly decryptor = opensslevp.decryptor("AES-256-CBC", KEY, ...options...) Bill

1 0

working with Python threads from C extension module?
by Bill Janssen Sept. 8, 2007

Sept. 8, 2007

Reading through the C API documentation, I find: ``This is done so that dynamically loaded extensions compiled with thread support enabled can be loaded by an interpreter that was compiled with disabled thread support.'' I've currently got the set-up-SSL-threading code in _ssl.c surrounded by a "#ifdef HAVE_THREAD" bracket. It sounds like that might not be sufficient. It sounds like I need a runtime test for thread availability, instead, like this: #ifdef HAVE_THREAD if (PyEval_ThreadsInitialized()) _setup_ssl_threads(); #endif Seem right? So what happens when someone loads the _ssl module, initializes the threads, and tries to use SSL? It's going to start failing again. I think I need my own version of Py_BEGIN_ALLOW_THREADS and Py_END_ALLOW_THREADS, don't I? Which also checks to see if the SSL threading support has been initialized, in addition to the Python threading support. Something like #define SSL_ALLOW_THREADS {if (_ssl_locks != NULL) { Py_BEGIN_ALLOW_THREADS }} #define SSL_DISALLOW_THREADS {if (_ssl_locks != NULL) { Py_BEGIN_ALLOW_THREADS }} Any comments? Bill

2 5

Re: [Python-Dev] working with Python threads from C extension module?
by Gustavo Carneiro Sept. 8, 2007

Sept. 8, 2007

On 08/09/2007, Bill Janssen <janssen(a)parc.com> wrote: > > > Be convoluted yourself and do this: > > > > #define PySSL_BEGIN_ALLOW_THREADS { if (_ssl_locks) { > Py_BEGIN_ALLOW_THREADS > > #define PySSL_END_ALLOW_THREADS Py_END_ALLOW_THREADS } } > > > > (Untested, but I think it should work.) > > Yes, that had occurred to me. We want the code inside the braces > still to run if the locks aren't held, so something more like > > #define PySSL_BEGIN_ALLOW_THREADS { \ > PyThreadState *_save; \ > if (_ssl_locks_count>0) {_save = > PyEval_SaveThread();} > #define PySSL_BLOCK_THREADS if > (_ssl_locks_count>0){PyEval_RestoreThread(_save)}; > #define PySSL_UNBLOCK_THREADS if (_ssl_locks_count>0){_save = > PyEval_SaveThread()}; > #define PySSL_END_ALLOW_THREADS if > (_ssl_locks_count>0){PyEval_RestoreThread(_save);} \ > } > > would do the trick. Unfortunately, this doesn't deal with the macro > behaviour. The user has "turned on" threading; they expect reads and > writes to yield the GIL so that other threads can make progress. But > the fact that threading has been "turned on" after the SSL module has > been initialized, means that threads don't work inside the SSL code. > So the user's understanding of the system will be broken. > > No, I don't see any good way to fix this except to add a callback > chain inside PyThread_init_thread, which is run down when threads are > initialized. Any module which needs to set up threads registers itself > on that chain, and gets called as part of PyThread_init_thread. But > I'm far from the smartest person on this list :-), so perhaps someone > else will see a good solution. I think this is a helpful additional tool to solve threading problems. Doesn't solve everything, but it certainly helps :-) For instance, one thing it doesn't solve is when a library being wrapped can be initialized with multithreading support, but only allows such initialization as a very first API call; you can't initialize threading at any arbitrary time during application runtime. Unfortunately I don't think there is any sane way to fix this problem :-( This has got to be a problem with other extension modules linked to > libraries which have their own threading abstractions. Yes. Another problem is that python extensions may not wish to incur performance penalty of python threading calls. For instance, pyorbit has these macros: #define pyorbit_gil_state_ensure() (PyEval_ThreadsInitialized()? (PyGILState_Ensure()) : 0) #define pyorbit_gil_state_release(state) G_STMT_START { \ if (PyEval_ThreadsInitialized()) \ PyGILState_Release(state); \ } G_STMT_END #define pyorbit_begin_allow_threads \ G_STMT_START { \ PyThreadState *_save = NULL; \ if (PyEval_ThreadsInitialized()) \ _save = PyEval_SaveThread(); #define pyorbit_end_allow_threads \ if (PyEval_ThreadsInitialized()) \ PyEval_RestoreThread(_save); \ } G_STMT_END They all call PyEval_ThreadsInitialized() before doing anything thread related to save some performance. The other reason to do it this way is that the Python API calls themselves abort if they are called with threading not initialized. It would be nice the upstream python GIL macros were more like pyorbit and became no-ops when threading is not enabled. -- Gustavo J. A. M. Carneiro INESC Porto, Telecommunications and Multimedia Unit "The universe is always one step beyond logic." -- Frank Herbert

1 0

Buildbot upgraded to 0.7.5
by "Martin v. Löwis" Sept. 8, 2007

Sept. 8, 2007

I just upgraded the buildbot master to 0.7.5. If you see any problems, please let me know. Neal: buildbot now supports reloading of configurations, without interrupting builds. Try "buildbot reconfig" when you make a change (certain changes would still require a restart). Regards, Martin

1 0

PEP 362: Signature objects
by Jim Jewett Sept. 8, 2007

Sept. 8, 2007

Brett Cannon wrote: > A Signature object has the following structure attributes: > * name : str > Name of the function. This is not fully qualified because > function objects for methods do not know the class they are > contained within. This makes functions and methods > indistinguishable from one another when passed to decorators, > preventing proper creation of a fully qualified name. (1) Would this change with the new static __class__ attribute used for the new super? (2) What about functions without a name? Do you want to say str or NoneType, or is that assumed? (3) Is the Signature object live or frozen? (name is writable ... will the Signature object reflect the new name, or the name in use at the time it was created?) > * var_annotations: dict(str, object) > Dict that contains the annotations for the variable parameters. > The keys are of the variable parameter with values of the Is there a special key for the "->" returns annotation, or is that available as a separate property? > The structure of the Parameter object is: > * name : (str | tuple(str)) > The name of the parameter as a string if it is not a tuple. If > the argument is a tuple then a tuple of strings is used. What is used for unnamed arguments (typically provided by C)? I like None, but I see the arguments for both "" and missing attribute. > * position : int > The position of the parameter within the signature of the > function (zero-indexed). For keyword-only parameters the position > value is arbitrary while not conflicting with positional > parameters. Is this just a property/alias for signature.parameters.index(self) ? What should a "parameter" object not associated with a specific signature return? -1, None, or missing attribute? Is there a way to get the associated Signature, or is it "compiled out" when the Signature and its child Parameters are first constructed? (I think the position property is the only attribute that would use it, unless you want some of the other attributes -- like annotations -- to be live.) ... I would also like to see a * value : object attribute; this would be missing on most functions, but might be filled in on a Signature representing a closure, or an execution frame. > When to construct the Signature object? > --------------------------------------- > The Signature object can either be created in an eager or lazy > fashion. In the eager situation, the object can be created during > creation of the function object. Since most code doesn't need it, I would expect it to be optimized out at least as often as docstrings are. > In the lazy situation, one would > pass a function object to a function and that would generate the > Signature object and store it to ``__signature__`` if > needed, and then return the value of ``__signature__``. Why store it? Do you expect many use cases to need the signature more than once (but not to save it themselves)? If there is a __signature__ attribute on a object, you have to specify whether it can be replaced, which parts of it are writable, how that will affect the function's own behavior, etc. I also suspect it might become a source of heisenbugs, like the "reference leaks" that were really DUMMY items in a dict. If the Signature is just a snapshot no longer attached to the original function, then people won't expect changes to the Signature to affect the callable. > Should ``Signature.bind`` return Parameter objects as keys? (see above) If a Signature is a snapshot (rather than a live part of the function), then it might make more sense to just add a value attribute to Parameter objects. > Provide a mapping of parameter name to Parameter object? > -------------------------------------------------------- > While providing access to the parameters in order is handy, it might > also be beneficial to provide a way to retrieve Parameter objects from > a Signature object based on the parameter's name. Which style of > access (sequential/iteration or mapping) will influence how the > parameters are stored internally and whether __getitem__ accepts > strings or integers. I think it should accept both. What storage mechanism to use is an internal detail that should be left to the implementation. I wouldn't expect Signature inspection to be inside a tight loop anyhow, unless it were part of a Generic Function dispatch engine ... and those authors (just PJE?) can optimize on what they actually need. > Remove ``has_*`` attributes? > ---------------------------- > If an EAFP approach to the API is taken, Please leave them; it is difficult to catch Exceptions in a list comprehension. > Have ``var_args`` and ``_var_kw_args`` default to ``None``? Makes sense to me, particularly since it should probably be consistent with function name, and that should probably be None. -jJ

3 2