Mailman 3 September 2007 - Python-Dev

Re: [Python-Dev] working with Python threads from C extension module?
by Guido van Rossum Sept. 7, 2007

Sept. 7, 2007

Well, one shouldn't be bothering with threads unless the user intends to create threads. So I think it's not kosher. Once threads are initialized, everything runs a tad slower because the GIL manipulations actually cost time (even if there are no other threads). On 9/7/07, Bill Janssen <janssen(a)parc.com> wrote: > > So what happens when someone loads the _ssl module, initializes the > > threads, and tries to use SSL? It's going to start failing again. I > > Which turns out to be exactly what test_ssl.py does. I'm tempted > to have the _ssl module call PyEval_InitThreads(). Would that be kosher? -- --Guido van Rossum (home page: http://www.python.org/~guido/)

2 1

OpenSSL thread safety when reading files?
by Bill Janssen Sept. 7, 2007

Sept. 7, 2007

I'm seeing a number of malloc (actully, free) errors, now that I'm pounding on the OpenSSL server/client setup with lots of server threads and client threads. They all look like either (gdb) bt #0 0x9010b807 in malloc_printf () #1 0x900058ad in szone_free () #2 0x90005588 in free () #3 0x9194e508 in CRYPTO_free () #4 0x91993e77 in ERR_clear_error () #5 0x919b1884 in PEM_X509_INFO_read_bio () #6 0x9197a692 in X509_load_cert_crl_file () #7 0x9197a80e in by_file_ctrl () #8 0x919d6e2e in X509_STORE_load_locations () [...] or (much more frequently) (gdb) bt #0 0x9010b807 in malloc_printf () #1 0x900058ad in szone_free () #2 0x90005588 in free () #3 0x9194e508 in CRYPTO_free () #4 0x91993e77 in ERR_clear_error () #5 0x949fcf11 in SSL_CTX_use_certificate_chain_file () [...] Always in ERR_clear_error(), always from some frame that's reading a certificate file for some purpose. If I disable Py_BEGIN_ALLOW_THREADS/Py_END_ALLOW_THREADS around the places where the C code reads the certificate files, all these free errors go away. ERR_clear_error() is supposed to be thread-safe; it operates on a per-thread error state structure (which I make sure is initialized in my C code). But it sure looks like the client and server threads are both working with the same error state. Bill

1 2

any tips on malloc debugging?
by Bill Janssen Sept. 7, 2007

Sept. 7, 2007

I've been expanding the SSL test suite, and found something like this cropping up, not always, but maybe 30% of the time. So I run it under gdb, but the "szone_error" breakpoint never gets hit. Any other malloc debugging tips I should know about? (gdb) info break Num Type Disp Enb Address What 1 breakpoint keep y 0x900f2e56 <szone_error+6> (gdb) (gdb) run ./Lib/test/regrtest.py -R :4: -u all test_ssl Starting program: /local/python/trunk/src/python.exe ./Lib/test/regrtest.py -R :4: -u all test_ssl test_ssl [...] python.exe(22696,0xa000d000) malloc: *** error for object 0x650800: double free python.exe(22696,0xa000d000) malloc: *** set a breakpoint in szone_error to debug test test_ssl failed -- Traceback (most recent call last): File "/local/python/trunk/src/Lib/test/test_ssl.py", line 304, in testSSL3 CERTFILE2, CERTFILE3) File "/local/python/trunk/src/Lib/test/test_ssl.py", line 203, in serverParamsTest raise test_support.TestFailed("Unexpected SSL error: " + str(x)) TestFailed: Unexpected SSL error: (8, '_ssl.c:394: EOF occurred in violation of protocol') 1 test failed: test_ssl [23436 refs] Program exited with code 01. (gdb) Bill

2 2

Re: [Python-Dev] any tips on malloc debugging?
by Guido van Rossum Sept. 7, 2007

Sept. 7, 2007

I think there's a way to enable heavier malloc debugging than the normal --with-pydebug. You'll have to enable it manually by editing Python.h I believe. Though it may already be on if you define Py_DEBUG. (Is WITH_PYMALLOC always on?) Ther may also be a libmalloc that enables heavier debugging; the malloc man page would have info. On 9/7/07, Bill Janssen <janssen(a)parc.com> wrote: > I've been expanding the SSL test suite, and found something like this > cropping up, not always, but maybe 30% of the time. So I run it under > gdb, but the "szone_error" breakpoint never gets hit. Any other > malloc debugging tips I should know about? > > (gdb) info break > Num Type Disp Enb Address What > 1 breakpoint keep y 0x900f2e56 <szone_error+6> > (gdb) (gdb) run ./Lib/test/regrtest.py -R :4: -u all test_ssl > Starting program: /local/python/trunk/src/python.exe ./Lib/test/regrtest.py -R :4: -u all test_ssl > test_ssl > [...] > python.exe(22696,0xa000d000) malloc: *** error for object 0x650800: double free > python.exe(22696,0xa000d000) malloc: *** set a breakpoint in szone_error to debug > test test_ssl failed -- Traceback (most recent call last): > File "/local/python/trunk/src/Lib/test/test_ssl.py", line 304, in testSSL3 > CERTFILE2, CERTFILE3) > File "/local/python/trunk/src/Lib/test/test_ssl.py", line 203, in serverParamsTest > raise test_support.TestFailed("Unexpected SSL error: " + str(x)) > TestFailed: Unexpected SSL error: (8, '_ssl.c:394: EOF occurred in violation of protocol') > > 1 test failed: > test_ssl > [23436 refs] > > Program exited with code 01. > (gdb) > > Bill > _______________________________________________ > Python-Dev mailing list > Python-Dev(a)python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (home page: http://www.python.org/~guido/)

1 0

Google spreadsheet to collaborate on backporting Py3K stuff to 2.6
by Brett Cannon Sept. 7, 2007

Sept. 7, 2007

Neal, Anthony, Thomas W., and I have a spreadsheet that was started to keep track of what needs to be done in what needs to be done in 2.6 for Py3K transitioning: http://spreadsheets.google.com/pub?key=pCKY4oaXnT81FrGo3ShGHGg . I am opening the spreadsheet up to everyone so that others can help maintain it. There is a sheet in the Python 3000 Tasks spreadsheet that should be merged into this spreadsheet and then deleted. If anyone wants to help with that it would be great (once something has been moved from "Python 3000 Tasks" to "Python 2 -> 3 transition" just delete it from "Python 3000 Tasks"). Because Neal created this spreadsheet he is the only one who can open editing to everyone. If you would like to have edit abilities to the spreadsheet just reply to this email saying you want an invite and I will add you manually (and if you want a different address added just say so). -Brett

3 2

PEP 362: Signature objects
by Brett Cannon Sept. 7, 2007

Sept. 7, 2007

Neal Becker over on python-3000 said that the Boost people could use this. Figured it was time to present it officially to the list to see if I can get it added for 2.6/3.0. The implementation in the sandbox works in both 2.6 and 3.0 out of the box (no 2to3 necessary) so feel free to play with it. --------------------------------------------------------- Abstract ======== Python has always supported powerful introspection capabilities, including that for functions and methods (for the rest of this PEP the word "function" refers to both functions and methods). Taking a function object, you can fully reconstruct the function's signature. Unfortunately it is a little unruly having to look at all the different attributes to pull together complete information for a function's signature. This PEP proposes an object representation for function signatures. This should help facilitate introspection on functions for various uses (e.g., decorators). The introspection information contains all possible information about the parameters in a signature (including Python 3.0 features). This object, though, is not meant to replace existing ways of introspection on a function's signature. The current solutions are there to make Python's execution work in an efficient manner. The proposed object representation is only meant to help make application code have an easier time to query a function on its signature. Signature Object ================ The overall signature of an object is represented by the Signature object. This object is to store a `Parameter object`_ for each parameter in the signature. It is also to store any information about the function itself that is pertinent to the signature. A Signature object has the following structure attributes: * name : str Name of the function. This is not fully qualified because function objects for methods do not know the class they are contained within. This makes functions and methods indistinguishable from one another when passed to decorators, preventing proper creation of a fully qualified name. * var_args : str Name of the variable positional parameter (i.e., ``*args``), if present, or the empty string. * var_kw_args : str Name of the variable keyword parameter (i.e., ``**kwargs``), if present, or the empty string. * var_annotations: dict(str, object) Dict that contains the annotations for the variable parameters. The keys are of the variable parameter with values of the annotation. If an annotation does not exist for a variable parameter then the key does not exist in the dict. * parameters : list(Parameter) List of the parameters of the function as represented by Parameter objects in the order of its definition (keyword-only arguments are in the order listed by ``code.co_varnames``). * bind(\*args, \*\*kwargs) -> dict(str, Parameter) Create a mapping from arguments to parameters. The keys are the names of the parameter that an argument maps to with the value being the value the parameter would have if this function was called with the given arguments. The Signature object is stored in the ``__signature__`` attribute of a function. When it is to be created is discussed in `Open Issues`_. Parameter Object ================ A function's signature is made up of several parameters. Python's different kinds of parameters is quite large and rich and continues to grow. Parameter objects represent any possible parameter. Originally the plan was to represent parameters using a list of parameter names on the Signature object along with various dicts keyed on parameter names to disseminate the various pieces of information one can know about a parameter. But the decision was made to incorporate all information about a parameter in a single object so as to make extending the information easier. This was originally put forth by Talin and the preferred form of Guido (as discussed at the 2006 Google Sprint). The structure of the Parameter object is: * name : (str | tuple(str)) The name of the parameter as a string if it is not a tuple. If the argument is a tuple then a tuple of strings is used. * position : int The position of the parameter within the signature of the function (zero-indexed). For keyword-only parameters the position value is arbitrary while not conflicting with positional parameters. The suggestion of setting the attribute to None or -1 to represent keyword-only parameters was rejected to prevent variable type usage and as a possible point of errors, respectively. * has_default : bool True if the parameter has a default value, else False. * default_value : object The default value for the parameter, if present, else the attribute does not exist. This is done so that the attribute is not accidentally used if no default value is set as any default value could be a legitimate default value itself. * keyword_only : bool True if the parameter is keyword-only, else False. * has_annotation : bool True if the parameter has an annotation, else False. * annotation Set to the annotation for the parameter. If ``has_annotation`` is False then the attribute does not exist to prevent accidental use. Implementation ============== An implementation can be found in Python's sandbox [#impl]_. There is a function named ``signature()`` which returns the value stored on the ``__signature__`` attribute if it exists, else it creates the Signature object for the function and sets ``__signature__``. For methods this is stored directly on the im_func function object since that is what decorators work with. Open Issues =========== When to construct the Signature object? --------------------------------------- The Signature object can either be created in an eager or lazy fashion. In the eager situation, the object can be created during creation of the function object. In the lazy situation, one would pass a function object to a function and that would generate the Signature object and store it to ``__signature__`` if needed, and then return the value of ``__signature__``. Should ``Signature.bind`` return Parameter objects as keys? ----------------------------------------------------------- Instead of returning a dict with keys consisting of the name of the parameters, would it be more useful to instead use Parameter objects? The name of the argument can easily be retrieved from the key (and the name would be used as the hash for a Parameter object). Provide a mapping of parameter name to Parameter object? -------------------------------------------------------- While providing access to the parameters in order is handy, it might also be beneficial to provide a way to retrieve Parameter objects from a Signature object based on the parameter's name. Which style of access (sequential/iteration or mapping) will influence how the parameters are stored internally and whether __getitem__ accepts strings or integers. One possible compromise is to have ``__getitem__`` provide mapping support and have ``__iter__`` return Parameter objects based on their ``position`` attribute. This allows for getting the sequence of Parameter objects easily by using the ``__iter__`` method on Signature object along with the sequence constructor (e.g., ``list`` or ``tuple``). Remove ``has_*`` attributes? ---------------------------- If an EAFP approach to the API is taken, both ``has_annotation`` and ``has_default`` are unneeded as the respective ``annotation`` and ``default_value`` attributes are simply not set. It's simply a question of whether to have a EAFP or LBYL interface. Have ``var_args`` and ``_var_kw_args`` default to ``None``? ------------------------------------------------------------ It has been suggested by Fred Drake that these two attributes have a value of ``None`` instead of empty strings when they do not exist. Deprecate ``inspect.getargspec()`` and ``.formatargspec()``? ------------------------------------------------------------- Since the Signature object replicates the use of ``getargspec()`` from the ``inspect`` module it might make sense to deprecate it in 2.6. ``formatargspec()`` could also go if Signature objects gained a __str__ representation. Issue with that is types such as ``int``, when used as annotations, do not lend themselves for output (e.g., ``"type 'int'>"`` is the string represenation for ``int``). The repr representation of types would need to change in order to make this reasonable. References ========== .. [#impl] pep362 directory in Python's sandbox (http://svn.python.org/view/sandbox/trunk/pep362/) Copyright ========= This document has been placed in the public domain.

1 0

Python access to data fields of SSL connection peer certificate
by Bill Janssen Sept. 6, 2007

Sept. 6, 2007

After a great deal of discussion, under the Subject line of "frozenset C API?" (you may have missed it :-), I'm coming to the conclusion that in revealing the fields of an SSL certificate, less is more. >From one of the messages in that thread: I'm trying to give the application the ability to do some level of authorization without requiring either of those packages. Like being able to tell who's on the other side of the connection :-). Right now, I think the right fields to expose are "subject" (I see little point to exposing "issuer"), "notAfter" (you're always guaranteed to be after "notBefore", or the cert wouldn't validate, so I see little point to exposing that, but "notAfter" can be used after the connection has been established), subjectAltName if present, and perhaps the certificate's serial number. Remember that the cert has already been validated, so I don't see how the other fields in the cert can be profitably used for authorization and/or accounting, which is the purpose of this interface. Anything else you want, you can pull over the DER blob and look into it with some other crypto package; I'll provide a way to pull the full binary form of the certificate into Python as a bytes string (as soon as the bytes API gets backported into the trunk). Under those rules, the samples in the current documentation would look like {'notAfter': 'May 8 23:59:59 2009 GMT', 'serialNumber': '6A4AC31B3110E6EB48F0FC51A39A171F', 'subject': ((('serialNumber', u'2497886'),), (('1.3.6.1.4.1.311.60.2.1.3', u'US'),), (('1.3.6.1.4.1.311.60.2.1.2', u'Delaware'),), (('countryName', u'US'),), (('postalCode', u'94043'),), (('stateOrProvinceName', u'California'),), (('localityName', u'Mountain View'),), (('streetAddress', u'487 East Middlefield Road'),), (('organizationName', u'VeriSign, Inc.'),), (('organizationalUnitName', u'Production Security Services'),), (('organizationalUnitName', u'Terms of use at www.verisign.com/rpa (c)06'),), (('commonName', u'www.verisign.com'),))} and {'notAfter': 'Feb 16 16:54:50 2013 GMT', 'serialNumber': 'FFAA4ADBF570818D', 'subject': ((('countryName', u'US'),), (('stateOrProvinceName', u'Delaware'),), (('localityName', u'Wilmington'),), (('organizationName', u'Python Software Foundation'),), (('organizationalUnitName', u'SSL'),), (('commonName', u'somemachine.python.org'),))} The server cert at https://www.dcl.hpi.uni-potsdam.de/ would look like {'notAfter': 'Mar 17 13:02:27 2008 GMT', 'serialNumber': '2567F168000300000678', 'subject': ((('countryName', u'DE'),), (('stateOrProvinceName', u'Brandenburg'),), (('localityName', u'Potsdam'),), (('organizationName', u'Hasso-Plattner-Institut'),), (('organizationalUnitName', u'Operating Systems & Middleware'),), (('commonName', u'www.dcl.hpi.uni-potsdam.de'),)), 'subjectAltName': ('DNS:www.dcl.hpi.uni-potsdam.de', 'DNS:www', 'DNS:dfw', 'DNS:dfw.dcl.hpi.uni-potsdam.de', 'IP Address:141.89.224.164')} Thanks to Martin for suggesting it. Bill

1 0

Re: [Python-Dev] frozenset C API?
by Guido van Rossum Sept. 6, 2007

Sept. 6, 2007

I guess nobody has tried to create frozenset instances from C code before. Almost everyone uses set anyway. What are you trying to do? On 9/4/07, Bill Janssen <janssen(a)parc.com> wrote: > I'm looking at building a "frozenset" instance as a return value from > a C function, and the C API seems ridiculously clumsy. Maybe I'm > misunderstanding it. Apparently, I need to create a list object, then > pass that to PyFrozenSet_New(), then decref the list object. > > Is that correct? > > What I'd like is something more like > > PyFrozenSet_NEW(int) => PySetObject * > PyFrozenSet_SET_ITEM(s, i, v) > > Any idea why these aren't part of the API? > > Bill > _______________________________________________ > Python-Dev mailing list > Python-Dev(a)python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (home page: http://www.python.org/~guido/)

3 24

x86 XP trunk failure
by Alan McIntyre Sept. 5, 2007

Sept. 5, 2007

Hi all, My build slave (http://www.python.org/dev/buildbot/trunk/x86%20XP%20trunk) keeps failing because of a crash that appears to be in the bsddb module. I assume the master deems the slave to be lost because it's sitting there waiting on me to make a choice on the "debug/abort" dialog box. I can provide details if anybody needs them. I just figured somebody might want to know that this is actual build/test problem instead of some kind of issue with the internet connection here. Thanks, Alan

3 7

Product function patch [issue 1093]
by Ryan Freckleton Sept. 5, 2007

Sept. 5, 2007

Hello, At one time Guido mentioned adding a built-in product() function to cover some of the remaining use cases of the built-in reduce(). I don't know if this function is still wanted or needed, but I've created an implementation with tests and documentation at http://bugs.python.org/issue1093 . If it is still wanted, could someone review it and give me feedback on it? Thanks, -- ===== --Ryan E. Freckleton

8 24