Mailman 3 September 2002 - Python-Dev

mysterious hangs in socket code
by Jeremy Hylton 04 Sep '02

04 Sep '02

I've been running a small, multi-threaded program to retrieve web pages today. The entire program appears to hang when I perform a slow DNS operation, even there is no application-level coordinate between the threads. The motivation comes from http://www.python.org/sf/591349, but I ended up writing a similar small test script, which I've attached. When I run this program with Python 2.1, it produces a steady stream of output -- urls and the time it took to load them. Most of the pages take less than a second, but some take a very long time. If I run this program with Python 2.2 or 2.3, it produces little bursts of output, then pauses for a long time, then repeats. I believe that the problem relates to DNS lookups, but not in a way I fully understand. If I connect gdb to any of the threads while the program is hung, it is always inside getaddrinfo(). My first realization was that the socketmodule stopped wrapping DNS lookups in By_BEGIN/END_ALLOW_THREADS calls when the IPv6 changes were integrated. But if I restore these calls -- see http://www.python.org/sf/604210 -- I don't see any change in behavior. The program still hangs periodically. One possibility is that the Linux getaddrinfo() is thread-safe, but only by way of a lock that only allows one request to be outstanding at a time. Not sure what the other possibilities are, but the current behavior is awful. Jeremy --------------------------------------------------------------------- import httplib import Queue import random import sys import threading import time import traceback import urlparse headers = {"Accept": "text/plain, text/html, image/jpeg, image/jpg, " "image/gif, image/png, */*"} class URLThread(threading.Thread): def __init__(self, queue): threading.Thread.__init__(self) self._queue = queue self._stopevent = threading.Event() def stop(self): self._stopevent.set() def run(self): while not self._stopevent.isSet(): self.fetch() def fetch(self): url = self._queue.get() t0 = time.time() try: self._fetch(url) except: etype, value, tb = sys.exc_info() L = ["Error occurred fetching %s\n" % url, "%s: %s\n" % (etype, value), ] L += traceback.format_tb(tb) sys.stderr.write("".join(L)) t1 = time.time() print url, round(t1 - t0, 2) def _fetch(self, url): parts = urlparse.urlparse(url) host = parts[1] path = parts[2] h = httplib.HTTPConnection(host) h.connect() h.request("GET", path, headers=headers) r = h.getresponse() r.read() h.close() urls = """\ http://www.andersen.com/ http://www.google.com/ http://www.google.com/images/logo.gif http://www.microsoft.com/ http://www.microsoft.com/homepage/gif/bnr-microsoft.gif http://www.microsoft.com/homepage/gif/1ptrans.gif http://www.microsoft.com/library/toolbar/images/curve.gif http://www.yahoo.com/ http://www.sourceforge.net/ http://www.slashdot.org/ http://www.kuro5hin.org/ http://www.intel.com/ http://www.aol.com/ http://www.amazon.com/ http://www.cnn.com/ http://money.cnn.com/ http://www.expedia.com/ http://www.tripod.com/ http://www.hotmail.com/ http://www.angelfire.com/ http://www.excite.com/ http://www.verisign.com/ http://www.riaa.com/ http://www.enron.com/ http://www.securityspace.com/ http://www.directv.com/ http://www.att.com/ http://www.qwest.com/ http://www.covad.com/ http://www.sprint.com/ http://www.mci.com/ http://www.worldcom.com/ """ urls = [u for u in urls.split("\n") if u] REPEAT = 10 THREADS = 8 class RandomQueue: def __init__(self, L): self.list = L def get(self): return random.choice(self.list) if __name__ == "__main__": urlq = RandomQueue(urls) sys.setcheckinterval(10) threads = [] for i in range(THREADS): t = URLThread(urlq) t.start() threads.append(t) while 1: try: time.sleep(30) except: break print "Shutting down threads..." for t in threads: t.stop() for t in threads: t.join()

4 3

Re: [Python-Dev] Should KeyError use repr() on its argument?
by Guido van Rossum 04 Sep '02

04 Sep '02

> > > The KeyError exception doesn't apply repr() to its argument. That's > > > annoying in cases like this: > > > > > > >>> a = {} > > > >>> a[''] > > > Traceback (most recent call last): > > > File "<stdin>", line 1, in ? > > > KeyError > > > >>> > > > > > > Should this be fixed? How? (I guess we could add a KeyError__str__ > > > method to exceptions.c that applies repr().) > > > > > > I've got a feeling this is a feature, but not a very useful one. > > > > I take it back. args[0] being the actual key that failed is a > > feature. str() not using repr() on args[0] is a bug. I'll fix it. > > > > What is args[0]? args is the name of the instance variable that most exceptions use to store the arguments that were passed to them in the raise statement (or equivalent C API). It is a tuple. Examples: >>> a = KeyError() >>> a.args () >>> a = KeyError(1) >>> a.args (1,) >>> a = KeyError(1,2,3) >>> a.args (1, 2, 3) >>> try: {}[''] except KeyError, k: print k.args ('',) >>> > Are you saying that dicts use repr() instead of str() to > get the key value when accessing? No, I'm saying that str(KeyError('foo')) should return repr('foo') rather than 'foo' as it does now. See current CVS. :-) --Guido van Rossum (home page: http://www.python.org/~guido/)

1 0

Nit about `setdefault' documentation
by pinard＠iro.umontreal.ca 04 Sep '02

04 Sep '02

Quite a small nit. Reading: ----------------------------------------------------------------------> >>> help({}.setdefault) Help on built-in function setdefault: setdefault(...) D.setdefault(k[,d]) -> D.get(k,d), also set D[k]=d if not D.has_key(k) ----------------------------------------------------------------------< I wonder if writing the last line as: ----------------------------------------------------------------------> D.setdefault(k[,d]) -> D.get(k,d), also set D[k]=d if k not in D ----------------------------------------------------------------------< would not better represent Python current fashion. :-) -- François Pinard http://www.iro.umontreal.ca/~pinard

1 0

Should KeyError use repr() on its argument?
by Guido van Rossum 03 Sep '02

03 Sep '02

(SF bug 598451.) The KeyError exception doesn't apply repr() to its argument. That's annoying in cases like this: >>> a = {} >>> a[''] Traceback (most recent call last): File "<stdin>", line 1, in ? KeyError >>> Should this be fixed? How? (I guess we could add a KeyError__str__ method to exceptions.c that applies repr().) I've got a feeling this is a feature, but not a very useful one. --Guido van Rossum (home page: http://www.python.org/~guido/)

1 1

Get rid of etype struct
by Christian Tismer 03 Sep '02

03 Sep '02

Hi Guido, I think I have a solution for this one, see the attached diff. I did what you suggested: Make the adressing of the members dependant from the metatype. The etype struct has lost its members[1] field, to make it easier to extend the structure. Instead, the allocator always adds one to the size, to have the sentinel in place. I did not yet publish the etype stucture, since I didn't find a good name and place for it. Testing was also not very thorow. I just checked that types work from Python and that I can add __slots__ to them. Will re-port this stuff to my Py2.2 Stackless base and try it out as base type for my own C types. It took me the whole day to understand how it must work, and then just an hour to get it to work. This is quite some stuff :-) Can somebody please have a look, if there are subtle errors? ciao - chris ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=591586&group_i… cvs -z9 diff -u dist/src/Objects/typeobject.c Index: dist/src/Objects/typeobject.c =================================================================== RCS file: /cvsroot/python/python/dist/src/Objects/typeobject.c,v retrieving revision 2.179 diff -u -r2.179 typeobject.c --- dist/src/Objects/typeobject.c 16 Aug 2002 17:01:08 -0000 2.179 +++ dist/src/Objects/typeobject.c 3 Sep 2002 18:04:39 -0000 @@ -20,9 +20,12 @@ see add_operators() below. */ PyBufferProcs as_buffer; PyObject *name, *slots; - PyMemberDef members[1]; + /* here are optional user slots, followed by the members. */ } etype; +#define GET_MEMBERS(etype) \ + ((PyMemberDef *)(((char *)etype) + (etype)->type.ob_type->tp_basicsize)-1) + static PyMemberDef type_members[] = { {"__basicsize__", T_INT, offsetof(PyTypeObject,tp_basicsize),READONLY}, {"__itemsize__", T_INT, offsetof(PyTypeObject, tp_itemsize), READONLY}, @@ -213,7 +216,8 @@ PyType_GenericAlloc(PyTypeObject *type, int nitems) { PyObject *obj; - const size_t size = _PyObject_VAR_SIZE(type, nitems); + const size_t size = _PyObject_VAR_SIZE(type, nitems+1); + /* note that we need to add one, for the sentinel */ if (PyType_IS_GC(type)) obj = _PyObject_GC_Malloc(size); @@ -253,7 +257,7 @@ PyMemberDef *mp; n = type->ob_size; - mp = ((etype *)type)->members; + mp = GET_MEMBERS((etype *)type); for (i = 0; i < n; i++, mp++) { if (mp->type == T_OBJECT_EX) { char *addr = (char *)self + mp->offset; @@ -318,7 +322,7 @@ PyMemberDef *mp; n = type->ob_size; - mp = ((etype *)type)->members; + mp = GET_MEMBERS((etype *)type); for (i = 0; i < n; i++, mp++) { if (mp->type == T_OBJECT_EX && !(mp->flags & READONLY)) { char *addr = (char *)self + mp->offset; @@ -1125,7 +1129,8 @@ /* Are slots allowed? */ nslots = PyTuple_GET_SIZE(slots); - if (nslots > 0 && base->tp_itemsize != 0) { + if (nslots > 0 && base->tp_itemsize != 0 && !PyType_Check(base)) { + /* for the special case of meta types, allow slots */ PyErr_Format(PyExc_TypeError, "nonempty __slots__ " "not supported for subtype of '%s'", @@ -1334,7 +1339,7 @@ } /* Add descriptors for custom slots from __slots__, or for __dict__ */ - mp = et->members; + mp = GET_MEMBERS(et); slotoffset = base->tp_basicsize; if (slots != NULL) { for (i = 0; i < nslots; i++, mp++) { @@ -1366,7 +1371,7 @@ } type->tp_basicsize = slotoffset; type->tp_itemsize = base->tp_itemsize; - type->tp_members = et->members; + type->tp_members = GET_MEMBERS(et); type->tp_getset = subtype_getsets; /* Special case some slots */ *****CVS exited normally with code 1*****

1 0

PyString_DecodeEscape and PEP293
by Walter Dörwald 03 Sep '02

03 Sep '02

A recent checkin added a function PyString_DecodeEscape() to stringobject.c. To make this function PEP293 compatible it would need access to unicode_decode_call_errorhandler which is defined static in unicodeobject.c. Does PyString_DecodeEscape() really need an errors argument? If yes, we could either move it to unicodeobject.c or make unicode_decode_call_errorhandler externally visible. Another problem that I noticed is that string-escape can't be used for encoding Unicode objects: >>> u"\u0100".encode("string-escape") Traceback (most recent call last): File "<stdin>", line 1, in ? TypeError: escape_encode() argument 1 must be str, not unicode Bye, Walter Dörwald

2 4

Re: PEP 218 (sets); moving set.py to Lib
by Michael Chermside 03 Sep '02

03 Sep '02

>> Hmm, I intended to have s1.refresh() return a new object for use in >> s2 while leaving s1 alone (being immutable and all). Now, I wonder >> if that was the right thing to do. The answer lies in use cases for >> algorithms that need sets of sets. If anyone knows off the top of >> their head that would be great; otherwise, I seem to remember that >> some of that business was found in compiler algorithms and graph >> packages. > > Let's call YAGNI on this one. > Furthermore, what if I create a BIG set like this: s = ImmutableSet( range(2**x) ) Now, not only do I use lots of memory for s, I ALSO keep around lots of memory to preserve a temporary list which I never wanted to keep anyhow! -- Michael Chermside

1 0

HAVE_CONFIG_H
by Guido van Rossum 03 Sep '02

03 Sep '02

I see no references to HAVE_CONFIG_H in the source code (except one #undef in readline.c), yet we #define it on the command line. Is that still necessary? --Guido van Rossum (home page: http://www.python.org/~guido/)

6 16

RE: [Python-Dev] Python-dev summary for 2002-08-15 - 2002-09-01
by Delaney, Timothy 02 Sep '02

02 Sep '02

> From: Brett Cannon [mailto:bac@OCF.Berkeley.EDU] > > I think I will go with adding http:// to all addresses and > putting them on their own line. May I suggest that this may be a good test document for reStructuredText? Especially if it is going to such places as slashdot ... Tim Delaney

1 0

To commit or not to commit
by Walter Dörwald 02 Sep '02

02 Sep '02

I'm ready to commit the PEP293 implementation. I've added LaTeX documentation in libcodecs.tex and libexcs.tex. There are only a few minor open issues (reflecting exception attribute modifications in args, PyString_DecodeEscape), but I guess we'll fix/document those in time. Any objections against committing the patch? Bye, Walter Dörwald

5 9