Mailman 3 March 2007 - Python-Dev

Re: [Python-Dev] Extended Buffer Interface/Protocol
by Carl Banks March 27, 2007

March 27, 2007

(cc'ing back to Python-dev; the original reply was intended for it by I had an email malfunction.) Travis Oliphant wrote: >Carl Banks wrote: >> 3. Allow getbuffer to return an array of "derefence offsets", one for >> each dimension. For a given dimension i, if derefoff[i] is >> nonnegative, it's assumed that the current position (base pointer + >> indexing so far) is a pointer to a subarray, and derefoff[i] is the >> offest in that array where the current position goes for the next >> dimension. If derefoff[i] is negative, there is no dereferencing. >> Here is an example of how it'd work: > > > This sounds interesting, but I'm not sure I totally see it. I probably > need a picture to figure out what you are proposing. I'll get on it sometime. For now I hope an example will do. > The derefoff > sounds like some-kind of offset. Is that enough? Why not just make > derefoff[i] == 0 instead of negative? I may have misunderstood something. I had thought the values exported by getbuffer could change as the view narrowed, but I'm not sure if it's the case now. I'll assume it isn't for now, because it simplifies things and demonstrates the concept better. Let's start from the beginning. First, change the prototype to this: typedef PyObject *(*getbufferproc)(PyObject *obj, void **buf, Py_ssize_t *len, int *writeable, char **format, int *ndims, Py_ssize_t **shape, Py_ssize_t **strides, int **isptr) "isptr" is a flag indicating whether, for a certain dimension, the positision we've strided to so far is a pointer that should be followed before proceeding with the rest of the strides. Now here's what a general "get_item_pointer" function would look like, given a set of indices: void* get_item_pointer(int ndim, void* buf, Py_ssize_t* strides, Py_ssize_t* derefoff, Py_ssize_t *indices) { char* pointer = (char*)buf; int i; for (i = 0; i < ndim; i++) { pointer += strides[i]*indices[i]; if (isptr[i]) { pointer = *(char**)pointer; } } return (void*)pointer; } > I don't fully understand the PIL example you gave. Yeah. How about more details. Here is a hypothetical image data object structure: struct rgba { unsigned char r, g, b, a; }; struct ImageObject { PyObject_HEAD; ... struct rgba** lines; Py_ssize_t height; Py_ssize_t width; Py_ssize_t shape_array[2]; Py_ssize_t stride_array[2]; Py_ssize_t view_count; }; "lines" points to malloced 1-D array of (struct rgba*). Each pointer in THAT block points to a seperately malloced array of (struct rgba). Got that? In order to access, say, the red value of the pixel at x=30, y=50, you'd use "lines[50][30].r". So what does ImageObject's getbuffer do? Leaving error checking out: PyObject* getbuffer(PyObject *self, void **buf, Py_ssize_t *len, int *writeable, char **format, int *ndims, Py_ssize_t **shape, Py_ssize_t **strides, int **isptr) { static int _isptr[2] = { 1, 0 }; *buf = self->lines; *len = self->height*self->width; *writable = 1; *ndims = 2; self->shape_array[0] = height; self->shape_array[1] = width; *shape = &self->shape_array; self->stride_array[0] = sizeof(struct rgba*); /* yep */ self->stride_array[1] = sizeof(struct rgba); *strides = &self->stride_array; *isptr = _isptr; self->view_count ++; /* create and return view object here, but for what? */ } There are three essential differences from a regular, contiguous array. 1. buf is set to point at the array of pointers, not directly to the data. 2. The isptr thing. isptr[0] is true to indicate that the first dimension is an array of pointers, not the actual data. 3. stride[0] is sizeof(struct rgba*), not self->width*sizeof(struct rgba) like it would be for a contiguous array. This is because your first stride is through an array of pointers, not the data itself. So let's examine what "get_item_pointer" above will do given these values. Once again, we're looking for the pixel at x=30, y=50. First, we set pointer to buf, that is, self->lines. Then we take the first stride: we add index[0]+strides[0], that is, 50*4=200, to poitner. pointer now equals &self->lines[50]. Now, we check isptr[0]. We see that it is true. Thus, the position we've strided to is, in fact, a pointer to a subarray where the actual data is. So we follow it: pointer = *pointer. pointer now equals self->lines[50] which equals &self->lines[50][0]. Next dimension. We take the second stride: we add index[1]+strides[1], that is, 30*4=120, to pointer. pointer now equals &self->lines[50][30]. Now, we check isptr[1]. It's false. No dereferencing this step. We're done. Return pointer. >> By the way, has anyone signed up to modify the standard library >> modules? I could do those when the protocol is finalized. And if >> you're implementing the new buffer protocol in 2.6 (while deprecating >> but not removing the old protocol, I presume), will the modules also >> be updated for 2.6? >> > Nobody has signed up for anything. I'm willing for anyone to help. > Many of the standard library modules will need to be modified. And > yes, I do want to implement the new protocol for 2.6 (adding it to the > current one). Updating the modules for 2.6 would not be high priority > (except the struct module), but is a desirable. Ok, then, consider me available for it. > Thanks for the ideas. Ok, I have two questions, now. First, I'm not sure why getbuffer needs to return a view object. I expect most views of data to be created separately--for instance, a view of an image is likely to be created in Python using something like this: imgview = ImageView(image,(left,right),(top,bottom)) I'd expect the ImageView object would call getbuffer and use the data returned in buf, len, writable, etc., and would have no need for a type-specific view object. Furthermore, I would expect in many cases different views are desirable, and some cases where the viewer is unknown to the exporter. And, if it does have to return a view for some reason, why bother returning buf, len, and friends in the function? Just return those values in the view object. Second question: what happens if a view wants to re-export the buffer? Do the views of the buffer ever change? Example, say you create a transposed view of a Numpy array. Now you want a slice of the transposed array. What does the transposed view's getbuffer export? Naively, I'd expect the "strides" and "shape" array to have rearranged indices, but it looks like you might be trying to get rid of this complexity. The reason I ask is: if things like "buf" and "strides" and "shape" could change when a buffer is re-exported, then it can complicate things for PIL-like buffers. (How would you account for offsets in a dimension that's in a subarray?) Carl Banks

5 6

RFC - GoogleSOC proposal -cleanupurllib
by Senthil March 26, 2007

March 26, 2007

Hi All, I have written a proposal to cleanup urllib as part of Google SoC. I am attaching the file 'soc1' with this email. Requesting you to go through the proposal and provide any feedback which I can incorporate in my submission. Thanks, Senthil -- O.R.Senthil Kumaran http://phoe6.livejournal.com

4 5

Final (final?) update to patch about socket timeout...
by Facundo Batista March 25, 2007

March 25, 2007

...in socket.py and httplib.py, with tests and docs. The patch is #1676823. Basically what I did now is: - Just put a timeout default to None. If None, skip settimeout() call. - Copy the exceptions behaviour that we have actually in the higher level libraries, to be sure we aren't breaking anything. - The function is now public, named "create_connection". Regards, -- . Facundo . Blog: http://www.taniquetil.com.ar/plog/ PyAr: http://www.python.org/ar/

2 6

Hindsight on Py_UNICODE_WIDE?
by Jason Orendorff March 24, 2007

March 24, 2007

Scheme is adding Unicode support in an upcoming standard: (DRAFT) http://www.r6rs.org/document/lib-html/r6rs-lib-Z-H-3.html I have two questions for the python-dev team about Python's Unicode experiences. If it's convenient, please take a moment to reply. Thanks in advance. 1. In hindsight, what do you think about PEP 261, the Py_UNICODE_WIDE build option? On balance, has this been good, bad, or indifferent? What's good/bad about it? 2. The idea of multiple string representations has come up (that is, where all strings are Unicode, but in memory some are 8-bit, some 16-bit, and some 32-bit--each string uses the narrowest possible representation). This has been discussed here for Python 3000. My question is: Is this for real? How far along is it? How likely is it? Thanks, Jason

4 3

Extended Buffer Interface/Protocol
by Travis Oliphant March 24, 2007

March 24, 2007

I'm soliciting feedback on my extended buffer protocol that I am proposing for inclusion in Python 3000. As soon as the Python 3.0 implementation is complete, I plan to back-port the result to Python 2.6, therefore, I think there may be some interest on this list. Basically, the extended buffer protocol seeks to allow memory sharing with 1) information about what is "in" the memory (float, int, C-structure, etc.) 2) information about the "shape" of the memory (if any) 3) information about discontiguous memory segments Number 3 is where I could use feedback --- especially from PIL users and developers. Strides are a common way to think about a possibly discontiguous chunk of memory (which appear in NumPy when you select a sub-region from a larger array). The strides vector tells you how many bytes to skip in each dimension to get to the next memory location for that dimension. Because NumPy uses this memory model as do several compute libraries (like BLAS and LAPACK), it makes sense to allow this memory model to be shared between objects in Python. Currently, the proposed buffer interface eliminates the multi-segment option (for Python 3.0) which I think was originally put in place because of the PIL. However, I don't know if it is actually used by any extension types. This is a big reason why Guido wants to drop the multi-segment interface option. The question is should we eliminate the possibility of sharing memory for objects that store data basically as "arrays" of arrays (i.e. true C-style arrays). That is what I'm currently proposing, but I could also see an argument that states that if we are going to support strided memory access, we should also support array of array memory access. If this is added, then it would be another function-call that gets a array-of-array-style memory from the object. What do others think of these ideas? One possible C-API call that Python could grow with the current buffer interface is to allow contiguous-memory mirroring of discontiguous memory, or an iterator object that iterates through every element of any object that exposes the buffer protocol. Thanks for any feedback, -Travis Oliphant

4 10

Exceptions for readonly attributes
by Raymond Hettinger March 24, 2007

March 24, 2007

Re: www.python.org/sf/1687163 I noticed that RO members raise a TypeError upon an attempted write. In contrast, we get an AttributeError when writing to a readonly property or to a readonly method (such as those for builtin types). IMO, the TypeError should really be an AttributeError. However, changing it makes four of the tests fail. The question for you guys is whether or not to make the fix. Raymond

2 1

regexp in Python
by Bartłomiej Wołowiec March 23, 2007

March 23, 2007

For some time I'm interested in regular expressions and Finite State Machine. Recently, I saw that Python uses "Secret Labs' Regular Expression Engine", which very often works too slow. Its pesymistic time complexity is O(2^n), although other solutions, with time complexity O(n*m) ( O(n*m^2), m is the length of the regular expression and n is the length of the text, introduction to problem: http://swtch.com/~rsc/regexp/regexp1.html ) For example: $ cat test.py import re import sys if sys.argv[1:]: n = int(sys.argv[1]) regexp = 'a?'*n + 'a'*n print 'regexp=', regexp str = 'a'*n + 'b' + 'a'*n print 'str=', str print re.findall(regexp, str) $ time python test.py 20 regexp= a?a?a?a?a?a?a?a?a?a?a?a?a?a?a?a?a?a?a?a?aaaaaaaaaaaaaaaaaaaa str= aaaaaaaaaaaaaaaaaaaabaaaaaaaaaaaaaaaaaaaa ['aaaaaaaaaaaaaaaaaaaa', 'aaaaaaaaaaaaaaaaaaaa'] real 0m0.592s user 0m0.508s sys 0m0.028s $ time python test.py 21 regexp= a?a?a?a?a?a?a?a?a?a?a?a?a?a?a?a?a?a?a?a?a?aaaaaaaaaaaaaaaaaaaaa str= aaaaaaaaaaaaaaaaaaaaabaaaaaaaaaaaaaaaaaaaaa ['aaaaaaaaaaaaaaaaaaaaa', 'aaaaaaaaaaaaaaaaaaaaa'] real 0m1.018s user 0m1.000s sys 0m0.004s $ time python test.py 22 regexp= a?a?a?a?a?a?a?a?a?a?a?a?a?a?a?a?a?a?a?a?a?a?aaaaaaaaaaaaaaaaaaaaaa str= aaaaaaaaaaaaaaaaaaaaaabaaaaaaaaaaaaaaaaaaaaaa ['aaaaaaaaaaaaaaaaaaaaaa', 'aaaaaaaaaaaaaaaaaaaaaa'] real 0m2.062s user 0m1.992s sys 0m0.028s $ time python test.py 23 regexp= a?a?a?a?a?a?a?a?a?a?a?a?a?a?a?a?a?a?a?a?a?a?a?aaaaaaaaaaaaaaaaaaaaaaa str= aaaaaaaaaaaaaaaaaaaaaaabaaaaaaaaaaaaaaaaaaaaaaa ['aaaaaaaaaaaaaaaaaaaaaaa', 'aaaaaaaaaaaaaaaaaaaaaaa'] real 0m4.159s user 0m4.092s sys 0m0.024s Unfortuntely not for all regular expressions quick implementation may be used (e.g. when it uses backreferences), but for major part of regular expressions this implementation works much faster. In addition to Google Summer of Code I would willingly write a faster implementation of regular expressions in Python. Naturally, the implementation would be 100% compatible with the existing regexp engine. What do you think about my proposition? -- Bartlomiej Wolowiec

4 3

Re: [Python-Dev] [Python-checkins] r54539 - in python/trunk: Lib/string.py Misc/NEWS Objects/typeobject.c
by Jim Jewett March 23, 2007

March 23, 2007

(Please note that most replies should trim at least one list from the Cc) On 3/23/07, guido.van.rossum <python-checkins(a)python.org> wrote: > Note: without the change to string.py, lots of spurious warnings happen. > What's going on there? I assume it was a defensive measure for subclasses of both Template and X, where X has its own metaclass (which might have an __init__ that shouldn't be ignored). This is where cooperative classes get ugly. You could argue that the "correct" code is to not bother making the super call if it would go all the way to object (or even type), but there isn't a good way to spell that. > + # A super call makes no sense since type() doesn't define __init__(). > + # (Or does it? And should type.__init__() accept three args?) > + # super(_TemplateMetaclass, cls).__init__(name, bases, dct) In this particular case, you could define a type.__init__ that did nothing. (If the signature were wrong, type.__new__ would have already caught it. If __new__ and __init__ are seeing something different, then the change was probably intentional.) The problem isn't really limited to type.__init__ though. You'll sometimes see similar patterns for __del__, close, save, etc. The main difference is that they have to either catch an error or check first, since object doesn't have an implementation of those methods. object.__init__ doesn't really do anything either, except check for errors. Getting rid of it should have the same effect as complaining about parameters. The ideal solution (discussion of which probably ought to stay on python-ideas) might be to replace object.__init__ with some sort of PlaceHolder object that raises an error *unless* called through a cooperative super. This PlaceHolder would also be useful for AbstractBaseClasses/Interfaces. PlaceHolder still wouldn't deal with the original concern of verifying that all arguments had already been stripped and used; but the ABCs might be able to. -jJ > Modified: python/trunk/Lib/string.py > ============================================================================== > --- python/trunk/Lib/string.py (original) > +++ python/trunk/Lib/string.py Fri Mar 23 05:58:42 2007 > @@ -108,7 +108,9 @@ > """ > > def __init__(cls, name, bases, dct): > - super(_TemplateMetaclass, cls).__init__(name, bases, dct) > + # A super call makes no sense since type() doesn't define __init__(). > + # (Or does it? And should type.__init__() accept three args?) > + # super(_TemplateMetaclass, cls).__init__(name, bases, dct) > if 'pattern' in dct: > pattern = cls.pattern > else:

1 0

deprecate commands.getstatus()
by Georg Brandl March 23, 2007

March 23, 2007

I'd like to deprecate commands.getstatus() in 2.6. Reason: there is getoutput() and getstatusoutput(). In the latter, "status" means the exit code. getstatus(), however, returns the output of "ls -ld <file>" which is completely nonobvious and confusing. Perhaps the whole commands module can be deprecated in favor of subprocess. Georg

16 36

Adding timeout to socket.py and httplib.py
by Facundo Batista March 23, 2007

March 23, 2007

I studied Skip patch, and I think he is in good direction: add a NetworkConnection object to socket.py, and then use it from the other modules. This NetworkConnection basically does what all the other modules do once and again, so no mistery about it (basically calls getaddrinfo over the received address, and try to open a socket to that address). I opened a new patch (#1676823) with the changes I propose regarding socket.py, because the goal from the two patches are different (my plan is go with the basic: first the change in socket.py and httplib, and no in all the other modules at this time). I do not know what to do with the previous patch (#723312), I guess it'll remain open until all the other modules get the timeout. Here're the differences between Skip patch and mine: - I only left changes regarding httplib and socket modules (both .py, docs, and NEWS). - I even removed a change in Python-ast.c (regarding __version__), but I don't know what's that for, so please enlighten me (thank you). - The NetworkConnection won't have a ``get_family`` method, if you need the family of the open socket, just ask the socket. - Added some test cases to test_socket.py regarding attributes, timeout and family; and a nice threaded test to actually try the timeout. - Added tests cases to test_httplib.py Feel free to review the patch, and commit it if you want (or tell me to do it after the review, it's just a command for me). Regards, -- . Facundo . Blog: http://www.taniquetil.com.ar/plog/ PyAr: http://www.python.org/ar/

7 30