Mailman 3 February 2009 - Python-Dev

Re: [Python-Dev] C API for appending to arrays
by Hrvoje Niksic 04 Feb '09

04 Feb '09

Mike Klaas wrote: > Do you need to append, or are you just looking to create/manipulate an > array with a bunch of c-float values? Mostly real-life examples I've seen of this were creating an array from C values obtained from an external source, such as an on-disk file, or another process. The code example was a (simplified and de-C++-ized) snippet of actual code. > I find As{Write/Read}Buffer sufficient for most of these tasks. They improve things, as shown in the second example, but they're still cumbersome to use for appending/initialization of the array. > (Note that you can > get the size of the resulting c array more easily than you are by > using PyObject_Length). Note that AsWriteBuffer always gives you the buffer size anyway -- you can't pass bufsize==NULL. Since I have to call AsWriteBuffer in each iteration (because I don't know when the buffer will resize), calling PyObject_Length in addition to that doesn't buy much, if anything. > I've included some example pyrex code that populates a new > array.array at c speed. [...] > cdef int NA > NA = len(W1) > W0 = array('d', [colTotal]) * NA The thing is, when reading values from a file or a general iterator, you typically don't know the number of values in advance. If I did, I would probably use an approach similar to yours. Thanks for the code -- even if it doesn't help in this case, I appreciate it as an instructing example of the advanced usage of Pyrex.

1 0

teaching the new urllib
by Brad Miller 04 Feb '09

04 Feb '09

I'm just getting ready to start the semester using my new book (Python Programming in Context) and noticed that I somehow missed all the changes to urllib in python 3.0. ARGH to say the least. I like using urllib in the intro class because we can get data from places that are more interesting/motivating/relevant to the students. Here are some of my observations on trying to do very basic stuff with urllib: 1. urllib.urlopen is now urllib.request.urlopen 2. The object returned by urlopen is no longer iterable! no more for line in url. 3. read, readline, readlines now return bytes objects or arrays of bytes instead of a str and array of str 4. Taking the naive approach to converting a bytes object to a str does not work as you would expect. >>> import urllib.request >>> page = urllib.request.urlopen('http://knuth.luther.edu/test.html') >>> page <addinfourl at 16419792 whose fp = <socket.SocketIO object at 0xfa8570>> >>> line = page.readline() >>> line b'<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"\n' >>> str(line) 'b\'<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"\\n\'' >>> As you can see from the example the 'b' becomes part of the string! It seems like this should be a bug, is it? Here's the iteration problem: 'b\'<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"\\n\'' >>> for line in page: print(line) Traceback (most recent call last): File "<pyshell#10>", line 1, in <module> for line in page: TypeError: 'addinfourl' object is not iterable Why is this not iterable anymore? Is this too a bug? What the heck is an addinfourl object? 5. Finally, I see that a bytes object has some of the same methods as strings. But the error messages are confusing. >>> line b' "http://www.w3.org/TR/html4/loose.dtd">\n' >>> line.find('www') Traceback (most recent call last): File "<pyshell#18>", line 1, in <module> line.find('www') TypeError: expected an object with the buffer interface >>> line.find(b'www') 11 Why couldn't find take string as a parameter? If folks have advice on which, if any, of these are bugs please let me know and I'll file them, and if possible work on fixes for them too. If you have advice on how I should better be teaching this new urllib that would be great to hear as well. Thanks, Brad -- Brad Miller Assistant Professor, Computer Science Luther College

8 9

C API for appending to arrays
by Hrvoje Niksic 04 Feb '09

04 Feb '09

The array.array type is an excellent type for storing a large amount of "native" elements, such as integers, chars, doubles, etc., without involving the heavy machinery of numpy. It's both blazingly fast and reasonably efficient with memory. The one thing missing from the array module is the ability to directly access array values from C. This might seem superfluous, as it's perfectly possible to manipulate array contents from Python/C using PyObject_CallMethod and friends. The problem is that it requires the native values to be marshalled to Python objects, only to be immediately converted back to native values by the array code. This can be a problem when, for example, a numeric array needs to be filled with contents, such as in this hypothetical example: /* error checking and refcounting subtleties omitted for brevity */ PyObject *load_data(Source *src) { PyObject *array_type = get_array_type(); PyObject *array = PyObject_CallFunction(array_type, "c", 'd'); PyObject *append = PyObect_GetAttrString(array, "append"); while (!source_done(src)) { double num = source_next(src); PyObject *f = PyFloat_FromDouble(num); PyObject *ret = PyObject_CallFunctionObjArgs(append, f, NULL); if (!ret) return NULL; Py_DECREF(ret); Py_DECREF(f); } Py_DECREF(array_type); return array; } The inner loop must convert each C double to a Python Float, only for the array to immediately extract the double back from the Float and store it into the underlying array of C doubles. This may seem like a nitpick, but it turns out that more than half of the time of this function is spent creating and deleting those short-lived floating-point objects. Float creation is already well-optimized, so opportunities for speedup lie elsewhere. The array object exposes a writable buffer, which can be used to store values directly. For test purposes I created a faster "append" specialized for doubles, defined like this: int array_append(PyObject *array, PyObject *appendfun, double val) { PyObject *ret; double *buf; Py_ssize_t bufsize; static PyObject *zero; if (!zero) zero = PyFloat_FromDouble(0); // append dummy zero value, created only once ret = PyObject_CallFunctionObjArgs(appendfun, zero, NULL); if (!ret) return -1; Py_DECREF(ret); // append the element directly at the end of the C buffer PyObject_AsWriteBuffer(array, (void **) &buf, &bufsize)); buf[bufsize / sizeof(double) - 1] = val; return 0; } This hack actually speeds up array creation by a significant percentage (30-40% in my case, and that's for code that was producing the values by parsing a large text file). It turns out that an even faster method of creating an array is by using the fromstring() method. fromstring() requires an actual string, not a buffer, so in C++ I created an std::vector<double> with a contiguous array of doubles, passed that array to PyString_FromStringAndSize, and called array.fromstring with the resulting string. Despite all the unnecessary copying, the result was much faster than either of the previous versions. Would it be possible for the array module to define a C interface for the most frequent operations on array objects, such as appending an item, and getting/setting an item? Failing that, could we at least make fromstring() accept an arbitrary read buffer, not just an actual string?

3 2

Partial function application 'from the right'
by Ben North 03 Feb '09

03 Feb '09

Hi, I find 'functools.partial' useful, but occasionally I'm unable to use it because it lacks a 'from the right' version. E.g., to create a function which splits a string on commas, you can't say # Won't work when called: split_comma = partial(str.split, sep = ',') and to create a 'log to base 10' function, you can't say # Won't work when called: log_10 = partial(math.log, base = 10.0) because str.split and math.log don't take keyword arguments. PEP-309, which introduces functools.partial, mentions For completeness, another object that appends partial arguments after those supplied in the function call (maybe called rightcurry) has been suggested. 'Completeness' by itself doesn't seem to have been a compelling reason to introduce this feature, but the above cases show that it would be of real use. I've created a patch which adds a 'partial_right' function. The two examples above: >>> import functools, math >>> split_comma = functools.partial_right(str.split, ',') >>> split_comma('a,b,c') ['a', 'b', 'c'] >>> log_10 = functools.partial_right(math.log, 10.0) >>> log_10(100.0) 2.0 and a general illustrative one: >>> def all_args(*args): return args ... >>> functools.partial_right(all_args, 1, 2)(3, 4) (3, 4, 1, 2) I was prompted to look at this by a recent message on python-dev: Xavier Morel <catch-all(a)masklinn.net>, Thu, 22 Jan 2009 14:44:41 +0100: > [...] drop(iterable, n) has to be written islice(iterable, n, None) > (and of course the naming isn't ideal), and you can't really use > functools.partial since the iterator is the first argument (unless > there's a way to partially apply only the tail args without kwargs). Xavier's case becomes: >>> import functools, itertools >>> drop = functools.partial_right(itertools.islice, None, None) >>> list(drop(range(10), 5)) [5, 6, 7, 8, 9] The patch adds a 'from_right' member to partial objects, which can be True for the new from-the-right behaviour, or False for the existing from-the-left behaviour. It's quite small, only c.40 lines, plus a c.70-line change to test_functools.py. I imagine a documention patch would be c.20 lines. Would there be any interest in this? Ben.

12 20

teaching the new urllib
by Brad Miller 03 Feb '09

03 Feb '09

I'm just getting ready to start the semester using my new book (Python Programming in Context) and noticed that I somehow missed all the changes to urllib in python 3.0. ARGH to say the least. I like using urllib in the intro class because we can get data from places that are more interesting/motivating/relevant to the students. Here are some of my observations on trying to do very basic stuff with urllib: 1. urllib.urlopen is now urllib.request.urlopen 2. The object returned by urlopen is no longer iterable! no more for line in url. 3. read, readline, readlines now return bytes objects or arrays of bytes instead of a str and array of str 4. Taking the naive approach to converting a bytes object to a str does not work as you would expect. >>> import urllib.request >>> page = urllib.request.urlopen('http://knuth.luther.edu/test.html') >>> page <addinfourl at 16419792 whose fp = <socket.SocketIO object at 0xfa8570>> >>> line = page.readline() >>> line b'<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"\n' >>> str(line) 'b\'<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"\\n\'' >>> As you can see from the example the 'b' becomes part of the string! It seems like this should be a bug, is it? Here's the iteration problem: 'b\'<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"\\n\'' >>> for line in page: print(line) Traceback (most recent call last): File "<pyshell#10>", line 1, in <module> for line in page: TypeError: 'addinfourl' object is not iterable Why is this not iterable anymore? Is this too a bug? What the heck is an addinfourl object? 5. Finally, I see that a bytes object has some of the same methods as strings. But the error messages are confusing. >>> line b' "http://www.w3.org/TR/html4/loose.dtd">\n' >>> line.find('www') Traceback (most recent call last): File "<pyshell#18>", line 1, in <module> line.find('www') TypeError: expected an object with the buffer interface >>> line.find(b'www') 11 Why couldn't find take string as a parameter? If folks have advice on which, if any, of these are bugs please let me know and I'll file them, and if possible work on fixes for them too. If you have advice on how I should better be teaching this new urllib that would be great to hear as well. Thanks, Brad -- Brad Miller Assistant Professor, Computer Science Luther College -- Brad Miller Assistant Professor, Computer Science Luther College

1 0

Should there be a source-code checksum in module objects?
by Rocky Bernstein 03 Feb '09

03 Feb '09

As I've mentioned, I've been re-examining from ground up the whole state of affairs in writing a debugger. One of the challenges of a debugger or any source-code analysis tool is verifying that the source-code that the tool is reporting on corresponds to the compiled object under execution. (For debuggers, this problem becomes more likely to occur when you are debugging on a computer that isn't the same as the computer where the code is running.) Is there a checksum of the source text computed and stored in compilation? If not, should there be one?

5 5

Re: [Python-Dev] C API for appending to arrays
by Hrvoje Niksic 03 Feb '09

03 Feb '09

Raymond Hettinger wrote: > [Hrvoje Niksic] >> The one thing missing from the array >> module is the ability to directly access array values from C. > > Please put a feature request on the bug tracker. Done, http://bugs.python.org/issue5141

1 0

Removing tp_compare?
by Mark Dickinson 02 Feb '09

02 Feb '09

Here's a question (actually, three questions) for python-dev that came up in the issue 1717 (removing cmp) discussion. Once the cmp removal is complete, the type object's tp_compare slot will no longer be used. The current plan is to rename it to tp_reserved, change its type to (void *), and raise TypeError when initializing any type that attempts to put something nonzero into that slot. But another possibility would be to remove it entirely. So... Questions: (1) Is it desirable to remove tp_compare entirely, instead of just renaming it? (2) If so, for which Python version should that removal take place? 3.0.1? 3.1.0? 4.0? and the all-important bikeshed question: (3) In the meantime, what should the renamed slot be called? tp_reserved? In the issue 1717 discussion, Raymond suggested tp_deprecated_compare. Any thoughts? My own opinion is that it really doesn't matter that much if the slot is left in; it's just a little annoying to have such backwards-compatibility baggage already present in the shiny new 3.0 series. A little like finding a big scratch on your brand-new bright yellow Hummer H3. Or not. N.B. The same questions apply to nb_reserved (which used to be nb_long) in the PyNumberMethods structure. Mark

7 14

python 3.0, tp_compare not used for == test?
by Campbell Barton 02 Feb '09

02 Feb '09

Hi, I have been writing a new C api that can build with both python 2.6 and 3.0 I found that when building with python 2.6, doing "a==b" between 2 different pyrna_struct_Type's would run tp_compare But with python 3.0, "a==b" will always be false and tp_compare function would not even run. The only way to get it to run the tp_compare function was to do "cmp(a, b)" however Id like to be able to use == still. The only way I could get "a==b" to work in python 3.0 was to add a tp_richcompare function which runs tp_compare with Py_CmpToRich() anyway Is this the intended behavior? - if so its important for external C API's to be updated to support this. Included the PyType below with some relevant details. I looked at other Python 3.0 types but could not see why this would happen. // ----snip, // initialized with PyType_Ready if( PyType_Ready( &pyrna_struct_Type ) < 0 ) return NULL; // header definition for BPy_StructRNA typedef struct { PyObject_HEAD /* required python macro */ PointerRNA ptr; int freeptr; /* needed in some cases if ptr.data is created on the fly, free when deallocing */ } BPy_StructRNA; // PyType /*-----------------------BPy_StructRNA method def------------------------------*/ PyTypeObject pyrna_struct_Type = { PyVarObject_HEAD_INIT(NULL, 0) "StructRNA", /* tp_name */ sizeof( BPy_StructRNA ), /* tp_basicsize */ 0, /* tp_itemsize */ /* methods */ ( destructor ) pyrna_struct_dealloc,/* tp_dealloc */ NULL, /* printfunc tp_print; */ NULL, /* getattrfunc tp_getattr; */ NULL, /* setattrfunc tp_setattr; */ ( cmpfunc ) pyrna_struct_compare, /* tp_compare */ ( reprfunc ) pyrna_struct_repr, /* tp_repr */ /* Method suites for standard classes */ NULL, /* PyNumberMethods *tp_as_number; */ NULL, /* PySequenceMethods *tp_as_sequence; */ NULL, /* PyMappingMethods *tp_as_mapping; */ /* More standard operations (here for binary compatibility) */ ( hashfunc )pyrna_struct_hash, /* hashfunc tp_hash; */ NULL, /* ternaryfunc tp_call; */ NULL, /* reprfunc tp_str; */ ( getattrofunc ) pyrna_struct_getattro, /* getattrofunc tp_getattro; */ ( setattrofunc ) pyrna_struct_setattro, /* setattrofunc tp_setattro; */ /* Functions to access object as input/output buffer */ NULL, /* PyBufferProcs *tp_as_buffer; */ /*** Flags to define presence of optional/expanded features ***/ Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, /* long tp_flags; */ NULL, /* char *tp_doc; Documentation string */ /*** Assigned meaning in release 2.0 ***/ /* call function for all accessible objects */ NULL, /* traverseproc tp_traverse; */ /* delete references to contained objects */ NULL, /* inquiry tp_clear; */ /*** Assigned meaning in release 2.1 ***/ /*** rich comparisons ***/ (richcmpfunc)pyrna_struct_richcmp, /* richcmpfunc tp_richcompare; */ /*** weak reference enabler ***/ 0, /* long tp_weaklistoffset; */ /*** Added in release 2.2 ***/ /* Iterators */ NULL, /* getiterfunc tp_iter; */ NULL, /* iternextfunc tp_iternext; */ /*** Attribute descriptor and subclassing stuff ***/ pyrna_struct_methods, /* struct PyMethodDef *tp_methods; */ NULL, /* struct PyMemberDef *tp_members; */ NULL, /* struct PyGetSetDef *tp_getset; */ NULL, /* struct _typeobject *tp_base; */ NULL, /* PyObject *tp_dict; */ NULL, /* descrgetfunc tp_descr_get; */ NULL, /* descrsetfunc tp_descr_set; */ 0, /* long tp_dictoffset; */ NULL, /* initproc tp_init; */ NULL, /* allocfunc tp_alloc; */ pyrna_struct_new, /* newfunc tp_new; */ /* Low-level free-memory routine */ NULL, /* freefunc tp_free; */ /* For PyObject_IS_GC */ NULL, /* inquiry tp_is_gc; */ NULL, /* PyObject *tp_bases; */ /* method resolution order */ NULL, /* PyObject *tp_mro; */ NULL, /* PyObject *tp_cache; */ NULL, /* PyObject *tp_subclasses; */ NULL, /* PyObject *tp_weaklist; */ NULL }; -- - Campbell

5 7

3.0.1/3.1.0 summary
by Brett Cannon 02 Feb '09

02 Feb '09

This is my attempt to summarize what everyone has been saying so we can get this resolved. >From what I can tell, most people like the idea of doing a 3.0.1 release ASAP (like "in a week or so" fast) with the stuff that should have been removed from 3.0.0 in the first place removed. People also seem to support doing a 3.1 release April/May where new stuff (e.g. io in C, new shelve back-end for sqlite3) is introduced to the rest of the world. This timeline has the benefit of allowing us to do an alpha release at PyCon and puts us at a six month release cycle which does not portray 3.0 or 3.1 as rushed releases. The sticky points I see are: 1. Barry, who is the release manager for 3.0.1, does not like the idea of the cruft that is being proposed removed from 3.0.1. Personally I say we continue to peer pressure him as I think a new major release is not like our typical minor release, but I am not about to force Barry to go against what he thinks is reasonable unless I am willing to step up as release manager (and I am not since I simply don't have the time to learn the process fast enough along with just a lack of time with other Python stuff). 2. Do we label 3.0.x as experimental? I say no since it isn't experimental; we basically had some bugs slip through that happen to be compatibility problems that were overlooked. I for one never viewed 3.0.x as experimental, just not the best we could necessarily do without more input from the community and our own experience with 3.x in general. Let's see if we can get these two points squared away so we can get 3.0.1 in whatever state it is meant to be in out the door quickly. -Brett

6 15