Mailman 3 April 2007 - Python-Dev

RELEASED Python 2.5.1, FINAL
by Anthony Baxter April 19, 2007

April 19, 2007

On behalf of the Python development team and the Python community, I'm happy to announce the release of Python 2.5.1 (FINAL) This is the first bugfix release of Python 2.5. Python 2.5 is now in bugfix-only mode; no new features are being added. According to the release notes, over 150 bugs and patches have been addressed since Python 2.5, including a fair number in the new AST compiler (an internal implementation detail of the Python interpreter). This is a production release of Python, and … [View More]

1 0

Re: [Python-Dev] PEP 3118: Extended buffer protocol (new version)
by Travis Oliphant April 19, 2007

April 19, 2007

Carl Banks wrote: > Travis Oliphant wrote: >> Carl Banks wrote: >>> Ok, I've thought quite a bit about this, and I have an idea that I >>> think will be ok with you, and I'll be able to drop my main >>> objection. It's not a big change, either. The key is to explicitly >>> say whether the flag allows or requires. But I made a few other >>> changes as well. >> I'm good with using an identifier to differentiate between an >… [View More]> "allowed" flag and a "require" flag. I'm not a big fan of >> VERY_LONG_IDENTIFIER_NAMES though. Just enough to understand what it >> means but not so much that it takes forever to type and uses up >> horizontal real-estate. > > That's fine with me. I'm not very particular about spellings, as long > as they're not misleading. > >>> Now, here is a key point: for these functions to work (indeed, for >>> PyObject_GetBuffer to work at all), you need enough information in >>> bufinfo to figure it out. The bufferinfo struct should be >>> self-contained; you should not need to know what flags were passed >>> to PyObject_GetBuffer in order to know exactly what data you're >>> looking at. >> Naturally. >> >>> Therefore, format must always be supplied by getbuffer. You cannot >>> tell if an array is contiguous without the format string. (But see >>> below.) >> >> No, I don't think this is quite true. You don't need to know what >> "kind" of data you are looking at if you don't get strides. If you >> use the SIMPLE interface, then both consumer and exporter know the >> object is looking at "bytes" which always has an itemsize of 1. > > But doesn't this violate the above maxim? Suppose these are the > contents of bufinfo: > > ndim = 1 > len = 20 > shape = (10,) > strides = (2,) > format = NULL In my thinking, format/itemsize is necessary if you have strides (as you do here) but not needed if you don't have strides information (i.e. you are assuming a C_CONTIGUOUS memory-chunk). The intent of the simple interface is to basically allow consumers to mimic the old buffer protocol, very easily. > > How does it know whether it's looking at contiguous array of 10 > two-byte objects, or a discontiguous array of 10 one-byte objects, > without having at least an item size? Since item size is now in the > mix, it's moot, of course. My only real concern is to have some way to tell the exporter that it doesn't need to "figure out" the format if the consumer doesn't care about it. Given the open-ended nature of the format string, it is possible that a costly format-string construction step could be under-taken even when the consumer doesn't care about it. I can see you are considering the buffer structure as a self-introspecting structure where I was considering it only in terms of how the consumer would be using its members (which implied it knew what it was asking for and wouldn't touch anything else). How about we assume FORMAT will always be filled in but we add a Py_BUF_REQUIRE_PRIMITIVE flag that will only return "primitive" format strings (i.e. basic c-types)? An exporter receiving this flag will have to return complicated data-types as 'bytes'. I would add this to the Py_BUF_SIMPLE default. > > The idea that Py_BUF_SIMPLE implies bytes is news to me. What if you > want a contiguous, one-dimensional array of an arbitrary type? I was > thinking this would be acceptable with Py_BUF_SIMPLE. Unsigned bytes are just the lowest common denominator. They represent the old way of sharing memory. Doesn't an "arbitrary type" mean bytes? Or did you mean what if you wanted a contiguous, one-dimensional array of a *specific* type? > It seems you want to require Py_BUF_FORMAT for that, which would > suggest to me that > But it now it seems even more unnecessary than it did before. > Wouldn't any consumer that just wants to look at a chunk of bytes > always use Py_BUF_FORMAT, especially if there's danger of a > presumptuous exporter raising an exception? > I'll put in the REQUIRE_PRIMITIVE_FORMAT idea in the next update to the PEP. I can just check in my changes to SVN, so it should show up by Friday. Thanks again, -Travis [View Less]

1 0

PEP 3118: Extended buffer protocol (new version)
by Carl Banks April 19, 2007

April 19, 2007

Travis Oliphant wrote: > Carl Banks wrote: >> Ok, I've thought quite a bit about this, and I have an idea that I >> think will be ok with you, and I'll be able to drop my main >> objection. It's not a big change, either. The key is to explicitly >> say whether the flag allows or requires. But I made a few other >> changes as well. > I'm good with using an identifier to differentiate between an "allowed" > flag and a "require" flag. I'm not a big … [View More]fan of > VERY_LONG_IDENTIFIER_NAMES though. Just enough to understand what it > means but not so much that it takes forever to type and uses up > horizontal real-estate. That's fine with me. I'm not very particular about spellings, as long as they're not misleading. >> Now, here is a key point: for these functions to work (indeed, for >> PyObject_GetBuffer to work at all), you need enough information in >> bufinfo to figure it out. The bufferinfo struct should be >> self-contained; you should not need to know what flags were passed to >> PyObject_GetBuffer in order to know exactly what data you're looking at. > Naturally. > >> Therefore, format must always be supplied by getbuffer. You cannot >> tell if an array is contiguous without the format string. (But see >> below.) > > No, I don't think this is quite true. You don't need to know what > "kind" of data you are looking at if you don't get strides. If you use > the SIMPLE interface, then both consumer and exporter know the object is > looking at "bytes" which always has an itemsize of 1. But doesn't this violate the above maxim? Suppose these are the contents of bufinfo: ndim = 1 len = 20 shape = (10,) strides = (2,) format = NULL How does it know whether it's looking at contiguous array of 10 two-byte objects, or a discontiguous array of 10 one-byte objects, without having at least an item size? Since item size is now in the mix, it's moot, of course. The idea that Py_BUF_SIMPLE implies bytes is news to me. What if you want a contiguous, one-dimensional array of an arbitrary type? I was thinking this would be acceptable with Py_BUF_SIMPLE. It seems you want to require Py_BUF_FORMAT for that, which would suggest to me that Py_BUF_ALLOW_ND amd Py_BUF_ALLOW_NONCONTIGUOUS, etc., would imply Py_BUF_FORMAT? IOW, pretty much anything that's not SIMPLE implies FORMAT? If that's the case, then most of the issues I brought up about item size don't apply. Also, if that's the case, you're right that Py_BUF_FORMAT makes more sense than Py_BUF_DONT_NEED_FORAMT. But it now it seems even more unnecessary than it did before. Wouldn't any consumer that just wants to look at a chunk of bytes always use Py_BUF_FORMAT, especially if there's danger of a presumptuous exporter raising an exception? > I'll update the PEP with my adaptation of your suggestions in a little > while. Ok. Thanks for taking the lead, and for putting up with my verbiose nitpicking. :) Carl Banks [View Less]

1 0

Weekly Python Patch/Bug Summary
by Kurt B. Kaiser April 19, 2007

April 19, 2007

Patch / Bug Summary ___________________ Patches : 357 open ( +8) / 3745 closed ( +8) / 4102 total (+16) Bugs : 958 open (+19) / 6657 closed ( +9) / 7615 total (+28) RFE : 251 open ( +2) / 280 closed ( +2) / 531 total ( +4) New / Reopened Patches ______________________ Help with Python codebase (2007-04-11) CLOSED http://python.org/sf/1698723 opened by Munawar Update to Macintosh library documentation (2007-04-11) http://python.org/sf/1698768 opened by Kevin … [View More]

1 0

PEP 3118: Extended buffer protocol (new version)
by Jim Jewett April 19, 2007

April 19, 2007

Reading this message without the entire PEP in front of me showed some confusing usage. (Details below) Most (but not all) I could resolve from the PEP itself, but they could be clarified with different constant names. Counter Proposal at bottom, and specific questions in between. Travis Oliphant wrote: > Carl Banks wrote: >> Some of the flags RESTRICT the kind of buffers that can be >> exported (Py_BUF_WRITABLE); other flags EXPAND the >> kind of buffers that can be … [View More]exported (Py_BUF_INDIRECT). >> That is highly confusing and I'm -1 >> on any proposal that includes both behaviors. > Basically, every flag corresponds to a different property of > the buffer that the consumer is requesting: I had trouble seeing it like that. Could you at least rename them something like SHAPE_VALID? But I would prefer the change suggested at the bottom. > Py_BUF_SIMPLE --- you are requesting the simplest possible (0x00) ??? Does this automatically assume a readable buffer? (No hardware write buffers?) (This one I couldn't tell from the PEP) > Py_BUF_WRITEABLE -- get a writeable buffer (0x01) > Py_BUF_READONLY -- get a read-only buffer (0x02) Do Py_BUF_READONLY promise that *this* consumer won't try to change it, or request that it be immutable (and no one else will change it either)? (From the PEP itself, I think you wanted IMMUTABLE.) When it comes back, does it mean "you, the consumer, promised not to change this", or "I, the exporter, won't authorize anyone to change it, including myself." If it is really a request for immutability, and the exporter can't make that promise, then should the buffer protocol itself make and return an immutable copy? For this grouping to make sense, I have to assume that you really mean Py_BUF_READABLE (0 - no way to mark it write-only) Py_BUF_WRITABLE (1 - meaning *this* consumer can write) Py_BUF_IMMUTABLE (2 - meaning no one can change it.) so 3 => compile-time error but that still doesn't mesh with your later statement that: > ... most people request read-or-write buffers > i.e. Py_BUF_SIMPLE. > Py_BUF_FORMAT -- get a "formatted" buffer. (0x04) Is this saying "I the consumer will respect your formatting" or "I the consumer will fail if you don't tell me the formatting"? If this flag comes back, does that mean that understanding the formatting is mandatory, or is it just informational? ??? To make this concrete, there are libraries that sniff and guess a format. Should they pass this flag or not? > Py_BUF_SHAPE -- get a buffer with shape information (0x08) Wait -- what is the difference between these two again? Is "format" the internal format of a single element, and shape the dimensions of an array? Should Py_BUF_FORMAT be Py_BUF_ELT_FORMAT? Do you pass Py_BUF_SHAPE to indicate that you'll accept N-Dim arrays, or to say that you prefer them or somehow need them? ??? If Py_BUF_SHAPE was requested, but the buffer really is 1-dimensional, should this still be set on the way back? (Presumably setting the "shape" variable to point to a (one-element) array containing len/itemsize? > Py_BUF_STRIDES -- get a buffer with stride information (and shape) (0x18) > Py_BUF_OFFSET -- get a buffer with suboffsets (and strides and shape) (0x38) I assume that Py_BUF_OFFSET doesn't make sense without Py_BUF_STRIDES, and Py_BUF_STRIDES doesn't make sense without Py_BUF_SHAPE. What happens if someone *does* pass 0x20? If you want to avoid that, it might make sense to treat this as 4 enumerated values (1-D, n-D, n-D with strides, n-D with strides and offsets) instead of 3 flags. If Py_BUF_OFFSET was requested, but the return value is a single continuous 1D array, should Py_BUF_OFFSET still come back but have the zeros filled in? Also, why are all negative offsets invalid? It seems like they might be useful for some re-orderings, and using an offset of 0 has the same effect as marking the offset unused. > For me, the most restrictive requests are > PY_BUF_WRITEABLE | Py_BUF_FORMAT and > Py_BUF_READONLY | Py_BUF_FORMAT > The most un-restrictive request (the largest circle in my mental Venn > diagram) is > Py_BUF_OFFSETS followed by Py_BUF_STRIDES followed by Py_BUF_SHAPE I think this is what he meant when he said that the flags had opposite meanings. Ideally, 0 and 0xFF should be shortcuts to either the most restrictive or the least restrictive. > Think of them as turning "on" members of the bufferinfo structure. If the senses can't be reversed, could you at least rename them to indicate this? Something like Py_BUF_VALID_SUBOFFSETS? ================================ Counter Proposal: 0x00: Py_BUF_RWLOCK: "Consumer is a new owner. It can read and write; no other code can." (most restrictive) 0x01: Py_BUF_READONLY: "Consumer doesn't need to (cannot) write." 0x02: Py_BUF_MUTABLE: "Other code can write" (implies) 0x03 = PY_BUF_UR: Py_BUF_READONLY | Py_BUF_MUTABLE : "Unconfirmed Read: Consumer won't write, but other code might." (skipping 0x04 just to get the shapes all in the second digit) 0x08: Py_BUF_ELT_FORMAT: "If you send the element format, I'll look at it." ??? What does this mean in a response? Just that the *format isn't NULL? # Note that these are four shapes alternatives, # rather than two orthogonal flags 0x0n: Py_BUF_1D: "No shape information." (in response, means none required, -- should the shape pointer still be set?) 0x1n: Py_BUF_SHAPE: "continous n-D arrays acceptable (to consumer) or used (by exporter)" 0x2n: Py_BUF_STRIDE: " ? How to best describe strides? Srides acceptable (to consumer) or used (by exporter)" 0x3n: Py_BUF_OFFSET: " ? How to best describe offsets? offests acceptable (to consumer) or used (by exporter)" -jJ [View Less]

2 1

SystemErrors in generator (just happened in 2.5)
by Neal Norwitz April 19, 2007

April 19, 2007

There were some SystemErrors on one of the Windows build slaves. Does anyone have any ideas what might be the cause? I looked through about 5 previous logs on the same slave and didn't see the same problem. I haven't seen this myself and I don't know if it's WIndows specific. I don't know if this has happened on the trunk or just 2.5. For the full log see: http://python.org/dev/buildbot/2.5/x86%20XP%202.5/builds/197/step-test/0 n -- *** Previous context (I don't think this really … [View More]matters): Re-running test 'test_profilehooks' in verbose mode test_caught_exception (test.test_profilehooks.ProfileHookTestCase) ... ok test_caught_nested_exception (test.test_profilehooks.ProfileHookTestCase) ... ok test_distant_exception (test.test_profilehooks.ProfileHookTestCase) ... ok test_exception (test.test_profilehooks.ProfileHookTestCase) ... ok test_exception_in_except_clause (test.test_profilehooks.ProfileHookTestCase) ... ok test_exception_propogation (test.test_profilehooks.ProfileHookTestCase) ... ok test_generator (test.test_profilehooks.ProfileHookTestCase) ... ok test_nested_exception (test.test_profilehooks.ProfileHookTestCase) ... ok test_raise (test.test_profilehooks.ProfileHookTestCase) ... ok test_raise_reraise (test.test_profilehooks.ProfileHookTestCase) ... ok test_raise_twice (test.test_profilehooks.ProfileHookTestCase) ... ok test_simple (test.test_profilehooks.ProfileHookTestCase) ... ok test_stop_iteration (test.test_profilehooks.ProfileHookTestCase) ... ok test_basic_exception (test.test_profilehooks.ProfileSimulatorTestCase) ... ok test_caught_exception (test.test_profilehooks.ProfileSimulatorTestCase) ... ok test_distant_exception (test.test_profilehooks.ProfileSimulatorTestCase) ... ok test_simple (test.test_profilehooks.ProfileSimulatorTestCase) ... ok ---------------------------------------------------------------------- Ran 17 tests in 0.016s OK *** Here is the problem: Exception exceptions.SystemError: 'error return without exception set' in <generator object at 0x00CCCDF8> ignored Exception exceptions.SystemError: 'error return without exception set' in <generator object at 0x010891F8> ignored Exception exceptions.SystemError: 'error return without exception set' in <generator object at 0x01089578> ignored Exception exceptions.SystemError: 'error return without exception set' in <generator object at 0x01089838> ignored Exception exceptions.SystemError: 'error return without exception set' in <generator object at 0x00D3BFB8> ignored *** The following messages occur in other successful tests too: a DOS box should flash briefly ... find_library('c') -> None find_library('m') -> None C:\buildbot_py25\2.5.mcintyre-windows\build\lib\test\test_unicode_file.py:103: UnicodeWarning: Unicode equal comparison failed to convert both arguments to Unicode - interpreting them as being unequal filename1==filename2 C:\buildbot_py25\2.5.mcintyre-windows\build\lib\shutil.py:36: UnicodeWarning: Unicode equal comparison failed to convert both arguments to Unicode - interpreting them as being unequal os.path.normcase(os.path.abspath(dst))) warning: DBTxn aborted in destructor. No prior commit() or abort(). [View Less]

3 2

Proposal: Allow any mapping after ** in calls
by Alexander Belopolsky April 19, 2007

April 19, 2007

Python allows arbitrary sequences after * in calls, but an expression following ** must be a (subclass of) dict. I believe * and ** should be treated similarly and since f(*UserList(..)) is valid, f(**UserDict(..)) should be valid as well. Of course, I can work around this limitation by writing f(**dict(UserDict(..))), but this is unnatural given that I don't have to do f(*tuple(UserList(..))). I have submitted a patch on SF that makes Python accept arbitrary mappings after **. See <https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1686487&group_…>.

3 2

minidom -> new-style classes?
by Jason Orendorff April 18, 2007

April 18, 2007

I'm working on minidom's DOM Level 1 compliance, targeting Python 2.6. We have some bugs involving DOM property behavior. For example, setting the nodeValue attribute of an Element is supposed to have no effect. We don't implement this. The right way to implement these quirks is using new-style classes and properties. Right now minidom uses old-style classes and lots of hackery, and it's pretty broken. (Another example--there is an Attr._set_prefix method, but it is *not* called from … [View More]

4 5

distutils mixes 32/64bit binaries
by Alexander Belopolsky April 18, 2007

April 18, 2007

On the platforms that can run both 32 and 64bit python, such as x86_64, distutils builds both 32 and 64 bit libraries in the same directory such as build/lib.linux-x86_64-2.5. This can be easily fixed by placing 64 bit libraries in build/lib64.linux-x86_64-2.5 instead. On the other hand it may be time to revisit use of os.uname in distutils' get_platform to determine the target directory. In the environments that can run python interpreters compiled for different architectures, distutils' … [View More]

1 0

test_normalization failures across community buildbots
by Grig Gheorghiu April 18, 2007

April 18, 2007

Almost all community buildbots have failed the test step due to a failure in test_normalization. Here's a link to the community farm for the trunk: http://www.python.org/dev/buildbot/community/trunk/ And here's an example of a failure: http://www.python.org/dev/buildbot/community/trunk/x86%20OSX%20trunk/builds… test test_normalization failed -- Traceback (most recent call last): File "/Users/builder/pybots/pybot/trunk.osaf-x86/build/Lib/test/test_normalization.py", line 36, in test_main … [View More]

4 6