Mailman 3 August 2002 - Python-Dev

PEP 298, final (?) version
by Thomas Heller Aug. 4, 2002

Aug. 4, 2002

Here is PEP 298 in it's near final version (not yet checked in). It seems to me we have to end the discussion and I'm quite happy with it. If accepted in this form, I can start the implementation right after the end of my vacation, second half of August. The only thing I consider worth changing is to rename the whole stuff from 'fixed buffer interface' to 'locked buffer interface', which makes more sense at the current state. Thomas ----- PEP: 298 Title: The Fixed Buffer Interface Version: $Revision: 1.4 $ Last-Modified: $Date: 2002/07/31 18:48:36 $ Author: Thomas Heller <theller(a)python.net> Status: Draft Type: Standards Track Created: 26-Jul-2002 Python-Version: 2.3 Post-History: 30-Jul-2002, 1-Aug-2002 Abstract This PEP proposes an extension to the buffer interface called the 'fixed buffer interface'. The fixed buffer interface fixes the flaws of the 'old' buffer interface [1] as defined in Python versions up to and including 2.2, and has the following semantics: The lifetime of the retrieved pointer is clearly defined and controlled by the client. The buffer size is returned as a 'size_t' data type, which allows access to large buffers on platforms where sizeof(int) != sizeof(void *). (Guido comments: This second sounds like a change we could also make to the "old" buffer interface, if we introduce another flag bit that's *not* part of the default flags.) Specification The fixed buffer interface exposes new functions which return the size and the pointer to the internal memory block of any python object which chooses to implement this interface. Retrieving a buffer from an object puts this object in a locked state during which the buffer may not be freed, resized, or reallocated. The object must be unlocked again by releasing the buffer if it's no longer used by calling another function in the fixed buffer interface. If the object never resizes or reallocates the buffer during it's lifetime, this function may be NULL. Failure to call this function (if it is != NULL) is a programming error and may have unexpected results. The fixed buffer interface omits the memory segment model which is present in the old buffer interface - only a single memory block can be exposed. Implementation Define a new flag in Include/object.h: /* PyBufferProcs contains bf_acquirefixedreadbuffer, bf_acquirefixedwritebuffer, and bf_releasefixedbuffer */ #define Py_TPFLAGS_HAVE_FIXEDBUFFER (1L<<15) This flag would be included in Py_TPFLAGS_DEFAULT: #define Py_TPFLAGS_DEFAULT ( \ .... Py_TPFLAGS_HAVE_FIXEDBUFFER | \ .... 0) Extend the PyBufferProcs structure by new fields in Include/object.h: typedef size_t (*acquirefixedreadbufferproc)(PyObject *, const void **); typedef size_t (*acquirefixedwritebufferproc)(PyObject *, void **); typedef void (*releasefixedbufferproc)(PyObject *); typedef struct { getreadbufferproc bf_getreadbuffer; getwritebufferproc bf_getwritebuffer; getsegcountproc bf_getsegcount; getcharbufferproc bf_getcharbuffer; /* fixed buffer interface functions */ acquirefixedreadbufferproc bf_acquirefixedreadbuffer; acquirefixedwritebufferproc bf_acquirefixedwritebuffer; releasefixedbufferproc bf_releasefixedbuffer; } PyBufferProcs; The new fields are present if the Py_TPFLAGS_HAVE_FIXEDBUFFER flag is set in the object's type. The Py_TPFLAGS_HAVE_FIXEDBUFFER flag implies the Py_TPFLAGS_HAVE_GETCHARBUFFER flag. The acquirefixedreadbufferproc and acquirefixedwritebufferproc functions return the size in bytes of the memory block on success, and fill in the passed void * pointer on success. If these functions fail - either because an error occurs or no memory block is exposed - they must set the void * pointer to NULL and raise an exception. The return value is undefined in these cases and should not be used. If calls to these functions succeed, eventually the buffer must be released by a call to the releasefixedbufferproc, supplying the original object as argument. The releasefixedbufferproc cannot fail. Usually these functions aren't called directly, they are called through convenience functions declared in Include/abstract.h: int PyObject_AquireFixedReadBuffer(PyObject *obj, const void **buffer, size_t *buffer_len); int PyObject_AcquireFixedWriteBuffer(PyObject *obj, void **buffer, size_t *buffer_len); void PyObject_ReleaseFixedBuffer(PyObject *obj); The former two functions return 0 on success, set buffer to the memory location and buffer_len to the length of the memory block in bytes. On failure, or if the fixed buffer interface is not implemented by obj, they return -1 and set an exception. The latter function doesn't return anything, and cannot fail. Backward Compatibility The size of the PyBufferProcs structure changes if this proposal is implemented, but the type's tp_flags slot can be used to determine if the additional fields are present. Reference Implementation Will be uploaded to the SourceForge patch manager by the author. Additional Notes/Comments Python strings, unicode strings, mmap objects, and array objects would expose the fixed buffer interface. mmap and array objects would actually enter a locked state while the buffer is active, this is not needed for strings and unicode objects. Resizing locked array objects is not allowed and will raise an exception. Whether closing a locked mmap object is an error or will only be deferred until the lock count reaches zero is an implementation detail. Community Feedback Greg Ewing doubts the fixed buffer interface is needed at all, he thinks the normal buffer interface could be used if the pointer is (re)fetched each time it's used. This seems to be dangerous, because even innocent looking calls to the Python API like Py_DECREF() may trigger execution of arbitrary Python code. The first version of this proposal didn't have the release function, but it turned out that this would have been too restrictive: mmap and array objects wouldn't have been able to implement it, because mmap objects can be closed anytime if not locked, and array objects could resize or reallocate the buffer. Credits Scott Gilbert came up with the name 'fixed buffer interface'. References [1] The buffer interface http://mail.python.org/pipermail/python-dev/2000-October/009974.html [2] The Buffer Problem http://www.python.org/peps/pep-0296.html Copyright This document has been placed in the public domain. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 End:

6 15

Adding popen2 like functionality to pty.py
by Rasjid Wilcox Aug. 3, 2002

Aug. 3, 2002

Dear Python Developers, I have submited a patch to add a popen2 like function to the pty.py library. It is just a first draft, and I'm happy to develop it further if there is interest. If so, I will do some docs and have a look at the test library for it. I would also be looking for some guidance on the best way to resolve some issues. I'm new to Python and its development process, so I'm hoping I have not broken any rules by not waiting a response via the patch manager before posting to python-dev. I would like to contribute to Python, as I think it is a truly delightful language. I don't have a computer science background as such, more pure mathematics (set theory, group theory, logic etc). I don't know C (yet), so I would be looking to work on the pure Python libraries, or help create new ones. I'm also willing to help with documentation. Cheers, Rasjid.

1 0

dbm module, whichdb, test_whichdb
by Skip Montanaro Aug. 3, 2002

Aug. 3, 2002

Folks, I just checked in a modified dbmmodule.c, whichdb.py and a new regression test file, test_whichdb.py. The change to dbmmodule.c accommodates linkage with Berkeley DB. The change to whichdb catches this case (opening "foo" actually creates "foo.db"). The new test_whichdb.py file simply adds regression tests for the whole mess. Please have a look, try it out, and let me know if it gives your system heartburn. Having messed around with this stuff off-and-on for awhile, I have no illusions about this going in without tickling some platform dependency. Jack, you're especially on alert, because I know you had problems with some earlier bsddb-related changes to setup.py. My iMac w/ MacOS X is still in Michigan with Ellen for the summer. Skip

2 2

Re: [Python-checkins] python/dist/src/Lib heapq.py,NONE,1.1
by David Goodger Aug. 3, 2002

Aug. 3, 2002

Guido wrote: > I was originally going to make it an unnamed string > literal -- maybe that's better? In PEP 258 I call those "Additional Docstrings": Many programmers would like to make extensive use of docstrings for API documentation. However, docstrings do take up space in the running program, so some of these programmers are reluctant to "bloat up" their code. Also, not all API documentation is applicable to interactive environments, where __doc__ would be displayed. The docstring processing system's extraction tools will concatenate all string literal expressions which appear at the beginning of a definition or after a simple assignment. Only the first strings in definitions will be available as __doc__, and can be used for brief usage text suitable for interactive sessions; subsequent string literals and all attribute docstrings are ignored by the Python bytecode compiler and may contain more extensive API information. Example:: def function(arg): """This is __doc__, function's docstring.""" """ This is an additional docstring, ignored by the bytecode compiler, but extracted by the Docutils. """ pass (Original idea from Moshe Zadka.) -- David Goodger <goodger(a)users.sourceforge.net> Open-source projects: - Python Docutils: http://docutils.sourceforge.net/ (includes reStructuredText: http://docutils.sf.net/rst.html) - The Go Tools Project: http://gotools.sourceforge.net/

2 1

Re: [Python-checkins] python/dist/src/Lib heapq.py,NONE,1.1
by Neil Schemenauer Aug. 2, 2002

Aug. 2, 2002

gvanrossum(a)users.sourceforge.net wrote: > Adding the heap queue algorithm, per discussion in python-dev last > week. Cool. > __about__ = """Heap queues [...] Is this going to become a "blessed" special name or do you consider it harmless abuse of the namespace? Neil

2 1

Weird error handling in os._execvpe
by Zack Weinberg Aug. 2, 2002

Aug. 2, 2002

While testing my tempfile.py rewrite I ran into this mess in os.py: def _execvpe(file, args, env=None): # ... if not _notfound: if sys.platform[:4] == 'beos': # Process handling (fork, wait) under BeOS (up to 5.0) # doesn't interoperate reliably with the thread interlocking # that happens during an import. The actual error we need # is the same on BeOS for posix.open() et al., ENOENT. try: unlink('/_#.# ## #.#') except error, _notfound: pass else: import tempfile t = tempfile.mktemp() # Exec a file that is guaranteed not to exist try: execv(t, ('blah',)) except error, _notfound: pass exc, arg = error, _notfound for dir in PATH: fullname = path.join(dir, file) try: apply(func, (fullname,) + argrest) except error, (errno, msg): if errno != arg[0]: exc, arg = error, (errno, msg) raise exc, arg This appears to be an overcomplicated, unreliable way of writing import errno def _execvpe(file, args, env=None): # ... for dir in PATH: fullname = path.join(dir, file) try: apply(func, (fullname,) + argrest) except error, (err, msg): if err != errno.ENOENT: # and err != errno.ENOTDIR, maybe raise raise error, (err, msg) Can anyone explain why it is done this way? zw

3 9

Re: [Python-Dev] seeing off SET_LINENO
by Michael Hudson Aug. 2, 2002

Aug. 2, 2002

Guido van Rossum <guido(a)python.org> writes: > > I've found another annoying problem. I'm not really expecting someone > > here to sovle it for me, but writing it down might help me think > > clearly. > > > > This is about the function epilogues that always get generated. I.e: > [...] > > The debugger stopping on the "print 1" is confusing. > > > > There's an "obvious" solution to this: check it we're less than 4 > > bytes from the end of the code string and don't do anything if we are. > > Um, I think that's less than reliable. I believe we just discussed > this when Oren's patch for yield in try/finally did a similar thing > (and weren't you the one who mentioned that your bytecodehacks can > cause this assumption to fail? :-). Good point. > I'm not actually sure that this needs fixing. Surely the --Return-- > should be a sufficient hint. I note that without your patch it also > stops at a confusing place, albeit a different one (on the "if a:" > line). The problem is that when we jump into the epilogue, a 'line' trace event gets generated before the 'return' one. So there is no --Return-- hint. > > This would be easy, except that for some bonkers reason, we support > > arbitrary buffer objects for code strings! (see _PyCode_GETCODEPTR in > > Include/compile.h -- though at least you can't create a code object > > with an array code string from python, the getreadbuffer failing will > > cause the interpreter to unceremoniously crash and burn). > > That went a little too fast. Can you explain that parenthetical > remark more clearly? 1) Don't you find the idea of type(co.co_code) == types.ArrayType at least a little scary? Mainly due to resizes -- having mutable code might be nice for development environments and such. 2) I thought it was possible for bf_getreadbuffer to fail (maybe I'm wrong here). _PyCode_GETCODEPTR does no error checking. > > I guess I can store the length somewhere -- _PyCode_GETCODEPTR returns > > this, more by accident than design I suspect -- or call > > bf_getsegcount(frame->f_code->co_code, &length) or something. > > > > Does anyone actually *use* this feature? I see Guido checked it in > > and the patch was written by Greg Stein. Anyone remember motivations > > from the time? > > Yes, Greg insisted that he might want to store Python bytecode in > Flash ROM, and that this way the bytecode would not have to be copied > to RAM. I see. > But I don't think this ever happened Gosh. > (well, maybe the now-dead Pippy port to PalmOS used it???). Maybe. Somehow doubt it, though. > I'd be happy to kill it as a YAGNI. That's nice, but if... > But that still doesn't mean I approve checking for "4 bytes from the > end". ...it doesn't actually help. Does anyone have any better ideas for not generating 'line' trace events in the epilogue? Cheers, M. -- I also feel it essential to note, [...], that Description Logics, non-Monotonic Logics, Default Logics and Circumscription Logics can all collectively go suck a cow. Thank you. -- http://advogato.org/person/Johnath/diary.html?start=4

3 2

Re: [Python-Dev] seeing off SET_LINENO
by Michael Hudson Aug. 2, 2002

Aug. 2, 2002

Guido van Rossum <guido(a)python.org> writes: > > pystone is a mystery. It's a fair bit slower but also much more > > variable with my patch. Moving trace code out of line helps quite a > > bit but it's still ~1% slower. > > Hm. For me (with your latest patch which moves the trace code out of > line) pystone is actually *less* variable with your patch than > without, and it's also faster with -O than before. > > So I wouldn't lose any sleep over pystone (leave that to Tim :-). I wasn't going to. > Maybe you should increase LOOPS in pystone.py; I usually set it to 40K > or even 100K. Did that. I thought measuring 0.8 secs was a bit on the thin side. [...] > > Also, it seems Lib/compiler currently works by generating SET_LINENO > > and then builds co_lnotab by scanning for them afterwards. That's not > > going to work in the new world, so I should probably think about how > > to change it... > > Or wait for Jeremy. Yes. > (I suppose you still *support* the SET_LINENO opcode?) No. Do you think I should? > BTW, you should change the .pyc magic number in your patch. Really? It's already changed since the last released Python. Easy enough to change again, though, and it makes testing easier. Cheers, M. -- You have run into the classic Dmachine problem: your machine has become occupied by a malevolent spirit. Replacing hardware or software will not fix this - you need an exorcist. -- Tim Bradshaw, comp.lang.lisp

2 2

Re: [Python-Dev] seeing off SET_LINENO
by Michael Hudson Aug. 2, 2002

Aug. 2, 2002

Guido van Rossum <guido(a)python.org> writes: > > Here goes. Everything is relative to 221-base, which is 2.2.1 from Sean's > > RPM. This is the slowest, so all percentages are negative, and more > > negative is better. I hope the names are obvious. > > > > 221-base +0.00% (obviously) > > 221-O-base: -9.69% > > CVS-base: -15.43% > > CVS-O-base: -23.56% > > CVS-hacked: -23.66% > > CVS-O-hacked: -23.70% > > > > (Nearly 25% speed up since 221? Boggle. Some of this may be compilation > > options, I guess) > > No, pymalloc sped us up quite a bit. Yes, this occurred to me after I posted. pystone is a mystery. It's a fair bit slower but also much more variable with my patch. Moving trace code out of line helps quite a bit but it's still ~1% slower. > > Anyway, it seems I haven't slowed -O down. At some point I might try > > moving the trace code out of line and see if that has any effect. Not > > today. Did do this yesterday, in fact. As I said, it helped pystone a bit, so I'll upload a separate patch to sf. [...] > What's the next step? I haven't had time to review your code. Do you > want to check it in without further review, or do you want to wait > until someone can give it a serious look? (Tim's on vacation this > week so it might be a while.) I think I'd like to wait for serious review. I'd be surprised if the patch went out of date at all quickly. Also, it seems Lib/compiler currently works by generating SET_LINENO and then builds co_lnotab by scanning for them afterwards. That's not going to work in the new world, so I should probably think about how to change it... Cheers, M. -- Finding a needle in a haystack is a lot easier if you burn down the haystack and scan the ashes with a metal detector. -- the Silicon Valley Tarot (another one nicked from David Rush)

2 1

PEP 298, __buffer__
by Scott Gilbert Aug. 2, 2002

Aug. 2, 2002

Tonight, I remember another thought that I've had for a while. There isn't currently a way for a class object created from Python script to indicate that it wishes to implement the buffer interface. In the Numeric source, I've seen them use self.__buffer__ for this purpose, but this isn't actually an officially sanctioned magic name. Now that classes can derive from builtin types, perhaps there is less of a need for this, but I still think we would want it. There are times when you want inheritance, and others when you want containment. With a slight modification to the PyObject_*Buffer functions (in the failure branches), an instance of a class could use containment of a PyBufferProcs supporting object and publish the buffer interface as its own. I'm thinking one of: class OneWay(object): def __init__(self): self.__buffer__ = bytes(1000) Or: class SomeOther(object): def __init__(self): self._private = bytes(1000) def __buffer__(self): return self._private I believe the first one is the way it's done in Numeric (Numarray too?). (Maybe Todd Miller will comment on this and whether it's useful to him.) If this is worthwhile, it could be added to PEP 298 or as a new mini PEP. In either case, I'm willing to do the work. Cheers, -Scott __________________________________________________ Do You Yahoo!? Yahoo! Health - Feel better, live better http://health.yahoo.com

3 2