Mailman 3 November 2000 - Python-Dev

[development doc updates]
by Fred L. Drake Nov. 29, 2000

Nov. 29, 2000

The development version of the documentation has been updated: http://python.sourceforge.net/devel-docs/

1 0

Python embedded in Exim
by Jeffrey C. Ollie Nov. 28, 2000

Nov. 28, 2000

Well, I had an epiphany over the US Thanksgiving holiday weekend (or maybe it was a brain aneurism) so I've spent some late nights and some time stolen from work and come up with a first pass at embedding Python in Exim. I imagine that embedding Python in Exim will be interesting to those folks writing virus scanners or for VERY tight integration of Mailman with Exim. You can also do all of those things that you can do with the Perl string expansions. Note: this code is VERY lightly tested. It compiles on my home hacked-up RedHat Linux system and works on a couple of simple tests that I've done, but no real e-mails have passed through this code yet. I'm tossing this out for public comment on the design and implementation... It's been a while since I've done a lot of C coding. Unfortunately, you must embed Python 2.0. This is because previous versions of Python used a hacked copy of PCRE as it's regular expression engine that conflicts with the copy of PCRE in Exim. Starting in Python 2.0 the default regular expression is a completely new one called SRE that supports Unicode. The PCRE-based regular expression engine is still included in Python 2.0 but I get around that by creating a private copy of the Python library and delete the offending object modules. You could do that in versions of Python prior to 2.0 but a lot of the standard library depends on regular expressions I didn't think that there was much point in trying to make the embedding work with 1.5.2 or 1.6. Well, on to the patch itself. The patch is made against Exim v3.20. After you've patched the source code, you need to set four variables in the Local/Makefile: EXIM_PYTHON=python.o PYTHON_INC=-I/usr/local/include/python2.0 PYTHON_LIB=/usr/local/lib/python2.0/config/libpython2.0.a PYTHON_EXTRA_LIBS=-lpthread -ldl -lutil Then build Exim as usual. There are three runtime directives that control the embedding, all of which are optional: python_at_start boolean - force startup of Python interpreter python_module_paths colon separated list of paths to append to sys.path python_initial_import colon separated list of modules to import at interpreter initialization time There are also two new command line switches -ys and -yd that will force an immediate startup of Python or delay startup, overriding python_at_start. Then you use it: ${python{<module>.<function>}{<arg1>}...} You must specify the module name and the function name. The named module will be imported automatically. And currently you must specify a function that returns a string (I'll look into adding accessing attributes, methods of classes, and converting non-strings into strings). There can be up to eight arguments. Each argument will be expanded by Exim before it's passed to the Python interpreter. Your python code has access to a module called "exim", which currently defines two things: exim.expand_string(s) which calls back to Exim to expand a string and exim.error which is an exception object used by expand_string One simple example: ${python{string.upper}{$local_part}} Error reporting isn't the best right now, I'm going to look into formatting exceptions and tracebacks for more meaningful error messages. Well, I think that's the gist of it. I'll work on patches to the documentation when this thing becomes more stable. As I said before I've only lightly tested this... beware! Comments, criticisms, suggestions, flames welcome... Jeff

1 0

Re: [Python-Dev] Re: [Bug #121013] Bug in <stringobject>.join(<unicodestring>)
by Michael Hudson Nov. 28, 2000

Nov. 28, 2000

"M.-A. Lemburg" <mal(a)lemburg.com> writes: > > Date: 2000-Nov-27 10:12 > > By: mwh > > > > Comment: > > I hope you're all suitably embarrassed - please see patch #102548 for the trivial fix... > > Hehe, that was indeed a trivial patch. What was that about trees > in a forest... The way I found it was perhaps instructive. I was looking at the function, and thought "that's a bit complicated" so I rewrote it (My rewrite also seems to be bit quicker so I'll upload it as soon as make test has finished[*]). In the course of rewriting it, I saw the line my patch touched and went "duh!". > I still think that the PySequence_Fast_GETITEM() macro should at > least include a fall-back type check which causes some exception in > case the used object was not "fixed" using PySequence_Fast() (which > I only noticed in string_join() just now). It's hard to see how; you're not going to check each invocation of PySequence_Fast_GETITEM for a NULL return, are you? It's possible that PySequence_Fast should raise an exception on being passed a string or Unicode object... but probably not. > Fredrik's PySequence_Fast_* APIs look interesting, BTW. Should be > used more often :-) Yes. But they're pretty new, aren't they? I find them a bit unsatisfactory that it's not possible to hoist the type check out of the inner loop. Still, it seems my PII's branch predictor nails that one... (i.e. changing it so that it didn't check inside the loop made naff-all difference to the running time). Cheers, M. [*] which reminds me: test_unicodedata is failing for me at the moment. Anyone else seeing this? It looks like a python regrtest.py -g is all that's needed... -- 7. It is easier to write an incorrect program than understand a correct one. -- Alan Perlis, http://www.cs.yale.edu/homes/perlis-alan/quotes.html

5 7

Re: [Python-Dev] Re: [Bug #121013] Bug in <stringobject>.join(<unicodestring>)
by Michael Hudson Nov. 28, 2000

Nov. 28, 2000

"M.-A. Lemburg" <mal(a)lemburg.com> writes: > Michael Hudson wrote: > > > > "M.-A. Lemburg" <mal(a)lemburg.com> writes: > > > > > > Date: 2000-Nov-27 10:12 > > > > By: mwh > > > > > > > > Comment: > > > > I hope you're all suitably embarrassed - please see patch #102548 for the trivial fix... > > > > > > Hehe, that was indeed a trivial patch. What was that about trees > > > in a forest... > > > > The way I found it was perhaps instructive. I was looking at the > > function, and thought "that's a bit complicated" so I rewrote it (My > > rewrite also seems to be bit quicker so I'll upload it as soon as make > > test has finished[*]). In the course of rewriting it, I saw the line > > my patch touched and went "duh!". > > Yeah. The bug must have sneaked in there when the function was > updated to the PySequence_Fast_* implementation. > > BTW, could you also add a patch for the test_string.py and > test_unicode.py tests ? Here's an effort, but it's a memory scribbling bug. If I change test_unicode.py thus: Index: test_unicode.py =================================================================== RCS file: /cvsroot/python/python/dist/src/Lib/test/test_unicode.py,v retrieving revision 1.22 diff -c -r1.22 test_unicode.py *** test_unicode.py 2000/10/23 17:22:08 1.22 --- test_unicode.py 2000/11/28 08:49:11 *************** *** 70,76 **** # join now works with any sequence type class Sequence: ! def __init__(self): self.seq = 'wxyz' def __len__(self): return len(self.seq) def __getitem__(self, i): return self.seq[i] --- 70,76 ---- # join now works with any sequence type class Sequence: ! def __init__(self): self.seq = [u'w',u'x',u'y',u'z'] def __len__(self): return len(self.seq) def __getitem__(self, i): return self.seq[i] *************** *** 78,83 **** --- 78,87 ---- test('join', u'', u'abcd', (u'a', u'b', u'c', u'd')) test('join', u' ', u'w x y z', Sequence()) test('join', u' ', TypeError, 7) + test('join', ' ', u'a b c d', [u'a', u'b', u'c', u'd']) + test('join', '', u'abcd', (u'a', u'b', u'c', u'd')) + test('join', ' ', u'w x y z', Sequence()) + test('join', ' ', TypeError, 7) class BadSeq(Sequence): def __init__(self): self.seq = [7, u'hello', 123L] and back out the fix for the join bug, this happens: ... ... Testing Unicode formatting strings... done. Testing builtin codecs... Traceback (most recent call last): File "test_unicode.py", line 378, in ? assert unicode('hello','utf8') == u'hello' File "/usr/local/src/python/dist/build/Lib/encodings/__init__.py", line 30, in ? import codecs,aliases SystemError: compile.c:185: bad argument to internal function Segmentation fault i.e. it crashes miles away from the problem. I'll reply to the other stuff later - no time now. Cheers, M. -- The only problem with Microsoft is they just have no taste. -- Steve Jobs, (From _Triumph of the Nerds_ PBS special) and quoted by Aahz Maruch on comp.lang.python

2 3

Re: [Python-Dev] Re: [Bug #121013] Bug in <stringobject>.join(<unicodestring>)
by Michael Hudson Nov. 28, 2000

Nov. 28, 2000

Michael Hudson <mwh21(a)cam.ac.uk> writes: > "M.-A. Lemburg" <mal(a)lemburg.com> writes: > > > > Date: 2000-Nov-27 10:12 > > > By: mwh > > > > > > Comment: > > > I hope you're all suitably embarrassed - please see patch #102548 for the trivial fix... > > > > Hehe, that was indeed a trivial patch. What was that about trees > > in a forest... > > The way I found it was perhaps instructive. I was looking at the > function, and thought "that's a bit complicated" so I rewrote it (My > rewrite also seems to be bit quicker so I'll upload it as soon as make > test has finished[*]). It's at https://sourceforge.net/patch/index.php?func=detailpatch&patch_id=102549&gr… I want a way of submitting patches by email. Is roundup in a usable state yet? Cheers, M. -- "The future" has arrived but they forgot to update the docs. -- R. David Murray, 9 May 2000

4 5

Re: [Python-Dev] Re: [Bug #121013] Bug in <stringobject>.join(<unicodestring>)
by Fredrik Lundh Nov. 27, 2000

Nov. 27, 2000

> your patch changes the behaviour of join on non-standard sequences. > > if you use join on a long synthetic sequence, your new version may run > out of memory where the old one worked just fine. looking again, this was obviously only true for the unicode version (which uses PySequence_GetItem), not for the 8-bit string version (which uses the PySequence_Fast API). maybe we should change the 8-bit implementation instead? if not, +1. </F>

1 0

Re: [Python-Dev] PEP-0218
by Moshe Zadka Nov. 27, 2000

Nov. 27, 2000

> > Well, another proof of the bugs in the PEP process -- my remarks were lost, > > so I'll send them here. > > Where did you send your comments? To you personally? -- Moshe Zadka <sig(a)zadka.site.co.il> This is a signature anti-virus. Please stop the spread of signature viruses!

3 2

A house upon the sand
by Mark Lutz Nov. 27, 2000

Nov. 27, 2000

Hi again Guido, On a totally unrelated topic: has anyone pushed the idea of a Python language standard lately? Things seem to be changing too fast for many developers to keep up, and it seems to me that a formal standard doc might help ease a few fears I've seen out there. Books used to be a sort of de facto standard, but even they aren't reliable anymore; and the manuals aren't useful as a standard if they are open to arbitrary change every few months. Frankly, some people in my classes are very concerned by the rapid pace of Python change, and I think their fear is justified. I get burned a little almost every time a new Python release rolls out too. Most recently, some new book examples that worked in 1.5.2 this summer no longer work under 2.0 this fall; I understand that most changes are improvements (and minor), but this is not a great story to tell. A prime example: the string module, used in almost every Python program ever written by the half-million Python users out there, has suddenly been marked as deprecated. I expect that it won't really go away, but has anyone considered the impact of even the suggestion of its deprecation on Python's acceptance? If using Python requires that programmers invest lots of time tracking the whims of python-dev, then Python will become much less useful, imo. Most developers just don't have the extra time to spare. A formal standard doc could give us at least a baseline Python definition that developers could cling to. Companies need to see a solid and reliable foundation. Unfortunately, I don't have the time or interest in pushing this idea through to fruition myself. Do you have any ideas along these lines? Maybe this task belongs in whatever body eventually takes over ownership. I'm copying this to python-dev in the hopes that it might trigger some sort of discussion. Cheers, --Mark Lutz (http://rmi.net/~lutz)

7 8

Re: [Bug #121013] Bug in <stringobject>.join(<unicodestring>)
by M.-A. Lemburg Nov. 27, 2000

Nov. 27, 2000

> Date: 2000-Nov-27 10:12 > By: mwh > > Comment: > I hope you're all suitably embarrassed - please see patch #102548 for the trivial fix... Hehe, that was indeed a trivial patch. What was that about trees in a forest... I still think that the PySequence_Fast_GETITEM() macro should at least include a fall-back type check which causes some exception in case the used object was not "fixed" using PySequence_Fast() (which I only noticed in string_join() just now). Fredrik's PySequence_Fast_* APIs look interesting, BTW. Should be used more often :-) -- Marc-Andre Lemburg ______________________________________________________________________ Company: http://www.egenix.com/ Consulting: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/

1 0

A house upon the sand
by Martin v. Loewis Nov. 26, 2000

Nov. 26, 2000

> the-lang-ref-is-the-only-std-you're-likely-to-get-ly y'rs - tim Since Java has been cited as an example of a language that has a "language definition" - it is just a language reference book, either. Sun did a number of attempts to make Java an Internation Standard, and they withdrew when they finally understood what it means. For Python, it would most likely mean that the only implementations don't conform to the standard. Regards, Martin

1 0