Mailman 3 July 2008 - Python-Dev

patch review request: float.hex and float.fromhex
by Mark Dickinson July 11, 2008

July 11, 2008

Does anyone have time to review the patch http://bugs.python.org/file10876/hex_float5.patch for issue 3008 (float <-> hexadecimal string conversion): http://bugs.python.org/issue3008 ? It would be really good if this could go in before next week's beta. Guido's approved the idea in principle, but I still need to: - get permission from Barry to check in a new feature this late in the release cycle, and - persuade some other developer to review the patch. I'll gladly 'pay' for a patch review by reviewing one or more of someone else's patches. Mark

2 1

Re: [Python-Dev] [Python-checkins] buildbot failure in sparc Debian 3.0
by Jeremy Hylton July 11, 2008

July 11, 2008

Does anyone have a clue about why this test fails only on this platform? The test is question is verifying that URLError gets raised. From the traceback, it appears that there is an uncaught exception (URLError) but it fails in an assertRaises() check for URLError. That doesn't make much sense unless the variable URLError refers to different objects, but both modules use "from urllib.error import URLError". And, of course, the test works fine on other platforms. Jeremy On Wed, Jul 2, 2008 at 4:23 PM, <buildbot(a)python.org> wrote: > The Buildbot has detected a new failure of sparc Debian 3.0. > Full details are available at: > http://www.python.org/dev/buildbot/all/sparc%20Debian%203.0/builds/330 > > Buildbot URL: http://www.python.org/dev/buildbot/all/ > > Buildslave for this Build: klose-debian-sparc > > Build Reason: > Build Source Stamp: [branch branches/py3k] HEAD > Blamelist: benjamin.peterson > > BUILD FAILED: failed test > > Excerpt from the test logfile: > 1 test failed: > test_urllib2 > > ====================================================================== > ERROR: test_badly_named_methods (test.test_urllib2.OpenerDirectorTests) > ---------------------------------------------------------------------- > > Traceback (most recent call last): > File "/home/pybot/buildarea-sid/3.0.klose-debian-sparc/build/Lib/test/test_urllib2.py", line 408, in test_badly_named_methods > self.assertRaises(URLError, o.open, scheme+"://example.com/") > File "/home/pybot/buildarea-sid/3.0.klose-debian-sparc/build/Lib/unittest.py", line 311, in failUnlessRaises > callableObj(*args, **kwargs) > File "/home/pybot/buildarea-sid/3.0.klose-debian-sparc/build/Lib/urllib/request.py", line 356, in open > response = self._open(req, data) > File "/home/pybot/buildarea-sid/3.0.klose-debian-sparc/build/Lib/urllib/request.py", line 379, in _open > 'unknown_open', req) > File "/home/pybot/buildarea-sid/3.0.klose-debian-sparc/build/Lib/urllib/request.py", line 334, in _call_chain > result = func(*args) > File "/home/pybot/buildarea-sid/3.0.klose-debian-sparc/build/Lib/urllib/request.py", line 1102, in unknown_open > raise URLError('unknown url type: %s' % type) > urllib.error.URLError: <urlopen error unknown url type: do> > > make: *** [buildbottest] Error 1 > > sincerely, > -The Buildbot > > _______________________________________________ > Python-checkins mailing list > Python-checkins(a)python.org > http://mail.python.org/mailman/listinfo/python-checkins >

3 2

Community buildbots and Python release quality metrics
by glyph＠divmod.com July 8, 2008

July 8, 2008

Today on planetpython.org, Doug Hellman announced the June issue of Python magazine. The cover story this month is about Pybots, "the fantastic automation system that has been put in place to make sure new releases of Python software are as robust and stable as possible". Last week, there was a "beta" release of Python which, according to the community buildbots, cannot run any existing python software. Normally I'd be complaining here about Twisted, but in fact Twisted is doing relatively well right now; only 80 failing tests. Django apparently cannot even be imported. The community buildbots have been in a broken state for months now[1]. I've been restraining myself from whinging about this, but now that it's getting close to release, it's time to get these into shape, or it's time to get rid of them. There are a few separate issues here. One is, to what extent should Python actually be compatible between releases? There's not much point in saying "this release is compatible with python 2.5" if all python 2.5 programs need to be modified in order to run on python 2.6. Of course, it's quite likely that there are some 3rd-party Python programs that continue to work, but python.org provides links to 2 projects that don't and none that do. Another way to phrase this question is, "whose responsibility is it to make Python 2.5 programs run on Python 2.6"? Or, "what happens when the core team finds out that a change they have made has broken some python software 'in the wild'"? Here are a couple of ways to answer these questions: 1) It's the python core team's responsibility. There should be Python buildbots for lots of different projects and they should all be green before the code can be considered [one of: allowed in trunk, alpha, beta, RC]. 2) It's the python core team's responsibility as long as the major projects are using public APIs. The code should not be considered [alpha, beta, RC] if there are any known breakages in public APIs. 3) It's only the python core team's responsibility to notify projects of incompatible changes of which they are aware, which may occur in public or private APIs. Major projects with breakages will be given a grace period before a [beta, RC, final] to release a forward-compatible version. 4) The status quo: every new Python release can (and will, at least speaking in terms of 2.6) break lots of popular software, with no warning aside from the beta period. There are no guarantees. For obvious reasons, I'd prefer #1, but any of the above is better than #4. I don't think #4 is an intentional choice, either; I think it's the result of there being no clear consequence for community buildbots failing. Or, for that matter, of core buildbots failing. Investigating the history is tricky (I see a bunch of emails from Barry here, but I'm not sure which is the final state) but I believe the beta went out with a bunch of buildbots offline or failing. A related but independent issue is: should the community buildbots continue to exist? An automated test is only as good as the consequences of its failure. As it stands, the community buildbots seem to be nothing but an embarrassment. I don't mean this to slam Grig, or the Python team - I mean, literally, if there is no consequence of their failure then the only use I can see for the data the buildbots currently provide (i.e. months and months of failures) is to embarrass the Python core developers. python.org's purpose should not be to provide negative press for Python ;), so if this continues to be the case, they probably shouldn't be linked. This isn't just an issue with changes to Python breaking stuff; the pybots' configuration is apparently not well maintained, since the buildbots which say "2.5" aren't passing either, and that's not a problem with the code, it's just that the slaves are actually running against trunk@HEAD. Ultimately these tools only exist to ensure the quality of Python releases. The really critical question here is, what does it mean to have a Python release that is high-quality? There are some obvious things: it shouldn't crash, it should conform to its own documentation. Personally I think "it passes all of its own tests" and "it runs existing Python code" are important metrics too, but the most important thing I'm trying to get across here is that there should be a clear understanding of which goals the release / QA / buildbot / test process is trying to accomplish. For me, personally, it really needs to be clear when I can say "you guys screwed up and broke stuff", and when I just have to suck it up and deal with the consequences of a new version of Python in Twisted. It's definitely bad for all of us if the result is that new releases of Python just break everything. Users don't care where the responsibility lies, they just know that stuff doesn't work, so we should decide on a process which allows Twisted (and other popular projects, like Django, Plone, pytz, PyGTK, pylons, ...) to (mostly) run when new versions of Python are released. [1]: http://mail.python.org/pipermail/python-dev/2008-May/079212.html

16 57

__module__ not found on ported Python
by Pree Raj July 8, 2008

July 8, 2008

Hi, I am trying to port Python to ThreadX. I have managed to get the prompt. However when I try "import sys" or any built in module I get an error __import__ not found. initmain() in the Initialization code is commented out at present because of some errors. Could it be because of this ? Also, I would like to know which are the MUST HAVE built in modules to be included for normal working of my ported version of Python. Thanks, Priya

3 3

bytearray and array.array are not thread-safe
by Antoine Pitrou July 7, 2008

July 7, 2008

Hello, Short story: bytearray and array.array by construction allow user code to reallocate their internal memory buffer. But a raw pointer to the said buffer can also be obtained by another thread, and used after releasing the GIL (for CPU-intensive operations like compression). As a consequence, the interpreter crashes. Was it envisioned? I see no warning in the docs for the array.array type (although it has been there for quite some time). See http://bugs.python.org/issue3139 (reported by Amaury) Regards Antoine.

3 4

Play with fuzzing
by Victor Stinner July 7, 2008

July 7, 2008

Hi, I wrote a fuzzing "framework" called Fusil and this week I wrote a fuzzer for Python. The idea is quite simple: for a module, - list all functions, classes and class methods - call a function with random arguments (of random types) - instanciate a class with random arguments - if the class is created correctly, call methods with random arguments Example: --------------------- 8< ----------------------------------- print "Call 39/40: linuxaudiodev.open()" try: linuxaudiodev.open( # argument 1/2 u"\u62C0\uFBD7\uB46A\u55E0\uFB7B\uD392\u7CEE", # argument 2/2 52.682, ) except Exception, err: print >>stderr, "ERROR: %s" % err --------------------- 8< ----------------------------------- I tried it on CPython 2.5 and then on CPython trunk (future 2.6). I found some bugs, see last bug entries in Python bugtracker. Just an example: http://bugs.python.org/issue3304 -> invalid call to PyMem_Free() in fileio_init() Most bugs crash with a segmentation fault, abort or a denial of service. If you would like to try my fuzzer, use: (1) svn co http://fusil.hachoir.org/svn/trunk fusil (2) cd fusil (3) ./run_fusil.sh -p projects/python.py --fast --remove ALL The option --fast goes faster, --remove does remove session directory even if Python generated some files, and "ALL" test all modules. FUSIL IS NOT SAFE! So run it under a different user using to avoid dangerous call to os.unlink(). The module list is hardcoded: it's the list of CPython modules written in C. More informations about Fusil: http://fusil.hachoir.org/trac -- Victor Stinner aka haypo http://www.haypocalc.com/blog/

3 2

Packing and unpacking integers
by Josiah Carlson July 6, 2008

July 6, 2008

A few years ago (yes, it's been that long), I proposed adding a new format code to struct that would pack integers as strings, similar to the 's' format code. In particular, struct.pack('>60G', v) would be a 60-byte big-endian unsigned integer as a string. The feature request is http://bugs.python.org/issue1023290 . Shortly thereafter, it was decided that it wouldn't become a struct format code, but instead would find itself as part of binhex. Raymond Hettinger was supposed to write the function a couple years ago for, I believe, Python 2.4 . It never happened. It still hasn't happened for Python 2.5 or 2.6 . I believe there is still a need for packing integers as strings and unpacking strings as integers, more specifically, offering to Python an interface to _PyLong_FromByteArray() and _PyLong_AsByteArray(). I would be happy to write the functionality and unittests this coming week for 2.6 and 3.0 if I get the ok. If not, I can write it for 2.7 and 3.1 . - Josiah

2 1

Second betas tomorrow
by Barry Warsaw July 5, 2008

July 5, 2008

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Wow, I bet this one crept up on you as quickly as it did me! We have our second planned beta releases for 2.6 and 3.0 tomorrow. As usual I will start looking at blockers and buildbots tomorrow afternoon (UTC-4 time) with a plan to start building things at about 6pm. Also, I will of course be in #python-dev on freenode to answer any questions, or get second opinions. PEP 361 claims that these will be the last betas. Whether that's true or not depends on how well the beta2's go. Please help review code or fix bugs. If you know of things that absolutely must go into beta2, be sure there is an open release-blocker bug on the issue. Thanks, - -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (Darwin) iQCVAwUBSGqQpnEjvBPtnXfVAQLdagP/VooK8+AoPrb1bR7xAxGqg0vC1HOKw5qZ 8VQArzgldz1OnoG24PuKGdaEw7PbHjCMkD0/CyZWjH8/yWawcxV7hKl6RYHJ3GX9 keroo7wz3/NaptJtA9ldoKA5ekV8WVVC5OElgtjKr+v6HorPQSHzUgJiDHYUS1FW A8fdHipyZds= =vwYy -----END PGP SIGNATURE-----

6 15

C99 code in the Python core?
by Mark Dickinson July 5, 2008

July 5, 2008

I have a general question and a specific question. First the general one: (1) When is it okay to use C99 code in the Python core? More particularly, is it considered acceptable to use widely-implemented library functions that are specified in C99 but not ANSI C, or widely-implemented features that are new to C99? Or is C99 code now acceptable pretty much anywhere? If so, should PEP 7 be updated? It currently says: """Use ANSI/ISO standard C (the 1989 version of the standard).""" I think there are some C99 features that still aren't implemented everywhere, even on major platforms. (Examples are the inverse hyperbolic trig functions in math.h.) And the specific question: (2) Is it okay to use the '%a' format specifier for sprintf, sscanf and friends. Are there major platforms where this isn't implemented? (Using '%a' would make the issue implementation much simpler.) Mark

3 2

UCS2/UCS4 default
by Jeroen Ruigrok van der Werven July 5, 2008

July 5, 2008

Guido (and others of course), back in 2001 you pointed out that you wanted to move to UCS4 completely as the ideal situation (http://mail.python.org/pipermail/i18n-sig/2001-June/001107.html) over the current default UCS2. Given 3.0 will use Unicode strings as the default, would it also not make sense to make the switch at this point as well? The current situation with UCS2 is particularly bad now that the CJK ideographs Extension B. has been produced (and C is under ballot and D is under development). Personally I use nothing else but UCS4 compiled Python binaries for the past years. See also http://www.python.org/dev/peps/pep-0261/ for background for the 2001 options. -- Jeroen Ruigrok van der Werven <asmodai(-at-)in-nomine.org> / asmodai イェルーンラウフロックヴァンデルウェルヴェン http://www.in-nomine.org/ | http://www.rangaku.org/ | GPG: 2EAC625B Expansion of happiness is the purpose of life...

17 53