Mailman 3 September 2007 - Python-Dev

Adding concat function to itertools
by Bruce Frederiksen Sept. 29, 2007

Sept. 29, 2007

7 7

Summary of Tracker Issues
by Tracker Sept. 29, 2007

Sept. 29, 2007

ACTIVITY SUMMARY (09/21/07 - 09/28/07) Tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue number. Do NOT respond to this message. 1278 open (+16) / 11424 closed (+17) / 12702 total (+33) Open issues with patches: 415 Average duration of open issues: 680 days. Median duration of open issues: 670 days. Open Issues Breakdown open 1273 (+16) pending 5 ( +0) Issues Created Or Reopened (33) _______________________________ … [View More]

1 0

Python 3.0a documentation
by scav＠blueyonder.co.uk Sept. 28, 2007

Sept. 28, 2007

I'd like to help out cleaning up the Python3.0 documentation. There are a lot of little leftovers from 2.x that are no longer true. (mentions of long, callable() etc.) Ideally (especially in the tutorial), we should only refer to 3.0 features and syntax, and keep the special cases and "other ways to do it" to a minimum. Before I dive in and start submitting patches, what does everyone else think? How much reference to previous python versions should be left in? Does it make sense to keep … [View More]

5 6

Iterating over objects of unknown length
by Oleg Broytmann Sept. 27, 2007

Sept. 27, 2007

Hello! (This seems like a "developing with Python" question and partially it is but please read on.) I have a class that represents SQL queries. Instances of the class can be iterated over. As an SQL query doesn't know in advance if it will produce any row the class doesn't implement __len__(). Moreover, users of the class sometimes write if sqlQuery: for row in sqlQuery: ... else: # no rows which is a bug (the query doesn't know if it's True or False; to find it out the user … [View More]

4 5

Summary of Tracker Issues
by Tracker Sept. 22, 2007

Sept. 22, 2007

ACTIVITY SUMMARY (09/14/07 - 09/21/07) Tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue number. Do NOT respond to this message. 1269 open (+12) / 11400 closed (+11) / 12669 total (+23) Open issues with patches: 412 Average duration of open issues: 678 days. Median duration of open issues: 655 days. Open Issues Breakdown open 1264 (+12) pending 5 ( +0) Issues Created Or Reopened (23) _______________________________ … [View More]

1 0

Better unittest failures
by Ron Adam Sept. 21, 2007

Sept. 21, 2007

The value of a unittest test is not in how well they pass, but in how well they fail. While looking at possibly helping with the str_uni branch when that was going on I found that in some cases unittest failure results can take a little bit (or a lot) of work to figure out just what was failing, where and why. While helping Eric test the new format function and class I came up with a partial solution which may be a bases for further improvements. Eric told me it did help quite a bit. … [View More]So I think it's worth looking into. Since we were running over a hundred different options over several different implementations to make sure they all passed and failed in the same way, we were using data based test cases so we could easily test the same data with each version. Unfortunately that has a drawback that the traceback doesn't show what data was used when testing exceptions. Additionally when something did fail it was not always obvious what and why it was failing. One of the conclusions I came to is it would be better if tests did not raise standard python exceptions unless the test itself has a problem. By having tests raise special *Test_Only* exceptions, it can make the output of the test very much clearer. Here are the added Test_Only Excepitons. These would only be in the unittest module to catch the following situations. Wrong_Result_Returned Unexpected_Exception_Raised No_Exception_Raised Wrong_Exception_Raised And two new functions that use them. assertTestReturns(expect, test, message) assertTestRaises(expect, test, message) These additions would not effect any existing tests. To use these requires the code to be tested to be wrapped in a function with no arguments. And it is the same format for both assertTestReturns and assertTestRaises. for data in testdata: expect, a, b, c = data def test(): return foo(a, b, c) assertTestReturns(expect, test, repr(data)) Replacing all existing tests with this form isn't reasonable but adding this as an option for those who want to use it is very easy to do. The test file I used to generate the following output is attached. Cheers, Ron ### # # Test output using standard assertEquals and assertRaises. # * The data has the form [(ref#, expect, args, kwds), ...] * The ref# is there to help find the failing test for situation where you may have dozens of almost identical data. It's not required but helpful to have. * I didn't include actual bad testcase tests in these examples, but if some generated exceptions similar to the that of the failing tests, I think it could add a bit more confusion to the situation than the not too confusing example here. $ python ut_test.py EEFFFFFF ====================================================================== ERROR: test_A (__main__.test1_normal_failures) ---------------------------------------------------------------------- Traceback (most recent call last): File "ut_test.py", line 100, in test_A result = some_function(*args, **kwds) File "ut_test.py", line 62, in some_function baz = kwds['baz'] KeyError: 'baz' # # This fails as a test "error" instead of a test "fail". # What was args and kwds here? # ====================================================================== ERROR: test_B (__main__.test1_normal_failures) ---------------------------------------------------------------------- Traceback (most recent call last): File "ut_test.py", line 108, in test_B self.assertRaises(expect, test, args, kwds) File "unittest.py", line 320, in failUnlessRaises callableObj(*args, **kwargs) File "ut_test.py", line 107, in test return some_function(*args, **kwds) File "ut_test.py", line 62, in some_function baz = kwds['baz'] KeyError: 'baz' # # Same as above. Fails as a test "error", unkown arguments # values for some_function(). # ====================================================================== FAIL: test_C (__main__.test1_normal_failures) ---------------------------------------------------------------------- Traceback (most recent call last): File "ut_test.py", line 114, in test_C self.assertRaises(expect, test, args, kwds) AssertionError: KeyError not raised # # What was args, and kwds values? # ====================================================================== FAIL: test_D (__main__.test1_normal_failures) ---------------------------------------------------------------------- Traceback (most recent call last): File "ut_test.py", line 120, in test_D repr((n, expect, args, kwds))) AssertionError: (8, ('Total baz:', 4), [1, 2], {'baz': 'Total baz:'}) # # This one is ok. # ### # # Test output using the added methods and test only exceptions with # the same test data. # * Test errors only occur on actual test "errors". * The reason for the fail is explained in all cases for test "fails". * The only time you get an actual python exception is when the test it self has a problem. Otherwise you get an test_exception that refers to the exception in the actual code. ====================================================================== FAIL: test_A (__main__.test2_new_failures) ---------------------------------------------------------------------- Traceback (most recent call last): File "ut_test.py", line 131, in test_A repr((n, expect, args, kwds))) File "ut_test.py", line 36, in assertTestReturns result = test() File "ut_test.py", line 129, in test return some_function(*args, **kwds) File "ut_test.py", line 62, in some_function baz = kwds['baz'] Unexpected_Exception_Raised: KeyError('baz',) Reference: (2, ('Total baz:', 3), [1, 2], {'raz': 'Total baz:'}) ====================================================================== FAIL: test_B (__main__.test2_new_failures) ---------------------------------------------------------------------- Traceback (most recent call last): File "ut_test.py", line 138, in test_B repr((n, expect, args, kwds))) File "ut_test.py", line 45, in assertTestRaises result = test() File "ut_test.py", line 136, in test return some_function(*args, **kwds) File "ut_test.py", line 62, in some_function baz = kwds['baz'] Wrong_Exception_Raised: KeyError('baz',) Reference: (4, <type 'exceptions.IndexError'>, [1, 2], {'raz': 'Total baz:'}) ====================================================================== FAIL: test_C (__main__.test2_new_failures) ---------------------------------------------------------------------- Traceback (most recent call last): File "ut_test.py", line 145, in test_C repr((n, expect, args, kwds))) File "ut_test.py", line 52, in assertTestRaises raise self.No_Exception_Raised(result, ref) No_Exception_Raised: returned -> ('Total baz:', 3) Reference: (6, <type 'exceptions.KeyError'>, [1, 2], {'baz': 'Total baz:'}) ====================================================================== FAIL: test_D (__main__.test2_new_failures) ---------------------------------------------------------------------- Traceback (most recent call last): File "ut_test.py", line 152, in test_D repr((n, expect, args, kwds))) File "ut_test.py", line 41, in assertTestReturns raise self.Wrong_Result_Returned(result, ref) Wrong_Result_Returned: ('Total baz:', 3) Reference: (8, ('Total baz:', 4), [1, 2], {'baz': 'Total baz:'}) ---------------------------------------------------------------------- Ran 8 tests in 0.004s FAILED (failures=6, errors=2) [View Less]

1 0

Building Python with CMake
by Alexander Neundorf Sept. 21, 2007

Sept. 21, 2007

Hi, as I wrote in my previous email, I'm currently porting Python to some more unusual platforms, namely to a super computer (http://www.research.ibm.com/bluegene/) and a tiny embedded operating system (http://ecos.sourceware.org), which have more or less surprisingly quite similar properties. To do this, I added CMake files to Python, in order to use CMake to cross compile Python to these platforms. CMake (http://www.cmake.org) is a buildsystem in scope similar to autotools, but it's … [View More]

4 8

SSL module backport package ready for more testing
by Bill Janssen Sept. 21, 2007

Sept. 21, 2007

I've posted an sdist version of the 'ssl' module for Pythons 2.3.5 to 2.5.x, at http://www.parc.com/janssen/transient/ssl-1.3.tar.gz. I think this is 'gold master', but before I upload it to the Cheeseshop, I'd like to get more testing on a broader variety of platforms. The intent of this package is to allow code development with older versions of Python that will continue to work on Python 2.6 and 3.x. To build, python setup.py build To test, python setup.py test I'd appreciate … [View More]

1 0

Exploration PEP : Concurrency for moderately massive (4 to 32 cores) multi-core architectures
by Krishna Sankar Sept. 20, 2007

Sept. 20, 2007

Folks, As a follow-up to the py3k discussions started by Bruce and Guido, I pinged Brett and he suggested I submit an exploratory proposal. Would appreciate insights, wisdom, the good, the bad and the ugly. A) Does it make sense ? B) Which application sets should we consider in designing the interfaces and implementations C) In this proposal, parallelism and concurrency are used in an interchangeable fashion. Thoughts ? D) Please suggest pertinent links, discussions and … [View More]insights. E) I have kept the proposal to a minimum to start the discussions and to explore if this is the right thing to do. Collaboratively, as we zero-in on one or two approaches, the idea is to expand it to a crisp and clear PEP. Need to do some more formatting as well. Cheers <k/> P.S : I had sent this to python-ideas couple of days ago and received two comments (Thanks Leonardo, Thanks Adam) I haven't incorporated their comments yet. Folks who are on both lists, pardon me for the spam. ------------------------------------------------------------------------------------------------------------ PEP: xxxxxxxx Title: Concurrency for moderately massive (4 to 32 cores) multi-core architectures Version: $Revision$ Last-Modified: $Date$ Author: Krishna Sankar <ksankar (at) doubleclix.net>, Status: Wandering ! (as in "Not all those who wander are lost ..." -J.R.R.Tolkien) Type: Process Content-Type: text/x-rst Created: 15-Sep-2007 Abstract -------- This proposal aims at leveraging the multi-core capability as an embedded mechanism in python. It is not whether python is slow or fast, but of performance and control of parallelism/concurrency in a moderately massive parallelism world. The aim is 4 to 32 cores. The proposal advocates two mechanisms - one for task parallelism and another for data intensive parallelism. Scientific computing and web 2.0 frameworks are the forefront users for this proposal. Other applications would benefit as well. Rationale --------- Multicore architectures need no introductions and their ubiquity is evident. It is imperative that Python has one or more standard ways of leveraging multi-core architectures. OTOH, traditional thread based concurrency and lock based exclusions are becoming more and more difficult to program correctly. First of all, the question is not whether py is slow or fast but performance of a system written in py. Which means, ability to leverage multi-core architectures as well as control. Control in term of things like ability to pin one process/task to a core, ability to pin one or more homogeneous tasks to specific cores et al, as well as not wait for a global lock and similar primitives. (Before anybody jumps into a conclusion, this is not about GIL by any means ;o)) Second, it is clear that we need a good solution (not THE solution) for moderately massive parallelism in multi-core architectures (i.e. 8-32 cores). Share nothing might not be optimal; we need some form of memory sharing, not just copy all data via messages. May be functional programming based on the blackboard pattern would work, who knows. I have seen systems saturated still having only ~25% of CPU utilization (in a 4 core system!). It is because we didn't leverage multi-cores and parallelism. So while py3k will not be slow, lack of a cohesive multi-core strategy will show up in system performance and byte us later(pun intended!). At least, in my mind, this is not an exercise about exposing locks and mutexes or threads in Python. I do believe that the GIL will be refactored to more granularity in the coming months (similar to the Global Locks in Linux) and most probably we will get microThreads et al. As we all know, architecture is constraining as well as liberating. The language primitives influence greatly how we think about a problem. In the discussions, Guido is right in insisting on speed, and Bruce is right in asking for language constructs. Without pragmatic speed, folks won't use it; same is the case without the required constructs. Both are barriers to adoption. We have an opportunity to offer a solution for multi-core architectures and let us seize it - we will rush in where angels fear to tread! Programming Models ------------------ There are at least 3 possible paradigms A. conventional threading model B. Functional model, Erlang being the most appropriate C. Some form of limited shared memory model (message passing but pass pointers, blackboard model) D. Others, like Transactional Memory [2] There is enough literature out there, so do not plan to explain these here. (<KS> Do we need more explanation? </KS>) Pragmatic proposal ------------------ May I suggest we embed two primitives in Python 3K: A) A functional style share-nothing set of interfaces (and implementations thereof) - provides the task parallelism/concurrency capability, "small messages, big computations" as Joe Armstrong calls it[3] B) A limited shared memory based model for data intensive parallelism Most probably this would be part of stdlib. While Guido is almost right in saying that this is a (std)library problem, it is not fully so. We would need a few primitives from the underlying PVM substrate. Possibly one reason for Guido's position is the lack of clarity as to what needs to be changed and why. IMHO, just saying take GIL off does not solve the problem either. The Zen of Python parallelism ----------------------------- I draw inspiration for the very timely article by James Reinders in DDJ [1]. It embodies what we should be doing viz.: 1. Refactor the problem into parallel tasks. We cannot help if the domain is sequential 2. Program to abstraction & program chores not cores. Writing correct program using raw threads et al is difficult. Let the underlying substrate decide how best to optimize 3. Design for scale 4. Have an option to turn concurrency off, for debugging 5. Declarative parallelism based mechanisms (?) Related Efforts --------------- The good news is there are at least 2 or 3 paradigms with implementations and rough benchmarks. Parallel python http://www.artima.com/weblogs/viewpost.jsp?thread=214303 http://cheeseshop.python.org/pypi/parallel Processing http://cheeseshop.python.org/pypi/processing http://code.google.com/p/papyros/ Discussions ----------- There are at least four thread sets (pardon the pun !) I am aware of: 1. The GIL discussions in python-dev and Guido's blog on GIL http://www.artima.com/weblogs/viewpost.jsp?thread=214235 2. The py3k topics started by Bruce http://www.artima.com/weblogs/viewpost.jsp?thread=214112, response by Guide http://www.artima.com/weblogs/viewpost.jsp?thread=214325 and reply to reply by Bruce http://www.artima.com/weblogs/viewpost.jsp?thread=214480 3. Python and concurrency http://mail.python.org/pipermail/python-ideas/2007-March/000338.html 4. Adam's reply in python-ideas http://mail.python.org/pipermail/python-ideas/2007-September/000972.html References [1]http://www.ddj.com/architect/201804248 [2]Transaction http://acmqueue.com/modules.php?name=Content&pa=showpage&pid=444 [3]Programming Erlang by Joe Armstrong [View Less]

5 9

Summary of Tracker Issues
by Tracker Sept. 20, 2007

Sept. 20, 2007

ACTIVITY SUMMARY (09/12/07 - 09/19/07) Tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue number. Do NOT respond to this message. 1266 open (+13) / 11396 closed (+11) / 12662 total (+24) Open issues with patches: 408 Average duration of open issues: 678 days. Median duration of open issues: 643 days. Open Issues Breakdown open 1262 (+13) pending 4 ( +0) Issues Created Or Reopened (24) _______________________________ … [View More]

1 0