Mailman 3 March 2013 - Python-Dev

Post-PyCon updates to PyParallel
by Trent Nelson 27 Mar '13

27 Mar '13

[ python-dev: I've set up a new list for pyparallel discussions: https://lists.snakebite.net/mailman/listinfo/pyparallel. This e-mail will be the last I'll send to python-dev@ regarding the on-going pyparallel work; please drop python-dev@ from the CC and just send to pyparallel(a)lists.snakebite.net -- I'll stay on top of the posts-from-unsubscribed-users moderation for those that want to reply to this e-mail but not subscribe. ] Hi folks, Wanted to give a quick update on the parallel work both during and after PyCon. During the language summit when I presented the slides I uploaded to speakerdeck.com, the majority of questions from other developers revolved around the big issues like data integrity and what happens when parallel objects interact with main-thread objects and vice-versa. So, during the sprints, I explored putting guards in place to throw an exception if we detect that a user has assigned a parallel object to a non-protected main-thread object. (I describe the concept of 'protection' in my follow up posts to python-dev last week: http://mail.python.org/pipermail/python-dev/2013-March/124690.html. Basically, protecting a main-thread object allows code like this to work without crashing: d = async.dict() def foo(): # async.rdtsc() is a helper method # that basically wraps the result of # the assembly RDTSC (read time- # stamp counter) instruction into a # PyLong object. So, it's handy when # I need to test the very functionality # being demonstrated here (creating # an object within a parallel context # and persisting it elsewhere). d['foo'] = async.rdtsc() def bar(): d['bar'] = async.rdtsc() async.submit_work(foo) async.submit_work(bar) ) It was actually pretty easy, far easier than I expected. It was achieved via Px_CHECK_PROTECTION(): https://bitbucket.org/tpn/pyparallel/commits/f3fe082668c6f3f699db990f046291… Various new tests related to the protection functionality: https://bitbucket.org/tpn/pyparallel/commits/f3fe082668c6f3f699db990f046291… The type of changes I had to make to other parts of CPython to perform the protection checks: https://bitbucket.org/tpn/pyparallel/commits/f3fe082668c6f3f699db990f046291… That was all working fine... until I started looking at adding support for lists (i.e. appending a parallel thread object to a protected, main-thread list). The problem is that appending to a list will often involve a list resize, which is done via PyMem_REALLOC() and some custom fiddling. That would mean if a parallel thread attempts to append to a list and it needs resizing, all the newly realloc'd memory would be allocated from the parallel context's heap. Now, this heap would stick around as long as the parallel objects have a refcount > 0. However, as soon as the last parallel object's refcount hits 0, the entire context will be scheduled for the cleanup/release/free dance, which will eventually blow away the entire heap and all the memory allocated against that heap... which means all the **ob_item stuff that was reallocated as part of the list resize. Not particularly desirable :-) As I was playing around with ways to potentially pre-allocate lists, it occurred to me that dicts would be affected in the exact same way; I just hadn't run into it yet because my unit tests only ever assigned a few (<5) objects to the protected dicts. Once the threshold gets reached (10?), a "dict resize" would take place, which would involve lots of PyMem_REALLOCs, and we get into the exact same situation mentioned above. So, at that point, I concluded that whole async protection stuff was not a viable long term solution. (In fact, the reason I first added it was simply to have an easy way to test things in unit tests.) The new solution I came up with: new thread-safe, interlocked data types that are *specifically* designed for this exact use case; transferring results from computation in a parallel thread back to a main thread 'container' object. First up is a new list type: xlist() (PyXListObject/PyXList_Type). I've just committed the work-in-progress stuff I've been able to hack out whilst traveling the past few days: https://bitbucket.org/tpn/pyparallel/commits/5b662eba4efe83e94d31bd9db4520a… It's not finished, and I'm pretty sure it doesn't even compile yet, but the idea is something like this: results = xlist() def worker1(input): # do work result = useful_work1() results.push(result) def worker2(input): # do work result = useful_work2() results.push(result) data = data_to_process() async.submit_work(worker1, data[:len(data)]) async.submit_work(worker2, data[len(data):]) async.run() for result in results: print(result) The big change is what happens during xlist.push(): https://bitbucket.org/tpn/pyparallel/commits/5b662eba4efe83e94d31bd9db4520a… +PyObject * +xlist_push(PyObject *obj, PyObject *src) +{ + PyXListObject *xlist = (PyXListObject *)obj; + assert(src); + + if (!Py_PXCTX) + PxList_PushObject(xlist->head, src); + else { + PyObject *dst; + _PyParallel_SetHeapOverride(xlist->heap_handle); + dst = PyObject_Clone(src, "objects of type %s cannot " + "be pushed to xlists"); + _PyParallel_RemoveHeapOverride(); + if (!dst) + return NULL; + PxList_PushObject(xlist->head, dst); + } + + /* + if (Px_CV_WAITERS(xlist)) + ConditionVariableWakeOne(&(xlist->cv)); + */ + + Py_RETURN_NONE; } Note the heap override and PyObject_Clone(), which currently looks like this: +PyObject * +PyObject_Clone(PyObject *src, const char *errmsg) +{ + int valid_type; + PyObject *dst; + PyTypeObject *tp; + + tp = Py_TYPE(src); + + valid_type = ( + PyBytes_CheckExact(src) || + PyByteArray_CheckExact(src) || + PyUnicode_CheckExact(src) || + PyLong_CheckExact(src) || + PyFloat_CheckExact(src) + ); + + if (!valid_type) { + PyErr_Format(PyExc_ValueError, errmsg, tp->tp_name); + return NULL; + } + + if (PyLong_CheckExact(src)) { + + } else if (PyFloat_CheckExact(src)) { + + } else if (PyUnicode_CheckExact(src)) { + + } else { + assert(0); + } + + +} Initially, I just want to get support working for simple types that are easy to clone. Any sort of GC/container types will obviously take a lot more work as they need to be deep-copied. You might also note the Px_CV_WAITERS() bit; these interlocked lists could quite easily function as producer/consumer queues, so, maybe you could do something like this: queue = xlist() def consumer(input): # do work ... def producer(): for i in xrange(100): queue.push(i) async.submit_queue(queue, consumer) async.submit_work(producer) Oh, forgot to mention the heap-override specifics: each xlist() gets its own heap handle -- when the "pushing" is done and the parallel object needs to be copied, the new memory is allocated against the xlist's heap. That heap will stick around until the xlist's refcnt hits 0, then everything will be blown away in one fell swoop. (Which means I'll need to tweak the memory/refcnt intercepts to handle this new concept -- like I had to do to support the notion of persisted contexts. Not a big deal.) I really like this approach; much more so than the persisted context stuff and the even-more-convoluted promotion stuff (yet to be written). Encapsulating all the memory associated with parallel to main-thread object transitions in the very object that is used to effect the transition just feels right. So, that means there are three main "memory alloc override"-type modes currently: - Normal. (Main-thread stuff, ref counting, PyMalloc stuff.) - Purely parallel. (Context-specific heap stuff, very fast.) - Parallel->main-thread transitions. (The stuff above.) (Or rather, there will be, once I finish this xlist stuff. That'll allow me to deprecate the TLS override stuff and the Context persistence stuff, both of which were nice experiments but fizzled out in practice.) ....the Px_CHECK_PROTECTION() work was definitely useful though and will need to be expanded to all objects. This will allow us to raise an exception if someone attempts to assign a parallel object to a normal main thread object (instead of one of the approved interlocked/parallel objects (like xlist)). Regards, Trent.

1 1

Google Summer of Code - Organization Deadline Approaching - March 29
by Brian Curtin 27 Mar '13

27 Mar '13

Just an FYI that there are under 3 days to apply to Google Summer of Code for mentoring organizations: http://www.google-melange.com/gsoc/homepage/google/gsoc2013. The student application deadline is later on in May. If you run a project that is interested in applying under the Python umbrella organization, contact Terri Oda at terri(a)zone12.com Is anyone here interested in leading CPython through GSOC? Anyone have potential students to get involved, or interested in being a mentor?

2 1

astoptimizer: static optimizer working on the AST
by Victor Stinner 27 Mar '13

27 Mar '13

Hi, I made progress since last August on my astoptimizer project (read the Changelog). Previous email thread: http://mail.python.org/pipermail/python-dev/2012-August/121286.html The astoptimizer project is an optimizer rewriting Python AST. It executes as much code as possible during the compilation. The optimizer itself is not designed to be fast, but to emit faster code. https://bitbucket.org/haypo/astoptimizer/ Some optimizations are not "pythonic": don't respect the Python language by making some assumptions (ex: on the namespace), which may be wrong in some cases. astoptimizer is written for adults which know which optimizations can be enabled, and which ones must be disabled in their application (application, not module). I'm trying to write a safe and "pythonic" default configuration. For example, len("abc") is not replaced with 3 by default. You have to enable explicitly the "builtin_funcs" configuration feature. astoptimizer can be used as a pythonic preprocessor: it replaces os.name, sys.platform and also your own constants by their value. It removes dead code and so may be used to remove completly the overhead of checks on the Python version or an the platform (if python3: ... else: ...). I would like to improve the integration of astoptimizer in Python 3.4. Brett Canon asked me to use an hook in importlib, I proposed to add a generic AST hook which would also be called by eval(), compile(), etc. I would like to allow anyone to use its own AST modifier, and simplify the usage of astoptimizer. => http://bugs.python.org/issue17515 Open issues: * what should be the name of "pyc" files? * how to handle different configuration of astoptimizer: generate different "pyc" files? * no sys.getasthook(): it would permit to chain different AST hooks -- There are many open issues proposing to write a better Python optimizer. Some of them are stuck because implementing them using the bytecode optimizer is hard. * http://bugs.python.org/issue1346238: A constant folding optimization pass for the AST * http://bugs.python.org/issue2181: optimize out local variables at end of function * http://bugs.python.org/issue2499: Fold unary + and not on constants * http://bugs.python.org/issue4264: Patch: optimize code to use LIST_APPEND instead of calling list.append * http://bugs.python.org/issue7682: Optimisation of if with constant expression * http://bugs.python.org/issue10399: AST Optimization: inlining of function calls * http://bugs.python.org/issue11549: Build-out an AST optimizer, moving some functionality out of the peephole optimizer * http://bugs.python.org/issue17068: peephole optimization for constant strings * http://bugs.python.org/issue17430: missed peephole optimization astoptimizer implements many optimizations listed in these issues. Read the README file to the list of optimizations already implemented, and the TODO file for ideas of new optimizations. -- The project is still experimental. I ran Python 2.7 and 3.4 test suites. Some tests are failing because the AST or the bytecode is different (which is expected). More tests are failing if you enable more agressive optimizations (especially if you enable the agressive mode to remove dead code). At least, Python does not crash :-) (It occurs sometimes when astoptimizer generates invalid AST!) Victor

2 1

[RELEASE] Python 2.7.4 release candidate 1
by Benjamin Peterson 27 Mar '13

27 Mar '13

I'm happy to announce the first release candidate of 2.7.4. 2.7.4 will be the latest maintenance release in the Python 2.7 series. It includes hundreds of bugfixes to the core language and standard library. There has recently been a lot of discussion about XML-based denial of service attacks. Specifically, certain XML files can cause XML parsers, including ones in the Python stdlib, to consume gigabytes of RAM and swamp the CPU. 2.7.4 does not include any changes in Python XML code to address these issues. Interested parties should examine the defusedxml package on PyPI: https://pypi.python.org/pypi/defusedxml 2.7.4 release candidate 1 is a testing release. Deploying it in production is not recommended. However, please download it and test with your libraries and applications, reporting any bugs you may find. Assuming no horrible bugs rear their heads, a final release of 2.7.4 will occur in 2 weeks. Downloads are at http://python.org/download/releases/2.7.4/ As always, please report bugs to http://bugs.python.org/ Enjoy, Benjamin Peterson 2.7 Release Manager

13 17

Re: [Python-Dev] cpython (2.7): Issue 17538: Document XML vulnerabilties
by Antoine Pitrou 26 Mar '13

26 Mar '13

On Tue, 26 Mar 2013 17:53:39 +0100 (CET) christian.heimes <python-checkins(a)python.org> wrote: > + > +The XML processing modules are not secure against maliciously constructed data. > +An attacker can abuse vulnerabilities for e.g. denial of service attacks, to > +access local files, to generate network connections to other machines, or > +to or circumvent firewalls. Really? Where does the "access local files, generate network connections, circumvent firewalls" allegation come from? Regards Antoine.

3 2

[RELEASED] Python 3.2.4 rc 1 and Python 3.3.1 rc 1
by Georg Brandl 25 Mar '13

25 Mar '13

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On behalf of the Python development team, I am pleased to announce the first release candidates of Python 3.2.4 and 3.3.1. Python 3.2.4 will be the last regular maintenance release for the Python 3.2 series, while Python 3.3.1 is the first maintenance release for the 3.3 series. Both releases include hundreds of bugfixes. There has recently been a lot of discussion about XML-based denial of service attacks. Specifically, certain XML files can cause XML parsers, including ones in the Python stdlib, to consume gigabytes of RAM and swamp the CPU. These releases do not include any changes in Python XML code to address these issues. Interested parties should examine the defusedxml package on PyPI: https://pypi.python.org/pypi/defusedxml These are testing releases: Please consider trying them with your code and reporting any bugs you may notice to: http://bugs.python.org/ To download Python 3.2.4 or Python 3.3.1, visit: http://www.python.org/download/releases/3.2.4/ or http://www.python.org/download/releases/3.3.1/ respectively. Enjoy! - -- Georg Brandl, Release Manager georg at python.org (on behalf of the entire python-dev team and all contributors) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.19 (GNU/Linux) iEYEARECAAYFAlFRRIoACgkQN9GcIYhpnLD6jACgnzYdYRKZ4kwkKeN3zSLSZ3Zr M/IAn17vlpxI3a3xk+i/ODOrCkMnRZro =B5sA -----END PGP SIGNATURE-----

1 0

Simple IDLE issues to commit before Python 2.7.4 release in two weeks on 4/6/2013
by Todd Rovito 25 Mar '13

25 Mar '13

Python 2.7.4 release candidate was just announced and is ready for testing here: http://python.org/download/releases/2.7.4/ Now the clock is ticking and we have two weeks to get IDLE issues pushed into CPython before the final release of 2.7.4. Below is my list of low risk issues that would be great to get pushed into CPython. I hope this email will encourage CPython Core Developers to make the commits or tell us what we can do, like more testing or better documentation, to get these issues cleaned up and committed before 2.7.4 ships. PEP-434 is still being discussed, I asked PEP authors to post latest version on 3/24, but I think most folks agree with the principle. IDLE might have more issues that could be fixed easily so feel free to add to the list. Thanks for everybody’s hard work to make IDLE better!!!!! http://bugs.python.org/issue7136 Idle File Menu Option Improvement http://bugs.python.org/issue17390 display python version on idle title bar http://bugs.python.org/issue17511 Idle find function closes after each find operation http://bugs.python.org/issue6699 IDLE: Warn user about overwriting a file that has a newer version on filesystem http://bugs.python.org/issue10747 Include version info in Windows shortcuts the issue is for 3.x but perhaps simple to implement for 2.7? The issue below has been talked about in the recent past on idle-dev (http://mail.python.org/pipermail/idle-dev/2013-March/003228.html) it will take some work to get committed because of the deletion of the Option menu on the Mac. But even if it gets committed then is disabled for the Mac with a simple sys.platform it would be a huge victory for IDLE in my mind. http://bugs.python.org/issue2704 IDLE: Patch to make PyShell behave more like a Terminal interface

7 9

IDLE in the stdlib
by Eli Bendersky 25 Mar '13

25 Mar '13

Interesting writeup about PyCon 2013 young coder education: http://therealkatie.net/blog/2013/mar/19/pycon-2013-young-coders/ Quote: "We used IDLE because it's already on Raspian's desktop. Personally, I like IDLE as a teaching tool. It's included in the standard library, it does tab completion and color coding, and it even has a text editor included so you don't have to start your class off by teaching everyone about paths. Too bad it's broke as hell." Personally, I think that IDLE reflects badly on Python in more ways than one. It's badly maintained, quirky and ugly. It serves a very narrow set of uses, and does it badly. Being part of Python *distributions* and being part of core Python standard library are two different things. The former may make sense, the latter IMHO makes no sense whatsoever. Outside the Python core IDLE can be maintained more freely, with less restrictions on contributors and hopefully become a better tool. Eli

30 77

wsgi validator with asynchronous handlers/servers
by Luca Sbardella 23 Mar '13

23 Mar '13

Hi, I have an asynchronous wsgi application handler which yields empty bytes before it is ready to yield the response body and, importantly, to call start_response. Something like this: def wsgi_handler(environ, start_response): body = generate_body(environ) body = maybe_async(body) while is_async(body): yield b'' start_response(...) ... I started using wsgiref.validator recently, nice little gem in the standard lib, and I discovered that the above handler does not validate! Disaster. Reading pep 3333 "the application *must* invoke the start_response() callable before the iterable yields its first body bytestring, so that the server can send the headers before any body content. However, this invocation *may* be performed by the iterable's first iteration, so servers *must not* assume that start_response() has been called before they begin iterating over the iterable." The pseudocode above does yields bytes before start_response, but they are not *body* bytes, they are empty bytes so that the asynchronous wsgi server releases the eventloop and call back at the next eventloop iteration. I'm I misinterpreting the pep, or the wsgi validator should be fixed accordingly? Regards, Luca

3 2

PEP 405 (venv) - why does it copy the DLLs on Windows
by Paul Moore 23 Mar '13

23 Mar '13

PEP 405 has this note: """ On Windows, it is necessary to also copy or symlink DLLs and pyd files from compiled stdlib modules into the env, because if the venv is created from a non-system-wide Python installation, Windows won't be able to find the Python installation's copies of those files when Python is run from the venv. """ I don't understand what this is saying - can someone clarify the reason behind this statement? What is different about a "non-system-wide installation" that causes this issue (I assume "non-system-wide" means "not All Users")? The reason I ask is that virtualenv doesn't do this, and I'm not clear if this is because of a potential bug lurking in virtualenv (in which case, I'd like to find out how to reproduce it) or because virtualenv takes a different approach which avoids this issue somehow. Thanks, Paul.

5 6