Mailman 3 November 2013 - Python-Dev

Mixed up core/module source file locations in CPython
by Eric Snow Nov. 19, 2013

Nov. 19, 2013

If you look at the Python and Modules directories in the cpython repo, you'll find modules in Python/ and core files (like python.c and main.c) in Modules/. (It's like parking on a driveway and driving on a parkway. <wink>) It's not that big a deal and not that hard to figure out (so I'm fine with the status quo), but it is a bit surprising. When I was first getting familiar with the code base a few years ago (as a C non-expert), it was a not insignificant but not major stumbling block. The situation is mostly a consequence of history, if I understand correctly. The subject has come up before and I don't recall any objections to doing something about it. I haven't had the time to track down those earlier discussions, though I remember Benjamin having some comment about it. Would it be too disruptive (churn, etc.) to clean this up in 3.5? I see it similarly to when I moved a light switch from outside my bathroom to inside. For a while, but not that long, I kept unconsciously reaching for the switch that was no longer there on the outside. Regardless I'm glad I did it. Likewise, moving the handful of files around is a relatively inconsequential change that would make the project just a little less surprising, particularly for new contributors. -eric p.s. Either way I'll probably take some time (it shouldn't take long) after the PEP 451 implementation is done to put together a patch that moves the files around, just to see what difference it makes.

4 6

Re: [Python-Dev] [Python-checkins] cpython: Close #17828: better handling of codec errors
by Walter Dörwald Nov. 19, 2013

Nov. 19, 2013

On 13.11.13 14:51, nick.coghlan wrote: > http://hg.python.org/cpython/rev/854a2cea31b9 > changeset: 87084:854a2cea31b9 > user: Nick Coghlan <ncoghlan(a)gmail.com> > date: Wed Nov 13 23:49:21 2013 +1000 > summary: > Close #17828: better handling of codec errors > > - output type errors now redirect users to the type-neutral > convenience functions in the codecs module > - stateless errors that occur during encoding and decoding > will now be automatically wrapped in exceptions that give > the name of the codec involved Wouldn't it be better to add an annotation API to the exceptions classes? This would allow to annotate all exceptions without having to replace the exception object. I.e. BaseException would have an additional method annotate(): try: dostuff(param) except Exception as exc: exc.annotate("while doing stuff with {}".format(param)) annotate() would simply append the message to an internal list attribute. BaseException.__str__() could then use this to format an appropriate exception message. Servus, Walter

4 9

[RELEASED] Python 3.3.3 final
by Georg Brandl Nov. 19, 2013

Nov. 19, 2013

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On behalf of the Python development team, I'm very happy to announce the release of Python 3.3.3. Python 3.3.3 includes several security fixes and over 150 bug fixes compared to the Python 3.3.2 release. Importantly, a security bug in CGIHTTPServer was fixed [1]. Thank you to those who tested the 3.3.3 release candidate! This release fully supports OS X 10.9 Mavericks. In particular, this release fixes an issue that could cause previous versions of Python to crash when typing in interactive mode on OS X 10.9. Python 3.3 includes a range of improvements of the 3.x series, as well as easier porting between 2.x and 3.x. In total, almost 500 API items are new or improved in Python 3.3. For a more extensive list of changes in the 3.3 series, see http://docs.python.org/3.3/whatsnew/3.3.html To download Python 3.3.3 rc2 visit: http://www.python.org/download/releases/3.3.3/ This is a production release, please report any bugs to http://bugs.python.org/ Enjoy! - -- Georg Brandl, Release Manager georg at python.org (on behalf of the entire python-dev team and 3.3's contributors) [1] http://bugs.python.org/issue19435 -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iEYEARECAAYFAlKLDGYACgkQN9GcIYhpnLCjewCfQ+EJHpzGzIyTvYOrYmsRmnbv n40AniVM0UuQWpPrlCu349fu7Edt+d4+ =WSIj -----END PGP SIGNATURE-----

1 0

New to the community - Introduction
by Avichal Dayal Nov. 18, 2013

Nov. 18, 2013

Hello, I'm Avichal Dayal and I'm new to this community rather the whole open source world. I've been "lurking" around for a while, reading bugs, code and asking relevant doubts on the core-mentorship mailing list. Now I intend to see how the development goes on and be a part of it. Hope to be a part of this community and contribute! Regards, Avichal

2 1

unicode Exception messages in py2.7
by Chris Barker Nov. 18, 2013

Nov. 18, 2013

Folks, (note this is about 2.7 -- sorry, but a lot of us still use that! I can only assume that in 3.* this is a non-issue) I just discovered an issue that's been around a long time: If you create an Exception with a unicode object for the message, the message can be silently ignored if it can not be encoded to ASCII (or, more properly, the default encoding). In my use-case, I was parsing a text file (utf-8), and wanted a bit of that text to be part of the Exception message (an error reading the file, I wanted the user to know what the text was surrounding the ill-formated part of the text file). What I got was a blank message, and it took a lot of poking at it to figure out why. My solution was: msg = u"Problem with line %i: %s This is not a valid time slot"%(linenum, line) raise ValueError(msg.encode('ascii', 'ignore')) which is really pretty painfully clunky. This is an issue brought up in various tutorial and blog posts, and all the solutions I've seen involve some similar clunkiness. I also found this issue in the issue tracker: http://bugs.python.org/issue2517 Which was resolved years ago, but as far as I can tell, only solved the problem of being able to do: unicode(an_exception) and get the proper unicode message object. But we still can't raise the darn thing and expect the user to see the message. Why is this the case? I can print a unicode object to the terminal, why can't raising an Exception print a unicode object? I can imagine for backward compatibility, or maybe for non-unicode terminals, or ??? Exceptions do need to print as ascii. However, having a message simply get swallowed up and disappear seems like the wrong solution. - auto-conversion to a default encoding is fraught with problems all over the board -- I know that. I also know that too much code would break too often if we didn't have auto-conversion. - for the most part, the auto-conversion uses 'strict' mode -- I generally dislike this, as it means code crashes when odd stuff gets introduced after testing, but I can see why it is done. - However, I can see why for raising Exceptions, the decision was made to swallow that error, so that the actual Exception intended is raised, rather than a new UnicodeEncodeError. - But combining 'strict' with ignoring the encoding exception seems like the worst of both worlds. So a proposal: Use 'replace" mode for the encoding to the default, and at least the user would see SOMETHING of the message. In a common case, it would be a lot of ascii, and in the worse case it would be a lot of question marks -- still better than a totally blank message. Another option would be to use the str(repr(the_message)) so the user would get the escaped version. Though I think that would be more ugly. What am I missing? This seems so obvious, and easy to do (though maybe it's buried in the C implementation of Exceptions) -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker(a)noaa.gov

12 25

NTPath or WindowsPath?
by Antoine Pitrou Nov. 18, 2013

Nov. 18, 2013

Hello, In a (private) discussion about PEP 428 and pathlib, Guido proposed that maybe NTPath should be renamed to WindowsPath, since the name is more likely to stay relevant in the middle term. What do you think? Regards Antoine.

7 7

Re: [Python-Dev] PyParallel: alternate async I/O and GIL removal
by Terry Reedy Nov. 17, 2013

Nov. 17, 2013

On 11/16/2013 8:39 PM, Guido van Rossum wrote: > Trent, I watched your video and read your slides. I only read the slides. > (Does the word "motormouth" mean anything to you? :-) The extra background (and repetition) was helpful to me in filling in things, especially about Windows, that I could not follow in the discussion a year ago. But I noticed that it might be tedious to people already familiar with the subject. > Clearly your work isn't ready for python-dev -- it is just too > speculative. I've moved python-dev to BCC and added python-ideas. > > It possibly doesn't even belong on python-ideas -- if you are serious > about wanting to change Linux or other *NIX variants, you'll have to go > find a venue where people who do forward-looking kernel work hang out. A working and useful Windows-only PyParallel (integrated with CPython or not) might help stimulate open-source *nix improvements to match the new Windows stuff. > Finally, I'm not sure why you are so confrontational about the way > Twisted and Tulip do things. We are doing things the only way they *can* > be done without overhauling the entire CPython implementation (which you > have proven will take several major release cycles, probably until 4.0). > It's fine that you are looking further forward than most of us. I don't > think it makes sense that you are blaming the rest of us for writing > libraries that can be used today. Just reading the slides, I did not pick up as much confrontation and blame as you did. I certainly appreciate that asyncio is working *now*, and on all three major target systems. I look forward to looking at the doc when it is added. I will understand a bit better for having read Trent's slides. -- Terry Jan Reedy

1 0

PyParallel: alternate async I/O and GIL removal
by Trent Nelson Nov. 17, 2013

Nov. 17, 2013

Hi folks, Video of the presentation I gave last weekend at PyData NYC regarding PyParallel just went live: https://vimeo.com/79539317 Slides are here: https://speakerdeck.com/trent/pyparallel-how-we-removed-the-gil-and-exploit… The work was driven by the async I/O discussions around this time last year on python-ideas. That resulted in me sending this: http://markmail.org/thread/kh3qgjbydvxt3exw#query:+page:1+mid:arua62vllzugj… ....where I attempted to argue that there was a better way of doing async I/O on Windows than the status quo of single-threaded, non-blocking I/O with an event multiplex syscall. I wasn't successful in convincing anyone at the time; I had no code to back it up and I didn't articulate my plans for GIL removal at the time either (figuring the initial suggestion would be met with enough scepticism as is). So, in the video above, I spend a lot of time detailing how IOCP works on Windows, how it presents us with a better environment than UNIX for doing asynchronous I/O, and how it paired nicely with the other work I did on coming up with a way for multiple threads to execute simultaneously across all cores without introducing any speed penalties. I'm particularly interested to hear if the video/slides helped UNIX-centric people gain a better understanding of how Windows does IOCP and why it would be preferable when doing async I/O. The reverse is also true: if you still think single-threaded, non- blocking synchronous I/O via kqueue/epoll is better than the approach afforded by IOCP, I'm interested in hearing why. As crazy as it sounds, my long term goal would be to try and influence Linux and BSD kernels to implement thread-agnostic I/O support such that an IOCP-like mechanism could be exposed; Solaris and AIX already do this via event ports and AIX's verbatim copy of Windows' IOCP API. (There is some promising work already being done on Linux; see recent MegaPipe paper for an example.) Regards, Trent.

2 1

What's the story on Py_FrozenMain?
by Eric Snow Nov. 16, 2013

Nov. 16, 2013

While looking at something unrelated, I happened to peek at Python/frozenmain.c and found Py_FrozenMain(). I kind of get the idea of it, but am curious what motivated the addition and who might be using it. The function is not documented and doesn't have much explanation. I'm guessing that not many are familiar with it (e.g. http://bugs.python.org/issue15893). FWIW the function was added quite a while ago (and hasn't been touched a whole lot since): changeset: 1270:14369a5e61679364deeae9a9a0deedbd593a72e0 branch: legacy-trunk user: Guido van Rossum <guido(a)python.org> date: Thu Apr 01 20:59:32 1993 +0000 summary: Support for frozen scripts; added -i option. -eric

3 3

The pysandbox project is broken
by Victor Stinner Nov. 16, 2013

Nov. 16, 2013

Hi, After having work during 3 years on a pysandbox project to sandbox untrusted code, I now reached a point where I am convinced that pysandbox is broken by design. Different developers tried to convinced me before that pysandbox design is unsafe, but I had to experience it myself to be convineced. It would also be nice to help developers looking for a sandbox for their application. Please tell me if you know sandbox projects for Python so I can redirect users of pysandbox to a safer solution. I already know PyPy sandbox. I would like to share my experience because I know that other developers are using sandboxes in production and that there is a real need for sandboxing. Origin of pysandbox =================== In 2010, a developper called Tav wrote a sandbox called "safelite.py": the sandbox hides sensitive attributes to separate a trusted namespace and an untrusted namespace. Tav challenged Python core developers to break his sandbox and... the sandbox was quickly broken. Even if it was quickly broken, I was conviced that Tav found something interesting and that there is a real need for sandboxing Python. I continued his work by putting more protections on the untrusted namespace. I published pysandbox 1.0 in june 2010. History of pysandbox ==================== pysandbox was used to build an IRC bot on a french Python channel. The bot executed Python code in the sandbox. The bot was mainly used by hackers to test the sandbox to try to find a vulnerability. It was nice to have such IRC bot on an Python help channel. Three month later after the release of pysandbox 1.0, the first vulnerability was found: it was possible to modify the __builtins__ dictionary to hack the sandbox functions and so escape from the sandbox. I had to blacklist common instructions like "dict.pop()" or "del dict[key]" to protect the __builtins__ dictionary. I had prefer to use a custom type for __builtins__ but CPython requires a real dictionary: Python/ceval.c has inlined version of PyDict_GetItem. For your information, I modified CPython 3.3 to accept arbitrary mapping types for __builtins__. Just after this fix, another vulnerability was found: it was still possible to modify __builtins__ using dict.__init__() method. The access to this method was also blocked. Seven months later, new vulnerabilities. The "timeout" protection was removed because it is not effective on CPU intensive functions implemented in C. And to workaround a known bug in CPython crashing the interpreter, the access to the type.__bases__ attribute was also blocked. But this protection has to be disabled on CPython 2.5 because of another CPython bug... The access to func_defaults/__defaults__ attributes of a function was also blocked to protect the sandbox, even if it was not exploitable to escape from the sandbox. Recent events ============== A few weeks ago, a security challenge targeted pysandbox. In less then one day, two vulnerabilities were found. First, the compile() builtin function was used to read line by line of an arbitrary file on the disk using a syntax error: the line is displayed in the traceback. Second, a context manager was used to retrieve a traceback object: from traceback.tb_frame, it was possible to navigate in the frames (using frame.f_back) to retrieve a frame of the trusted namespace, and then use f_globals attribute of the frame to retrieve a global name. Game over. I fixed these two vulnerabilities in pysandbox 1.5.1: compile() is now blocked by default, and the access to traceback.tb_frame, frame.f_back and frame.f_globals has been blocked. I also started to work on a new design of pysandbox (version currently called "pysandbox 1.6", might become pysandbox 2.0 later): run untrusted code in a subprocess to have a safer design. Using a subprocess, it becomes easier to limit the memory usage, setup a real timeout, limit bytes written to stdout, limit the size of data send to and received from the child process, etc. But my main motivation was to not crash the whole application if the untrusted code exploits a know Python bug to crash the process. They are (too) many ways to crash Python using common types and functions... The problem is that after each release it becomes harder to write Python code in the sandbox. For example it becomes very hard to give access to objects from the trusted namespace to the untrusted namespace, because the whole object must be serialized to be passed to the child process. It becomes also harder to debug bugs in the sandboxeded code because the traceback feature doesn't work well in the sandbox. Pysandbox is broken =================== In my opinion, the compile() vulnerabilty is the proof that it is not possible to put a sandbox in CPython. Blocking access to the open() builtin function and the file type constructor are not enough if unrelated functions can give access indirectly to the file system. Having read access on the file system is a critical vulnerability in pysandbox and modifying CPython to not print the source code line in a traceback is also not acceptable. I now agree that putting a sandbox in CPython is the wrong design. There are too many ways to escape the untrusted namespace using the various introspection features of the Python language. To guarantee the safetely of a security product, the code should be carefuly audited and the code to review must be as small as possible. Using pysandbox, the "code" is the whole Python core which is a really huge code base. For example, the Python and Objects directories of Python 3.4 contain more than 126,000 lines of C code. The security of pysandbox is the security of its weakest part. A single bug is enough to escape the whole sandbox. Attackers had original and different ideas like hacking __builtins__, using warnings, context manager, syntax errors, arbitrary bytecode, etc. It is hard to protect the untrusted namespace for all these different Python features. It might be possible to invest a lot of time to put enough protections to protect the untrusted namespace, but it leads to my second point: pysandbox cannot be used in practice. pysandbox cannot be used in practice ==================================== To protect the untrusted namespace, pysandbox installs a lot of different protections. Because of all these protections, it becomes hard to write Python code. Basic features like "del dict[key]" are denied. Passing an object to a sandbox is not possible to sandbox, pysandbox is unable to proxify arbitary objects. For something more complex than evaluating "1+(2*3)", pysandbox cannot be used in practice, because of all these protections. Individual protections cannot be disabled, all protections are required to get a secure sandbox. So what should be used to sandbox Python? ========================================= I developed pysandbox for fun in my free time. But I was contacted by different companies interested to use pysandbox in production on their web application. So I think that there is a real need to execute arbitrary untrusted code. I now think that putting a sandbox directly in Python cannot be secure. To build a secure sandbox, the whole Python process must be put in an external sandbox. There are for example projects using Linux SECCOMP security feature to isolate the Python process. PyPy has a similar design, it implemented something similar to SECCOMP but in a portable way. Please tell me if you know sandbox projects for Python so I can redirect users of pysandbox to a safer solution. I already know PyPy sandbox. Victor

9 13