Mailman 3 November 2013 - Python-Dev

Re: [Python-Dev] The pysandbox project is broken
by Josiah Carlson Nov. 16, 2013

Nov. 16, 2013

Python-dev is for the development of the Python core language, the CPython runtime, and libraries. Your sandbox, despite using and requiring deep knowledge of the runtime, is not developing those things. If you had a series of requests for the language or runtime that would make your job easier, then your thread would be on-topic. I replied off-list because I didn't want to contribute to the off-topic posting, but if posting on-list is required for you to pay attention, so be it. - Josiah On Nov 12, 2013 2:51 PM, "Victor Stinner" <victor.stinner(a)gmail.com> wrote: > 2013/11/12 Josiah Carlson <josiah.carlson(a)gmail.com>: > > I'm replying off-list because I didn't want to bother the other folks in > > python-dev (also, your post might have been better on python-list, but I > > digress). > > I don't understand why you are writing to me directly. I won't reply > if you don't write publicly on python-dev. > > Summary of my email: it's not possible to write a sandbox in CPython. > So it's very specific to CPython internals. I'm not subscribed to > python-list. > > Victor > > > > > Long story short, I think that you are right, and I think that you are > > wrong. > > > > I think that you are right that your current pysandbox implementation is > > likely broken by design. You are starting from a completely working > Python > > runtime, then eliminating/hiding/blocking certain features. This makes > it a > > game of whack-a-mole, for every vulnerability you fix, a new one comes up > > later. The only way to fix this problem is to change your design. > > > > If you wanted to do it right, instead of removing things that are > > vulnerable, start by defining what is safe, and expose only those safe > > things. As an example, you did the right thing by splitting your main and > > subprocess into two pieces. But you don't need to serialize your objects > > from the trusted namespace to give access to the sandbox (that exposes > your > > "trusted" objects to the sandbox in a raw manner, in obvious preparation > for > > exploitation). Instead you would just expose a proxy object whose method > > calls/attribute references are made across your pipe (or socket, or > > whatever) to the trusted controlling process. Is it slower? Yes. Does it > > matter? Not if it keeps the sandbox secure. > > > > Now if you start by saying, "what is allowed?", the most obvious > destination > > is that you will more or less end up writing your own Python runtime. > That's > > not necessarily a bad thing, as if you know that a new runtime is your > > destination, you can look for a viable alternate-language runtime to > begin > > with to short-circuit your work. The best option that I can come up with > at > > this point is Javascript as a destination language, as there are several > > Python to Javascript compilers out there, Javascript is sandboxed by > design, > > and you can arbitrarily eliminate portions of the py->js compilation > > opportunities to eliminate attack vectors (specifically keeping only > those > > that you know won't lead to an attack). > > > > Another option is Lua, though I don't really know of any viable Python to > > Lua transpilers out there. > > > > Good luck with whatever you decide to do. > > > > Regards, > > - Josiah > > > > > > > > On Tue, Nov 12, 2013 at 1:16 PM, Victor Stinner < > victor.stinner(a)gmail.com> > > wrote: > >> > >> Hi, > >> > >> After having work during 3 years on a pysandbox project to sandbox > >> untrusted code, I now reached a point where I am convinced that > >> pysandbox is broken by design. Different developers tried to convinced > >> me before that pysandbox design is unsafe, but I had to experience it > >> myself to be convineced. > >> > >> It would also be nice to help developers looking for a sandbox for > >> their application. Please tell me if you know sandbox projects for > >> Python so I can redirect users of pysandbox to a safer solution. I > >> already know PyPy sandbox. > >> > >> I would like to share my experience because I know that other > >> developers are using sandboxes in production and that there is a real > >> need for sandboxing. > >> > >> > >> Origin of pysandbox > >> =================== > >> > >> In 2010, a developper called Tav wrote a sandbox called "safelite.py": > >> the sandbox hides sensitive attributes to separate a trusted namespace > >> and an untrusted namespace. Tav challenged Python core developers to > >> break his sandbox and... the sandbox was quickly broken. Even if it > >> was quickly broken, I was conviced that Tav found something > >> interesting and that there is a real need for sandboxing Python. I > >> continued his work by putting more protections on the untrusted > >> namespace. I published pysandbox 1.0 in june 2010. > >> > >> > >> History of pysandbox > >> ==================== > >> > >> pysandbox was used to build an IRC bot on a french Python channel. The > >> bot executed Python code in the sandbox. The bot was mainly used by > >> hackers to test the sandbox to try to find a vulnerability. It was > >> nice to have such IRC bot on an Python help channel. > >> > >> Three month later after the release of pysandbox 1.0, the first > >> vulnerability was found: it was possible to modify the __builtins__ > >> dictionary to hack the sandbox functions and so escape from the > >> sandbox. I had to blacklist common instructions like "dict.pop()" or > >> "del dict[key]" to protect the __builtins__ dictionary. I had prefer > >> to use a custom type for __builtins__ but CPython requires a real > >> dictionary: Python/ceval.c has inlined version of PyDict_GetItem. For > >> your information, I modified CPython 3.3 to accept arbitrary mapping > >> types for __builtins__. > >> > >> Just after this fix, another vulnerability was found: it was still > >> possible to modify __builtins__ using dict.__init__() method. The > >> access to this method was also blocked. > >> > >> Seven months later, new vulnerabilities. The "timeout" protection was > >> removed because it is not effective on CPU intensive functions > >> implemented in C. And to workaround a known bug in CPython crashing > >> the interpreter, the access to the type.__bases__ attribute was also > >> blocked. But this protection has to be disabled on CPython 2.5 because > >> of another CPython bug... The access to func_defaults/__defaults__ > >> attributes of a function was also blocked to protect the sandbox, even > >> if it was not exploitable to escape from the sandbox. > >> > >> > >> Recent events > >> ============== > >> > >> A few weeks ago, a security challenge targeted pysandbox. In less then > >> one day, two vulnerabilities were found. First, the compile() builtin > >> function was used to read line by line of an arbitrary file on the > >> disk using a syntax error: the line is displayed in the traceback. > >> Second, a context manager was used to retrieve a traceback object: > >> from traceback.tb_frame, it was possible to navigate in the frames > >> (using frame.f_back) to retrieve a frame of the trusted namespace, and > >> then use f_globals attribute of the frame to retrieve a global name. > >> Game over. > >> > >> I fixed these two vulnerabilities in pysandbox 1.5.1: compile() is now > >> blocked by default, and the access to traceback.tb_frame, frame.f_back > >> and frame.f_globals has been blocked. > >> > >> I also started to work on a new design of pysandbox (version currently > >> called "pysandbox 1.6", might become pysandbox 2.0 later): run > >> untrusted code in a subprocess to have a safer design. Using a > >> subprocess, it becomes easier to limit the memory usage, setup a real > >> timeout, limit bytes written to stdout, limit the size of data send to > >> and received from the child process, etc. But my main motivation was > >> to not crash the whole application if the untrusted code exploits a > >> know Python bug to crash the process. They are (too) many ways to > >> crash Python using common types and functions... > >> > >> The problem is that after each release it becomes harder to write > >> Python code in the sandbox. For example it becomes very hard to give > >> access to objects from the trusted namespace to the untrusted > >> namespace, because the whole object must be serialized to be passed to > >> the child process. It becomes also harder to debug bugs in the > >> sandboxeded code because the traceback feature doesn't work well in > >> the sandbox. > >> > >> > >> Pysandbox is broken > >> =================== > >> > >> In my opinion, the compile() vulnerabilty is the proof that it is not > >> possible to put a sandbox in CPython. Blocking access to the open() > >> builtin function and the file type constructor are not enough if > >> unrelated functions can give access indirectly to the file system. > >> Having read access on the file system is a critical vulnerability in > >> pysandbox and modifying CPython to not print the source code line in a > >> traceback is also not acceptable. > >> > >> I now agree that putting a sandbox in CPython is the wrong design. > >> There are too many ways to escape the untrusted namespace using the > >> various introspection features of the Python language. To guarantee > >> the safetely of a security product, the code should be carefuly > >> audited and the code to review must be as small as possible. Using > >> pysandbox, the "code" is the whole Python core which is a really huge > >> code base. For example, the Python and Objects directories of Python > >> 3.4 contain more than 126,000 lines of C code. > >> > >> The security of pysandbox is the security of its weakest part. A > >> single bug is enough to escape the whole sandbox. > >> > >> Attackers had original and different ideas like hacking __builtins__, > >> using warnings, context manager, syntax errors, arbitrary bytecode, > >> etc. It is hard to protect the untrusted namespace for all these > >> different Python features. > >> > >> It might be possible to invest a lot of time to put enough protections > >> to protect the untrusted namespace, but it leads to my second point: > >> pysandbox cannot be used in practice. > >> > >> > >> pysandbox cannot be used in practice > >> ==================================== > >> > >> To protect the untrusted namespace, pysandbox installs a lot of > >> different protections. Because of all these protections, it becomes > >> hard to write Python code. Basic features like "del dict[key]" are > >> denied. Passing an object to a sandbox is not possible to sandbox, > >> pysandbox is unable to proxify arbitary objects. > >> > >> For something more complex than evaluating "1+(2*3)", pysandbox cannot > >> be used in practice, because of all these protections. Individual > >> protections cannot be disabled, all protections are required to get a > >> secure sandbox. > >> > >> > >> So what should be used to sandbox Python? > >> ========================================= > >> > >> I developed pysandbox for fun in my free time. But I was contacted by > >> different companies interested to use pysandbox in production on their > >> web application. So I think that there is a real need to execute > >> arbitrary untrusted code. > >> > >> I now think that putting a sandbox directly in Python cannot be > >> secure. To build a secure sandbox, the whole Python process must be > >> put in an external sandbox. There are for example projects using Linux > >> SECCOMP security feature to isolate the Python process. > >> > >> PyPy has a similar design, it implemented something similar to SECCOMP > >> but in a portable way. > >> > >> Please tell me if you know sandbox projects for Python so I can > >> redirect users of pysandbox to a safer solution. I already know PyPy > >> sandbox. > >> > >> > >> Victor > >> _______________________________________________ > >> Python-Dev mailing list > >> Python-Dev(a)python.org > >> https://mail.python.org/mailman/listinfo/python-dev > >> Unsubscribe: > >> > https://mail.python.org/mailman/options/python-dev/josiah.carlson%40gmail.c… > > > > >

16 26

Reminder: Python 3.4 feature freeze in a week
by Larry Hastings Nov. 16, 2013

Nov. 16, 2013

We're on schedule to tag Python 3.4 Beta 1 next Saturday. And when that happens we go into "feature freeze" on Python trunk; no new features will be accepted in trunk until we branch the 3.4 release branch next February. Time to get those checkins in folks. Last call, everyone, //arry/

1 0

Re: [Python-Dev] cpython: Issue #19544 and Issue #6516: Restore support for --user and --group parameters
by Christian Heimes Nov. 15, 2013

Nov. 15, 2013

Am 15.11.2013 19:07, schrieb jason.coombs: > http://hg.python.org/cpython/rev/b9c9c4b2effe > changeset: 87119:b9c9c4b2effe > user: Andrew Kuchling <amk(a)amk.ca> > date: Fri Nov 15 13:01:52 2013 -0500 > summary: > Issue #19544 and Issue #6516: Restore support for --user and --group parameters to sdist command as found in Python 2.7 and originally slated for Python 3.2 but accidentally rolled back as part of the distutils2 rollback. Closes Issue #6516. > Your commit has broken the build: ./python -E -S -m sysconfig --generate-posix-vars Could not find platform dependent libraries <exec_prefix> Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>] Traceback (most recent call last): File "./setup.py", line 11, in <module> from distutils.core import Extension, setup File "/home/heimes/dev/python/cpython/Lib/distutils/core.py", line 18, in <module> from distutils.cmd import Command File "/home/heimes/dev/python/cpython/Lib/distutils/cmd.py", line 9, in <module> from distutils import util, dir_util, file_util, archive_util, dep_util File "/home/heimes/dev/python/cpython/Lib/distutils/archive_util.py", line 27, in <module> from grp import getgrnam ImportError: No module named 'grp' The grp module is built later by setup.py.

1 0

Finding overlapping matches with re assertions: bug or feature?
by Tim Peters Nov. 15, 2013

Nov. 15, 2013

I was surprised to find that "this works": if you want to find all _overlapping_ matches for a regexp R, wrap it in (?=(R)) and feed it to (say) finditer. Here's a very simple example, finding all overlapping occurrences of "xx": pat = re.compile("(?=(xx))") for it in pat.finditer("xxxx"): print(it.span(1)) That displays: (0, 2) (1, 3) (2, 4) Is that a feature? Or an accident? It's very surprising to find a non-empty match inside an empty match (the outermost lookahead assertion). If it's intended behavior, it's just in time for the holiday season; e.g., to generate ASCII art for half an upside-down Christmas tree: pat = re.compile("(?=(x+))") for it in pat.finditer("xxxxxxxxxx"): print(it.group(1))

2 2

Summary of Python tracker Issues
by Python tracker Nov. 15, 2013

Nov. 15, 2013

ACTIVITY SUMMARY (2013-11-08 - 2013-11-15) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 4265 (+38) closed 27119 (+45) total 31384 (+83) Open issues with patches: 1979 Issues opened (74) ================== #1180: Option to ignore or substitute ~/.pydistutils.cfg http://bugs.python.org/issue1180 reopened by jason.coombs #6466: duplicate get_version() code between cygwinccompiler and emxcc http://bugs.python.org/issue6466 reopened by jason.coombs #6516: reset owner/group to root for distutils tarballs http://bugs.python.org/issue6516 reopened by jason.coombs #16286: Use hash if available to optimize a==b and a!=b for bytes and http://bugs.python.org/issue16286 reopened by haypo #17354: TypeError when running setup.py upload --show-response http://bugs.python.org/issue17354 reopened by berker.peksag #19466: Clear state of threads earlier in Python shutdown http://bugs.python.org/issue19466 reopened by haypo #19531: Loading -OO bytecode files if -O was requested can lead to pro http://bugs.python.org/issue19531 opened by Sworddragon #19532: compileall -f doesn't force to write bytecode files http://bugs.python.org/issue19532 opened by Sworddragon #19533: Unloading docstrings from memory if -OO is given http://bugs.python.org/issue19533 opened by Sworddragon #19534: normalize() in locale.py fails for sr_RS.UTF-8@latin http://bugs.python.org/issue19534 opened by mfabian #19535: Test failures with -OO http://bugs.python.org/issue19535 opened by serhiy.storchaka #19536: MatchObject should offer __getitem__() http://bugs.python.org/issue19536 opened by brandon-rhodes #19537: Fix misalignment in fastsearch_memchr_1char http://bugs.python.org/issue19537 opened by schwab #19538: Changed function prototypes in the PEP 384 stable ABI http://bugs.python.org/issue19538 opened by theller #19539: The 'raw_unicode_escape' codec buggy + not appropriate for Pyt http://bugs.python.org/issue19539 opened by zuo #19541: ast.dump(indent=True) prettyprinting http://bugs.python.org/issue19541 opened by techtonik #19542: WeakValueDictionary bug in setdefault()&pop() http://bugs.python.org/issue19542 opened by arigo #19543: Add -3 warnings for codec convenience method changes http://bugs.python.org/issue19543 opened by ncoghlan #19544: Port distutils as found in Python 2.7 to Python 3.x. http://bugs.python.org/issue19544 opened by jason.coombs #19545: time.strptime exception context http://bugs.python.org/issue19545 opened by Claudiu.Popa #19546: configparser leaks implementation detail http://bugs.python.org/issue19546 opened by Claudiu.Popa #19547: HTTPS proxy support missing without warning http://bugs.python.org/issue19547 opened by 02strich #19548: 'codecs' module docs improvements http://bugs.python.org/issue19548 opened by zuo #19549: PKG-INFO is created with CRLF on Windows http://bugs.python.org/issue19549 opened by techtonik #19550: PEP 453: Windows installer integration http://bugs.python.org/issue19550 opened by ncoghlan #19551: PEP 453: Mac OS X installer integration http://bugs.python.org/issue19551 opened by ncoghlan #19552: PEP 453: venv module and pyvenv integration http://bugs.python.org/issue19552 opened by ncoghlan #19553: PEP 453: "make install" and "make altinstall" integration http://bugs.python.org/issue19553 opened by ncoghlan #19554: Enable all freebsd* host platforms http://bugs.python.org/issue19554 opened by wg #19555: "SO" config var not getting set http://bugs.python.org/issue19555 opened by Marc.Abramowitz #19557: ast - docs for every node type are missing http://bugs.python.org/issue19557 opened by techtonik #19558: Provide Tcl/Tk linkage information for extension module builds http://bugs.python.org/issue19558 opened by ned.deily #19561: request to reopen Issue837046 - pyport.h redeclares gethostnam http://bugs.python.org/issue19561 opened by risto3 #19562: Added description for assert statement http://bugs.python.org/issue19562 opened by thatiparthy #19563: Changing barry's email to barry(a)python.org http://bugs.python.org/issue19563 opened by thatiparthy #19564: test_multiprocessing_spawn hangs http://bugs.python.org/issue19564 opened by haypo #19565: test_multiprocessing_spawn: RuntimeError and assertion error o http://bugs.python.org/issue19565 opened by haypo #19566: ERROR: test_close (test.test_asyncio.test_unix_events.FastChil http://bugs.python.org/issue19566 opened by haypo #19568: bytearray_setslice_linear() leaves the bytearray in an inconsi http://bugs.python.org/issue19568 opened by haypo #19569: Use __attribute__((deprecated)) to warn usage of deprecated fu http://bugs.python.org/issue19569 opened by haypo #19570: distutils' Command.ensure_dirname fails on Unicode http://bugs.python.org/issue19570 opened by saschpe #19572: Report more silently skipped tests as skipped http://bugs.python.org/issue19572 opened by zach.ware #19573: Fix the docstring of inspect.Parameter and the implementation http://bugs.python.org/issue19573 opened by Antony.Lee #19575: subprocess.Popen with multiple threads: Redirected stdout/stde http://bugs.python.org/issue19575 opened by Bernt.Røskar.Brenna #19576: "Non-Python created threads" documentation doesn't mention PyE http://bugs.python.org/issue19576 opened by haypo #19577: memoryview bind (the opposite of release) http://bugs.python.org/issue19577 opened by mpb #19578: del list[a:b:c] doesn't handle correctly list_resize() failure http://bugs.python.org/issue19578 opened by haypo #19581: PyUnicodeWriter: change the overallocation factor for Windows http://bugs.python.org/issue19581 opened by haypo #19582: Tkinter is not working with Py_SetPath http://bugs.python.org/issue19582 opened by Debarshi.Goswami #19584: IDLE fails - Python V2.7.6 - 64b on Win7 64b http://bugs.python.org/issue19584 opened by bob7wake #19585: Frame annotation http://bugs.python.org/issue19585 opened by doerwalter #19586: distutils: Remove assertEquals and assert_ deprecation warning http://bugs.python.org/issue19586 opened by Gregory.Salvan #19587: Remove empty tests in test_bytes.FixedStringTest http://bugs.python.org/issue19587 opened by zach.ware #19588: Silently skipped test in test_random http://bugs.python.org/issue19588 opened by zach.ware #19590: Use specific asserts in test_email http://bugs.python.org/issue19590 opened by serhiy.storchaka #19591: Use specific asserts in ctype tests http://bugs.python.org/issue19591 opened by serhiy.storchaka #19593: Use specific asserts in importlib tests http://bugs.python.org/issue19593 opened by serhiy.storchaka #19594: Use specific asserts in unittest tests http://bugs.python.org/issue19594 opened by serhiy.storchaka #19595: Silently skipped test in test_winsound http://bugs.python.org/issue19595 opened by zach.ware #19596: Silently skipped tests in test_importlib http://bugs.python.org/issue19596 opened by zach.ware #19597: Add decorator for tests not yet implemented http://bugs.python.org/issue19597 opened by zach.ware #19599: Failure of test_async_timeout() of test_multiprocessing_spawn: http://bugs.python.org/issue19599 opened by haypo #19600: Use specific asserts in distutils tests http://bugs.python.org/issue19600 opened by serhiy.storchaka #19601: Use specific asserts in sqlite3 tests http://bugs.python.org/issue19601 opened by serhiy.storchaka #19602: Use specific asserts in tkinter tests http://bugs.python.org/issue19602 opened by serhiy.storchaka #19603: Use specific asserts in test_decr http://bugs.python.org/issue19603 opened by serhiy.storchaka #19604: Use specific asserts in array tests http://bugs.python.org/issue19604 opened by serhiy.storchaka #19605: Use specific asserts in datetime tests http://bugs.python.org/issue19605 opened by serhiy.storchaka #19606: Use specific asserts in http.cookiejar tests http://bugs.python.org/issue19606 opened by serhiy.storchaka #19607: Use specific asserts in weakref tests http://bugs.python.org/issue19607 opened by serhiy.storchaka #19608: devguide needs pictures http://bugs.python.org/issue19608 opened by techtonik #19610: TypeError in distutils.command.upload http://bugs.python.org/issue19610 opened by berker.peksag #19611: inspect.getcallargs doesn't properly interpret set comprehensi http://bugs.python.org/issue19611 opened by nedbat #19612: test_subprocess: sporadic failure of test_communicate_epipe() http://bugs.python.org/issue19612 opened by haypo Most recent 15 issues with no replies (15) ========================================== #19612: test_subprocess: sporadic failure of test_communicate_epipe() http://bugs.python.org/issue19612 #19607: Use specific asserts in weakref tests http://bugs.python.org/issue19607 #19606: Use specific asserts in http.cookiejar tests http://bugs.python.org/issue19606 #19605: Use specific asserts in datetime tests http://bugs.python.org/issue19605 #19604: Use specific asserts in array tests http://bugs.python.org/issue19604 #19603: Use specific asserts in test_decr http://bugs.python.org/issue19603 #19602: Use specific asserts in tkinter tests http://bugs.python.org/issue19602 #19601: Use specific asserts in sqlite3 tests http://bugs.python.org/issue19601 #19600: Use specific asserts in distutils tests http://bugs.python.org/issue19600 #19599: Failure of test_async_timeout() of test_multiprocessing_spawn: http://bugs.python.org/issue19599 #19594: Use specific asserts in unittest tests http://bugs.python.org/issue19594 #19591: Use specific asserts in ctype tests http://bugs.python.org/issue19591 #19584: IDLE fails - Python V2.7.6 - 64b on Win7 64b http://bugs.python.org/issue19584 #19578: del list[a:b:c] doesn't handle correctly list_resize() failure http://bugs.python.org/issue19578 #19573: Fix the docstring of inspect.Parameter and the implementation http://bugs.python.org/issue19573 Most recent 15 issues waiting for review (15) ============================================= #19610: TypeError in distutils.command.upload http://bugs.python.org/issue19610 #19607: Use specific asserts in weakref tests http://bugs.python.org/issue19607 #19606: Use specific asserts in http.cookiejar tests http://bugs.python.org/issue19606 #19605: Use specific asserts in datetime tests http://bugs.python.org/issue19605 #19604: Use specific asserts in array tests http://bugs.python.org/issue19604 #19603: Use specific asserts in test_decr http://bugs.python.org/issue19603 #19602: Use specific asserts in tkinter tests http://bugs.python.org/issue19602 #19601: Use specific asserts in sqlite3 tests http://bugs.python.org/issue19601 #19600: Use specific asserts in distutils tests http://bugs.python.org/issue19600 #19597: Add decorator for tests not yet implemented http://bugs.python.org/issue19597 #19596: Silently skipped tests in test_importlib http://bugs.python.org/issue19596 #19594: Use specific asserts in unittest tests http://bugs.python.org/issue19594 #19593: Use specific asserts in importlib tests http://bugs.python.org/issue19593 #19591: Use specific asserts in ctype tests http://bugs.python.org/issue19591 #19590: Use specific asserts in test_email http://bugs.python.org/issue19590 Top 10 most discussed issues (10) ================================= #18864: Implementation for PEP 451 (importlib.machinery.ModuleSpec) http://bugs.python.org/issue18864 25 msgs #19575: subprocess.Popen with multiple threads: Redirected stdout/stde http://bugs.python.org/issue19575 16 msgs #19566: ERROR: test_close (test.test_asyncio.test_unix_events.FastChil http://bugs.python.org/issue19566 12 msgs #5815: locale.getdefaultlocale() missing corner case http://bugs.python.org/issue5815 10 msgs #19544: Port distutils as found in Python 2.7 to Python 3.x. http://bugs.python.org/issue19544 9 msgs #19568: bytearray_setslice_linear() leaves the bytearray in an inconsi http://bugs.python.org/issue19568 9 msgs #18861: Problems with recursive automatic exception chaining http://bugs.python.org/issue18861 7 msgs #19565: test_multiprocessing_spawn: RuntimeError and assertion error o http://bugs.python.org/issue19565 7 msgs #19533: Unloading docstrings from memory if -OO is given http://bugs.python.org/issue19533 6 msgs #19534: normalize() in locale.py fails for sr_RS.UTF-8@latin http://bugs.python.org/issue19534 6 msgs Issues closed (43) ================== #6683: smtplib authentication - try all mechanisms http://bugs.python.org/issue6683 closed by python-dev #9878: Avoid parsing pyconfig.h and Makefile by autogenerating extens http://bugs.python.org/issue9878 closed by barry #9922: subprocess.getstatusoutput can fail with utf8 UnicodeDecodeErr http://bugs.python.org/issue9922 closed by tim.golden #10262: Add --soabi option to `configure` http://bugs.python.org/issue10262 closed by barry #10734: test_ttk test_heading_callback fails with newer Tk 8.5 http://bugs.python.org/issue10734 closed by serhiy.storchaka #11473: upload command no longer accepts repository by section name http://bugs.python.org/issue11473 closed by jason.coombs #11513: chained exception/incorrect exception from tarfile.open on a n http://bugs.python.org/issue11513 closed by barry #11677: make test has horrendous performance on an ecryptfs http://bugs.python.org/issue11677 closed by barry #12828: xml.dom.minicompat is not documented http://bugs.python.org/issue12828 closed by python-dev #13674: crash in datetime.strftime http://bugs.python.org/issue13674 closed by tim.golden #15440: multiprocess fails to re-raise exception which has mandatory a http://bugs.python.org/issue15440 closed by sbt #16685: audioop functions shouldn't accept strings http://bugs.python.org/issue16685 closed by serhiy.storchaka #16803: Make test_importlib run tests under both _frozen_importlib and http://bugs.python.org/issue16803 closed by brett.cannon #17823: 2to3 fixers for missing codecs http://bugs.python.org/issue17823 closed by ncoghlan #17828: More informative error handling when encoding and decoding http://bugs.python.org/issue17828 closed by python-dev #17874: ProcessPoolExecutor in interactive shell doesn't work in Windo http://bugs.python.org/issue17874 closed by sbt #18923: Use the new selectors module in the subprocess module http://bugs.python.org/issue18923 closed by neologix #19227: test_multiprocessing_xxx hangs under Gentoo buildbots http://bugs.python.org/issue19227 closed by benjamin.peterson #19249: Enumeration.__eq__ http://bugs.python.org/issue19249 closed by ethan.furman #19261: Add support for 24-bit in the sunau module http://bugs.python.org/issue19261 closed by serhiy.storchaka #19406: PEP 453: add the ensurepip module http://bugs.python.org/issue19406 closed by python-dev #19429: OSError constructor does not handle errors correctly http://bugs.python.org/issue19429 closed by haypo #19440: Clean up test_capi http://bugs.python.org/issue19440 closed by zach.ware #19490: Problem installing matplotlib 1.3.1 with Python 2.7.6rc1 and 3 http://bugs.python.org/issue19490 closed by ned.deily #19499: "import this" is cached in sys.modules http://bugs.python.org/issue19499 closed by barry #19515: Share duplicated _Py_IDENTIFIER identifiers in C code http://bugs.python.org/issue19515 closed by haypo #19530: cross thread shutdown of UDP socket exhibits unexpected behavi http://bugs.python.org/issue19530 closed by neologix #19540: PEP339: Fix link to Zephyr ASDL paper http://bugs.python.org/issue19540 closed by benjamin.peterson #19556: A missing link to Python-2.7.6.tar.bz2 in Download page. http://bugs.python.org/issue19556 closed by ned.deily #19559: Interactive interpreter crashes after any two commands http://bugs.python.org/issue19559 closed by ezio.melotti #19560: PEP 8 operator precedence across parens http://bugs.python.org/issue19560 closed by ezio.melotti #19567: mimetypes.init() raise unhandled excption in windows http://bugs.python.org/issue19567 closed by r.david.murray #19571: urlparse.parse_qs with empty string http://bugs.python.org/issue19571 closed by r.david.murray #19574: Negative integer division error http://bugs.python.org/issue19574 closed by r.david.murray #19579: test_asyncio: test__run_once timings should be relaxed http://bugs.python.org/issue19579 closed by haypo #19580: Got resource warning when running test_base_events (test_async http://bugs.python.org/issue19580 closed by haypo #19583: time.strftime fails to use %:z time formatter of the underlyin http://bugs.python.org/issue19583 closed by deronnax #19589: Use specific asserts in test_asyncio http://bugs.python.org/issue19589 closed by serhiy.storchaka #19592: Use specific asserts in lib2to3 tests http://bugs.python.org/issue19592 closed by serhiy.storchaka #19598: Timeout in test_popen() of test_asyncio.test_windows_utils.Pop http://bugs.python.org/issue19598 closed by haypo #19609: Codec exception chaining shouldn't cover the initial codec loo http://bugs.python.org/issue19609 closed by ncoghlan #1097797: Encoding for Code Page 273 used by EBCDIC Germany Austria http://bugs.python.org/issue1097797 closed by akuchling #1109602: Need some setup.py sanity http://bugs.python.org/issue1109602 closed by ned.deily

1 0

"*zip-bomb" via codecs
by Serhiy Storchaka Nov. 14, 2013

Nov. 14, 2013

It is possible make a DDoS using the fact that codecs registry provides access to gzip and bzip2 decompressor. Someone can send HTTP request or email message with specified "gzip_codec" or "bzip2_codec" as content encoding and great well compressed gzip- or bzip2-file as a content. Naive server will use the bytes.decode() method to decompress a content. It is possible to create small compressed files which require very much time and memory to decompress. Of course bytes.decode() will fail becouse decoder returns bytes instead string, but time and memory are already wasted. I have no working example but I'm sure it will be easy to create it. I suspect many services will be vulnerable for this attack. Simple solution for this problem is check any foreign encoding that it is conteined in a special set of safe encodings. But every program should check it explicitly. For more general solution bytes.decode() should reject encoding *before* starting of decoding. I.e. either all bytes->str decoders should be registered in separated registry, or all codecs should have additional attributes which determines input and output type.

1 0

Re: [Python-Dev] peps: PEP 456: add some of the new implementation details to the PEP's text
by Antoine Pitrou Nov. 14, 2013

Nov. 14, 2013

On Wed, 13 Nov 2013 23:33:02 +0100 (CET) christian.heimes <python-checkins(a)python.org> wrote: > > > +Small string optimization > +========================= > + > +Hash functions like SipHash24 have a costly initialization and finalization > +code that can dominate speed of the algorithm for very short strings. On the > +other hand Python calculates the hash value of short strings quite often. A > +simple and fast function for especially for hashing of small strings can make > +a measurably impact on performance. For example these measurements were taken > +during a run of Python's regression tests. Additional measurements of other > +code have shown a similar distribution. Well, the text above talks about a "measurably (typo?) impact on performance", but you aren't giving any performance numbers, which doesn't help the reader of those lines. Regards Antoine.

2 1

Re: [Python-Dev] [Python-checkins] Daily reference leaks (784a02ec2a26): sum=522
by Benjamin Peterson Nov. 14, 2013

Nov. 14, 2013

2013/11/14 Antoine Pitrou <solipsis(a)pitrou.net>: > On Thu, 14 Nov 2013 22:01:32 +1000 > Nick Coghlan <ncoghlan(a)gmail.com> wrote: >> On 14 Nov 2013 21:58, "Nick Coghlan" <ncoghlan(a)gmail.com> wrote: >> > >> > >> > On 14 Nov 2013 13:52, <solipsis(a)pitrou.net> wrote: >> > > >> > > results for 784a02ec2a26 on branch "default" >> > > -------------------------------------------- >> > > >> > > test_codeccallbacks leaked [40, 40, 40] references, sum=120 >> > > test_codeccallbacks leaked [40, 40, 40] memory blocks, sum=120 >> > > test_codecs leaked [38, 38, 38] references, sum=114 >> > > test_codecs leaked [24, 24, 24] memory blocks, sum=72 >> > > test_email leaked [16, 16, 16] references, sum=48 >> > > test_email leaked [16, 16, 16] memory blocks, sum=48 >> > >> > Hmm, it appears I have a reference leak somewhere. >> >> Ah, Benjamin fixed it already. Thanks! :) > > The reference leak task has been running for quite some time on my > personal machine and I believe it has proven useful. I have no problem > continuing running it on the same machine (which is mostly sitting idle > anyway), but maybe it should rather be hosted on our CI infrastructure? > Any suggestions? Thank you very much for running that, btw. I'm sure we would have released a lot of horribly leaking stuff without it. > > (the script is quite rough with hardcoded stuff, but beating it into > better shape could be a nice target for first-time contributors) Perhaps someone can figure out how to run it on one of the the buildbots? -- Regards, Benjamin

1 0

Re: [Python-Dev] [Python-checkins] Daily reference leaks (784a02ec2a26): sum=522
by Nick Coghlan Nov. 14, 2013

Nov. 14, 2013

On 14 Nov 2013 13:52, <solipsis(a)pitrou.net> wrote: > > results for 784a02ec2a26 on branch "default" > -------------------------------------------- > > test_codeccallbacks leaked [40, 40, 40] references, sum=120 > test_codeccallbacks leaked [40, 40, 40] memory blocks, sum=120 > test_codecs leaked [38, 38, 38] references, sum=114 > test_codecs leaked [24, 24, 24] memory blocks, sum=72 > test_email leaked [16, 16, 16] references, sum=48 > test_email leaked [16, 16, 16] memory blocks, sum=48 Hmm, it appears I have a reference leak somewhere. Cheers, Nick. > > > Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogx2QIb_', '-x'] > _______________________________________________ > Python-checkins mailing list > Python-checkins(a)python.org > https://mail.python.org/mailman/listinfo/python-checkins

1 1

Re: [Python-Dev] [Python-checkins] cpython: Issue #17828: va_start() must be accompanied by va_end()
by Nick Coghlan Nov. 13, 2013

Nov. 13, 2013

On 14 Nov 2013 10:40, "christian.heimes" <python-checkins(a)python.org> wrote: > > http://hg.python.org/cpython/rev/99ba1772c469 > changeset: 87089:99ba1772c469 > user: Christian Heimes <christian(a)cheimes.de> > date: Thu Nov 14 01:39:35 2013 +0100 > summary: > Issue #17828: va_start() must be accompanied by va_end() > CID 1128793: Missing varargs init or cleanup (VARARGS) Today I learned... :) Thanks! Cheers, Nick. > > files: > Objects/exceptions.c | 13 +++++++------ > 1 files changed, 7 insertions(+), 6 deletions(-) > > > diff --git a/Objects/exceptions.c b/Objects/exceptions.c > --- a/Objects/exceptions.c > +++ b/Objects/exceptions.c > @@ -2632,12 +2632,6 @@ > PyObject *new_exc, *new_val, *new_tb; > va_list vargs; > > -#ifdef HAVE_STDARG_PROTOTYPES > - va_start(vargs, format); > -#else > - va_start(vargs); > -#endif > - > PyErr_Fetch(&exc, &val, &tb); > caught_type = (PyTypeObject *) exc; > /* Ensure type info indicates no extra state is stored at the C level */ > @@ -2690,7 +2684,14 @@ > * types as well, but that's quite a bit trickier due to the extra > * state potentially stored on OSError instances. > */ > + > +#ifdef HAVE_STDARG_PROTOTYPES > + va_start(vargs, format); > +#else > + va_start(vargs); > +#endif > msg_prefix = PyUnicode_FromFormatV(format, vargs); > + va_end(vargs); > if (msg_prefix == NULL) > return NULL; > > > -- > Repository URL: http://hg.python.org/cpython > > _______________________________________________ > Python-checkins mailing list > Python-checkins(a)python.org > https://mail.python.org/mailman/listinfo/python-checkins >

1 0