Mailman 3 February 2010 - Python-Dev

embedding Python interpreter in non-console windows application
by stephen Feb. 17, 2010

Feb. 17, 2010

Hello, THE PROBLEM: I am having a problem that I have seen asked quite a bit on the web, with little to no follow up. The problem is essentially this. When embedding (LoadLibraryA()) the python interpreter dll in a non-windows application the developer must first create a console for python to do output/input with. I properly initialize the CRT and AllocConsole() to do this. I then GetSTDHandle() for stdin and stdout accordingly and open those handles with the requisite flags "read" for STDIN and "write" for stdout. This all works great and is then verified and tested to work by printf() and fgets(). This issue however happens when attempting to PyRun_InteractiveLoop() and PyRun_SimpleString(). A PyRun_SimpleString("print 'test'") displays nothing in my freshly allocated console window. Similarly a PyRun_InteractiveLoop(stdin, NULL); yields nothing either even though the line printf("testing"); directly ahead of it works just fine. Does anyone have insight on how I can make this work with the freshly allocated console's stdin/stdout/stderr? SPECULATION: That is the question, so now on to the speculation. I suspect that something in the python runtime doesn't "get handles" correctly for STDIN and STDOUT upon initialization. I have perused the source code to find out exactly how this is done and I suspect that it starts in PyInitializeEx with calls to PySys_GetObject("stdin") and "stdout" accordingly. However I don't actually see where this translates into the Python runtime checking with the C-runtime for the "real" handles to STDIN and STDOUT. I dont ever see the Python runtime "ask the system" where his handles to STDIN and STDOUT are. SUBSEQUENT QUESTION: Is there anything I can do to initialize the Python interpreter (running as a dll) pointing him at his appropriate STDIN and STDOUT handles?

3 2

Re: [Python-Dev] __file__ is not always an absolute path
by exarkun＠twistedmatrix.com Feb. 17, 2010

Feb. 17, 2010

On 6 Feb, 11:53 pm, guido(a)python.org wrote: >On Sat, Feb 6, 2010 at 3:22 PM, <exarkun(a)twistedmatrix.com> wrote: >>On 10:29 pm, guido(a)python.org wrote: >>> >>>[snip] >>> >>>I haven't tried to repro this particular example, but the reason is >>>that we don't want to have to call getpwd() on every import nor do we >>>want to have some kind of in-process variable to cache the current >>>directory. (getpwd() is relatively slow and can sometimes fail >>>outright, and trying to cache it has a certain risk of being wrong.) >> >>Assuming you mean os.getcwd(): > >Yes. >>exarkun@boson:~$ python -m timeit -s 'def f(): pass' 'f()' >>10000000 loops, best of 3: 0.132 usec per loop >>exarkun@boson:~$ python -m timeit -s 'from os import getcwd' >>'getcwd()' >>1000000 loops, best of 3: 1.02 usec per loop >>exarkun@boson:~$ >>So it's about 7x more expensive than a no-op function call. I'd call >>this >>pretty quick. Compared to everything else that happens during an >>import, >>I'm not convinced this wouldn't be lost in the noise. I think it's at >>least >>worth implementing and measuring. > >But it's a system call, and its speed depends on a lot more than the >speed of a simple function call. It depends on the OS kernel, possibly >on the filesystem, and so on. Do you know of a case where it's actually slow? If not, how convincing should this argument really be? Perhaps we can measure it on a few platforms before passing judgement. For reference, my numbers are from Linux 2.6.31 and my filesystem (though I don't think it really matters) is ext3. I have eglibc 2.10.1 compiled by gcc version 4.4.1. >Also "os.getcwd()" abstracts away >various platform details that the C import code would have to >replicate. That logic can all be hidden behind a C API which os.getcwd() can then be implemented in terms of. There's no reason for it to be any harder to invoke from C than it is from Python. >Really, the approach of preprocessing sys.path makes much >more sense. If an app wants sys.path[0] to be an absolute path too >they can modify it themselves. That may turn out to be the less expensive approach. I'm not sure in what other ways it is the approach that makes much more sense. Quite the opposite: centralizing the responsibility for normalizing this value makes a lot of sense if you consider things like reducing code duplication and, in turn, removing the possibility for bugs. Adding better documentation for __file__ is another task which I think is worth undertaking, regardless of whether any change is made to how its value is computed. At the moment, the two or three sentences about it in PEP 302 are all I've been able to find, and they don't really get the job done. Jean-Paul

3 2

math.hypot, complex.__abs__, and documentation
by David DiCato Feb. 17, 2010

Feb. 17, 2010

I have a minor concern about certain corner cases with math.hypot and complex.__abs__, namely when one component is infinite and one is not a number. If we execute the following code: import math inf = float('inf') nan = float('nan') print math.hypot(inf, nan) print abs(complex(nan, inf)) ... then we see that 'inf' is printed in both cases. The standard library tests (for example, test_cmath.py:test_abs()) seem to test for this behavior as well, and FWIW, I personally agree with this convention. However, the math module's documentation for both 2.6 and 3.1 states, "All functions return a quiet NaN if at least one of the args is NaN." math.pow(1.0, nan) is another such exception to the rule. Perhaps the documentation should be updated to reflect this. Thanks, - David

4 9

Release timer for Core Development page
by anatoly techtonik Feb. 16, 2010

Feb. 16, 2010

I've got another idea of having a release timer on http://python.org/dev/ page together with link to generated release calendar. It will help to automatically monitor deadlines for feature fixes in alpha releases without manually monitoring this mailing list. There is already a navigation box on the right side where this information fits like a glove. Does anybody else find this feature useful for Python development? -- anatoly t.

2 1

setUpClass and setUpModule in unittest
by Michael Foord Feb. 16, 2010

Feb. 16, 2010

Hello all, The next 'big' change to unittest will (may?) be the introduction of class and module level setUp and tearDown. This was discussed on Python-ideas and Guido supported them. They can be useful but are also very easy to abuse (too much shared state, monolithic test classes and modules). Several authors of other Python testing frameworks spoke up *against* them, but several *users* of test frameworks spoke up in favour of them. ;-) I'm pretty sure I can introduce setUpClass and setUpModule without breaking compatibility with existing unittest extensions or backwards compatibility issues - with the possible exception of test sorting. Where you have a class level setUp (for example creating a database connection) you don't want the tearDown executed a *long* time after the setUp. In the presence of class or module level setUp /tearDown (but only if they are used) I would expect test sorting to only sort within the class or module [1]. I will introduce the setUp and tearDown as new 'tests' - so failures are reported separately, and all tests in the class / module will have an explicit skip in the event of a setUp failure. A *better* (more general) solution for sharing and managing resources between tests is to use something like TestResources by Robert Collins. http://pypi.python.org/pypi/testresources/ A minimal example of using test resources shows very little boilerplate overhead from what setUpClass (etc) would need, and with the addition of some helper functions could be almost no overhead. I've challenged Robert that if he can provide examples of using Test Resources to meet the class and module level use-cases then I would support bringing Test Resources into the standard library as part of unittest (modulo licensing issues which he is happy to work on). I'm not sure what response I expect from this email, and neither option will be implemented without further discussion - possibly at the PyCon sprints - but I thought I would make it clear what the possible directions are. All the best, Michael Foord [1] I *could* allow sorting of all tests within a module, inserting the setUpClass / tearDownClass in the right place after the sort. It would probably be better to group tests per class anyway and in fact the existing suite sorting support may do this already (in which case it isn't an issue) - I haven't looked into the implementation. -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (”BOGUS AGREEMENTS”) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (”BOGUS AGREEMENTS”) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.

15 73

3.1.2
by Benjamin Peterson Feb. 15, 2010

Feb. 15, 2010

It's about time for another 3.1 bug fix release. I propose this schedule: March 6: Release Candidate (same day as 2.7a4) March 20: 3.1.2 Final release -- Regards, Benjamin

5 5

pysandsox project
by Victor Stinner Feb. 15, 2010

Feb. 15, 2010

Hi, I'm working on a new sandbox project. The goal is to create an empty namespace and write strict rules for the interaction with the existing namespace (full featured Python namespace). By default, you cannot read a file, use print, import a module or exit Python. But you can enable some functions by using config "features". Example: "regex" feature allows you to import the re module which will contain a subset of the real re module, just enough to match a regex. To protect the sandbox namespace, some attributes are "hidden": function closure and globals, frame locals and type subclasses. __builtins__ is also replaced by a read-only dictionary. Objects are not directly injected in the sandbox namespace: a proxy is used to get a read-only view of the object. pysandbox is based on safelite.py, project written by tav one year ago (search tav in python-dev archive, February 2009). I tested RestrictedPython, but the approach is different (rewrite bytecode) and the project is not maintained since 3 or 4 years (only minor updates on the documentation or the copyright header). pysandbox is different than RestrictedPython because it blocks everything by default and has simply config options to enable a set of features. pysandbox is different than safelite.py because it contains unit tests (ensure that blocked features are really blocked). Only attributes/functions allowing to escape the sandbox are blocked. Eg. frames are still accessibles, only the frame locals are blocked. This blacklist policy is broken by design, but it's a nice way to quickly get a working sandbox without having to modify CPython too much. pysandbox status is closer to a proof-of-concept than a beta version, there are open issues (see above). Please test it and try to break it! -- To try pysandbox, download the last version using git clone or a tarball at: http://github.com/haypo/pysandbox/ You don't need to install it, use "python interpreter.py" or "python execfile.py yourscript.py". Use --help to get more options. I tested pysandbox on Linux with Python 2.5, 2.6 and 2.7. I guess that it should work on Python 3.0 with minor changes. -- The current syntax is: config = SandboxConfig(...) with Sandbox(config): ... execute untrusted code here ... This syntax has a problem: local frame variables are not protected by a proxy, nor removed from the sandbox namespace. I tried to remove the frame locals, but Python uses STORE_FAST/LOAD_FAST bytecodes in a function, and this fast cache is not accessible in Python. Clear this cache may introduce unexpected behaviours. pysandbox modify some structure attributes (frame.f_builtins and frame.f_tstate.interp.builtins) directly in memory using some ctypes tricks. I used that to avoid patching CPython and to get faster a working proof-of-concept. Set USE_CPYTHON_HACK to False (in sandbox/__init__.py) to disable these hacks, but they are needed to protect __builtins__ (see related tests). -- By default, pysandbox doesn't use CPython restricted mode, because this mode is too restrictive (it's not possible to read a file or import a module). But pysandbox can use it with SandboxConfig(cpython_restricted=True). -- See README file for more information and TODO file for a longer status. Victor

2 1

Google Groups Mirror
by anatoly techtonik Feb. 15, 2010

Feb. 15, 2010

What is the point in maintaining archive listed in http://mail.python.org/mailman/listinfo/python-dev if it is not searchable? Recently I needed to find old thread about adding tags to roundup but couldn't. GMane archive doesn't search - http://news.gmane.org/navbar.php?group=gmane.comp.python.devel&query=roundu… and Google Group archive is private http://groups.google.com/group/python-dev How about opening an official read-only Google Group mirror? Then it will be possible to subscribe to notifications to just interesting threads. -- anatoly t.

5 10

Tracker outage
by "Martin v. Löwis" Feb. 15, 2010

Feb. 15, 2010

Starting around 14:00 UTC today, we will take the trackers at bugs.python.org, bugs.jython.org, and psf.upfronthosting.co.za offline for a system upgrade. The outage should not last longer than four hours (probably much shorter). Regards, Martin

1 0

PEP 385: Auditing
by "Martin v. Löwis" Feb. 14, 2010

Feb. 14, 2010

I recently set up a Mercurial hosting solution myself, and noticed that there is no audit trail of who had been writing to the "master" clone. There are commit messages, but they could be fake (even misleading to a different committer). The threat I'm concerned about is that of a stolen SSH key. If that is abused to push suspicious changes into the repository, it is really difficult to find out whose key had been used. The solution I came up with is to define an "incoming" hook on the repository which will log the SSH user along with the pack ID of the pack being pushed. I'd like to propose that a similar hook is installed on repositories hosted at hg.python.org (unless Mercurial offers something better already). Whether or not this log should be publicly visible can be debated; IMO it would be sufficient if only sysadmins can inspect it in case of doubt. Alterntively, the email notification sent to python-checkins could could report who the pusher was. Dirkjan: if you agree to such a strategy, please mention that in the PEP. Regards, Martin

6 10