Mailman 3 January 2013 - Python-Dev

Re: [Python-Dev] cpython (2.7): Fix memory error in test_ucn.
by Antoine Pitrou 21 Jan '13

21 Jan '13

On Mon, 21 Jan 2013 12:06:25 +0100 (CET) serhiy.storchaka <python-checkins(a)python.org> wrote: > diff --git a/Lib/test/test_ucn.py b/Lib/test/test_ucn.py > --- a/Lib/test/test_ucn.py > +++ b/Lib/test/test_ucn.py > @@ -144,13 +144,14 @@ > # very very long bogus character name > try: > x = b'\\N{SPACE' + b'x' * int(_testcapi.UINT_MAX + 1) + b'}' > + self.assertEqual(len(x), len(b'\\N{SPACE}') + > + (_testcapi.UINT_MAX + 1)) > + self.assertRaisesRegexp(UnicodeError, > + 'unknown Unicode character name', > + x.decode, 'unicode-escape' > + ) > except MemoryError: > raise unittest.SkipTest("not enough memory") > - self.assertEqual(len(x), len(b'\\N{SPACE}') + (_testcapi.UINT_MAX + 1)) > - self.assertRaisesRegexp(UnicodeError, > - 'unknown Unicode character name', > - x.decode, 'unicode-escape' > - ) You can't just do that. The test may end up swapping a lot and make the machine grind to a halt, rather than raise MemoryError. This threatens to make life miserable for anyone who hasn't enough RAM, but has enough swap to fit the test's working set. Really, you have to use a bigmem decorator. Regards Antoine.

1 0

Re: [Python-Dev] cpython (3.2): Issue #16335: Fix integer overflow in unicode-escape decoder.
by Christian Heimes 21 Jan '13

21 Jan '13

Am 21.01.2013 10:46, schrieb serhiy.storchaka: > http://hg.python.org/cpython/rev/7625866f8127 > changeset: 81622:7625866f8127 > branch: 3.2 > parent: 81610:260a9afd999a > user: Serhiy Storchaka <storchaka(a)gmail.com> > date: Mon Jan 21 11:38:00 2013 +0200 > summary: > Issue #16335: Fix integer overflow in unicode-escape decoder. > > files: > Lib/test/test_ucn.py | 16 ++++++++++++++++ > Objects/unicodeobject.c | 3 ++- > 2 files changed, 18 insertions(+), 1 deletions(-) > > > diff --git a/Lib/test/test_ucn.py b/Lib/test/test_ucn.py > --- a/Lib/test/test_ucn.py > +++ b/Lib/test/test_ucn.py > @@ -8,6 +8,7 @@ > """#" > > import unittest > +import _testcapi > > from test import support > > @@ -141,6 +142,21 @@ > str, b"\\NSPACE", 'unicode-escape', 'strict' > ) > > + @unittest.skipUnless(_testcapi.INT_MAX < _testcapi.PY_SSIZE_T_MAX, > + "needs UINT_MAX < SIZE_MAX") > + def test_issue16335(self): > + # very very long bogus character name > + try: > + x = b'\\N{SPACE' + b'x' * (_testcapi.UINT_MAX + 1) + b'}' > + except MemoryError: > + raise unittest.SkipTest("not enough memory") > + self.assertEqual(len(x), len(b'\\N{SPACE}') + (_testcapi.UINT_MAX + 1)) > + self.assertRaisesRegex(UnicodeError, > + 'unknown Unicode character name', > + x.decode, 'unicode-escape' > + ) > + > + The test requires a lot of memory on my 64bit Linux box. Two of three times the test process was killed by the Kernel's out of memory manager although I have 8 GB physical RAM and 4 GB swap. Can you rewrite the test to use less memory?

1 0

Modules/socketmodule.c: avoiding second fcntl() call worth the effort?
by Peter Portante 20 Jan '13

20 Jan '13

Hello folks, I noticed while stracing a process that sock.setblocking() calls always result in pairs of fcntl() calls on Linux. Checking 2.6.8, 2.7.3, and 3.3.0 Modules/socketmodule.c, the code seems to use the following (unless I have missed something): delay_flag = fcntl(s->sock_fd, F_GETFL, 0); if (block) delay_flag &= (~O_NONBLOCK); else delay_flag |= O_NONBLOCK; fcntl(s->sock_fd, F_SETFL, delay_flag); Perhaps a check to see the flags changed might be worth making? int orig_delay_flag = fcntl(s->sock_fd, F_GETFL, 0); if (block) delay_flag = orig_delay_flag & (~O_NONBLOCK); else delay_flag = orig_delay_flag | O_NONBLOCK; if (delay_flag != orig_delay_flag) fcntl(s->sock_fd, F_SETFL, delay_flag); OpenStack Swift using the Eventlet module, which sets the accepted socket non-blocking, resulting in twice the number of fcntl() calls. Not a killer on performance, but it seems simple enough to save a system call here. Thanks for your consideration, -peter

4 5

PEP 433: Add cloexec argument to functions creating file descriptors
by Victor Stinner 18 Jan '13

18 Jan '13

HTML version: http://www.python.org/dev/peps/pep-0433/ *** PEP: 433 Title: Add cloexec argument to functions creating file descriptors Version: $Revision$ Last-Modified: $Date$ Author: Victor Stinner <victor.stinner(a)gmail.com> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 10-January-2013 Python-Version: 3.4 Abstract ======== This PEP proposes to add a new optional argument ``cloexec`` on functions creating file descriptors in the Python standard library. If the argument is ``True``, the close-on-exec flag will be set on the new file descriptor. Rationale ========= On UNIX, subprocess closes file descriptors greater than 2 by default since Python 3.2 [#subprocess_close]_. All file descriptors created by the parent process are automatically closed. ``xmlrpc.server.SimpleXMLRPCServer`` sets the close-on-exec flag of the listening socket, the parent class ``socketserver.BaseServer`` does not set this flag. There are other cases creating a subprocess or executing a new program where file descriptors are not closed: functions of the os.spawn*() family and third party modules calling ``exec()`` or ``fork()`` + ``exec()``. In this case, file descriptors are shared between the parent and the child processes which is usually unexpected and causes various issues. This PEP proposes to continue the work started with the change in the subprocess, to fix the issue in any code, and not just code using subprocess. Inherited file descriptors issues --------------------------------- Closing the file descriptor in the parent process does not close the related resource (file, socket, ...) because it is still open in the child process. The listening socket of TCPServer is not closed on ``exec()``: the child process is able to get connection from new clients; if the parent closes the listening socket and create a new listening socket on the same address, it would get an "address already is used" error. Not closing file descriptors can lead to resource exhaustion: even if the parent closes all files, creating a new file descriptor may fail with "too many files" because files are still open in the child process. Security -------- Leaking file descriptors is a major security vulnerability. An untrusted child process can read sensitive data like passwords and take control of the parent process though leaked file descriptors. It is for example a known vulnerability to escape from a chroot. Atomicity --------- Using ``fcntl()`` to set the close-on-exec flag is not safe in a multithreaded application. If a thread calls ``fork()`` and ``exec()`` between the creation of the file descriptor and the call to ``fcntl(fd, F_SETFD, new_flags)``: the file descriptor will be inherited by the child process. Modern operating systems offer functions to set the flag during the creation of the file descriptor, which avoids the race condition. Portability ----------- Python 3.2 added ``socket.SOCK_CLOEXEC`` flag, Python 3.3 added ``os.O_CLOEXEC`` flag and ``os.pipe2()`` function. It is already possible to set atomically close-on-exec flag in Python 3.3 when opening a file and creating a pipe or socket. The problem is that these flags and functions are not portable: only recent versions of operating systems support them. ``O_CLOEXEC`` and ``SOCK_CLOEXEC`` flags are ignored by old Linux versions and so ``FD_CLOEXEC`` flag must be checked using ``fcntl(fd, F_GETFD)``. If the kernel ignores ``O_CLOEXEC`` or ``SOCK_CLOEXEC`` flag, a call to ``fcntl(fd, F_SETFD, flags)`` is required to set close-on-exec flag. .. note:: OpenBSD older 5.2 does not close the file descriptor with close-on-exec flag set if ``fork()`` is used before ``exec()``, but it works correctly if ``exec()`` is called without ``fork()``. Scope ----- Applications still have to close explicitly file descriptors after a ``fork()``. The close-on-exec flag only closes file descriptors after ``exec()``, and so after ``fork()`` + ``exec()``. This PEP only change the close-on-exec flag of file descriptors created by the Python standard library, or by modules using the standard library. Third party modules not using the standard library should be modified to conform to this PEP. The new ``os.set_cloexec()`` function can be used for example. Impacted functions: * ``os.forkpty()`` * ``http.server.CGIHTTPRequestHandler.run_cgi()`` Impacted modules: * ``multiprocessing`` * ``socketserver`` * ``subprocess`` * ``tempfile`` * ``xmlrpc.server`` * Maybe: ``signal``, ``threading`` XXX Should ``subprocess.Popen`` set the close-on-exec flag on file XXX XXX descriptors of the constructor the ``pass_fds`` argument? XXX .. note:: See `Close file descriptors after fork`_ for a possible solution for ``fork()`` without ``exec()``. Proposal ======== This PEP proposes to add a new optional argument ``cloexec`` on functions creating file descriptors in the Python standard library. If the argument is ``True``, the close-on-exec flag will be set on the new file descriptor. Add a new function: * ``os.set_cloexec(fd: int, cloexec: bool)``: set or unset the close-on-exec flag of a file descriptor Add a new optional ``cloexec`` argument to: * ``open()``: ``os.fdopen()`` is indirectly modified * ``os.dup()``, ``os.dup2()`` * ``os.pipe()`` * ``socket.socket()``, ``socket.socketpair()``, ``socket.socket.accept()`` * Maybe also: ``os.open()``, ``os.openpty()`` * TODO: * ``select.devpoll()`` * ``select.poll()`` * ``select.epoll()`` * ``select.kqueue()`` * ``socket.socket.recvmsg()``: use ``MSG_CMSG_CLOEXEC``, or ``os.set_cloexec()`` The default value of the ``cloexec`` argument is ``False`` to keep the backward compatibility. The close-on-exec flag will not be set on file descriptors 0 (stdin), 1 (stdout) and 2 (stderr), because these files are expected to be inherited. It would still be possible to set close-on-exec flag explicitly using ``os.set_cloexec()``. Drawbacks: * Many functions of the Python standard library creating file descriptors are cannot be changed by this proposal, because adding a ``cloexec`` optional argument would be surprising and too many functions would need it. For example, ``os.urandom()`` uses a temporary file on UNIX, but it calls a function of Windows API on Windows. Adding a ``cloexec`` argument to ``os.urandom()`` would not make sense. See `Always set close-on-exec flag`_ for an incomplete list of functions creating file descriptors. * Checking if a module creates file descriptors is difficult. For example, ``os.urandom()`` creates a file descriptor on UNIX to read ``/dev/urandom`` (and closes it at exit), whereas it is implemented using a function call on Windows. It is not possible to control close-on-exec flag of the file descriptor used by ``os.urandom()``, because ``os.urandom()`` API does not allow it. Alternatives ============ Always set close-on-exec flag ----------------------------- Always set close-on-exec flag on new file descriptors created by Python. This alternative just changes the default value of the new ``cloexec`` argument. If a file must be inherited by child processes, ``cloexec=False`` argument can be used. ``subprocess.Popen`` constructor has an ``pass_fds`` argument to specify which file descriptors must be inherited. The close-on-exec flag of these file descriptors must be changed with ``os.set_cloexec()``. Example of functions creating file descriptors which will be modified to set close-on-exec flag: * ``os.urandom()`` (on UNIX) * ``curses.window.getwin()``, ``curses.window.putwin()`` * ``mmap.mmap()`` (if ``MAP_ANONYMOUS`` is not defined) * ``oss.open()`` * ``Modules/main.c``: ``RunStartupFile()`` * ``Python/pythonrun.c``: ``PyRun_SimpleFileExFlags()`` * ``Modules/getpath.c``: ``search_for_exec_prefix()`` * ``Modules/zipimport.c``: ``read_directory()`` * ``Modules/_ssl.c``: ``load_dh_params()`` * ``PC/getpathp.c``: ``calculate_path()`` * ``Python/errors.c``: ``PyErr_ProgramText()`` * ``Python/import.c``: ``imp_load_dynamic()`` * TODO: ``PC/_msi.c`` Many functions are impacted indirectly by this alternative. Examples: * ``logging.FileHandler`` Advantages of setting close-on-exec flag by default: * There are far more programs that are bitten by FD inheritance upon exec (see `Inherited file descriptors issues`_ and `Security`_) than programs relying on it (see `Applications using inherance of file descriptors`_). Drawbacks of setting close-on-exec flag by default: * The os module is written as a thin wrapper to system calls (to functions of the C standard library). If atomic flags to set close-on-exec flag are not supported (see `Appendix: Operating system support`_), a single Python function call may call 2 or 3 system calls (see `Performances`_ section). * Extra system calls, if any, may slow down Python: see `Performances`_. * It violates the principle of least surprise. Developers using the os module may expect that Python respects the POSIX standard and so that close-on-exec flag is not set by default. Backward compatibility: only a few programs rely on inherance of file descriptors, and they only pass a few file descriptors, usually just one. These programs will fail immediatly with ``EBADF`` error, and it will be simple to fix them: add ``cloexec=False`` argument or use ``os.set_cloexec(fd, False)``. The ``subprocess`` module will be changed anyway to unset close-on-exec flag on file descriptors listed in the ``pass_fds`` argument of Popen constructor. So it possible that these programs will not need any fix if they use the ``subprocess`` module. Add a function to set close-on-exec flag by default --------------------------------------------------- An alternative is to add also a function to change globally the default behaviour. It would be possible to set close-on-exec flag for the whole application including all modules and the Python standard library. This alternative is based on the `Proposal`_ and adds extra changes. Add new functions: * ``sys.getdefaultcloexec() -> bool``: get the default value of the close-on-exec flag for new file descriptor * ``sys.setdefaultcloexec(cloexec: bool)``: enable or disable close-on-exec flag, the state of the flag can be overriden in each function creating a file descriptor The major change is that the default value of the ``cloexec`` argument is ``sys.getdefaultcloexec()``, instead of ``False``. When ``sys.setdefaultcloexec(True)`` is called to set close-on-exec by default, we have the same drawbacks than `Always set close-on-exec flag`_ alternative. There are additionnal drawbacks of having two behaviours depending on ``sys.getdefaultcloexec()`` value: * It is not more possible to know if the close-on-exec flag will be set or not just by reading the source code. Close file descriptors after fork --------------------------------- This PEP does not fix issues with applications using ``fork()`` without ``exec()``. Python needs a generic process to register callbacks which would be called after a fork, see `Add an 'afterfork' module`_. Such registry could be used to close file descriptors just after a ``fork()``. Drawbacks: * This alternative does not solve the problem for programs using ``exec()`` without ``fork()``. * A third party module may call directly the C function ``fork()`` which will not call "atfork" callbacks. * All functions creating file descriptors must be changed to register a callback and then unregister their callback when the file is closed. Or a list of *all* open file descriptors must be maintained. * The operating system is a better place than Python to close automatically file descriptors. For example, it is not easy to avoid a race condition between closing the file and unregistering the callback closing the file. open(): add "e" flag to mode ---------------------------- A new "e" mode would set close-on-exec flag (best-effort). This alternative only solves the problem for ``open()``. socket.socket() and os.pipe() do not have a ``mode`` argument for example. Since its version 2.7, the GNU libc supports ``"e"`` flag for ``fopen()``. It uses ``O_CLOEXEC`` if available, or use ``fcntl(fd, F_SETFD, FD_CLOEXEC)``. With Visual Studio, fopen() accepts a "N" flag which uses ``O_NOINHERIT``. Applications using inherance of file descriptors ================================================ Most developers don't know that file descriptors are inherited by default. Most programs do not rely on inherance of file descriptors. For example, ``subprocess.Popen`` was changed in Python 3.2 to close all file descriptors greater than 2 in the child process by default. No user complained about this behavior change. Network servers using fork may want to pass the client socket to the child process. For example, on UNIX a CGI server pass the socket client through file descriptors 0 (stdin) and 1 (stdout) using ``dup2()``. This specific case is not impacted by this PEP because the close-on-exec flag is never set on file descriptors smaller than 3. To access a restricted resource like creating a socket listening on a TCP port lower than 1024 or reading a file containing sensitive data like passwords, a common practice is: start as the root user, create a file descriptor, create a child process, pass the file descriptor to the child process and exit. Security is very important in such use case: leaking another file descriptor would be a critical security vulnerability (see `Security`_). The root process may not exit but monitors the child process instead, and restarts a new child process and pass the same file descriptor if the previous child process crashed. Example of programs taking file descriptors from the parent process using a command line option: * gpg: ``--status-fd <fd>``, ``--logger-fd <fd>``, etc. * openssl: ``-pass fd:<fd>`` * qemu: ``-add-fd <fd>`` * valgrind: ``--log-fd=<fd>``, ``--input-fd=<fd>``, etc. * xterm: ``-S <fd>`` On Linux, it is possible to use ``"/dev/fd/<fd>"`` filename to pass a file descriptor to a program expecting a filename. Performances ============ Setting close-on-exec flag may require additional system calls for each creation of new file descriptors. The number of additional system calls depends on the method used to set the flag: * ``O_NOINHERIT``: no additionnal system call * ``O_CLOEXEC``: one addition system call, but only at the creation of the first file descriptor, to check if the flag is supported. If no, Python has to fallback to the next method. * ``ioctl(fd, FIOCLEX)``: one addition system call per file descriptor * ``fcntl(fd, F_SETFD, flags)``: two addition system calls per file descriptor, one to get old flags and one to set new flags XXX Benchmark the overhead for these 4 methods. XXX Implementation ============== os.set_cloexec(fd, cloexec) --------------------------- Best-effort by definition. Pseudo-code:: if os.name == 'nt': def set_cloexec(fd, cloexec=True): SetHandleInformation(fd, HANDLE_FLAG_INHERIT, int(cloexec)) else: fnctl = None ioctl = None try: import ioctl except ImportError: try: import fcntl except ImportError: pass if ioctl is not None and hasattr('FIOCLEX', ioctl): def set_cloexec(fd, cloexec=True): if cloexec: ioctl.ioctl(fd, ioctl.FIOCLEX) else: ioctl.ioctl(fd, ioctl.FIONCLEX) elif fnctl is not None: def set_cloexec(fd, cloexec=True): flags = fcntl.fcntl(fd, fcntl.F_GETFD) if cloexec: flags |= FD_CLOEXEC else: flags &= ~FD_CLOEXEC fcntl.fcntl(fd, fcntl.F_SETFD, flags) else: def set_cloexec(fd, cloexec=True): raise NotImplementedError( "close-on-exec flag is not supported " "on your platform") ioctl is preferred over fcntl because it requires only one syscall, instead of two syscalls for fcntl. .. note:: ``fcntl(fd, F_SETFD, flags)`` only supports one flag (``FD_CLOEXEC``), so it would be possible to avoid ``fcntl(fd, F_GETFD)``. But it may drop other flags in the future, and so it is safer to keep the two functions calls. .. note:: ``fopen()`` function of the GNU libc ignores the error if ``fcntl(fd, F_SETFD, flags)`` failed. open() ------ * Windows: ``open()`` with ``O_NOINHERIT`` flag [atomic] * ``open()`` with ``O_CLOEXEC flag`` [atomic] * ``open()`` + ``os.set_cloexec(fd, True)`` [best-effort] os.dup() -------- * ``fcntl(fd, F_DUPFD_CLOEXEC)`` [atomic] * ``dup()`` + ``os.set_cloexec(fd, True)`` [best-effort] os.dup2() --------- * ``dup3()`` with ``O_CLOEXEC`` flag [atomic] * ``dup2()`` + ``os.set_cloexec(fd, True)`` [best-effort] os.pipe() --------- * Windows: ``_pipe()`` with ``O_NOINHERIT`` flag [atomic] * ``pipe2()`` with ``O_CLOEXEC`` flag [atomic] * ``pipe()`` + ``os.set_cloexec(fd, True)`` [best-effort] socket.socket() --------------- * ``socket()`` with ``SOCK_CLOEXEC`` flag [atomic] * ``socket()`` + ``os.set_cloexec(fd, True)`` [best-effort] socket.socketpair() ------------------- * ``socketpair()`` with ``SOCK_CLOEXEC`` flag [atomic] * ``socketpair()`` + ``os.set_cloexec(fd, True)`` [best-effort] socket.socket.accept() ---------------------- * ``accept4()`` with ``SOCK_CLOEXEC`` flag [atomic] * ``accept()`` + ``os.set_cloexec(fd, True)`` [best-effort] Backward compatibility ====================== There is no backward incompatible change. The default behaviour is unchanged: the close-on-exec flag is not set by default. Appendix: Operating system support ================================== Windows ------- Windows has an ``O_NOINHERIT`` flag: "Do not inherit in child processes". For example, it is supported by ``open()`` and ``_pipe()``. The value of the flag can be modified using: ``SetHandleInformation(fd, HANDLE_FLAG_INHERIT, 1)``. ``CreateProcess()`` has an ``bInheritHandles`` argument: if it is FALSE, the handles are not inherited. It is used by ``subprocess.Popen`` with ``close_fds`` option. fcntl ----- Functions: * ``fcntl(fd, F_GETFD)`` * ``fcntl(fd, F_SETFD, flags | FD_CLOEXEC)`` Availability: AIX, Digital UNIX, FreeBSD, HP-UX, IRIX, Linux, Mac OS X, OpenBSD, Solaris, SunOS, Unicos. ioctl ----- Functions: * ``ioctl(fd, FIOCLEX, 0)`` sets close-on-exec flag * ``ioctl(fd, FIONCLEX, 0)`` unsets close-on-exec flag Availability: Linux, Mac OS X, QNX, NetBSD, OpenBSD, FreeBSD. Atomic flags ------------ New flags: * ``O_CLOEXEC``: available on Linux (2.6.23+), FreeBSD (8.3+), OpenBSD 5.0, QNX, BeOS, next NetBSD release (6.1?). This flag is part of POSIX.1-2008. * ``socket()``: ``SOCK_CLOEXEC`` flag, available on Linux 2.6.27+, OpenBSD 5.2, NetBSD 6.0. * ``fcntl()``: ``F_DUPFD_CLOEXEC`` flag, available on Linux 2.6.24+, OpenBSD 5.0, FreeBSD 9.1, NetBSD 6.0. This flag is part of POSIX.1-2008. * ``recvmsg()``: ``MSG_CMSG_CLOEXEC``, available on Linux 2.6.23+, NetBSD 6.0. On Linux older than 2.6.23, ``O_CLOEXEC`` flag is simply ignored. So we have to check that the flag is supported by calling ``fcntl()``. If it does not work, we have to set the flag using ``fcntl()``. XXX what is the behaviour on Linux older than 2.6.27 XXX with SOCK_CLOEXEC? XXX New functions: * ``dup3()``: available on Linux 2.6.27+ (and glibc 2.9) * ``pipe2()``: available on Linux 2.6.27+ (and glibc 2.9) * ``accept4()``: available on Linux 2.6.28+ (and glibc 2.10) If ``accept4()`` is called on Linux older than 2.6.28, ``accept4()`` returns ``-1`` (fail) and errno is set to ``ENOSYS``. Links ===== Links: * `Secure File Descriptor Handling <http://udrepper.livejournal.com/20407.html>`_ (Ulrich Drepper, 2008) * `win32_support.py of the Tornado project <https://bitbucket.org/pvl/gaeseries-tornado/src/c2671cea1842/tornado/win32_…>`_: emulate fcntl(fd, F_SETFD, FD_CLOEXEC) using ``SetHandleInformation(fd, HANDLE_FLAG_INHERIT, 1)`` Python issues: * `open() does not able to set flags, such as O_CLOEXEC <http://bugs.python.org/issue12105>`_ * `Add "e" mode to open(): close-and-exec (O_CLOEXEC) / O_NOINHERIT <http://bugs.python.org/issue16850>`_ * `TCP listening sockets created without FD_CLOEXEC flag <http://bugs.python.org/issue12107>`_ * `Use O_CLOEXEC in the tempfile module <http://bugs.python.org/issue16860>`_ * `Support accept4() for atomic setting of flags at socket creation <http://bugs.python.org/issue10115>`_ * `Add an 'afterfork' module <http://bugs.python.org/issue16500>`_ Ruby: * `Set FD_CLOEXEC for all fds (except 0, 1, 2) <http://bugs.ruby-lang.org/issues/5041>`_ * `O_CLOEXEC flag missing for Kernel::open <http://bugs.ruby-lang.org/issues/1291>`_: `commit reverted <http://bugs.ruby-lang.org/projects/ruby-trunk/repository/revisions/31643>`_ later Footnotes ========= .. [#subprocess_close] On UNIX since Python 3.2, subprocess.Popen() closes all file descriptors by default: ``close_fds=True``. It closes file descriptors in range 3 inclusive to ``local_max_fd`` exclusive, where ``local_max_fd`` is ``fcntl(0, F_MAXFD)`` on NetBSD, or ``sysconf(_SC_OPEN_MAX)`` otherwise. If the error pipe has a descriptor smaller than 3, ``ValueError`` is raised.

9 26

Summary of Python tracker Issues
by Python tracker 18 Jan '13

18 Jan '13

ACTIVITY SUMMARY (2013-01-11 - 2013-01-18) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 3829 ( +4) closed 24942 (+54) total 28771 (+58) Open issues with patches: 1668 Issues opened (40) ================== #15031: Split .pyc parsing from module loading http://bugs.python.org/issue15031 reopened by brett.cannon #16932: urlparse fails at parsing "www.python.org:80/" http://bugs.python.org/issue16932 reopened by georg.brandl #16937: -u (unbuffered I/O) command line option documentation mismatch http://bugs.python.org/issue16937 opened by mjpieters #16938: pydoc confused by __dir__ http://bugs.python.org/issue16938 opened by ronaldoussoren #16942: urllib still doesn't support persistent connections http://bugs.python.org/issue16942 opened by C19 #16945: rewrite CGIHTTPRequestHandler to always use subprocess http://bugs.python.org/issue16945 opened by neologix #16946: subprocess: _close_open_fd_range_safe() does not set close-on- http://bugs.python.org/issue16946 opened by haypo #16948: email.mime.text.MIMEText: QP encoding broken with charset!=ISO http://bugs.python.org/issue16948 opened by jwilk #16953: select module compile errors with broken poll() http://bugs.python.org/issue16953 opened by Jeffrey.Armstrong #16954: Add docstrings for ElementTree module http://bugs.python.org/issue16954 opened by serhiy.storchaka #16956: Allow signed line number deltas in the code object's line num http://bugs.python.org/issue16956 opened by Mark.Shannon #16957: shutil.which() shouldn't look in working directory on unix-y s http://bugs.python.org/issue16957 opened by takluyver #16958: The sqlite3 context manager does not work with isolation_level http://bugs.python.org/issue16958 opened by r.david.murray #16959: rlcompleter doesn't work if __main__ can't be imported http://bugs.python.org/issue16959 opened by Aaron.Meurer #16961: No regression tests for -E and individual environment vars http://bugs.python.org/issue16961 opened by ncoghlan #16962: _posixsubprocess module uses outdated getdents system call http://bugs.python.org/issue16962 opened by riku-voipio #16964: Add 'm' format specifier for mon_grouping etc. http://bugs.python.org/issue16964 opened by skrah #16965: 2to3 should rewrite execfile() to open in 'rb' mode http://bugs.python.org/issue16965 opened by barry #16967: Keyword only argument default values are evaluated before othe http://bugs.python.org/issue16967 opened by Kay.Hayen #16968: Fix test discovery for test_concurrent_futures.py http://bugs.python.org/issue16968 opened by zach.ware #16969: test_urlwithfrag fail http://bugs.python.org/issue16969 opened by Ry #16970: argparse: bad nargs value raises misleading message http://bugs.python.org/issue16970 opened by chris.jerdonek #16971: Refleaks in charmap decoder http://bugs.python.org/issue16971 opened by serhiy.storchaka #16972: Useless function call in site.py http://bugs.python.org/issue16972 opened by x746e #16974: when "python -c command" does a traceback, it open the file "< http://bugs.python.org/issue16974 opened by ericlammerts #16975: Broken error handling in codecs.escape_decode() http://bugs.python.org/issue16975 opened by serhiy.storchaka #16976: Asyncore/asynchat hangs when used with ssl sockets http://bugs.python.org/issue16976 opened by Anthony.Lozano #16977: argparse: mismatch between choices parsing and usage/error mes http://bugs.python.org/issue16977 opened by chris.jerdonek #16978: fix grammar in 'threading' documentation http://bugs.python.org/issue16978 opened by tshepang #16979: Broken error handling in codecs.unicode_escape_decode() http://bugs.python.org/issue16979 opened by serhiy.storchaka #16980: SystemError in codecs.unicode_escape_decode() http://bugs.python.org/issue16980 opened by serhiy.storchaka #16981: ImportError hides real error when there too many open files du http://bugs.python.org/issue16981 opened by jinty #16983: header parsing could apply postel's law to encoded words insid http://bugs.python.org/issue16983 opened by r.david.murray #16985: Docs reference a concrete UTC tzinfo, but none exists http://bugs.python.org/issue16985 opened by jason.coombs #16986: ElementTree incorrectly parses strings with declared encoding http://bugs.python.org/issue16986 opened by serhiy.storchaka #16988: argparse: PARSER option for nargs not documented http://bugs.python.org/issue16988 opened by Robert #16989: allow distutils debug mode to be enabled more easily http://bugs.python.org/issue16989 opened by chris.jerdonek #16991: Add OrderedDict written in C http://bugs.python.org/issue16991 opened by eric.snow #16993: shutil.which() should preserve path case http://bugs.python.org/issue16993 opened by serhiy.storchaka #16994: collections.Counter.least_common http://bugs.python.org/issue16994 opened by davidcoallier Most recent 15 issues with no replies (15) ========================================== #16994: collections.Counter.least_common http://bugs.python.org/issue16994 #16993: shutil.which() should preserve path case http://bugs.python.org/issue16993 #16989: allow distutils debug mode to be enabled more easily http://bugs.python.org/issue16989 #16985: Docs reference a concrete UTC tzinfo, but none exists http://bugs.python.org/issue16985 #16983: header parsing could apply postel's law to encoded words insid http://bugs.python.org/issue16983 #16978: fix grammar in 'threading' documentation http://bugs.python.org/issue16978 #16972: Useless function call in site.py http://bugs.python.org/issue16972 #16965: 2to3 should rewrite execfile() to open in 'rb' mode http://bugs.python.org/issue16965 #16964: Add 'm' format specifier for mon_grouping etc. http://bugs.python.org/issue16964 #16962: _posixsubprocess module uses outdated getdents system call http://bugs.python.org/issue16962 #16961: No regression tests for -E and individual environment vars http://bugs.python.org/issue16961 #16953: select module compile errors with broken poll() http://bugs.python.org/issue16953 #16948: email.mime.text.MIMEText: QP encoding broken with charset!=ISO http://bugs.python.org/issue16948 #16946: subprocess: _close_open_fd_range_safe() does not set close-on- http://bugs.python.org/issue16946 #16945: rewrite CGIHTTPRequestHandler to always use subprocess http://bugs.python.org/issue16945 Most recent 15 issues waiting for review (15) ============================================= #16994: collections.Counter.least_common http://bugs.python.org/issue16994 #16993: shutil.which() should preserve path case http://bugs.python.org/issue16993 #16991: Add OrderedDict written in C http://bugs.python.org/issue16991 #16981: ImportError hides real error when there too many open files du http://bugs.python.org/issue16981 #16980: SystemError in codecs.unicode_escape_decode() http://bugs.python.org/issue16980 #16979: Broken error handling in codecs.unicode_escape_decode() http://bugs.python.org/issue16979 #16978: fix grammar in 'threading' documentation http://bugs.python.org/issue16978 #16977: argparse: mismatch between choices parsing and usage/error mes http://bugs.python.org/issue16977 #16975: Broken error handling in codecs.escape_decode() http://bugs.python.org/issue16975 #16972: Useless function call in site.py http://bugs.python.org/issue16972 #16971: Refleaks in charmap decoder http://bugs.python.org/issue16971 #16970: argparse: bad nargs value raises misleading message http://bugs.python.org/issue16970 #16968: Fix test discovery for test_concurrent_futures.py http://bugs.python.org/issue16968 #16962: _posixsubprocess module uses outdated getdents system call http://bugs.python.org/issue16962 #16957: shutil.which() shouldn't look in working directory on unix-y s http://bugs.python.org/issue16957 Top 10 most discussed issues (10) ================================= #16468: argparse only supports iterable choices http://bugs.python.org/issue16468 17 msgs #10527: multiprocessing.Pipe problem: "handle out of range in select() http://bugs.python.org/issue10527 11 msgs #16893: Create IDLE help.txt from Doc/library/idle.rst http://bugs.python.org/issue16893 11 msgs #12939: Add new io.FileIO using the native Windows API http://bugs.python.org/issue12939 9 msgs #14468: Update cloning guidelines in devguide http://bugs.python.org/issue14468 7 msgs #16968: Fix test discovery for test_concurrent_futures.py http://bugs.python.org/issue16968 7 msgs #16969: test_urlwithfrag fail http://bugs.python.org/issue16969 7 msgs #11844: Update json to upstream simplejson latest release http://bugs.python.org/issue11844 6 msgs #16974: when "python -c command" does a traceback, it open the file "< http://bugs.python.org/issue16974 6 msgs #11983: Inconsistent hash and comparison for code objects http://bugs.python.org/issue11983 5 msgs Issues closed (50) ================== #3876: multiprocessing does not compile on systems which do not defin http://bugs.python.org/issue3876 closed by skrah #5066: IDLE documentation for Unix obsolete/incorrect http://bugs.python.org/issue5066 closed by asvetlov #7340: Doc for sys.exc_info has warning that is no longer valid http://bugs.python.org/issue7340 closed by python-dev #7468: PyErr_Format documentation doesn't mention all format codes http://bugs.python.org/issue7468 closed by serhiy.storchaka #8712: Skip libpthread related test failures on OpenBSD http://bugs.python.org/issue8712 closed by skrah #8714: Delayed signals in the REPL on OpenBSD (possibly libpthread re http://bugs.python.org/issue8714 closed by skrah #9720: zipfile writes incorrect local file header for large files in http://bugs.python.org/issue9720 closed by serhiy.storchaka #11290: ttk.Combobox['values'] String Conversion to Tcl http://bugs.python.org/issue11290 closed by serhiy.storchaka #11441: compile() raises SystemError if called from except clause http://bugs.python.org/issue11441 closed by brett.cannon #11729: libffi assembler relocation check is not robust, fails with cl http://bugs.python.org/issue11729 closed by skrah #12812: libffi does not build with clang on amd64 http://bugs.python.org/issue12812 closed by skrah #14110: FreeBSD: test_os fails if user is in the wheel group http://bugs.python.org/issue14110 closed by skrah #14377: Modify serializer for xml.etree.ElementTree to allow forcing t http://bugs.python.org/issue14377 closed by python-dev #14850: The inconsistency of codecs.charmap_decode http://bugs.python.org/issue14850 closed by serhiy.storchaka #15442: Expand the list of default dirs filecmp.dircmp ignores http://bugs.python.org/issue15442 closed by python-dev #15861: ttk.Treeview "unmatched open brace in list" http://bugs.python.org/issue15861 closed by serhiy.storchaka #16216: Arithmetic operations with NULL http://bugs.python.org/issue16216 closed by serhiy.storchaka #16259: Replace exec() in test.regrtest with __import__ http://bugs.python.org/issue16259 closed by r.david.murray #16398: deque.rotate() could be much faster http://bugs.python.org/issue16398 closed by rhettinger #16422: Decimal constants should be the same for py & c module version http://bugs.python.org/issue16422 closed by skrah #16613: ChainMap.new_child could use improvement http://bugs.python.org/issue16613 closed by python-dev #16730: _fill_cache in _bootstrap.py crashes without directory execute http://bugs.python.org/issue16730 closed by brett.cannon #16762: test_subprocess failure on OpenBSD/NetBSD buildbots http://bugs.python.org/issue16762 closed by neologix #16829: IDLE on POSIX can't print filenames with spaces http://bugs.python.org/issue16829 closed by serhiy.storchaka #16886: Doctests in test_dictcomp depend on dict order http://bugs.python.org/issue16886 closed by python-dev #16916: The Extended Iterable Unpacking (PEP-3132) for byte strings do http://bugs.python.org/issue16916 closed by python-dev #16923: test_ssl kicks up a lot of ResourceWarnings http://bugs.python.org/issue16923 closed by pitrou #16933: argparse: remove magic from examples http://bugs.python.org/issue16933 closed by chris.jerdonek #16936: Documentation for stat.S_IFMT inconsistent http://bugs.python.org/issue16936 closed by python-dev #16939: Broken link in 14. Cryptographic Service http://bugs.python.org/issue16939 closed by chris.jerdonek #16940: argparse 15.4.5.1. Sub-commands documentation missing indentat http://bugs.python.org/issue16940 closed by ezio.melotti #16941: TkInter won't update display on OS X if delay is too small http://bugs.python.org/issue16941 closed by ned.deily #16943: seriously? FileCookieJar can't really save ? save method is N http://bugs.python.org/issue16943 closed by nadeem.vawda #16944: German number separators not working using format language and http://bugs.python.org/issue16944 closed by skrah #16947: Search for "sherpa" on pypi leads to gitflow http://bugs.python.org/issue16947 closed by r.david.murray #16949: removal of string exceptions is already done http://bugs.python.org/issue16949 closed by python-dev #16950: the old raise syntax is not legal in Python 3 http://bugs.python.org/issue16950 closed by python-dev #16951: expand on meaning of 'string literals that rely on significant http://bugs.python.org/issue16951 closed by georg.brandl #16952: test_kqueue failure on NetBSD/OpenBSD http://bugs.python.org/issue16952 closed by neologix #16955: multiprocessing.connection poll() always returns false http://bugs.python.org/issue16955 closed by sbt #16960: Fix PEP8 errors everywhere http://bugs.python.org/issue16960 closed by ezio.melotti #16963: module html.parser HTMLParser's strict mode don't work http://bugs.python.org/issue16963 closed by pysean #16966: Publishing multiprocessing listener code http://bugs.python.org/issue16966 closed by sbt #16973: Extending the datetime package by adding a workDayMath module http://bugs.python.org/issue16973 closed by ezio.melotti #16982: _ssl not built --without-threads http://bugs.python.org/issue16982 closed by skrah #16984: idle problem with dark color schemes in kde http://bugs.python.org/issue16984 closed by serwy #16987: HP-UX: SHLIB_EXT is not set http://bugs.python.org/issue16987 closed by skrah #16990: re: match of nongreedy regex not grouping right http://bugs.python.org/issue16990 closed by r.david.murray #16992: signal.set_wakeup_fd(400) crashes on Windows http://bugs.python.org/issue16992 closed by python-dev #16922: ElementTree.findtext() returns empty bytes object instead of e http://bugs.python.org/issue16922 closed by eli.bendersky

1 0

DLLs folder on Windows
by Piotr Dobrogost 15 Jan '13

15 Jan '13

Hi! I'm curious how dlls from the DLLs folder on Windows are being loaded? As they are not placed in the same folder where python.exe resides I guess they must be loaded by giving the path explicitly but I'm not sure. I'm asking because there's no DLLs folder being created when creating virtualenv and I suspect it should be. This is a followup to my question "Why doesn't virtualenv create DLLs folder?" (http://stackoverflow.com/q/6657541/95735) and to the virtualenv's issue 87 - "Problems creating virtualenv on Windows when Python is not installed for all users.". Regards, Piotr Dobrogost

2 1

PEP-431 updated.
by Lennart Regebro 15 Jan '13

15 Jan '13

OK, here is the third major incarnation of PEP-431. I think it addresses all the concerns brought to light so far. https://raw.github.com/regebro/tz-pep/master/pep-04tz.txt Is there a simpler way of doing this than thus "cut and paste" updating? Is there a way to make pull requests to hg.python.org, for example? ------------ PEP: 431 Title: Time zone support improvements Version: $Revision$ Last-Modified: $Date$ Author: Lennart Regebro <regebro(a)gmail.com> BDFL-Delegate: Barry Warsaw <barry(a)python.org> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 11-Dec-2012 Post-History: 11-Dec-2012, 28-Dec-2012 Abstract ======== This PEP proposes the implementation of concrete time zone support in the Python standard library, and also improvements to the time zone API to deal with ambiguous time specifications during DST changes. Proposal ======== Concrete time zone support -------------------------- The time zone support in Python has no concrete implementation in the standard library outside of a tzinfo baseclass that supports fixed offsets. To properly support time zones you need to include a database over all time zones, both current and historical, including daylight saving changes. But such information changes frequently, so even if we include the last information in a Python release, that information would be outdated just a few months later. Time zone support has therefore only been available through two third-party modules, ``pytz`` and ``dateutil``, both who include and wrap the "zoneinfo" database. This database, also called "tz" or "The Olsen database", is the de-facto standard time zone database over time zones, and it is included in most Unix and Unix-like operating systems, including OS X. This gives us the opportunity to include the code that supports the zoneinfo data in the standard library, but by default use the operating system's copy of the data, which typically will be kept updated by the updating mechanism of the operating system or distribution. For those who have an operating system that does not include the zoneinfo database, for example Windows, the Python source distribution will include a copy of the zoneinfo database, and a distribution containing the latest zoneinfo database will also be available at the Python Package Index, so it can be easily installed with the Python packaging tools such as ``easy_install`` or ``pip``. This could also be done on Unices that are no longer receiving updates and therefore have an outdated database. With such a mechanism Python would have full time zone support in the standard library on any platform, and a simple package installation would provide an updated time zone database on those platforms where the zoneinfo database isn't included, such as Windows, or on platforms where OS updates are no longer provided. The time zone support will be implemented by making the ``datetime`` module into a package, and adding time zone support to ``datetime`` based on Stuart Bishop's ``pytz`` module. Getting the local time zone --------------------------- On Unix there is no standard way of finding the name of the time zone that is being used. All the information that is available is the time zone abbreviations, such as ``EST`` and ``PDT``, but many of those abbreviations are ambiguous and therefore you can't rely on them to figure out which time zone you are located in. There is however a standard for finding the compiled time zone information since it's located in ``/etc/localtime``. Therefore it is possible to create a local time zone object with the correct time zone information even though you don't know the name of the time zone. A function in ``datetime`` should be provided to return the local time zone. The support for this will be made by integrating Lennart Regebro's ``tzlocal`` module into the new ``datetime`` module. For Windows it will look up the local Windows time zone name, and use a mapping between Windows time zone names and zoneinfo time zone names provided by the Unicode consortium to convert that to a zoneinfo time zone. The mapping should be updated before each major or bugfix release, scripts for doing so will be provided in the ``Tools/`` directory. Ambiguous times --------------- When changing over from daylight savings time (DST) the clock is turned back one hour. This means that the times during that hour happens twice, once without DST and then once with DST. Similarly, when changing to daylight savings time, one hour goes missing. The current time zone API can not differentiate between the two ambiguous times during a change from DST. For example, in Stockholm the time of 2012-11-28 02:00:00 happens twice, both at UTC 2012-11-28 00:00:00 and also at 2012-11-28 01:00:00. The current time zone API can not disambiguate this and therefore it's unclear which time should be returned:: # This could be either 00:00 or 01:00 UTC: >>> dt = datetime(2012, 10, 28, 2, 0, tzinfo=zoneinfo('Europe/Stockholm')) # But we can not specify which: >>> dt.astimezone(zoneinfo('UTC')) datetime.datetime(2012, 10, 28, 1, 0, tzinfo=<UTC>) ``pytz`` solved this problem by adding ``is_dst`` parameters to several methods of the tzinfo objects to make it possible to disambiguate times when this is desired. This PEP proposes to add these ``is_dst`` parameters to the relevant methods of the ``datetime`` API, and therefore add this functionality directly to ``datetime``. This is likely the hardest part of this PEP as this involves updating the C version of the ``datetime`` library with this functionality, as this involved writing new code, and not just reorganizing existing external libraries. Implementation API ================== The zoneinfo database --------------------- The latest version of the zoneinfo database should exist in the ``Lib/tzdata`` directory of the Python source control system. This copy of the database should be updated before every Python feature and bug-fix release, but not for releases of Python versions that are in security-fix-only-mode. Scripts to update the database will be provided in ``Tools/``, and the release instructions will be updated to include this update. New configure options ``--enable-internal-timezone-database`` and ``--disable-internal-timezone-database`` will be implemented to enable and disable the installation of this database when installing from source. A source install will default to installing them. Binary installers for systems that have a system-provided zoneinfo database may skip installing the included database since it would never be used for these platforms. For other platforms, for example Windows, binary installers must install the included database. Changes in the ``datetime``-module ---------------------------------- The public API of the new time zone support contains one new class, one new function, one new exception and four new collections. In addition to this, several methods on the datetime object gets a new ``is_dst`` parameter. New class ``DstTzInfo`` ^^^^^^^^^^^^^^^^^^^^^^^^ This class provides a concrete implementation of the ``zoneinfo`` base class that implements DST support. New function ``zoneinfo(name=None, db_path=None)`` ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ This function takes a name string that must be a string specifying a valid zoneinfo time zone, i.e. "US/Eastern", "Europe/Warsaw" or "Etc/GMT". If not given, the local time zone will be looked up. If an invalid zone name is given, or the local time zone can not be retrieved, the function raises `UnknownTimeZoneError`. The function also takes an optional path to the location of the zoneinfo database which should be used. If not specified, the function will look for databases in the following order: 1. Use the database in ``/usr/share/zoneinfo``, if it exists. 2. Check if the `tzdata-update` module is installed, and then use that database. 3. Use the Python-provided database in ``Lib/tzdata``. If no database is found an ``UnknownTimeZoneError`` or subclass thereof will be raised with a message explaining that no zoneinfo database can be found, but that you can install one with the ``tzdata-update`` package. New parameter ``is_dst`` ^^^^^^^^^^^^^^^^^^^^^^^^ A new ``is_dst`` parameter is added to several methods to handle time ambiguity during DST changeovers. * ``tzinfo.utcoffset(dt, is_dst=False)`` * ``tzinfo.dst(dt, is_dst=False)`` * ``tzinfo.tzname(dt, is_dst=False)`` * ``datetime.astimezone(tz, is_dst=False)`` The ``is_dst`` parameter can be ``False`` (default), ``True``, or ``None``. ``False`` will specify that the given datetime should be interpreted as not happening during daylight savings time, i.e. that the time specified is after the change from DST. ``True`` will specify that the given datetime should be interpreted as happening during daylight savings time, i.e. that the time specified is before the change from DST. ``None`` will raise an ``AmbiguousTimeError`` exception if the time specified was during a DST change over. It will also raise a ``NonExistentTimeError`` if a time is specified during the "missing time" in a change to DST. New exceptions ^^^^^^^^^^^^^^ * ``UnknownTimeZoneError`` This exception is a subclass of KeyError and raised when giving a time zone specification that can't be found:: >>> datetime.Timezone('Europe/New_York') Traceback (most recent call last): ... UnknownTimeZoneError: There is no time zone called 'Europe/New_York' * ``InvalidTimeError`` This exception serves as a base for ``AmbiguousTimeError`` and ``NonExistentTimeError``, to enable you to trap these two separately. It will subclass from ValueError, so that you can catch these errors together with inputs like the 29th of February 2011. * ``AmbiguousTimeError`` This exception is raised when giving a datetime specification that is ambiguous while setting ``is_dst`` to None:: >>> datetime(2012, 11, 28, 2, 0, tzinfo=zoneinfo('Europe/Stockholm'), is_dst=None) >>> Traceback (most recent call last): ... AmbiguousTimeError: 2012-10-28 02:00:00 is ambiguous in time zone Europe/Stockholm * ``NonExistentTimeError`` This exception is raised when giving a datetime specification that is ambiguous while setting ``is_dst`` to None:: >>> datetime(2012, 3, 25, 2, 0, tzinfo=zoneinfo('Europe/Stockholm'), is_dst=None) >>> Traceback (most recent call last): ... NonExistentTimeError: 2012-03-25 02:00:00 does not exist in time zone Europe/Stockholm New collections ^^^^^^^^^^^^^^^ * ``all_timezones`` is the exhaustive list of the time zone names that can be used, listed alphabethically. * ``all_timezones_set`` is a set of the time zones in ``all_timezones``. * ``common_timezones`` is a list of useful, current time zones, listed alphabethically. * ``common_timezones_set`` is a set of the time zones in ``common_timezones``. The ``tzdata-update``-package ----------------------------- The zoneinfo database will be packaged for easy installation with ``easy_install``/``pip``/``buildout``. This package will not install any Python code, and will not contain any Python code except that which is needed for installation. It will be kept updated with the same tools as the internal database, but released whenever the ``zoneinfo``-database is updated, and use the same version schema. Differences from the ``pytz`` API ================================= * ``pytz`` has the functions ``localize()`` and ``normalize()`` to work around that ``tzinfo`` doesn't have is_dst. When ``is_dst`` is implemented directly in ``datetime.tzinfo`` they are no longer needed. * The ``timezone()`` function is called ``zoneinfo()`` to avoid clashing with the ``timezone`` class introduced in Python 3.2. * ``zoneinfo()`` will return the local time zone if called without arguments. * The class ``pytz.StaticTzInfo`` is there to provide the ``is_dst`` support for static time zones. When ``is_dst`` support is included in ``datetime.tzinfo`` it is no longer needed. * ``InvalidTimeError`` subclasses from ``ValueError``. Resources ========= * http://pytz.sourceforge.net/ * http://pypi.python.org/pypi/tzlocal * http://pypi.python.org/pypi/python-dateutil * http://unicode.org/cldr/data/common/supplemental/windowsZones.xml Copyright ========= This document has been placed in the public domain.

2 1

Should libpython.a go in eprefix?
by Ben Leslie 15 Jan '13

15 Jan '13

Hi all, This issue: http://bugs.python.org/issue9807 from a couple of years ago seems to have changed the LIBPL build variable from pointing to a directory inside EPREFIX, to a directory in PREFIX. This doesn't seem ideal as LIBPL is populated with platform specific files such as libpython.a. I'm wondering if this was intentional, or an oversight? It seems to dramatically lessen the utility of providing an --exec-prefix to configure. Thanks, Ben

1 0

Accept just PEP-0426
by Daniel Holth 14 Jan '13

14 Jan '13

I think this PEP is a significant improvement from its predecessor. It represents features like extras (provides-extra) and build requirements (setup-requires-dist) that are in use in the Python community but cannot be represented in older versions of the format, it finally specifies a UTF-8 encoding, removes RFC 822, provides an extension mechanism, and allows the description to be placed in the document payload. PEP 426 doesn't have anything to do with the Wheel PEPs 425 and 427, other than that its features are necessary to usefully represent a large number of existing Python packages. How about moving this one along to focus on the other two. I'm not sure what the Post-History should be. We have been talking about it for a while. Thanks, Daniel Holth PEP: 426 Title: Metadata for Python Software Packages 1.3 Version: $Revision$ Last-Modified: $Date$ Author: Daniel Holth <dholth(a)fastmail.fm> Discussions-To: Distutils SIG Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 30 Aug 2012 Abstract ======== This PEP describes a mechanism for adding metadata to Python distributions. It includes specifics of the field names, and their semantics and usage. This document specifies version 1.3 of the metadata format. Version 1.0 is specified in PEP 241. Version 1.1 is specified in PEP 314. Version 1.2 is specified in PEP 345. Version 1.3 of the metadata format adds fields designed to make third-party packaging of Python Software easier and defines a formal extension mechanism. The fields are "Setup-Requires-Dist" "Provides-Extra", and "Extension". This version also adds the `extra` variable to the `environment markers` specification and allows the description to be placed into a payload section. Metadata Files ============== The syntax defined in this PEP is for use with Python distribution metadata files. The file format is a simple UTF-8 encoded Key: value format with case-insensitive keys and no maximum line length, followed by a blank line and an arbitrary payload. It is parseable by the ``email`` module with an appropriate ``email.policy.Policy()``. When ``metadata`` is a Unicode string, ```email.parser.Parser().parsestr(metadata)`` is a serviceable parser. There are two standard locations for these metadata files: * the ``PKG-INFO`` file included in the base directory of Python source distribution archives (as created by the distutils ``sdist`` command) * the ``.dist-info/METADATA`` files in a Python installation database, as described in PEP 376. Other tools involved in Python distribution may also use this format. Encoding ======== Metadata 1.3 files are UTF-8 with the restriction that keys must be ASCII. Parser implementations should be aware that older versions of the Metadata specification do not specify an encoding. Fields ====== This section specifies the names and semantics of each of the supported metadata fields. In a single Metadata 1.3 file, fields marked with "(optional)" may occur 0 or 1 times. Fields marked with "(multiple use)" may be specified 0, 1 or more times. Only "Metadata-Version", "Name", "Version", and "Summary" must appear exactly once. The fields may appear in any order within the file. Metadata-Version :::::::::::::::: Version of the file format; "1.3" is the only legal value. Example:: Metadata-Version: 1.3 Name :::: The name of the distribution. Example:: Name: BeagleVote Version ::::::: A string containing the distribution's version number. This field must be in the format specified in PEP 386. Example:: Version: 1.0a2 Summary ::::::: A one-line summary of what the distribution does. Example:: Summary: A module for collecting votes from beagles. Platform (multiple use) ::::::::::::::::::::::: A Platform specification describing an operating system supported by the distribution which is not listed in the "Operating System" Trove classifiers. See "Classifier" below. Examples:: Platform: ObscureUnix Platform: RareDOS Supported-Platform (multiple use) ::::::::::::::::::::::::::::::::: Binary distributions containing a metadata file will use the Supported-Platform field in their metadata to specify the OS and CPU for which the binary distribution was compiled. The semantics of the Supported-Platform field are not specified in this PEP. Example:: Supported-Platform: RedHat 7.2 Supported-Platform: i386-win32-2791 Description (optional, deprecated) :::::::::::::::::::::::::::::::::: A longer description of the distribution that can run to several paragraphs. Software that deals with metadata should not assume any maximum size for this field. The contents of this field can be written using reStructuredText markup [1]_. For programs that work with the metadata, supporting markup is optional; programs can also display the contents of the field as-is. This means that authors should be conservative in the markup they use. Since a line separator immediately followed by another line separator indicates the end of the headers section, any line separators in the description must be suffixed by whitespace to indicate continuation. Since Metadata 1.3 the recommended place for the description is in the payload section of the document, after the last header. The description does not need to be reformatted when it is included in the payload. Keywords (optional) ::::::::::::::::::: A list of additional keywords to be used to assist searching for the distribution in a larger catalog. Example:: Keywords: dog puppy voting election Home-page (optional) :::::::::::::::::::: A string containing the URL for the distribution's home page. Example:: Home-page: http://www.example.com/~cschultz/bvote/ Download-URL (optional) ::::::::::::::::::::::: A string containing the URL from which this version of the distribution can be downloaded. (This means that the URL can't be something like ".../BeagleVote-latest.tgz", but instead must be ".../BeagleVote-0.45.tgz".) Author (optional) ::::::::::::::::: A string containing the author's name at a minimum; additional contact information may be provided. Example:: Author: C. Schultz, Universal Features Syndicate, Los Angeles, CA <cschultz(a)peanuts.example.com> Author-email (optional) ::::::::::::::::::::::: A string containing the author's e-mail address. It contains a name and e-mail address in the RFC 5322 recommended ``Address Specification`` format. Example:: Author-email: "C. Schultz" <cschultz(a)example.com> Maintainer (optional) ::::::::::::::::::::: A string containing the maintainer's name at a minimum; additional contact information may be provided. Note that this field is intended for use when a project is being maintained by someone other than the original author: it should be omitted if it is identical to ``Author``. Example:: Maintainer: C. Schultz, Universal Features Syndicate, Los Angeles, CA <cschultz(a)peanuts.example.com> Maintainer-email (optional) ::::::::::::::::::::::::::: A string containing the maintainer's e-mail address. It has the same format as ``Author-email``. Note that this field is intended for use when a project is being maintained by someone other than the original author: it should be omitted if it is identical to ``Author-email``. Example:: Maintainer-email: "C. Schultz" <cschultz(a)example.com> License (optional) :::::::::::::::::: Text indicating the license covering the distribution where the license is not a selection from the "License" Trove classifiers. See "Classifier" below. This field may also be used to specify a particular version of a license which is named via the ``Classifier`` field, or to indicate a variation or exception to such a license. Examples:: License: This software may only be obtained by sending the author a postcard, and then the user promises not to redistribute it. License: GPL version 3, excluding DRM provisions The full text of the license would normally be included in a separate file. Classifier (multiple use) ::::::::::::::::::::::::: Each entry is a string giving a single classification value for the distribution. Classifiers are described in PEP 301 [2]. Examples:: Classifier: Development Status :: 4 - Beta Classifier: Environment :: Console (Text Based) Requires-Dist (multiple use) :::::::::::::::::::::::::::: Each entry contains a string naming some other distutils project required by this distribution. The format of a requirement string is identical to that of a distutils project name (e.g., as found in the ``Name:`` field. optionally followed by a version declaration within parentheses. The distutils project names should correspond to names as found on the `Python Package Index`_. Version declarations must follow the rules described in `Version Specifiers`_ Examples:: Requires-Dist: pkginfo Requires-Dist: PasteDeploy Requires-Dist: zope.interface (>3.5.0) Setup-Requires-Dist (multiple use) :::::::::::::::::::::::::::::::::: Like Requires-Dist, but names dependencies needed while the distributions's distutils / packaging `setup.py` / `setup.cfg` is run. Commonly used to generate a manifest from version control. Examples:: Setup-Requires-Dist: custom_setup_command Dependencies mentioned in `Setup-Requires-Dist` may be installed exclusively for setup and are not guaranteed to be available at run time. Provides-Dist (multiple use) :::::::::::::::::::::::::::: Each entry contains a string naming a Distutils project which is contained within this distribution. This field *must* include the project identified in the ``Name`` field, followed by the version : Name (Version). A distribution may provide additional names, e.g. to indicate that multiple projects have been bundled together. For instance, source distributions of the ``ZODB`` project have historically included the ``transaction`` project, which is now available as a separate distribution. Installing such a source distribution satisfies requirements for both ``ZODB`` and ``transaction``. A distribution may also provide a "virtual" project name, which does not correspond to any separately-distributed project: such a name might be used to indicate an abstract capability which could be supplied by one of multiple projects. E.g., multiple projects might supply RDBMS bindings for use by a given ORM: each project might declare that it provides ``ORM-bindings``, allowing other projects to depend only on having at most one of them installed. A version declaration may be supplied and must follow the rules described in `Version Specifiers`_. The distribution's version number will be implied if none is specified. Examples:: Provides-Dist: OtherProject Provides-Dist: AnotherProject (3.4) Provides-Dist: virtual_package Obsoletes-Dist (multiple use) ::::::::::::::::::::::::::::: Each entry contains a string describing a distutils project's distribution which this distribution renders obsolete, meaning that the two projects should not be installed at the same time. Version declarations can be supplied. Version numbers must be in the format specified in `Version Specifiers`_. The most common use of this field will be in case a project name changes, e.g. Gorgon 2.3 gets subsumed into Torqued Python 1.0. When you install Torqued Python, the Gorgon distribution should be removed. Examples:: Obsoletes-Dist: Gorgon Obsoletes-Dist: OtherProject (<3.0) Requires-Python (optional) :::::::::::::::::::::::::: This field specifies the Python version(s) that the distribution is guaranteed to be compatible with. Version numbers must be in the format specified in `Version Specifiers`_. Examples:: Requires-Python: 2.5 Requires-Python: >2.1 Requires-Python: >=2.3.4 Requires-Python: >=2.5,<2.7 Requires-External (multiple use) :::::::::::::::::::::::::::::::: Each entry contains a string describing some dependency in the system that the distribution is to be used. This field is intended to serve as a hint to downstream project maintainers, and has no semantics which are meaningful to the ``distutils`` distribution. The format of a requirement string is a name of an external dependency, optionally followed by a version declaration within parentheses. Because they refer to non-Python software releases, version numbers for this field are **not** required to conform to the format specified in PEP 386: they should correspond to the version scheme used by the external dependency. Notice that there's is no particular rule on the strings to be used. Examples:: Requires-External: C Requires-External: libpng (>=1.5) Project-URL (multiple use) :::::::::::::::::::::::::: A string containing a label and a browsable URL for the project, separated by the last occurrence of comma and space ", ". Example:: Bug, Issue Tracker, http://bitbucket.org/tarek/distribute/issues/ The label is a free text. Provides-Extra (multiple use) ::::::::::::::::::::::::::::: A string containing the name of an optional feature. Must be printable ASCII, not containing whitespace, comma (,), or square brackets []. May be used to make a dependency conditional on whether the optional feature has been requested. Example:: Name: beaglevote Provides-Extra: pdf Requires-Dist: reportlab; extra == 'pdf' Requires-Dist: nose; extra == 'test' Requires-Dist: sphinx; extra == 'doc' A second distribution requires an optional dependency by placing it inside square brackets and can request multiple features by separating them with a comma (,). The full set of requirements is the union of the `Requires-Dist` sets evaluated with `extra` set to `None` and then to the name of each requested feature. Example:: Requires-Dist: beaglevote[pdf] -> requires beaglevote, reportlab Requires-Dist: beaglevote[test, doc] -> requires beaglevote, sphinx, nose Two feature names `test` and `doc` are reserved to mark dependencies that are needed for running automated tests and generating documentation, respectively. It is legal to specify `Provides-Extra` without referencing it in any `Requires-Dist`. It is an error to request a feature name that has not been declared with `Provides-Extra`. Extension (multiple use) :::::::::::::::::::::::: An ASCII string, not containing whitespace or the / character, that indicates the presence of extended metadata. Additional tags defined by an `Extension: Chili` must be of the form `Chili/Name`:: Extension: Chili Chili/Type: Poblano Chili/Heat: Mild An implementation might iterate over all the declared `Extension:` fields to invoke the processors for those extensions. As the order of the fields is not used, the `Extension: Chili` field may appear before or after its declared tags `Chili/Type:` etc. Version Specifiers ================== Version specifiers are a series of conditional operators and version numbers, separated by commas. Conditional operators must be one of "<", ">", "<=", ">=", "==" and "!=". Any number of conditional operators can be specified, e.g. the string ">1.0, !=1.3.4, <2.0" is a legal version declaration. The comma (",") is equivalent to the **and** operator. Each version number must be in the format specified in PEP 386. When a version is provided, it always includes all versions that starts with the same value. For example the "2.5" version of Python will include versions like "2.5.2" or "2.5.3". Pre and post releases in that case are excluded. So in our example, versions like "2.5a1" are not included when "2.5" is used. If the first version of the range is required, it has to be explicitly given. In our example, it will be "2.5.0". Notice that some projects might omit the ".0" prefix for the first release of the "2.5.x" series: - 2.5 - 2.5.1 - 2.5.2 - etc. In that case, "2.5.0" will have to be explicitly used to avoid any confusion between the "2.5" notation that represents the full range. It is a recommended practice to use schemes of the same length for a series to completely avoid this problem. Some Examples: - ``Requires-Dist: zope.interface (3.1)``: any version that starts with 3.1, excluding post or pre-releases. - ``Requires-Dist: zope.interface (==3.1)``: equivalent to ``Requires-Dist: zope.interface (3.1)``. - ``Requires-Dist: zope.interface (3.1.0)``: any version that starts with 3.1.0, excluding post or pre-releases. Since that particular project doesn't use more than 3 digits, it also means "only the 3.1.0 release". - ``Requires-Python: 3``: Any Python 3 version, no matter wich one, excluding post or pre-releases. - ``Requires-Python: >=2.6,<3``: Any version of Python 2.6 or 2.7, including post releases of 2.6, pre and post releases of 2.7. It excludes pre releases of Python 3. - ``Requires-Python: 2.6.2``: Equivalent to ">=2.6.2,<2.6.3". So this includes only Python 2.6.2. Of course, if Python was numbered with 4 digits, it would have include all versions of the 2.6.2 series. - ``Requires-Python: 2.5.0``: Equivalent to ">=2.5.0,<2.5.1". - ``Requires-Dist: zope.interface (3.1,!=3.1.3)``: any version that starts with 3.1, excluding post or pre-releases of 3.1 *and* excluding any version that starts with "3.1.3". For this particular project, this means: "any version of the 3.1 series but not 3.1.3". This is equivalent to: ">=3.1,!=3.1.3,<3.2". Environment markers =================== An **environment marker** is a marker that can be added at the end of a field after a semi-colon (";"), to add a condition about the execution environment. Here are some example of fields using such markers:: Requires-Dist: pywin32 (>1.0); sys.platform == 'win32' Obsoletes-Dist: pywin31; sys.platform == 'win32' Requires-Dist: foo (1,!=1.3); platform.machine == 'i386' Requires-Dist: bar; python_version == '2.4' or python_version == '2.5' Requires-External: libxslt; 'linux' in sys.platform The micro-language behind this is a simple subset of Python: it compares only strings, with the ``==`` and ``in`` operators (and their opposites), and with the ability to combine expressions. Parenthesis are supported for grouping. The pseudo-grammar is :: EXPR [in|==|!=|not in] EXPR [or|and] ... where ``EXPR`` belongs to any of those: - python_version = '%s.%s' % (sys.version_info[0], sys.version_info[1]) - python_full_version = sys.version.split()[0] - os.name = os.name - sys.platform = sys.platform - platform.version = platform.version() - platform.machine = platform.machine() - platform.python_implementation = platform.python_implementation() - a free string, like ``'2.4'``, or ``'win32'`` - extra = (name of requested feature) or None Notice that ``in`` is restricted to strings, meaning that it is not possible to use other sequences like tuples or lists on the right side. The fields that benefit from this marker are: - Requires-Python - Requires-External - Requires-Dist - Setup-Requires-Dist - Provides-Dist - Obsoletes-Dist - Classifier (The `extra` variable is only meaningful for Requires-Dist.) Summary of Differences From PEP 345 =================================== * Metadata-Version is now 1.3. * Values are now expected to be UTF-8. * A payload (containing the description) may appear after the headers. * Added `extra` to environment markers. * Most fields are now optional. * Changed fields: - Description - Project-URL - Requires-Dist * Added fields: - Extension - Provides-Extra - Setup-Requires-Dist References ========== This document specifies version 1.3 of the metadata format. Version 1.0 is specified in PEP 241. Version 1.1 is specified in PEP 314. Version 1.2 is specified in PEP 345. .. [1] reStructuredText markup: http://docutils.sourceforge.net/ .. _`Python Package Index`: http://pypi.python.org/pypi/ .. [2] PEP 301: http://www.python.org/dev/peps/pep-0301/ Appendix ======== Parsing and generating the Metadata 1.3 serialization format using Python 3.3:: # Metadata 1.3 demo from email.generator import Generator from email import header from email.parser import Parser from email.policy import Compat32 from email.utils import _has_surrogates class MetadataPolicy(Compat32): max_line_length = 0 continuation_whitespace = '\t' def _sanitize_header(self, name, value): if not isinstance(value, str): return value if _has_surrogates(value): raise NotImplementedError() else: return value def _fold(self, name, value, sanitize): body = ((self.linesep+self.continuation_whitespace) .join(value.splitlines())) return ''.join((name, ': ', body, self.linesep)) if __name__ == "__main__": import sys import textwrap pkg_info = """\ Metadata-Version: 1.3 Name: package Version: 0.1.0 Summary: A package. Description: Description =========== A description of the package. """ m = Parser(policy=MetadataPolicy()).parsestr(pkg_info) m['License'] = 'GPL' description = m['Description'] description_lines = description.splitlines() m.set_payload(description_lines[0] + '\n' + textwrap.dedent('\n'.join(description_lines[1:])) + '\n') del m['Description'] # Correct if sys.stdout.encoding == 'UTF-8': Generator(sys.stdout, maxheaderlen=0).flatten(m) Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 End:

20 111

Issue about LoadError
by lname name 14 Jan '13

14 Jan '13

http://bugs.python.org/issue16715 I assume that someone changes the LoadError class and goes into the revert function. How he can figure out that the cookies should return to the previous state if LoadError was raised ?

1 0