Mailman 3 November 2011 - Python-ideas

Re: [Python-ideas] A concurrency survey of sorts
by Weeble 06 Nov '11

06 Nov '11

> Regardless, I still think the GIL should be fixed or the threading module > removed. It's disingenuous to have a threading module when it doesn't work > as advertised due to an interpreter "feature" of dubious merit anyway. Is this a serious proposal? Without threads, is there any reasonable cross-platform way to communicate interactively with multiple subprocesses over pipes? Windows doesn't have select. I don't *like* using threads, but as far as I can tell they're the only available way to do this.

6 8

Need way to check if a `datetime.timedelta` is positive
by Ram Rachum 05 Nov '11

05 Nov '11

(Sorry for possible double-post, Google Groups was doing weird things.) Hey, I'm currently writing code that's manipulating `datetime.timedelta` objects. I get one and I need to check whether it's positive or negative. I can't really think of an elegant way to do this. Here's one: my_timedelta >= datetime.timedelta(0) Another one: my_timedelta.total_seconds() >= 0 Perhaps we should allow a more elegant way? Possibly `my_timedelta >= 0` for 0 only? Thanks, Ram.

4 3

Fwd: Concurrent safety?
by Mike Meyer 05 Nov '11

05 Nov '11

Sigh. Sent this only to bruce by accident. Sorry about the duplicates Bruce. ---------- Forwarded message ---------- From: Mike Meyer <mwm(a)mired.org> Date: Mon, Oct 31, 2011 at 10:30 PM Subject: Re: [Python-ideas] Concurrent safety? To: Bruce Leban <bruce(a)leapyear.org> On Mon, 31 Oct 2011 18:07:19 -0700 Bruce Leban <bruce(a)leapyear.org> wrote: > On Mon, Oct 31, 2011 at 4:19 PM, Mike Meyer <mwm(a)mired.org> wrote: > >> That would mean that all of my current code is broken. > > Pretty much, yes. It's like adding garbage collection and removing > > alloc*/free. It's going to break a *lot* of code. That's why I said "not in > > 3. and possibly never in cPython." > In order to make concurrent code slightly safer, you're going to break all > existing programs that don't use concurrency. That seems to me like a new > language, not Python. You've been on this list long enough to see the > attention that's paid to backward compatibility. The way adding automatic memory management just made pointer arithmetic slightly safer. But yeah, the first thing I said was "never in 3.x, possibly never in cPython." I've been on this list long enough to know that, while the community pays a lot of attention to backwards compatibility, it is willing to throw it out when there's enough benefit. As you point out, this is a hard problem. I know I haven't covered all the issues. That's why the second thing I said was that I'm hoping to get people smarter than me to look at things. The cpu manufacturers have switched to improving performance by adding more cores instead of cranking clock speeds. As time goes by, more and more programmers are going to want to leverage that in serious ways. I already do, and find that Python makes some design choices unavailable or very fragile(*). I'd like to make those choices available, and help Python get ready for the time when that desire is the norm instead of the exception. > > Furthermore, merely *reading* an object that isn't locked can cause > >> problems. This code is not thread-safe: > >> if element in dictionary: return dictionary[element] > >> so you have to decide how much safety you want and what cost we're > >> willing to pay for this. > > You're right - it's not thread safe. However, it also doesn't suffer from > > the problem I'm trying to deal with, where you mutate an object in a way > > that leaves things broken, but won't be detected at that point. If it > > breaks because someone mutates the object underneath it, it'll throw an > > exception at that point. I know you can construct cases where that isn't > > so. > I think the cases where non-thread-safe code won't throw an exception are > numerous, for example, the equally trivial: Again, I said such cases can be built. I *didn't* say they were exceptions, I proposed a change to deal with them. > If you're going to tackle thread safety, it should address more of the > problem. These bugs are in many ways worse than mutating "an object in a > way that leaves things broken, but won't be detected at that point." The > above bugs may *never* be detected. I've come across bugs like that that > were in code for many years before I found them (and I'm sure that's > happened to others on this list as well). Like me. That's part of why I want to get the interpreter to help find them. > The first thing to do is identify the problems you want to solve and make > sure that the problems are well understood. Then design some solutions. > Starting with a bad solution to a fraction of the problem isn't a good > start. I've identified the problem I want to solve: I want to make concurrent use of python objects "safe by default", so that doing unsafe things causes the programmer to have to do something explicit about making things safe. I believe this can be done at the mutation points (again, clojure shows that it can be done). I also want to preserve as much of Python's existing code as possible. It may be that Python's existing data structures mean my believe about mutation points is wrong. This may be the wrong solution. It may be that such a change is to large to be acceptable. But the only way to find out is to investigate it. This discussion has already generated significant changes in the original proposal, plus some implementation ideas. <mike *) I'll note that some of the alternatives make the choices available in Python unavailable in a similar way. -- Mike Meyer <mwm(a)mired.org> http://www.mired.org/ Independent Software developer/SCM consultant, email for more information. O< ascii ribbon campaign - stop html mail - www.asciiribbon.org

9 43

Re: [Python-ideas] PEP 3155 - Qualified name for classes and functions
by Joshua Landau 04 Nov '11

04 Nov '11

First of all - hello python-ideas. I liked the look of the archives, so I thought I'd chip in, if it's OK. > And what about: >> >> <f locals>.g >> and >> <<xyzzy locals>.f locals>.g >> > > In the end I like the non-nested version better. > xyzzy.<local f>.<local g> or, preferably, xyzzy.<instance.f>.<instance.g> It's non-nested, shows the locality and maps the locality to the variable. If not showing function instances is disliked, maybe drop the period between the function and "<locals>" from yours: xyzzy<locals>.f<locals>.g or, again, xyzzy<instance>.f<instance>.g I think this is just cleaner than: xyzzy.<locals>.f.<locals>.g because it doesn't imply that xyzzy has a single set of locals - especially using <instance>.

1 0

A concurrency survey of sorts
by Mike Meyer 04 Nov '11

04 Nov '11

In order to get a better idea of where things stand, I'd like to get answers to a few questions. This isn't a traditional broadbased survey, but an attempt to get answers from a few people who might know or have good ideas. This is probably where I should have started, but better late than never. 1) How much of the Python standard library is known to be thread safe? 2) How many packages in PyPI are known to be thread safe? 3) Can you suggest another approach to getting safe high-performance shared data in concurrent operation? I've already considered: a) I proposed making actions that mutate data require locked objects, because I've seen that work in other languages. I recognize that doesn't mean it will work in Python, but it's more than I can say about the alternatives I knew about then., b) Bertrand Meyer's SCOOPS system, designed for Eiffel. It has two major strikes against it: 1) it is based on type attributes on *variables*, andI could figure out how to translate that to a language where variables aren't typed. 2) I don't know that there's a working implementation. 4) Can you suggest a minor change that would move things toward safer concurrent code with high-performance shared data? I can see two possibilities: a) Audit any parts of the standard library that aren't already known to be thread safe, and flag those that aren't. Fixing them may want to wait on a better mechanism than posix locks. b) Add a high-level, high-performance shared object facility to the multiprocess package. Thanks, <mike

8 8

Testing with numbers
by Yuval Greenfield 03 Nov '11

03 Nov '11

Working on algorithms that don't have just the boolean states of "success" or "failure", I find myself wanting to have regular unit tests adjacent to where I put my scalar result of how well the algorithm worked. E.g. the percentage of test samples failed, an x minus y squared sum, etc. As I understand it unittest doesn't have the facilities for such test cases or results. One could use assertGreater to test for passing a threshold but usually the acceptable numbers are rapidly changing. When running a test, you'd want to see the resulting number instead of an F/E/. result. Another use case is pypy speed benchmarks. Are there others here who would/could use this? --Yuval Greenfield

2 1

Changing str(someclass) to return only the class name
by Éric Araujo 02 Nov '11

02 Nov '11

Hi everyone, I’ve sometimes wished that str(someclass) were the natural way to get a class name, for example when writing __repr__ methods. Guido expressed the same wish recently, so I’ve made a patch for Python 3.3. In the whole standard library, only one module (optparse) and three test files (which were parsing the output of str(someclass) or using doctests) had to be updated because of this change: http://bugs.python.org/issue13224 One can say that this change would break code and should thus be rejected; one could argue that the previous behavior of str(someclass) was undefined, or an implementation detail. What say you? Cheers

18 46

Draft PEP for the regularization of Python install layouts
by VanL 01 Nov '11

01 Nov '11

In part inspired by the virtualenv-in-the-stdlib PEP, I figured that it might be a good time to draft up a PEP to fix one of my regular annoyances: the ever-so-slightly different layouts for Python between platforms. For someone who develops on Windows and a Mac and deploys on Linux, this is a major pain in the rear - so much so that I always change my system Python install to have matching environments. I harbor no illusions that this is necessarily a thorn in anyone else's side, or that this will go anywhere. However, I have written it up and I would love any feedback. Thanks, Van ===================================== PEP: XXX Title: Consistent Python Environment Layout Across Platforms Version: $Revision$ Last-Modified: $Date$ Author: Van Lindberg <van.lindberg at gmail.com> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 28-Oct-2011 Python-Version: 3.3 Post-History: 28-Oct-2011 Abstract ======== Python currently uses different environment layouts based upon the underlying operating system and the interpreter implementation language. This PEP proposes to regularize the directory layout for Python environments. Motivation ========== One of Python's strengths is its cross-platform appeal. Carefully- written Python programs are frequently portable between operating systems and Python implementations with very few changes. Over the years, substantial effort has been put into maintaining platform parity and providing consistent interfaces to available functionality, even when different underlying implementations are necessary (such as with ntpath and posixpath). One place where Python is unnecessarily different, however, is in the layout and organization of the Python environment. This is most visible in the name of the directory for binaries on the Windows platform ("Scripts") versus the name of the directory for binaries on every other platform ("bin"), but a full listing of the layouts shows substantial differences in layout and capitalization across platforms. Sometimes the include is capitalized ("Include"), and sometimes not; and the python version may or may not be included in the path to the standard library or not. The differences between platforms become particularly noticable when attempting to do cross-platform development in an isolated environment like a `virtualenv`_ or the proposed Python Virtual Environments. Differences between platforms get hard-coded in or worked around so that the same code will just work without having to artificially take the platform into account. This information can also be made available to Python tools (like virtualenv, `distribute`_, or `pip`_) so that environment directories can be more easily found and changed. Regularizing the Python environment layout across Python versions will lower the differences between Python versions, making it more consistent for all users, and enforcing a unified API for virtual environments and installers rather than ad-hoc detection or hard-coded values. .. _virtualenv: http://www.virtualenv.org .. _distribute: http://packages.python.org/distribute/ .. _pip: http://www.pip-installer.org/ Specification ============= When the Python binary is executed, it imports both sysconfig and distutils.command.install. These two modules contain slightly different versions of the environment layout. They are: From sysconfig: _INSTALL_SCHEMES = { 'posix_prefix': { 'stdlib': '{base}/lib/python{py_version_short}', 'platstdlib': '{platbase}/lib/python{py_version_short}', 'purelib': '{base}/lib/python{py_version_short}/site-packages', 'platlib': '{platbase}/lib/python{py_version_short}/site-packages', 'include': '{base}/include/python{py_version_short}', 'platinclude': '{platbase}/include/python{py_version_short}', 'scripts': '{base}/bin', 'data': '{base}', }, 'posix_home': { 'stdlib': '{base}/lib/python', 'platstdlib': '{base}/lib/python', 'purelib': '{base}/lib/python', 'platlib': '{base}/lib/python', 'include': '{base}/include/python', 'platinclude': '{base}/include/python', 'scripts': '{base}/bin', 'data' : '{base}', }, 'nt': { 'stdlib': '{base}/Lib', 'platstdlib': '{base}/Lib', 'purelib': '{base}/Lib/site-packages', 'platlib': '{base}/Lib/site-packages', 'include': '{base}/Include', 'platinclude': '{base}/Include', 'scripts': '{base}/Scripts', 'data' : '{base}', }, 'os2': { 'stdlib': '{base}/Lib', 'platstdlib': '{base}/Lib', 'purelib': '{base}/Lib/site-packages', 'platlib': '{base}/Lib/site-packages', 'include': '{base}/Include', 'platinclude': '{base}/Include', 'scripts': '{base}/Scripts', 'data' : '{base}', }, 'os2_home': { 'stdlib': '{userbase}/lib/python{py_version_short}', 'platstdlib': '{userbase}/lib/python{py_version_short}', 'purelib': '{userbase}/lib/python{py_version_short}/site-packages', 'platlib': '{userbase}/lib/python{py_version_short}/site-packages', 'include': '{userbase}/include/python{py_version_short}', 'scripts': '{userbase}/bin', 'data' : '{userbase}', }, 'nt_user': { 'stdlib': '{userbase}/Python{py_version_nodot}', 'platstdlib': '{userbase}/Python{py_version_nodot}', 'purelib': '{userbase}/Python{py_version_nodot}/site-packages', 'platlib': '{userbase}/Python{py_version_nodot}/site-packages', 'include': '{userbase}/Python{py_version_nodot}/Include', 'scripts': '{userbase}/Scripts', 'data' : '{userbase}', }, 'posix_user': { 'stdlib': '{userbase}/lib/python{py_version_short}', 'platstdlib': '{userbase}/lib/python{py_version_short}', 'purelib': '{userbase}/lib/python{py_version_short}/site-packages', 'platlib': '{userbase}/lib/python{py_version_short}/site-packages', 'include': '{userbase}/include/python{py_version_short}', 'scripts': '{userbase}/bin', 'data' : '{userbase}', }, 'osx_framework_user': { 'stdlib': '{userbase}/lib/python', 'platstdlib': '{userbase}/lib/python', 'purelib': '{userbase}/lib/python/site-packages', 'platlib': '{userbase}/lib/python/site-packages', 'include': '{userbase}/include', 'scripts': '{userbase}/bin', 'data' : '{userbase}', }, } From distutils.command.install: if sys.version < "2.2": WINDOWS_SCHEME = { 'purelib': '$base', 'platlib': '$base', 'headers': '$base/Include/$dist_name', 'scripts': '$base/Scripts', 'data' : '$base', } else: WINDOWS_SCHEME = { 'purelib': '$base/Lib/site-packages', 'platlib': '$base/Lib/site-packages', 'headers': '$base/Include/$dist_name', 'scripts': '$base/Scripts', 'data' : '$base', } INSTALL_SCHEMES = { 'unix_prefix': { 'purelib': '$base/lib/python$py_version_short/site-packages', 'platlib': '$platbase/lib/python$py_version_short/site-packages', 'headers': '$base/include/python$py_version_short/$dist_name', 'scripts': '$base/bin', 'data' : '$base', }, 'unix_home': { 'purelib': '$base/lib/python', 'platlib': '$base/lib/python', 'headers': '$base/include/python/$dist_name', 'scripts': '$base/bin', 'data' : '$base', }, 'unix_user': { 'purelib': '$usersite', 'platlib': '$usersite', 'headers': '$userbase/include/python$py_version_short/$dist_name', 'scripts': '$userbase/bin', 'data' : '$userbase', }, 'nt': WINDOWS_SCHEME, 'nt_user': { 'purelib': '$usersite', 'platlib': '$usersite', 'headers': '$userbase/Python$py_version_nodot/Include/$dist_name', 'scripts': '$userbase/Scripts', 'data' : '$userbase', }, 'os2': { 'purelib': '$base/Lib/site-packages', 'platlib': '$base/Lib/site-packages', 'headers': '$base/Include/$dist_name', 'scripts': '$base/Scripts', 'data' : '$base', }, 'os2_home': { 'purelib': '$usersite', 'platlib': '$usersite', 'headers': '$userbase/include/python$py_version_short/$dist_name', 'scripts': '$userbase/bin', 'data' : '$userbase', }, } There is an API call (sysconfig.get_config_var) that is used to resolve these various variables, but there is no way to set them other than directly clobbering _INSTALL_SCHEMES/INSTALL_SCHEME dicts holding the information (in both sysconfig and in distutils.command.install). This PEP proposes to change the default installation layout as follows: - Define a new config var, py_binaries, and a corresponding module-level variable, _PY_BINARIES with the default value 'bin' - Change sysconfig's _INSTALL_SCHEMES to INSTALL_SCHEMES, and make it part of the public API. A default scheme equivalent to the current posix_prefix would be added and made to be the default. - To support virtual environments, a special key "current" would be added to INSTALL_SCHEMES. The value associated with "current" would be the name of the currently active scheme. During activation, a virtual environment could place a custom layout in the INSTALL_SCHEMES dict and set "current" to the custom scheme. This could also be used to support legacy or custom layouts. This would be done as follows:: from _collections import defaultdict _DEFAULT_SCHEME = { 'stdlib': '{userbase}/lib/python{py_version_short}', 'platstdlib': '{userbase}/lib/python{py_version_short}', 'purelib': '{userbase}/lib/python{py_version_short}/site-packages', 'platlib': '{userbase}/lib/python{py_version_short}/site-packages', 'include': '{userbase}/include/python{py_version_short}', 'headers': '{userbase}/include/python{py_version_short}', 'scripts': '{userbase}/{py_binaries}', 'data' : '{userbase}', } INSTALL_SCHEMES = defaultdict(lambda: _DEFAULT_SCHEME.copy()) INSTALL_SCHEMES['install'] = _DEFAULT_SCHEME INSTALL_SCHEMES['current'] = 'install' - Change distutils.command.install to use information from INSTALL_SCHEMES instead of embedding its own copy of the dict. By using a defaultdict, prior users of the API could transparently query the dict using the platform value and receive back a reasonable response. The current virtual environment would be retrievable by getting INSTALL_SCHEMES[INSTALL_SCHEMES['current']], and the installed scheme retrievable by getting INSTALL_SCHEMES['install']. Open Issues ========== By default, OS X Framework installs use a format that is different than the one specified above. Possible responses would be to change the layout just for OS X, but special-casing as little as possible. The double indirection to get the current install scheme is a little ugly, but I am not sure what would be better. This explicitly removes the compatibility code in distutils for 2.x install layouts (particularly layouts pre-2.3). This is not considered a major issue, as 2.2 code will largely not be compatible with 3.3, and an individual user could add a custom layout if needed. Based on actual use, some of the distinctions that appeared necessary when distutils and sysconfig were first written (platlib, purelib, include, headers) seem to be duplicative. As written above, the distinctions are maintained for ease of moving to the standardized layout, but it may be worth choosing just one name and sticking with it.

3 3

Concurrent safety?
by Mike Meyer 01 Nov '11

01 Nov '11

This is a blue sky idea. It can't happen in Python 3.x, and possibly not ever in cPython. I'm mostly hoping to get smarter people than me considering the issue. Synopsis Python, as a general rule, tries to be "safe" about things. If something isn't obviously correct, it tends to throw runtime errors to let you know that you need to be explicit about what you want. When there's not an obvious choice for type conversions, it raises an exception. You generally don't have to worry about resource allocation if you stay in python. And so on. The one glaring exception is in concurrent programs. While the tools python has for dealing with such are ok, there isn't anything to warn you when you fail to use those tools and should be. The goal of this proposal is to fix that, and get the Python interpreter to help locate code that isn't safe to use in concurrent programs. Existence Proof This is possible. Clojure is a dynamic language in the LISP family that will throw exceptions if you try mutating variables without properly protecting them against concurrent access. This is not to say that the Clojure solution is the solution, or even the right solution for Python. It's just to demonstrate that this can be done. Object Changes Object semantics don't need to change very much. The existing immutable types will work well in this environment exactly as is. The mutable types - well, we can no longer go changing them willy-nilly. But any language needs mutable types, and there's nothing wrong with the ones we have. Since immutable types don't require access protection *at all*, it might be worthwhile to add a new child of object, "immutable". Instances of this type would be immutable after creation. Presumably, the __new__ method of Python classes inheriting from immutable would be used to set the initial attributes, but the __init__ method might also be able to handle that role. However, this is a performance tweak, allowing user-written classes to skip any runtime checks for being mutated. Binding Changes One of the way objects are mutated is by changing their bindings. As such, some of the bindings might need to be protected. Local variables are fine. We normally can't export those bindings to other functions, just the values bound to them. So changing the binding can stay the same. The bound object can be exported to other threads of execution, but changing it will fall under the rules for changing objects. Ditto for nonlocals. On the other hand, rebindings of module and class and instance variables can be visible in other threads of execution, so they require protection, just like changing mutable objects. New Syntax The protection mechanism is the change to the language. I propose a single new keyword, "locking", that acts similar to the "try" keyword. The syntax is: 'locking' value [',' value]*':' suite The list of values are the objects that can be mutated in this lock. An immutable object showing up in the list of values is a TypeError. It's not clear that function calls should be allowed in the list of values. On the other hand, indexing and attributes clearly should be, and those can turn into function calls, so it's not clear they shouldn't be allowed, either. The locked values can be mutated during the body of the locking suite. For the builtin mutable types, this means invoking their mutating methods. For modules, classes and object instances, it means rebinding their attributes. Locked objects stay locked during function invocations in the suite. This means you can write utility functions that expect to be passed locked objects to mutate. A locking statement can be used inside of another locking statement. See the Implementation section for possible restrictions on this. Any attempt to mutate an object that isn't currently locked will raise an exception. Possibly ValueError, possibly a new exception class just for this purpose. This includes rebinding attributes of objects that aren't locked. Implementation There are at least two ways this can be implemented, both with different restrictions on the suite. While both of them can probably be optimized if it's know that there are no other threads of execution, checking for attempts to mutate unlocked objects should still happen. 1) Conventional locking All the objects being locked have locks attached to them, which are locked when before entering the suite. The implementation must order the locked object in some repeatable way, so that two locking statements that have more than one locked object in common will obtain the locks on those objects in the same order. This will prevent deadlocks. This method will require that the initial locking statement lock all objects that may be locked during the execution of it's suite. This may be a reason for allowing functions as locking values, as a way to get locks on objects that code called in the suite is going to need. Another downside is that the programmer needs to handle exceptions raised during the suite to insure that a set of related changes leaves the relevant objects in a consistent state. In this case, an optional 'except' clause should be added to the locking statement to hold such code. 2) Software Transactional Memory In an STM implementation, copies of the locked objects are created by the locking statement, and they original are "fingerprinted" in some way. The locking suite then runs. When the suite completes, the fingerprints of the originals are checked to see if some other thread of execution has changed them. If they haven't changed, they are replaced by the copies, and execution continues. If the originals have changed, the entire process starts over. In this implementation, the only actual locking is during the original fingerprinting process (to insure that a consistent state is captured) and at the end of the suite. FWIW, this is one of the models provided by Clojure. The restriction on the suite in this case is that running it twice - except for changes to the locked objects - needs to be acceptable. In this case, exceptions don't need to be handled by the programmer to insure consistency. If an exception happens during the execution of the suite, the original values are never replaced. <mike -- Mike Meyer <mwm(a)mired.org> http://www.mired.org/ Independent Software developer/SCM consultant, email for more information. O< ascii ribbon campaign - stop html mail - www.asciiribbon.org

8 16