Mailman 3 November 2011 - Python-ideas

Testing with numbers
by Yuval Greenfield Nov. 3, 2011

Nov. 3, 2011

Working on algorithms that don't have just the boolean states of "success" or "failure", I find myself wanting to have regular unit tests adjacent to where I put my scalar result of how well the algorithm worked. E.g. the percentage of test samples failed, an x minus y squared sum, etc. As I understand it unittest doesn't have the facilities for such test cases or results. One could use assertGreater to test for passing a threshold but usually the acceptable numbers are rapidly … [View More]

2 1

Changing str(someclass) to return only the class name
by Éric Araujo Nov. 2, 2011

Nov. 2, 2011

Hi everyone, I’ve sometimes wished that str(someclass) were the natural way to get a class name, for example when writing __repr__ methods. Guido expressed the same wish recently, so I’ve made a patch for Python 3.3. In the whole standard library, only one module (optparse) and three test files (which were parsing the output of str(someclass) or using doctests) had to be updated because of this change: http://bugs.python.org/issue13224 One can say that this change would break code and … [View More]

18 46

Draft PEP for the regularization of Python install layouts
by VanL Nov. 1, 2011

Nov. 1, 2011

In part inspired by the virtualenv-in-the-stdlib PEP, I figured that it might be a good time to draft up a PEP to fix one of my regular annoyances: the ever-so-slightly different layouts for Python between platforms. For someone who develops on Windows and a Mac and deploys on Linux, this is a major pain in the rear - so much so that I always change my system Python install to have matching environments. I harbor no illusions that this is necessarily a thorn in anyone else's side, or … [View More]that this will go anywhere. However, I have written it up and I would love any feedback. Thanks, Van ===================================== PEP: XXX Title: Consistent Python Environment Layout Across Platforms Version: $Revision$ Last-Modified: $Date$ Author: Van Lindberg <van.lindberg at gmail.com> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 28-Oct-2011 Python-Version: 3.3 Post-History: 28-Oct-2011 Abstract ======== Python currently uses different environment layouts based upon the underlying operating system and the interpreter implementation language. This PEP proposes to regularize the directory layout for Python environments. Motivation ========== One of Python's strengths is its cross-platform appeal. Carefully- written Python programs are frequently portable between operating systems and Python implementations with very few changes. Over the years, substantial effort has been put into maintaining platform parity and providing consistent interfaces to available functionality, even when different underlying implementations are necessary (such as with ntpath and posixpath). One place where Python is unnecessarily different, however, is in the layout and organization of the Python environment. This is most visible in the name of the directory for binaries on the Windows platform ("Scripts") versus the name of the directory for binaries on every other platform ("bin"), but a full listing of the layouts shows substantial differences in layout and capitalization across platforms. Sometimes the include is capitalized ("Include"), and sometimes not; and the python version may or may not be included in the path to the standard library or not. The differences between platforms become particularly noticable when attempting to do cross-platform development in an isolated environment like a `virtualenv`_ or the proposed Python Virtual Environments. Differences between platforms get hard-coded in or worked around so that the same code will just work without having to artificially take the platform into account. This information can also be made available to Python tools (like virtualenv, `distribute`_, or `pip`_) so that environment directories can be more easily found and changed. Regularizing the Python environment layout across Python versions will lower the differences between Python versions, making it more consistent for all users, and enforcing a unified API for virtual environments and installers rather than ad-hoc detection or hard-coded values. .. _virtualenv: http://www.virtualenv.org .. _distribute: http://packages.python.org/distribute/ .. _pip: http://www.pip-installer.org/ Specification ============= When the Python binary is executed, it imports both sysconfig and distutils.command.install. These two modules contain slightly different versions of the environment layout. They are: From sysconfig: _INSTALL_SCHEMES = { 'posix_prefix': { 'stdlib': '{base}/lib/python{py_version_short}', 'platstdlib': '{platbase}/lib/python{py_version_short}', 'purelib': '{base}/lib/python{py_version_short}/site-packages', 'platlib': '{platbase}/lib/python{py_version_short}/site-packages', 'include': '{base}/include/python{py_version_short}', 'platinclude': '{platbase}/include/python{py_version_short}', 'scripts': '{base}/bin', 'data': '{base}', }, 'posix_home': { 'stdlib': '{base}/lib/python', 'platstdlib': '{base}/lib/python', 'purelib': '{base}/lib/python', 'platlib': '{base}/lib/python', 'include': '{base}/include/python', 'platinclude': '{base}/include/python', 'scripts': '{base}/bin', 'data' : '{base}', }, 'nt': { 'stdlib': '{base}/Lib', 'platstdlib': '{base}/Lib', 'purelib': '{base}/Lib/site-packages', 'platlib': '{base}/Lib/site-packages', 'include': '{base}/Include', 'platinclude': '{base}/Include', 'scripts': '{base}/Scripts', 'data' : '{base}', }, 'os2': { 'stdlib': '{base}/Lib', 'platstdlib': '{base}/Lib', 'purelib': '{base}/Lib/site-packages', 'platlib': '{base}/Lib/site-packages', 'include': '{base}/Include', 'platinclude': '{base}/Include', 'scripts': '{base}/Scripts', 'data' : '{base}', }, 'os2_home': { 'stdlib': '{userbase}/lib/python{py_version_short}', 'platstdlib': '{userbase}/lib/python{py_version_short}', 'purelib': '{userbase}/lib/python{py_version_short}/site-packages', 'platlib': '{userbase}/lib/python{py_version_short}/site-packages', 'include': '{userbase}/include/python{py_version_short}', 'scripts': '{userbase}/bin', 'data' : '{userbase}', }, 'nt_user': { 'stdlib': '{userbase}/Python{py_version_nodot}', 'platstdlib': '{userbase}/Python{py_version_nodot}', 'purelib': '{userbase}/Python{py_version_nodot}/site-packages', 'platlib': '{userbase}/Python{py_version_nodot}/site-packages', 'include': '{userbase}/Python{py_version_nodot}/Include', 'scripts': '{userbase}/Scripts', 'data' : '{userbase}', }, 'posix_user': { 'stdlib': '{userbase}/lib/python{py_version_short}', 'platstdlib': '{userbase}/lib/python{py_version_short}', 'purelib': '{userbase}/lib/python{py_version_short}/site-packages', 'platlib': '{userbase}/lib/python{py_version_short}/site-packages', 'include': '{userbase}/include/python{py_version_short}', 'scripts': '{userbase}/bin', 'data' : '{userbase}', }, 'osx_framework_user': { 'stdlib': '{userbase}/lib/python', 'platstdlib': '{userbase}/lib/python', 'purelib': '{userbase}/lib/python/site-packages', 'platlib': '{userbase}/lib/python/site-packages', 'include': '{userbase}/include', 'scripts': '{userbase}/bin', 'data' : '{userbase}', }, } From distutils.command.install: if sys.version < "2.2": WINDOWS_SCHEME = { 'purelib': '$base', 'platlib': '$base', 'headers': '$base/Include/$dist_name', 'scripts': '$base/Scripts', 'data' : '$base', } else: WINDOWS_SCHEME = { 'purelib': '$base/Lib/site-packages', 'platlib': '$base/Lib/site-packages', 'headers': '$base/Include/$dist_name', 'scripts': '$base/Scripts', 'data' : '$base', } INSTALL_SCHEMES = { 'unix_prefix': { 'purelib': '$base/lib/python$py_version_short/site-packages', 'platlib': '$platbase/lib/python$py_version_short/site-packages', 'headers': '$base/include/python$py_version_short/$dist_name', 'scripts': '$base/bin', 'data' : '$base', }, 'unix_home': { 'purelib': '$base/lib/python', 'platlib': '$base/lib/python', 'headers': '$base/include/python/$dist_name', 'scripts': '$base/bin', 'data' : '$base', }, 'unix_user': { 'purelib': '$usersite', 'platlib': '$usersite', 'headers': '$userbase/include/python$py_version_short/$dist_name', 'scripts': '$userbase/bin', 'data' : '$userbase', }, 'nt': WINDOWS_SCHEME, 'nt_user': { 'purelib': '$usersite', 'platlib': '$usersite', 'headers': '$userbase/Python$py_version_nodot/Include/$dist_name', 'scripts': '$userbase/Scripts', 'data' : '$userbase', }, 'os2': { 'purelib': '$base/Lib/site-packages', 'platlib': '$base/Lib/site-packages', 'headers': '$base/Include/$dist_name', 'scripts': '$base/Scripts', 'data' : '$base', }, 'os2_home': { 'purelib': '$usersite', 'platlib': '$usersite', 'headers': '$userbase/include/python$py_version_short/$dist_name', 'scripts': '$userbase/bin', 'data' : '$userbase', }, } There is an API call (sysconfig.get_config_var) that is used to resolve these various variables, but there is no way to set them other than directly clobbering _INSTALL_SCHEMES/INSTALL_SCHEME dicts holding the information (in both sysconfig and in distutils.command.install). This PEP proposes to change the default installation layout as follows: - Define a new config var, py_binaries, and a corresponding module-level variable, _PY_BINARIES with the default value 'bin' - Change sysconfig's _INSTALL_SCHEMES to INSTALL_SCHEMES, and make it part of the public API. A default scheme equivalent to the current posix_prefix would be added and made to be the default. - To support virtual environments, a special key "current" would be added to INSTALL_SCHEMES. The value associated with "current" would be the name of the currently active scheme. During activation, a virtual environment could place a custom layout in the INSTALL_SCHEMES dict and set "current" to the custom scheme. This could also be used to support legacy or custom layouts. This would be done as follows:: from _collections import defaultdict _DEFAULT_SCHEME = { 'stdlib': '{userbase}/lib/python{py_version_short}', 'platstdlib': '{userbase}/lib/python{py_version_short}', 'purelib': '{userbase}/lib/python{py_version_short}/site-packages', 'platlib': '{userbase}/lib/python{py_version_short}/site-packages', 'include': '{userbase}/include/python{py_version_short}', 'headers': '{userbase}/include/python{py_version_short}', 'scripts': '{userbase}/{py_binaries}', 'data' : '{userbase}', } INSTALL_SCHEMES = defaultdict(lambda: _DEFAULT_SCHEME.copy()) INSTALL_SCHEMES['install'] = _DEFAULT_SCHEME INSTALL_SCHEMES['current'] = 'install' - Change distutils.command.install to use information from INSTALL_SCHEMES instead of embedding its own copy of the dict. By using a defaultdict, prior users of the API could transparently query the dict using the platform value and receive back a reasonable response. The current virtual environment would be retrievable by getting INSTALL_SCHEMES[INSTALL_SCHEMES['current']], and the installed scheme retrievable by getting INSTALL_SCHEMES['install']. Open Issues ========== By default, OS X Framework installs use a format that is different than the one specified above. Possible responses would be to change the layout just for OS X, but special-casing as little as possible. The double indirection to get the current install scheme is a little ugly, but I am not sure what would be better. This explicitly removes the compatibility code in distutils for 2.x install layouts (particularly layouts pre-2.3). This is not considered a major issue, as 2.2 code will largely not be compatible with 3.3, and an individual user could add a custom layout if needed. Based on actual use, some of the distinctions that appeared necessary when distutils and sysconfig were first written (platlib, purelib, include, headers) seem to be duplicative. As written above, the distinctions are maintained for ease of moving to the standardized layout, but it may be worth choosing just one name and sticking with it. [View Less]

3 3

Concurrent safety?
by Mike Meyer Nov. 1, 2011

Nov. 1, 2011

This is a blue sky idea. It can't happen in Python 3.x, and possibly not ever in cPython. I'm mostly hoping to get smarter people than me considering the issue. Synopsis Python, as a general rule, tries to be "safe" about things. If something isn't obviously correct, it tends to throw runtime errors to let you know that you need to be explicit about what you want. When there's not an obvious choice for type conversions, it raises an exception. You generally don't have to worry about … [View More]resource allocation if you stay in python. And so on. The one glaring exception is in concurrent programs. While the tools python has for dealing with such are ok, there isn't anything to warn you when you fail to use those tools and should be. The goal of this proposal is to fix that, and get the Python interpreter to help locate code that isn't safe to use in concurrent programs. Existence Proof This is possible. Clojure is a dynamic language in the LISP family that will throw exceptions if you try mutating variables without properly protecting them against concurrent access. This is not to say that the Clojure solution is the solution, or even the right solution for Python. It's just to demonstrate that this can be done. Object Changes Object semantics don't need to change very much. The existing immutable types will work well in this environment exactly as is. The mutable types - well, we can no longer go changing them willy-nilly. But any language needs mutable types, and there's nothing wrong with the ones we have. Since immutable types don't require access protection *at all*, it might be worthwhile to add a new child of object, "immutable". Instances of this type would be immutable after creation. Presumably, the __new__ method of Python classes inheriting from immutable would be used to set the initial attributes, but the __init__ method might also be able to handle that role. However, this is a performance tweak, allowing user-written classes to skip any runtime checks for being mutated. Binding Changes One of the way objects are mutated is by changing their bindings. As such, some of the bindings might need to be protected. Local variables are fine. We normally can't export those bindings to other functions, just the values bound to them. So changing the binding can stay the same. The bound object can be exported to other threads of execution, but changing it will fall under the rules for changing objects. Ditto for nonlocals. On the other hand, rebindings of module and class and instance variables can be visible in other threads of execution, so they require protection, just like changing mutable objects. New Syntax The protection mechanism is the change to the language. I propose a single new keyword, "locking", that acts similar to the "try" keyword. The syntax is: 'locking' value [',' value]*':' suite The list of values are the objects that can be mutated in this lock. An immutable object showing up in the list of values is a TypeError. It's not clear that function calls should be allowed in the list of values. On the other hand, indexing and attributes clearly should be, and those can turn into function calls, so it's not clear they shouldn't be allowed, either. The locked values can be mutated during the body of the locking suite. For the builtin mutable types, this means invoking their mutating methods. For modules, classes and object instances, it means rebinding their attributes. Locked objects stay locked during function invocations in the suite. This means you can write utility functions that expect to be passed locked objects to mutate. A locking statement can be used inside of another locking statement. See the Implementation section for possible restrictions on this. Any attempt to mutate an object that isn't currently locked will raise an exception. Possibly ValueError, possibly a new exception class just for this purpose. This includes rebinding attributes of objects that aren't locked. Implementation There are at least two ways this can be implemented, both with different restrictions on the suite. While both of them can probably be optimized if it's know that there are no other threads of execution, checking for attempts to mutate unlocked objects should still happen. 1) Conventional locking All the objects being locked have locks attached to them, which are locked when before entering the suite. The implementation must order the locked object in some repeatable way, so that two locking statements that have more than one locked object in common will obtain the locks on those objects in the same order. This will prevent deadlocks. This method will require that the initial locking statement lock all objects that may be locked during the execution of it's suite. This may be a reason for allowing functions as locking values, as a way to get locks on objects that code called in the suite is going to need. Another downside is that the programmer needs to handle exceptions raised during the suite to insure that a set of related changes leaves the relevant objects in a consistent state. In this case, an optional 'except' clause should be added to the locking statement to hold such code. 2) Software Transactional Memory In an STM implementation, copies of the locked objects are created by the locking statement, and they original are "fingerprinted" in some way. The locking suite then runs. When the suite completes, the fingerprints of the originals are checked to see if some other thread of execution has changed them. If they haven't changed, they are replaced by the copies, and execution continues. If the originals have changed, the entire process starts over. In this implementation, the only actual locking is during the original fingerprinting process (to insure that a consistent state is captured) and at the end of the suite. FWIW, this is one of the models provided by Clojure. The restriction on the suite in this case is that running it twice - except for changes to the locked objects - needs to be acceptable. In this case, exceptions don't need to be handled by the programmer to insure consistency. If an exception happens during the execution of the suite, the original values are never replaced. <mike -- Mike Meyer <mwm(a)mired.org> http://www.mired.org/ Independent Software developer/SCM consultant, email for more information. O< ascii ribbon campaign - stop html mail - www.asciiribbon.org [View Less]

8 16