Mailman 3 January 2009 - Python-Dev

subprocess crossplatformness and async communication
by anatoly techtonik 27 Jan '09

27 Jan '09

Greetings, This turned out to be a rather long post that in short can be summarized as: "please-please-please, include asynchronous process communication in subprocess module and do not allow "available only on ..." functionality", because it hurts the brain". Code to speak for itself: http://code.activestate.com/recipes/440554/ The subprocess module was a great step forward to unify various spawn and system and exec and etc. calls in one module, and more importantly - in one uniform API. But this API is partly crossplatform, and I believe I've seen recent commits to docs with more unix-only differences in this module. The main point of this module is to "allows you to spawn new processes, connect to their input/output/error pipes, and obtain their return codes". PEP 324 goal is also to make "make Python an even better replacement language for over-complicated shell scripts". Citing pre-subrocess PEP 324, "Currently, Python has a large number of different functions for process creation. This makes it hard for developers to choose." Now there is one class with many methods and many platform-specific comments and notices. To make thing worse people on Unix use subprocess with fcntl and people on windows tend not to use it at all, because it looks complicated and doesn't solve the problem with asynchronous communication. That I suggest is to add either support for async crossplatfrom read/write/probing of executed process or a comment at the top of module documentation that will warn that subprocess works in blocking mode. With async mode you can emulate blocking, the opposite is not possible. This will save python users a lot of time. Thanks for reading my rant. BTW, the proposed change is top 10 python recipe on ActiveState http://code.activestate.com/recipes/langs/python/ -- --anatoly t.

3 4

microsoft dlls apparently don't support data. implications: PyAPI functions required to access data across modules.
by Luke Kenneth Casson Leighton 27 Jan '09

27 Jan '09

according to the wikipedia entry on dlls, dlls do not support data, only functions. which would explain two things: 1) why certain modules are forcibly linked into pythonNN.dll 2) why attempts to move them out of pythonNN.dll cause runtime crashes. so i will continue the experiment, and remove all the "data" references from the pythonNN.def that i added, and deal with the knock-on consequences, which will involve adding "get" functions. for example, PyAPI_FUNC(char*) _PyStructSequence_Get_UnnamedField(void) use of such functions will allow various bits and pieces - such as PyStructSequence_UnnamedField - to be converted back to static in their respective c files. any objections, speak now, because this will involve quite a bit of work. l.

8 13

Incorrect documentation (and possibly implementation) for rlcompleter.Completer?
by Carl Johnson 26 Jan '09

26 Jan '09

The documentation at http://docs.python.org/library/rlcompleter.html claims that > Completer.complete(text, state)¶ > > Return the state*th completion for *text. > > If called for text that doesn’t include a period character > ('.'), it will complete from names currently defined in __main__, > __builtin__ and keywords (as defined by the keyword module). > > If called for a dotted name, it will try to evaluate anything > without obvious side-effects (functions will not be evaluated, but > it can generate calls to __getattr__()) up to the last part, and > find matches for the rest via the dir() function. Any exception > raised during the evaluation of the expression is caught, silenced > and None is returned. In other words, it claims to use dir(obj) as part of the tab completion process. This is not true (using Python 2.6.1 on OS X): >>> class B(object): ... def __dir__(self): return dir(u"") #Makes B objects look like strings ... >>> b = B() >>> dir(b) ['__add__', '__class__', '__contains__', '__delattr__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__getitem__', '__getnewargs__', '__getslice__', '__gt__', '__hash__', '__init__', '__le__', '__len__', '__lt__', '__mod__', '__mul__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__rmod__', '__rmul__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '_formatter_field_name_split', '_formatter_parser', 'capitalize', 'center', 'count', 'decode', 'encode', 'endswith', 'expandtabs', 'find', 'format', 'index', 'isalnum', 'isalpha', 'isdecimal', 'isdigit', 'islower', 'isnumeric', 'isspace', 'istitle', 'isupper', 'join', 'ljust', 'lower', 'lstrip', 'partition', 'replace', 'rfind', 'rindex', 'rjust', 'rpartition', 'rsplit', 'rstrip', 'split', 'splitlines', 'startswith', 'strip', 'swapcase', 'title', 'translate', 'upper', 'zfill'] >>> c = rlcompleter.Completer() >>> c.complete("b.", 0) #Notice that it does NOT return __add__ u'b.__class__(' >>> c.matches #Notice that this list is completely different from the list given by dir(b) [u'b.__class__(', u'b.__delattr__(', u'b.__doc__', u'b.__format__(', u'b.__getattribute__(', u'b.__hash__(', u'b.__init__(', u'b.__new__(', u'b.__reduce__(', u'b.__reduce_ex__(', u'b.__repr__(', u'b.__setattr__(', u'b.__sizeof__(', u'b.__str__(', u'b.__subclasshook__(', u'b.__class__(', u'b.__class__(', u'b.__delattr__(', u'b.__dict__', u'b.__dir__(', u'b.__doc__', u'b.__format__(', u'b.__getattribute__(', u'b.__hash__(', u'b.__init__(', u'b.__module__', u'b.__new__(', u'b.__reduce__(', u'b.__reduce_ex__(', u'b.__repr__(', u'b.__setattr__(', u'b.__sizeof__(', u'b.__str__(', u'b.__subclasshook__(', u'b.__weakref__', u'b.__class__(', u'b.__delattr__(', u'b.__doc__', u'b.__format__(', u'b.__getattribute__(', u'b.__hash__(', u'b.__init__(', u'b.__new__(', u'b.__reduce__(', u'b.__reduce_ex__(', u'b.__repr__(', u'b.__setattr__(', u'b.__sizeof__(', u'b.__str__(', u'b.__subclasshook__('] As I see it, there are two ways to fix the problem: Change the documentation or change rlcompleter.Complete. I think the latter option is preferable (although it might have to wait for Python 2.7/3.1), but I thought I would ask other people if I'm missing something and if not which fix is preferred. If other people agree that it's a bug, I'll file it. -- Carl Johnson

2 2

Operator module deprecations
by Raymond Hettinger 26 Jan '09

26 Jan '09

I would like to deprecate some outdated functions in the operator module. The isSequenceType(), isMappingType(), and isNumberType() functions never worked reliably and now their intended purpose has been largely fulfilled by ABCs. The isCallable() function has long been deprecated and I think it's finally time to rip it out. The repeat() function never really corresponded to an operator. Instead, it reflected an underlying implementation detail (namely the naming of the sq_repeat slot and the abstract C API function PySequence_Repeat). That functionality is already exposed by operator.mul: operator.mul('abc', 3) --> 'abcabcabc' Raymond

4 5

Changes needed for python-2.6.spec to build successfully
by Patrick Toal 26 Jan '09

26 Jan '09

Hello, I'm not subscribed to this list, but this is the best place I could think of to send this. Please feel free to ignore if this has already been addressed, or if I've approached it completely wrong. When trying to perform an rpmbuild of the python-2.6.1 tarball on my CentOS 5.1 system using the included python-2.6.spec file, I ran into a bunch of vexing problems. My solution to them is included in the diff to the specfile attached. Some of these fixes are probably not appropriate for everyone (eg: my need for shared libs, vs static). I hope this saves someone else a bit of time. :) Cheers, Pat ---- Patrick Toal ptoal(a)takeflight.ca

3 2

progress: compiling python2.5 under msys (specifically but not exclusively under wine) with msvcr80
by Luke Kenneth Casson Leighton 26 Jan '09

26 Jan '09

this is a progress report on compiling python using entirely free software tools, no proprietary compilers or operating systems involved, yet still linking and successfully running with msvcr80 assemblies. manifests and rc files, which are compiled to internal resources, have been added. various sections which are uniquely identifed by _MSC_VER >= 1400 etc have had to be enabled with corresponding MSVCRT_VERSION >= 0x0800 - in particular, signal handling (PyOS_getsig()). currently, under wine with msvcr80, there looks like there is a bug with a common theme related to threads, but here's a short list: test_array.py is blocking, test_bz2.py is hanging and test_cmd_line.py causes a segfault; test_ctypes is _still_ a bundle of fun. for those people who use native win32 platforms who are compiling up this code, you should have better luck. significantly, the wine developers have been absolutely fantastic, and have fixed several bugs in wine, sometimes within hours, that were found as a result of running the extremely comprehensive python regression tests. the python regression tests are a credit to the collaborative incremental improvement process of free software development. i look forward to seeing the same incremental improvement applied to the development of python, evidence of which would be clearly seen by the acceptance of one of the following patches, one of which is dated 2003: http://bugs.python.org/issue3754 http://bugs.python.org/issue841454 http://bugs.python.org/issue3871 http://bugs.python.org/issue4954 http://bugs.python.org/issue5010 for those people wishing to track and contribute to the development of python for win32 using entirely free software tools, either under wine or native windows, there is a git repository, here, slightly illogically named pythonwine because that's where i started from (cross-compiling python under wine, so i could get at the wine registry from python). obviously, since then, things have... moved on :) http://github.com/lkcl/pythonwine/tree/python_2.5.2_wine l.

3 5

Issue 4285 Patch
by Ross Light 26 Jan '09

26 Jan '09

Hello, python-dev! My name is Ross Light. I was a documentation contributor at GHOP a couple years back and I wanted to start contributing to the core interpreter. I found Issue 4285 and decided to write a patch for it. This is my first patch, so I'd like someone to review my patch and make sure I'm doing things right. http://bugs.python.org/issue4285 Thanks! Cheers, Ross Light

1 0

Issue 4285 Patch
by Ross Light 26 Jan '09

26 Jan '09

Hello, python-dev! My name is Ross Light. I was a documentation contributor at GHOP a couple years back and I wanted to start contributing to the core interpreter. I found Issue 4285 and decided to write a patch for it. This is my first patch, so I'd like someone to review my patch and make sure I'm doing things right. http://bugs.python.org/issue4285 Thanks! Cheers, Ross Light

1 0

Re: [Python-Dev] [Python-checkins] PEP 374
by Barry Warsaw 26 Jan '09

26 Jan '09

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Jan 25, 2009, at 11:49 AM, Antoine Pitrou wrote: > Barry Warsaw <barry <at> python.org> writes: >> >> Besides, certain developments like support for the svn wire protocol >> in bzr would make the WFC (we fear change :) argument moot. > > This is an argument *against* the usefulness of switching! Why? This simply allows those who are happy with the status quo to live peacefully with those who want the benefits of the new capabilities. Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (Darwin) iQCVAwUBSXyjW3EjvBPtnXfVAQLZHwP/UAwA/fcfaDDoaQI1Qa0F50u57AESc/GN bPIgUe6I93fwgAHx/+9jQWxgVJCjIOWlSavZqtLOr7nPR6gN4B27d4XpntLE7O47 3JXkV5QEZL0YDob0M33qAPSgPZsMv1++fWo9FDrk0o9SVzmsrP4OytsUsykRiOkC gkMtAPnzeAQ= =j2t2 -----END PGP SIGNATURE-----

2 2

future-proofing vector tables for python APIs: binary-module interoperability
by Luke Kenneth Casson Leighton 26 Jan '09

26 Jan '09

On Fri, Jan 23, 2009 at 10:48 PM, Roumen Petrov <bugtrack(a)roumenpetrov.info> wrote: >>> python.exe (say, the official one) loads >>> python25.dll. Then, an import is made of a ming-wine extension, say >>> foo.pyd, which is linked with libpython2.5.dll, which then gets loaded. >>> Voila, you have two interpreters in memory, with different type objects, >>> memory heaps, and so on. >> >> ok, there's a solution for that - the gist of the solution is already >> implemented in things like Apache Runtime and Apache2 (modules), and >> is an extremely common standard technique implemented in OS kernels. >> the "old school" name for it is "vector tables". >> > [SNIP] Did you think that this will escape python MSVC from "Assembly hell" > ? let me think about that.... write some things down, i might have an answer at the end :) but it would certainly mean that there would be both a future-proof path for binary modules from either msvc-compiled _or_ mingw-compiled 2.5, 2.6, 2.7 etc. to work with 2.5, 2.6, 2.7, 2.8 etc. _without_ a recompile. [forwards-future-proof-compatibility _is_ possible, but... it's a bit more... complicated. backwards-compatibility is easy]. what you do is you make sure that the vector-table is always and only "extended" - added to - never "removed from" or altered. if one function turns out to be a screw-up (inadequate, not enough parameters), you do NOT change its function parameters, you add an "Ex" version - or an "Ex1" version. just like microsoft does. [.... now you know _why_ they do that "ridiculous" thing of adding FunctionEx1 FunctionEx2 and if you look at the MSHTML specification i think they go up to _six_ revisions of the same function in one case!] to detect revisions of the vector-table you use a "negotiation" tactic. you add a bit-field at the beginning of the struct, and each bit expresses a "new revision" indicating that the vector table has been extended (and so needs to be typecast to a different struct - exactly the same as is done with PyObject being typecast to different structs). the first _function_ in the vector-table is one which the module must call (in its initXXXX()) to pass in the "version number" of the module, to the python runtime. just in case someone needs to know. but for the most part, the initiation - of function call-out - is done _from_ modules, so each and every module will never try to call something beyond what it understands. but basically, not only is this technique nothing new - it's in use in Apache RunTime, FreeDCE, the NT Kernel, the Linux Kernel - but also it's actually _already_ in use in one form in the way that python objects are typecast from PyObject to other types of structs! the difference is that a bit-field would make detection of revisions a bit easier but to be honest you could just as easily make it an int and increase the revision number. .... ok, i've thought about your question, and i think it might [save us from assembly hell]. what you would likely have to do is compile _individual modules_ with assemblies, should they need them. for example, the msvcrt module would obviously have to be.... hey, that'd be interesting, how about having different linked versions of the msvcrt module? coool :) in the mingw builds, it's not necessary to link in PC/msvcrtmodule.o into the python dll - so (and this confused the hell out of me for a minute so i had to do find . -name "msvcrt*") you end up with a Modules/msvcrt.pyd. surely, that should be the _only_ dll which gets _specifically_ linked against msvcr71.dll (or 90, or... whatever) and it would be even _better_ if that then got _named_ msvcr71.pyd, msvcr90.pyd etc. i'll do an experiment, later, to confirm that this actually _does_ work - i.e. creating an msvcr80.pyd with "mingw gcc -specs=msvcr80". the neat thing is that if it works, you wouldn't need to _force_ people to link to the python dll or the python exe with msvcr90 or any other version. and the mingw built python.exe or python dll would be interchangeable, as it would be _specific modules_ that required specific versions of the msvc runtime. l.

2 6