Mailman 3 October 2004 - Python-Dev

new member
by B H, Vijaya Kumar (Vijaya) 04 Oct '04

04 Oct '04

Hi all My name is Vijay I am new member of this group I have been working on python from last 1 year , i love this language for its simpicity , i also worked on jython which i used in few of my automation tasks Regards Vijaya Kumar B.H.

1 0

proposed struct module format code addition
by Josiah Carlson 04 Oct '04

04 Oct '04

Good day everyone, I have produced a patch against the latest CVS to add support for two new formatting characters in the struct module. It is currently an RFE, which I include a link to at the end of this post. Please read the email before you respond to it. Generally, the struct module is for packing and unpacking of binary data. It includes support to pack and unpack the c types: byte, char, short, long, long long, char[], *, and certain variants of those (signed/unsigned, big/little endian, etc.) Purpose ------- I had proposed two new formatting characters, 'g' and 'G' (for biGint or lonG int). There was one primary purpose, to offer users the opportunity to specify their own integer lengths (very useful for cryptography, and real-world applications that involve non-standard sized integers). Current solutions involve shifting, masking, and multiple passes over data. There is a secondary purpose, and that is that future n-byte integers (like 16-byte/128-bit integers as supported by SSE2) are already taken care of. It also places packing and unpacking of these larger integers in the same module as packing and packing of other integers, floats, etc. This makes documentation easy. Functionality-wise, it merely uses the two C functions _PyLong_FromByteArray() and _PyLong_ToByteArray(), with a few lines to handle interfacing with the pack and unpack functions in the struct module. An example of use is as follows: >>> struct.pack('>3g', -1) '\xff\xff\xff' >>> struct.pack('>3g', 2**23-1) '\x7f\xff\xff' >>> struct.pack('>3g', 2**23) Traceback (most recent call last): File "<stdin>", line 1, in ? OverflowError: long too big to convert >>> struct.pack('>3G', 2**23) '\x80\x00\x00' It follows the struct module standard 'lowercase for signed, uppercase for unsigned'. Arguments --------- There seem to be a few arguments against its inclusion into structmodule.c... Argument: The size specifier is variable, so you must know the size/magnitude of the thing you are (un)packing before you (un)pack it. My Response: All use cases I have for this particular mechanism involve not using 'variable' sized structs, but fixed structs with integers of non-standard byte-widths. Specifically, I have a project in which I use some 3 and 5 byte unsigned integers. One of my (un)pack format specifiers is '>H3G3G', and another is '>3G5G' (I have others, but these are the most concise). Certainly this does not fit the pickle/cPickle long (un)packing use-case, but that problem relies on truely variable long integer lengths, of which this specifier does not seek to solve. Really, the proposed 'g' and 'G' format specifiers are only as variable as the previously existing 's' format specifier. Argument: The new specifiers are not standard C types. My Response: Certainly they are not standard C types, but they are flexible enough to subsume all current integer C type specifiers. The point was to allow a user to have the option of specifying their own integer lengths. This supports use cases involving certain kinds of large dataset processing (my use case, which I may discuss after we release) and cryptography, specifically in the case of PKC... while 1: blk = get_block() iblk = struct.unpack('>128G', blk)[0] uiblk = pow(iblk, power, modulous) write_block(struct.pack('>128G', uiblk)) The 'p' format specifier is also not a standard C type, and yet it is included in struct, specifically because it is useful. Argument: You can already do the same thing with: pickle.encode_long(long_int) pickle.decode_long(packed_long) and some likely soon-to-be included additions to the binascii module. My Response: That is not the same. Nontrivial problems require multiple passes over your data with multiple calls. A simple: struct.unpack('H3G3G', st) becomes: pickle.decode_long(st[:2]) #or an equivalent struct call pickle.decode_long(st[2:5]) pickle.decode_long(st[5:8]) And has no endian or sign options, or requires the status quo using of masks and shifts to get the job done. As previously stated, one point of the module is to reduce the amount of bit shifting and masking required. Argument: We could just document a method for packing/unpacking these kinds of things in the struct module, if this really is where people would look for such a thing. My Response: I am not disputing that there are other methods of doing this, I am saying that the struct module includes a framework and documentation location that can include this particular modification with little issue, which is far better than any other proposed location for equivalent functionality. Note that functionality equivalent to pickle.encode/decode_long is NOT what this proposed enhancement is for. Argument: The struct module has a steep learning curve already, and this new format specifier doesn't help it. My Response: I can't see how a new format specifier would necessarily make the learning curve any more difficult, if it was even difficult in the first place. Why am I even posting --------------------- Raymond has threatened to close this RFE due to the fact that only I have been posting to state that I would find such an addition useful. If you believe this functionality is useful, or even if you think that I am full of it, tell us: http://python.org/sf/1023290 - Josiah

9 13

introducing
by Lars Blockken 04 Oct '04

04 Oct '04

Hello, I'm Lars and i'm knew for python so i hope you can answer my questions in the future. Greetz Lars

2 1

proposed struct module format code addition
by Josiah Carlson 03 Oct '04

03 Oct '04

Good day everyone, I have produced a patch against the latest CVS to add support for two new formatting characters in the struct module. It is currently an RFE, which I include a link to at the end of this post. Please read the email before you respond to it. Generally, the struct module is for packing and unpacking of binary data. It includes support to pack and unpack the c types: byte, char, short, long, long long, char[], *, and certain variants of those (signed/unsigned, big/little endian, etc.) Purpose ------- I had proposed two new formatting characters, 'g' and 'G' (for biGint or lonG int). There was one primary purpose, to offer users the opportunity to specify their own integer lengths (very useful for cryptography, and real-world applications that involve non-standard sized integers). Current solutions involve shifting, masking, and multiple passes over data. There is a secondary purpose, and that is that future n-byte integers (like 16-byte/128-bit integers as supported by SSE2) are already taken care of. It also places packing and unpacking of these larger integers in the same module as packing and packing of other integers, floats, etc. This makes documentation easy. Functionality-wise, it merely uses the two C functions _PyLong_FromByteArray() and _PyLong_ToByteArray(), with a few lines to handle interfacing with the pack and unpack functions in the struct module. An example of use is as follows: >>> struct.pack('>3g', -1) '\xff\xff\xff' >>> struct.pack('>3g', 2**23-1) '\x7f\xff\xff' >>> struct.pack('>3g', 2**23) Traceback (most recent call last): File "<stdin>", line 1, in ? OverflowError: long too big to convert >>> struct.pack('>3G', 2**23) '\x80\x00\x00' It follows the struct module standard 'lowercase for signed, uppercase for unsigned'. Arguments --------- There seem to be a few arguments against its inclusion into structmodule.c... Argument: The size specifier is variable, so you must know the size/magnitude of the thing you are (un)packing before you (un)pack it. My Response: All use cases I have for this particular mechanism involve not using 'variable' sized structs, but fixed structs with integers of non-standard byte-widths. Specifically, I have a project in which I use some 3 and 5 byte unsigned integers. One of my (un)pack format specifiers is '>H3G3G', and another is '>3G5G' (I have others, but these are the most concise). Certainly this does not fit the pickle/cPickle long (un)packing use-case, but that problem relies on truely variable long integer lengths, of which this specifier does not seek to solve. Really, the proposed 'g' and 'G' format specifiers are only as variable as the previously existing 's' format specifier. Argument: The new specifiers are not standard C types. My Response: Certainly they are not standard C types, but they are flexible enough to subsume all current integer C type specifiers. The point was to allow a user to have the option of specifying their own integer lengths. This supports use cases involving certain kinds of large dataset processing (my use case, which I may discuss after we release) and cryptography, specifically in the case of PKC... while 1: blk = get_block() iblk = struct.unpack('>128G', blk)[0] uiblk = pow(iblk, power, modulous) write_block(struct.pack('>128G', uiblk)) The 'p' format specifier is also not a standard C type, and yet it is included in struct, specifically because it is useful. Argument: You can already do the same thing with: pickle.encode_long(long_int) pickle.decode_long(packed_long) and some likely soon-to-be included additions to the binascii module. My Response: That is not the same. Nontrivial problems require multiple passes over your data with multiple calls. A simple: struct.unpack('H3G3G', st) becomes: pickle.decode_long(st[:2]) #or an equivalent struct call pickle.decode_long(st[2:5]) pickle.decode_long(st[5:8]) And has no endian or sign options, or requires the status quo using of masks and shifts to get the job done. As previously stated, one point of the module is to reduce the amount of bit shifting and masking required. Argument: We could just document a method for packing/unpacking these kinds of things in the struct module, if this really is where people would look for such a thing. My Response: I am not disputing that there are other methods of doing this, I am saying that the struct module includes a framework and documentation location that can include this particular modification with little issue, which is far better than any other proposed location for equivalent functionality. Note that functionality equivalent to pickle.encode/decode_long is NOT what this proposed enhancement is for. Argument: The struct module has a steep learning curve already, and this new format specifier doesn't help it. My Response: I can't see how a new format specifier would necessarily make the learning curve any more difficult, if it was even difficult in the first place. Why am I even posting --------------------- Raymond has threatened to close this RFE due to the fact that only I have been posting to state that I would find such an addition useful. If you believe this functionality is useful, or even if you think that I am full of it, tell us: http://python.org/sf/1023290 - Josiah

1 0

[OT] stats on type-inferencing atomic types in local variables in the stdlib
by Brett C. 03 Oct '04

03 Oct '04

My thesis (which, for those who don't know, was to come up with a way to do type inferencing in the compiler without requiring any semantic changes; basically type inferencing atomic types assigned to local variables) is now far enough long that I have the algorithm done and I can generate statistics on what opcodes are called with the most common types that I can specifically infer (can also do static type checking on occasion; only triggers were actual unit tests making sure TypeError was raised for certain things like ``~4.2`` and such). Thought some of you might get a kick out of this since the numbers are rather blatent for certain opcodes and methods. To read the stats, the number to the left is the number of times the opcode was compiled (not executed!) with the specific type(s) known for the opcode (if it took two args, then both types are listed; order was considered irrelevant). Now they are listed as integers, so here is the conversion:: Basestring 4 IntegralType 8 FloatType 16 ImagType 32 DictType 64 ListType 128 TupleType 256 For the things named "meth_<something>" that is the method being called immediately on the type. Now realize these numbers are only for opcodes where I could definitely infer the type; ones where it could be more than one type, regardless if those possibilities were very specific, I just ignored it and did not include in the stats. I also tweaked some opcodes knowing how they are more often used. So, for instance, BINARY_MODULO checks specifically for the case of when the left side is a basestring and then just doesn't worry about the other args. Other ones I just didn't bother with all the args since it was not interesting to me in terms of deciding what type-specific opcodes I want to come up with. Anyway, here are the numbers on Lib sans Lib/test (129,814 lines according to SLOCCount) for the ones above 100:: (101, ('BINARY_MULTIPLY', (8, 4))), (106, ('BINARY_SUBSCR', 128)), (118, ('GET_ITER', 128)), (124, ('BINARY_MODULO', None)), (195, ('meth_join', 4)), (204, ('BINARY_ADD', (8, 8))), (331, ('BINARY_ADD', (4, 4))), (513, ('BINARY_LSHIFT', (8, 8))), (840, ('meth_append', 128)), (1270, ('PRINT_ITEM', 4)), (1916, ('BINARY_MODULO', 4)), (12302, ('STORE_SUBSCR', 64))] We sure like our dictionaries (for those that don't know, dictionaries are created by making an empty dict and then basically doing an indivual assignment for each value). We also seem to love to use string interpolation, and printing stuff. Using list.append is also popular. Now the BINARY_LSHIFT is rather interesting, and that ties into the whole issue of how much I can actually infer; since binary work tends to be with all constants I can infer it really easily and so its frequency is rather high. Its actual frequency of use, though, compared to other things probably is not high, though. Plus I doubt Plone, for instance, uses ``<<`` very often so I suspect the opcode will get weeded out when I incorporate stats from the other apps I am taking stats from. As for the stuff I cut out, the surprising thing from those numbers was how few mathematical expressions could be inferred. I checked my numbers with grep and there really is only 3 times where a float constant is divided by a float constant (and they are all in colorsys). I was not expecting that at all. Guess global variables or object attributes tend to have them or I just can't infer the values. Either way I just wasn't expecting that. Anyway, as I said I just thought some people might find this interesting. Don't read into this too much since I am just using these numbers as guidelines for type-specific opcodes to write for use as a quantifiable measurement of the usefulness of type inferencing like this. -Brett P.S.: anyone who is *really* interested I can send you the full stats for the apps I have run my modified version of compile.c against.

2 2

python-dev Summary for 2004-09-01 through 2004-09-15 [draft]
by Brett C. 02 Oct '04

02 Oct '04

===================== Summary Announcements ===================== Python 2.4a3 has been released. Go to http://www.python.org/2.4/ and give it twirl. Sorry for this summary being so short, but school has started back up again so I am in the middle of suddenly having to switch back into homework mode after spending the summer just having a 9:00-17:00 job. And since it is a new school year I am going to abuse this space and say that anyone in San Luis Obispo, including students, should join the `SLO Meetup`_ coming up on October 14. .. _SLO Meetup: http://python.meetup.com/95/ ========= Summaries ========= -------------------- Movement in PEP Land -------------------- `PEP 334`_ (Simple Coroutines via SuspendIteration) came into existence. `PEP 328`_ (Relative Imports) got some discussion on postponing making imports absolute instead of the relative/absolute semantics they have now. As it stands it looks like the changeover might get pushed off. `PEP 292`_ (Simpler String Substitutions) seems to finally be done and locked down. `PEP 335`_ (Overloadable Boolean Operators) came into existence. .. _PEP 334: http://www.python.org/peps/pep-0334.html .. _PEP 328: http://www.python.org/peps/pep-0328.html .. _PEP 292: http://www.python.org/peps/pep-0292.html .. _PEP 335: http://www.python.org/peps/pep-0335.html Contributing threads: - `PEP 334 - Simple Coroutines via SuspendIteration <>`__ - `PEP 328 - Relative Imports <>`__ - `Re: Alternative Implementation for PEP 292:Simple String Substitutions <>`__ - `ANN: PEP 335: Overloadable Boolean Operators <>`__ ------------------------------------------------------ __str__, __unicode__, and how to have them play nicely ------------------------------------------------------ Did you know that __str__ methods are allowed to return Unicode objects? Well, it turns out they can, but that str() (which calls PyObject_Str()) automatically tries to convert the value returned by __str__ into ASCII. Basically __str__ shouldn't return Unicode if you can help it and you should use __unicode__ instead and reserve __str__ to return str objects only. Contributing threads: - `unicode and __str__ <>`__ ---------------------- Backporting C APIs bad ---------------------- Somebody (*cough* Guido *cough*) asked if the datetime C API could be backported to 2.3 . The argument was that the only person who would probably use it is the person who asked for it, the author of cx_Oracle. Well, pretty much everyone spoke up against this. The argument went that adding an API would just be bad since there would suddenly be a point in the 2.3 releases where backwards compatibility was broken. People brought up the point of 2.2 where in 2.2.1 booleans were added and how that has caused compatibility headaches for some people. In the end the API was not backported. Contributing threads: - `Re: [Python-checkins] python/dist/src/Modules threadmodule.c, 2.56, 2.56.8.1 <>`__ ---------------------------------------------------------------------- Got to love race conditions thanks to the filesystem and external apps ---------------------------------------------------------------------- Tim Peters found a race condition one can have on Windows if you have an app that uses the low-level hooks into the filesystem. If you create a file, delete it, and then try to delete the directory the directory deletion will fail since the file is not deleted yet. What can happen is an indexing program can still be indexing the file before the filesystem is allowed to delete it and thus the directory is not truly empty when the directory deletion is executed. Fun stuff. Contributing threads: - `Coernic Desktop Search versus shutil.rmtree <>`__ ------------------------------- Python 2.4a3 is out the door!!! ------------------------------- Go to http://www.python.org/2.4/ , download it (using the bz2 version if possible so as to save on bandwidth), and run it against your code, run the test suite, put it on your head and sell yourself to an art gallery, etc. Contributing threads: - `RELEASED Python 2.4, alpha 3 <>`__ ---------------------------- Cleaning the Exception House ---------------------------- The idea of reorganizing the exceptions hierarchy came up again (see http://www.python.org/dev/summary/2004-08-01_2004-08-15.html#an-exception-i… for a previous discussion on this). This time, the idea of separating the hierarchy into exceptions one would like to catch with a bare 'except' and those that you wouldn't was brought up. The idea is that some exceptions, such as MemoryError, one does not want to catch in a blanket statement usually. Chances of recovering from that kind of exception is low and should only be caught if you know what you are doing. So tweaking the exception hierarchy so that exceptions that were not near-catastrophic could inherit from an exception class that people could catch so as to allow the proper exceptions to propagate to the top-level without issue. Tim Peters even went as far as to suggest deprecating bare 'except' statements. This would force people to be explicit about what they want to catch, whether it be all "safe" exceptions or *all* exceptions. As it stands now no officially decision has been made for Python 3000 since that is about the only place this could happen. Contributing threads: - `Dangerous exceptions <>`__ --------------------------------------------------------------- Making decorators not look like decorators to the outside world --------------------------------------------------------------- Raymond Hettinger pointed out that a decorator does not, by default, look like the function that it is fiddling with (if that is the intent). Since most decorators will most likely be a wrapper function some things need to be set in the wrapper in order not to mask things in the wrapped function (doc string, argument parameters, etc.). So Raymond pointed out some things one can do. This also led to the suggestion of having a common name used to store a reference back to the wrapped function. There was also the mention that a decorator-oriented module in the stdlib will probably materialize in Python 2.5 . For now, though, stick recipes either in the Python Cookbook or in the Python wiki at http://www.python.org/cgi-bin/moinmoin/PythonDecoratorLibrary . Contributing threads: - `decorator support <>`__ =============== Skipped Threads =============== - random.py still broken wrt. urandom - random.py fixage - Re: [Python-checkins] python/dist/src/Lib/test test_compiler.py, 1.5, 1.6 test_decimal.py, 1.13, 1.14 - assert failure on obmalloc - Re: [Python-checkins] python/dist/src/Modules socketmodule.c, 1.304, 1.305 - Install-on-first-use vs. optional extensions - Console vs. GUI applications - Adding status code constants to httplib - PEP 292: method names - PEP 265 - Sorting dicts by value - httplib is not v6 compatible, is this going to be fixed? - OT: Unicode history - --with-tsc compile fails - tempfile.TemporaryFile on Windows NT - PyExc_UnicodeDecodeError - urllib.urlopen() vs IDNs, percent-encoded hosts, ':' - tabs in httplib.py and test_httplib.py There is now a vimrc file in the Misc directory that sets things up to follow PEPs 7 & 8 - strawman decision: @decorator won't change - unicode inconsistency?

1 1

Proposing a sys.special_exceptions tuple
by Nick Coghlan 01 Oct '04

01 Oct '04

I spent some time the other day looking at the use of bare except statements in the standard library. Many of them seemed to fall into the category of 'need to catch anything user code is likely to throw, but shouldn't be masking SystemExit, StopIteration, KeyboardInterrupt, MemoryError, etc'. Changing them to "except Exception:" doesn't help, since all of the above still fit into that category (Tim posted a message recently about rearranging the Exception heirarchy to fix this. Backwards compatibility woes pretty much killed the discussion though). However, another possibility occurred to me: try: # Do stuff except sys.special_exceptions: raise except: # Deal with all the mundane stuff With an appropriately defined tuple, that makes it easy for people to "do the right thing" with regards to critical exceptions. Such a tuple could also be useful for invoking isinstance() and issubclass(). Who knows? If something like this caught on, it might some day be possible to kill a Python script with a single press of Ctrl-C };> Cheers, Nick. -- Nick Coghlan Brisbane, Australia

6 9

Running a module as a script
by Nick Coghlan 01 Oct '04

01 Oct '04

Patch # 1035498 attempts to implement the semantics suggested by Ilya and Anthony and co. "python -m module" Means: - find the source file for the relevant module (using the standard locations for module import) - run the located script as __main__ (note that containing packages are NOT imported first - it's as if the relevant module was executed directly from the command line) - as with '-c' anything before the option is an argument to the interpreter, anything after is an argument to the script The allowed modules are those whose associated source file meet the normal rules for a command line script. I believe that means .py and .pyc files only (e.g. "python -m profile" works, but "python -m hotshot" does not). Special import hooks (such as zipimport) almost certainly won't work (since I don't believe they work with the current command line script mechanism). Cheers, Nick. -- Nick Coghlan Brisbane, Australia

4 7

Another test_compiler mystery
by Tim Peters 01 Oct '04

01 Oct '04

I've noticed several times now, in both debug and release builds, that if I run regrtest.py with -uall, *sometimes* it just stops after running test_compiler: $ python_d regrtest.py -uall test_grammar test_opcodes ... test_compare test_compile test_compiler $ There's no indication of error, it just ends. It's not consistent. Happened once when I was running with -v, and test_compiler's output ended here: ... compiling C:\Code\python\lib\test\test_operator.py compiling C:\Code\python\lib\test\test_optparse.py compiling C:\Code\python\lib\test\test_os.py compiling C:\Code\python\lib\test\test_ossaudiodev.py compiling C:\Code\python\lib\test\test_parser.py In particular, there's no Ran M tests in Ns output, so it doesn't look like unittest (let alone regrtest) ever got control back. Hmm. os.listdir() is in sorted order on NTFS, so test_compiler should be chewing over a lot more files after test_parser.py. *This* I could blame on a blown C stack -- although I'd expect a much nastier symptom then than just premature termination. Anyone else?

23 50