Mailman 3 April 2008 - Python-ideas

Ideas towards GIL removal
by Greg Ewing July 18, 2008

July 18, 2008

I've been thinking about some ideas for reducing the amount of refcount adjustment that needs to be done, with a view to making GIL removal easier. 1) Permanent objects In a typical Python program there are many objects that are created at the beginning and exist for the life of the program -- classes, functions, literals, etc. Refcounting these is a waste of effort, since they're never going to go away. So perhaps there could be a way of marking such objects as "permanent" or "immortal". Any refcount operation on a permanent object would be a no-op, so no locking would be needed. This would also have the benefit of eliminating any need to write to the object's memory at all when it's only being read. 2) Objects owned by a thread Python code creates and destroys temporary objects at a high rate -- stack frames, argument tuples, intermediate results, etc. If the code is executed by a thread, those objects are rarely if ever seen outside of that thread. It would be beneficial if refcount operations on such objects could be carried out by the thread that created them without locking. To achieve this, two extra fields could be added to the object header: an "owning thread id" and a "local reference count". (The existing refcount field will be called the "global reference count" in what follows.) An object created by a thread has its owning thread id set to that thread. When adjusting an object's refcount, if the current thread is the object's owning thread, the local refcount is updated without locking. If the object has no owning thread, or belongs to a different thread, the object is locked and the global refcount is updated. The object is considered garbage only when both refcounts drop to zero. Thus, after a decref, both refcounts would need to be checked to see if they are zero. When decrementing the local refcount and it reaches zero, the global refcount can be checked without locking, since a zero will never be written to it until it truly has zero non-local references remaining. I suspect that these two strategies together would eliminate a very large proportion of refcount-related activities requiring locking, perhaps to the point where those remaining are infrequent enough to make GIL removal practical. -- Greg

8 19

Re: [Python-ideas] A new .pyc file format
by gagsl-py2＠yahoo.com.ar April 29, 2008

April 29, 2008

En Mon, 28 Apr 2008 11:30:52 -0300, Calvin Spealman <ironfroggy-+z4E91SeU6Qg+LkckTzjXg(a)public.gmane.org> escribió: > I'll play my part here and toss out some ideas we could use this for. > I'm not really advocating it, yet, but I'll say I am +0. In either case, > if we did, here are some possible uses: > > We could break up the code into multiple sections and allow alternatives > for sections with different versions. Different versions could be used > for a few different things, including different optimization levels, > supporting multiple bytecode versions, or storing both pre and post AST > transformations of the code. Thanks for the examples! The idea was to allow storing such things, but I could not think of a concrete alternative example. > Meta-ish data like docstrings could be pulled out of the code objects > and injected in non-optimized modes. This might also include original > source for code, which could be helpful when you change the source and > still-running code tracebacks and gives you invalid lines. > > Bookkeeping data could sit in some sections, detailing things like call > stats (average call time, frequency, etc) and other information that > could be useful for optimizers and JIT compilers like psyco. Do you mean, to update the file after the code is executed, to store such statistics? When would it be done? -- Gabriel Genellina Tarjeta de crédito Yahoo! de Banco Supervielle. Solicitá tu nueva Tarjeta de crédito. De tu PC directo a tu casa. www.tuprimeratarjeta.com.ar

1 0

A new .pyc file format
by Gabriel Genellina April 29, 2008

April 29, 2008

Hello (Sorry if you get this twice, I can't see my original post from gmane) I want to propose a new .pyc file format. Currently .pyc files use a very simple format: - MAGIC number (4 bytes, little-endian) - last modification time of source file (4 bytes, little-endian) - code object (marshaled) The problem is that this format is *too* simple. It can't be changed, nor can accomodate other fields if desired. I propose using a more flexible ..pyc format (resembling RIFF files with multiple levels). The layout would be as follows: - A file contains a sequence of sections. - A section has an identifier (4 bytes, usually ASCII letters), followed by its size (4 bytes, not counting the section identifier nor the size itself), followed by the actual section content. - The layout inside each section is arbitrary, but it's suggested to use the same technique: a sequence of (identifier x 4 bytes, size x 4 bytes, actual value) The outer section is called "PYCO" (from Python Code, or a contraction of pyc+pyo) and contains at least 4 subsections: - "VERS": import's "MAGIC number", now seen as a "code version number" (4 bytes, same format as before) - "DATE": last modification time of source file (4 bytes, same format as before) - "COFL": the code.co_flags attribute (4 bytes) - "CODE": the marshaled code object 4 bytes 4 bytes +-----.-----.-----.-----+-----.-----.-----.-----+ | "P" | "Y" | "C" | "O" | size of whole section | +-----------------------+-----------------------+ +-----.-----.-----.-----+-----.-----.-----.-----+-----.-----.-----.-----+ | "V" | "E" | "R" | "S" | 4 | import "MAGIC number" | +-----.-----.-----.-----+-----.-----.-----.-----+-----.-----.-----.-----+ | "D" | "A" | "T" | "E" | 4 | source file st_mtime | +-----.-----.-----.-----+-----.-----.-----.-----+-----------------------+ | "C" | "O" | "F" | "L" | 4 | code.co_flags | +-----.-----.-----.-----+-----.-----.-----.-----+-----.-----.-----.-----+ | "C" | "O" | "D" | "E" | size of marshaled code| marshaled code object | +-----.-----.-----.-----+-----.-----.-----.-----+ ... ... ... | | ... ... ... | +-----------------------+ New sections -or subsections inside a section- can be defined in the future. No implied knowledge of section meanings or its structure is required to read the file; readers can safely skip over sections they don't understand, and never lost synchronism. Compared with the current format, it has an overhead of 44 bytes. The format above can replace the current format used for .pyc/.pyo files (but see below). Of course it's totally incompatible with the old format. Apart from changing every place where .pyc files are read or written in the Python sources (not so many, I've identified all of them), 3rd party libraries and tools using the old format would have to be updated. Perhaps a new module should be provided to read and write pyc files. Anyway the change is "safe", in the sense that any old code expecting the MAGIC number in the first 4 bytes will reject the new format as invalid and not process it. Due to this incompatibility, this should be aimed at Python 3.x; I hope we are on time to implement this for 3.0? A step further: Currently, the generated code object depends on the Python version and the optimize flag; it used to depend on the Unicode flag too, but that's not the case for Python 3. The Python version determines the base MAGIC number; the Unicode flag increments that number by 1; the optimize flag determines the file extension used (.pyc/.pyo). With this new format, there is no need to use two different extensions anymore: all of this can be gathered from the attributes above, so several variants of the same code object can be stored in a single file. The importer can choose which one to load based on those attributes. The selection can be made rather quickly, just the relevant attributes have to be read actually; all other subsections can be entirely skipped without further parsing. Some issues: - endianness: .pyc files currently store the modification time and magic number in little-endian; probably one should just stick to it. - making the size of all sections multiple of 4 may be a good idea, so marshaled code should be padded with up to 3 NUL bytes at the end. - section ordering, and subsection ordering inside a section: should not be relevant; what if one can't seek to an earlier part of the file? (Ok, unlikely, but currently import.c appears to handle such cases). If "CODE" comes before any of "VERS", "COFL", "DATE" it should be necesary to rewind the file to read the code section. The easy fix is to forbid that situation: "CODE" must come after all of those subsections. - The co_flags attribute of code objects is now externally visible; future Python versions should not redefine those flags. - There is no provision for explicit attribute types: "VERS" is a number, "CODE" is a marshaled code object... The reader has to *know* that (although it can completely skip over unknown attributes). No string attributes were defined (nor required). For the *current* needs, it's enough as it is. But perhaps in the future this reveals as a shortcoming, and the .pyc format has to be changed *again*, and I'd hate that. - Perhaps the source modification date should be stored in a more portable way? - a naming problem: currently, the code version number defined in import.c is called "MAGIC", and is written at the very beginning of the file. It identifies the file as having a valid code object. In the proposed format, the file will begin with the letters "PYCO" instead, and the current magic number is buried inside a subsection... it's not a "magic" anymore, just a version number, and the "magic" in the sense used by file(1) would be the 4 bytes "PYCO". So the name "MAGIC" should be changed everywhere... it seems too drastic. - 32 bits should be enough for all sizes (and 640k should be enough for all people...) Implementation: I don't have a complete implementation yet, but if this format is approved (as is or with any changes) I could submit a patch. I've made a small but incompatible modification in the currently used .pyc format in order to detect all places where this change would impact, and they're not so many actually. -- Gabriel Genellina

10 11

annoying dictionary problem, non-existing keys
by bvidinli April 29, 2008

April 29, 2008

i use dictionaries to hold some config data, such as: conf={'key1':'value1','key2':'value2'} and so on... when i try to process conf, i have to code every time like: if conf.has_key('key1'): if conf['key1']<>'': other commands.... this is very annoying. in php, i was able to code only like: if conf['key1']=='someth' in python, this fails, because, if key1 does not exists, it raises an exception. MY question: is there a way to directly get value of an array/tuple/dict item by key, as in php above, even if key may not exist, i should not check if key exist, i should only use it, if it does not exist, it may return only empty, just as in php.... i hope you understand my question... -- İ.Bahattin Vidinli Elk-Elektronik Müh. ------------------- iletisim bilgileri (Tercih sirasina gore): skype: bvidinli (sesli gorusme icin, www.skype.com) msn: bvidinli(a)iyibirisi.com yahoo: bvidinli +90.532.7990607 +90.505.5667711

2 1

Apologize: annoying dictionary problem, non-existing keys
by bvidinli April 26, 2008

April 26, 2008

Thank you all for your answers. i get many usefull answers and someone remembered me of avoiding cross-posting. i apologize from all of you, for cross posting and disturbing. that day was not a good day for me... it is my fault.. sory.... and have nice days... 26 Nisan 2008 Cumartesi 02:52 tarihinde Collin Winter <collinw(a)gmail.com> yazdı: > 2008/4/24 bvidinli <bvidinli(a)gmail.com>: > > I posted to so many lists because, > > > > this issue is related to all lists, > > this is an idea for python, > > this is related to development of python... > > > > why are you so much defensive ? > > > > i think ideas all important for development of python, software.... > > i am sory anyway.... hope will be helpful. > > Please consult the documentation first: > http://docs.python.org/lib/typesmapping.html . You're looking for the > get() method. > > This attribute of PHP is hardly considered a feature, and is not > something Python wishes to emulate. > > Collin Winter > > > 2008/4/24, Terry Reedy <tjreedy(a)udel.edu>: > > > > > > > Python-dev is for discussion of development of future Python. Use > > > python-list / comp.lang.python / gmane.comp.python.general for usage > > > questions. > > > > > > > > > > > > _______________________________________________ > > > Python-Dev mailing list > > > Python-Dev(a)python.org > > > http://mail.python.org/mailman/listinfo/python-dev > > > Unsubscribe: http://mail.python.org/mailman/options/python-dev/bvidinli%40gmail.com > > > > > > > > > > > -- > > İ.Bahattin Vidinli > > Elk-Elektronik Müh. > > ------------------- > > iletisim bilgileri (Tercih sirasina gore): > > skype: bvidinli (sesli gorusme icin, www.skype.com) > > msn: bvidinli(a)iyibirisi.com > > yahoo: bvidinli > > > > +90.532.7990607 > > +90.505.5667711 > > _______________________________________________ > > > > > > Python-Dev mailing list > > Python-Dev(a)python.org > > http://mail.python.org/mailman/listinfo/python-dev > > Unsubscribe: http://mail.python.org/mailman/options/python-dev/collinw%40gmail.com > > > -- İ.Bahattin Vidinli Elk-Elektronik Müh. ------------------- iletisim bilgileri (Tercih sirasina gore): skype: bvidinli (sesli gorusme icin, www.skype.com) msn: bvidinli(a)iyibirisi.com yahoo: bvidinli +90.532.7990607 +90.505.5667711

1 0

Re: [Python-ideas] [Python-Dev] annoying dictionary problem, non-existing keys
by bvidinli April 25, 2008

April 25, 2008

I posted to so many lists because, this issue is related to all lists, this is an idea for python, this is related to development of python... why are you so much defensive ? i think ideas all important for development of python, software.... i am sory anyway.... hope will be helpful. 2008/4/24, Terry Reedy <tjreedy(a)udel.edu>: > Python-dev is for discussion of development of future Python. Use > python-list / comp.lang.python / gmane.comp.python.general for usage > questions. > > > > _______________________________________________ > Python-Dev mailing list > Python-Dev(a)python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/bvidinli%40gmail.com > -- İ.Bahattin Vidinli Elk-Elektronik Müh. ------------------- iletisim bilgileri (Tercih sirasina gore): skype: bvidinli (sesli gorusme icin, www.skype.com) msn: bvidinli(a)iyibirisi.com yahoo: bvidinli +90.532.7990607 +90.505.5667711

2 1

Associative arrays in hardware
by Neil Toronto April 16, 2008

April 16, 2008

Full-on associative memory wouldn't be necessary. If there were a machine-level instruction in current chips that did a dict-style lookup almost as fast as a base+index*size, Python could *scream*. It might work like 80x86's lea (load effective address): hashaddr base, recsize, dictsize, value, destreg It would be very CISCy, and there are a whole host of practical problems dealing with hashing functions, good collision resolution, etc. But any pointer/index-based (e.g. interned string) lookup could go almost as fast as C++'s "this->". Mainstream is moving toward dynamic languages, and it makes sense to support that in hardware. Heck, I hear routers and switches already have things like this. Does anybody have the ear of an Intel chip designer? Neil

3 3

Re: [Python-ideas] Associative arrays in hardware
by Neil Toronto April 16, 2008

April 16, 2008

Mathias Panzenböck wrote: > However, there still could be a collision so you have to compare the key > by value anyway, so the performance might not by that much better. And > pythons strings are immutable so the hash code is cached anyway and has > not be calculated every time. But this instruction could only calculate > a hash of a sequence of bytes (like a string), not of a complex object > which has some fields that form the key (and ohters that dont). So it's > pretty useless in my opinion. (But I'm so tired I can hardly think, > maybe I'm missing something.) Yes, and that's that most language identifiers are "interned" strings: just a pointer to them is sufficient for equality comparison. (Ruby and every Lisp dialect on the planet call these "symbols".) Python is so very close to using nothing but interned strings for all namespace dicts that it wouldn't be that difficult to change it. It's entirely possible that a hash of the string pointer would be sufficient, so it wouldn't even be necessary to hash the string and send it along. > Neil Toronto schrieb: >> Full-on associative memory wouldn't be necessary. If there were a >> machine-level instruction in current chips that did a dict-style >> lookup almost as fast as a base+index*size, Python could *scream*. It >> might work like 80x86's lea (load effective address): >> >> hashaddr base, recsize, dictsize, value, destreg >> >> It would be very CISCy, and there are a whole host of practical >> problems dealing with hashing functions, good collision resolution, >> etc. But any pointer/index-based (e.g. interned string) lookup could >> go almost as fast as C++'s "this->". Mainstream is moving toward >> dynamic languages, and it makes sense to support that in hardware. >> >> Heck, I hear routers and switches already have things like this. Does >> anybody have the ear of an Intel chip designer? >> >> Neil >

1 0

stealing "var" from javascript
by Aaron Watters April 14, 2008

April 14, 2008

I know it's too late but as long as py3k will break everything, why not add explicit introduction of local variables? This is one place where Perl actually gets something right with "use strict;" and "my variable=...;" In particular there is a big conceptual difference between creating a new variable and changing a binding to an existing variable. You could make this difference explicit by saying # create local myvariable (error if exists) and assign value var myvariable=value; # rebind local myvariable (error if doesn't exist) myvariable = value; This would allow catching a lot of nameerrors at compile time, and also name mistakes which don't show up as nameerrors (like accidentally typing a variable name wrong in such a way that it doesn't fail the test cases, but only fails after deployment...). Any assignment with no associated "my" declaration would automatically be an error at compile time. As I say, I know it's too late. I also know it's probably not a new idea, but I couldn't locate it in a brief attempt. I also know that various python lint tools catch most name problems like the above... hmmm... I actually wouldn't mind the "my" keyword instead of "var" since it is shorter and probably less frequently used in existing code... (btw, "yield" broke every existing financial application :( ). Just a thought. -- Aaron Watters

8 7

adopt an enum type for the standard library?
by Mark Summerfield April 12, 2008

April 12, 2008

AFAIK if you want enumerations you must either create your own, or use a third party module, or use namedtuple: Files = collections.namedtuple("Files", "minimum maximum")(1, 200) ... x = Files.minimum Using: MINIMUM, MAXIMUM = 1, 200 is often inconvenient, since you might have several different ones. Of course you could do: MIN_FILES = 1 MIN_DIRS = 0 Personally, I like enums and consider them to be a fundamental, but I don't like the above approaches. There is an enum module in PyPI http://pypi.python.org/pypi/enum/ and there are several versions in the Python Cookbook. Wouldn't one of these be worth adopting for the standard library? -- Mark Summerfield, Qtrac Ltd., www.qtrac.eu

10 14