Mailman 3 June 2004 - Python-Dev

RE: [Python-Dev] Object free lists
by Eric Huss June 15, 2004

June 15, 2004

> I think you'd have better luck with a patch that bounded the sizes of these > free lists. That's not hard to do; e.g., look at the simple bounded free > list in frameobject.c. I wouldn't like a small limit on ints or floats, but > letting any subsystem suck up unbounded resource forever does cause > problems. Unfortunately the limit code would be much more complex than in frameobject because frameobject does not allocate in blocks. So, we would have to somehow determine if an int block is completely free before freeing it. Some different methods: 1) Scan each block and count the number of free integers. If the block contains only free integers, then free it. This would essentially be the code in PyInt_Fini(). We actually have code that calls PyInt_Fini when we are doing leak detection. 2) Alternate to #1, it could avoid scanning entire blocks if it detected a used int, and instead manually dissect the free_list iff the entire block is free instead of rebuilding free_list from scratch like PyInt_Fini() does. 3) Add a counter of the number of free ints in PyIntBlock. Free the block with this number equals the maximum number of ints in a block. The problem with #1 is that it is slow, and one would need to determine how often to call it. #2 has the same problem. #3 is a little nicer, at the cost of maintaining the counter (which should be negligible). Thoughts? -Eric

3 2

what about DECREF() implemented in python?
by alejandro david weil June 15, 2004

June 15, 2004

Did some version of python had DECREF() implemented in python? Or, at least, some callback / overload posibility from python, like, ob_type->tp_dealloc ? I think that with this, someone could implement, in python, the object-finalizators when an object goes out of scope.. Has sense that I make some stats to see if it's time expensive? See ya, david -- + There is no dark side of the moon really. Matter of fact it's all dark.

1 1

Re: [Python-Dev] Object free lists
by Bob Ippolito June 15, 2004

June 15, 2004

On Jun 15, 2004, at 10:52 PM, Tim Peters wrote: > [Eric Huss] >> The only problem I see with using pymalloc is that the objects won't >> be >> stored in memory in such a compact way. > > The difference can be large, because pymalloc aligns to 8-byte > boundaries. > So, e.g., on most boxes today a Python int object consumes 12 bytes, > but > would consume 16 if allocated with pymalloc. That's a 33% boost, and > that > can be significant for apps hanging on to lots of ints. > > OTOH, on most boxes there would be no real size difference for float > objects. The implementation of malloc in OS X always returns pointers aligned to 16-byte boundaries, so that the buffers can be reasonably used by AltiVec instructions. Would there be a difference in this case? :) -bob

2 1

Object free lists
by Eric Huss June 15, 2004

June 15, 2004

Is there a plan to switch the basic objects (int, float, method, etc.) to use pymalloc instead of the block allocation technique they currently use? The reason I ask is because the free_list in these objects is unbounded. It becomes a problem in a server environment if activity spikes cause different types of objects to start eating memory and never give it up. The only problem I see with using pymalloc is that the objects won't be stored in memory in such a compact way. I would imagine there would be no performance difference. Of course some optimizations (such as small_ints) would stay. I would be willing to submit a patch with the change if there is interest. -Eric

2 1

Re: Object finalization for local (ie function) scop es
by Jewett, Jim J June 15, 2004

June 15, 2004

Paul Moore: > Can you enumerate precisely how your proposal differs from PEP 310? > ... > 3. Derives the objects to call the "release" method on, based on some > sort of local variable scan (I don't follow this too well, so I > apologise if I've described it badly). My (perhaps faulty) understanding is that it depended only on the class of the object. PEP 310 says to call the __enter__ and __exit__ methods if an object is wrapped by a "with:" clause. The scope-guard proposal says to call those methods anytime they exist. Neither works well with the rare case when you do want the object to outlive its initial scope. I see a best-of-both-worlds by treating __exit__ as a mini-__del__. __exit__ would typically be called sooner, and it would always be called, even if this meant breaking cycles in an arbitrary order. Unfortunately, one reason __del__ cycles aren't broken this way is that people don't always follow the guidance for what should be done in a __del__ function. The meaning of __exit__ would be "okay, we'll run the cleanup like we should, but now messups are your own fault!". If that wasn't OK the first time (__del__), I'm not sure it will be OK the second. -jJ

1 0

Changing PCBuild/*.vcproj to text mode
by Löwis Martin von June 15, 2004

June 15, 2004

I changed all *.vcproj files to text mode, removing the CRLFs in the repository. In your sandboxes, you will still see the -kb flag, because it is a sticky tag. You need to perform cvs up -A on the PCBuild directory to remove the sticky tag from your sandbox. This causes a text check-out, which, when you are using CVSNT, will put in CRLFs into the files. Kind regards, Martin

1 0

Object finalization for local (ie function) scopes
by Oliver Schoenborn June 14, 2004

June 14, 2004

This is a module I made to test a concept of finalization *not* based on __del__. I'm curious to find out if the techniques look like they could be implemented directly in the Python interpreter to lessen the burden on the user, which is why I post this here. Summary: - Seems like can't rely on __del__ being called (standard says no guarantee) - Means a coder of a class for which instances need some sort of finalization (e.g. file objects need close()) must document textually the need for finalization - User of class must remember to use try/finally blocks and position them judiciously since several objects may require "finalization"; this is easy to forget if there are multiple (normal or exceptional) exit points. The technique is based on inheritance: a class that needs finalization is derived from scope.NeedsFinalization. This class adds info to a special function, called scope.ScopeGuardian, that wraps a try/finally around the function that you are interested in making "finalization safe" (ie the function in which you would normally have to worry about manually putting a try/finally). This seems to work well and has the advantage that: - Coder of class makes explicit by base class that finalizaiton is important, no need to remember to document - User of class knows just by looking at class def, can't overlook a sentence saying "call this-that when done with object" - Some error checking can be done to decrease likelyhood of user forgetting the try/finallies (which are no longer needed) or of wanting finalization before ref count goes to 0. The info added to the function, seems to me, could be done at the interpreter implementation level, e.g. by adding a new field to function objects (f_name, etc), such as f_scopedobjs, so it would be almost trivial to use even for beginners. There are a couple of limitations in the current implementation that are easy to fix (e.g. make it work for recursive functions, nested scope guarding, and methods), don't know yet about thread safety. Anyways, from having looked at the archives, doesn't seem like this technique has been used, and would't require addition of a new keyword (only a new class, .e.g as opposed to pep310 approach). Thanks for any feedback, Oliver

6 25

pydoc improvements
by Noam Raphael June 14, 2004

June 14, 2004

Hello, I've made a few improvements to the pydoc module. I've tried to deliver them to Ka-Ping Yee, but it seems that the e-mail wasn't transferred, so here I present them to python-dev, including Ping. I hope they will be reviewed and, if found to be appropriate, accepted. Pretty-Printed Source Code ========================== Many times, in order to understand exactly what a function does, I'm interested in its source code. This is especially true for Python, because of its readability. So I made pydoc produce, along with the HTML documentation of a module, a highlighted source code of the module, interlinked with its documentation. I added to the HTMLDoc class a new method, sourcemodule (you may think of a better name). It gets a filename and a few more parameters, and returns the pretty-printed source code, in HTML format. Class and method definitions are links to the description of them in the generated module documentation. I also made class and function definitions in the module documentation link to their definition in the source code. I've changed the second link on the right-top of a module documentation to point to the HTML source instead of the original file, because I think usually people are interested in seeing the code in their browser, not in downloading it. In case they are interesting in downloading it, I added a link called "download <filename>" to the HTML source, so people can click it and hopefully get the pure source code. The file name of the HTML source code is "<module name>-source.html". It is now generated automatically along with the "<module name>.html", and is served by the HTTP server. Public and Private ================== Public attributes are defined by the current pydoc version as attributes whose name doesn't start with an underscore, or, for module attributes where __all__ is defined, attributes which appear in __all__. Currently pydoc doesn't show private objects at all. This is usually what people want, but sometimes, when they are interested in what the module really does (for example when they want to define a subclass), they may want to view the documentation for private objects too. So I added a link to the module documentation header, which alternates between "Show Private" and "Hide Private", and shows or hides private attributes - both module attributes and class methods. To accomplish this, I defined two alternative style sheets: one which shows private attributes and one which doesn't. The "Show/Hide Private" link uses javascript to switch between these two styles. (The style can also be selected by a menu in the Mozilla browser.) I used a cookie to save the user's prefered style - if I hadn't, the style would've returned to the default every time a new documentation page is opened. The produces HTML is standard-compliant, to my knowledge, and works with both Mozilla and IE. The style-switching link doesn't work in konqueror, so one using it won't be able to see private attributes - not too bad. Many modules contain objects imported from other modules. Usually these are for the module's own use, and the module's user isn't interested in them. These are not even private - I don't think anyone will want to see their documentation. In the current version of pydoc, these are filtered by calling inspect.getmodule. This raises two problems: the first is that inspect.getmodule can't deduce the module of built-in objects, so the module documentation may be filled with such "garbage" (it actually happened to me, before I've used my version of pydoc). The second problem is that sometimes a module wants attributes of other modules to be considered as its own attributes too (for example, the tokenize module exports the constants of the token module). My solution is this: attributes which are not public attributes of a module, and are public attributes of another imported module, are never displayed. They are probably the result of an internal "from module import something" statement. The patch attached is against version 1.91, because I implemented using __all__ in a bit different way than version 1.92. More flexible HTML documentation of a complete directory ======================================================== Here's my actual situation: There's a package maintainance system, where you first create files in your home directory and then use the system to install them in a common place. For example, if a package is called foo, and includes the modules ya.py and chsia.py, the modules will be named /home/noam/foo/lib/python/{ya,chsia}.py. I want to automatically document them, so that the HTML files will sit in /home/noam/foo/doc. After installing the package, the python modules will be copied to /usr/local/lib/python/, and the documentation will be copied to /usr/local/doc/foo. I want to be able to type one command to generate all the documentation, but I don't want to waste time if I only need the documentation re-generated for one module. So I made "pydoc -w <dir>" produce documentation only if the target file was modified before the module was. (pydoc -w <filename> still write the documentation even if it seems up to date.) It is perfectly legitimate to write a HTML documentation for a module by your own. So I made pydoc never overwrite HTML files which weren't created by pydoc. To accomplish this, I added the tag <meta name="generator" content="pydoc"> to the head of all HTML files produced by pydoc. A warning is printed if a module was modified after a manually created documentation was modified. I wanted to be able to run "pydoc -w" from any directory, not just from /home/noam/foo/doc, so I added an option "-o <dir>", to specify the output directory. (If it isn't specified, the current directory is used.) I wanted the documentation to refer to the installed file (/usr/local/lib/python/foo.py), not to the file in my home directory. So I added an option "-d <destdir>", to specify the directory in which the files will reside after being installed. To summarise, the command which I should now run to produce the documentation is pydoc -w -o doc/ -d /usr/local/ lib/python/ (running it from /home/noam/foo). The -d and -o options are already available in compileall.py, for exactly the same reasons, I think. None of this should cause backward compatibility issues. Module loading ============== Currently, safeimport(), used by writedocs(), gets only a module name and imports it using __import__(). This means that when you document a complete directory, you can only produce HTML documentation for modules which are already installed, and that a documentation for the wrong version of a module may be produced. I changed safeimport() to get an optional list of directories in which to look for the module/package, in which case it uses imp.find_module and imp.load_module instead of __import__. This means that when producing documentation for a complete directory, the actual files in the directory are used, not installed modules of the same name. Misc. ===== I changed the docmodule methods, to use cStringIO.StringIO.write instead of s = s + something. in do_GET, forceload was set to True for some reason. It caused problems when documenting modules which pydoc itself uses, and I don't see why should modules be forced to load, so I changed it to False. The link to a file was "file:"+path. I changed it to "file://"+path, because my browser (Mozilla) treated the path as a relative path. Attached is a diff against version 1.91 of pydoc.py (see the section on public and private). I will be glad if after your review, at least some of these changes will be accepted. Have a happy day, Noam Raphael

3 2

LC_NUMERIC, gtk and Patch 774665
by Nick Coghlan June 14, 2004

June 14, 2004

A problem was recently reported on wxPython-dev that smells strongly of this issue with GTK changing the LC_NUMERIC locale to something other than 'C' causing subsequent severe problems in the handling of floats by the Python interpreter. I went looking for Christian's proposed PEP (to make Python more locale agnostic when it comes to LC_NUMERIC), and didn't find it listed in PEP 0. The last activity on Gustavo's patch was in December 2003, and the last comment in September. What's the current state of this? Regards, Nick. -- Nick Coghlan | Brisbane, Australia Email: ncoghlan(a)email.com | Mobile: +61 409 573 268

5 6

Re: [Python-Dev] VC 7.1 maintenance?
by Guido van Rossum June 14, 2004

June 14, 2004

> > therefore CVS won't even attempt merges. > > Can't say I care. The checkin msg showed a teensy context diff, and I > couldn't measure the time difference between CVS applying a patch or > replacing the file. I agree, but I didn't mean that. I meant when there are local changes and changes in the CVS repository, for text files CVS attempts to merge those changes, but for binaries it simply moves the local file aside and checks out a fresh copy. --Guido van Rossum (home page: http://www.python.org/~guido/)

1 0