Mailman 3 November 2005 - Python-Dev

Problems with the Python Memory Manager
by Travis Oliphant Nov. 24, 2005

Nov. 24, 2005

I know (thanks to Google) that much has been said in the past about the Python Memory Manager. My purpose in posting is simply to given a use-case example of how the current memory manager (in Python 2.4.X) can be problematic in scientific/engineering code. Scipy core is a replacement for Numeric. One of the things scipy core does is define a new python scalar object for ever data type that an array can have (currently 21). This has many advantages and is made feasible by the ability … [View More]

10 15

Re: [Python-Dev] a Python interface for the AST (WAS: DRAFT: python-dev...)
by Phillip J. Eby Nov. 24, 2005

Nov. 24, 2005

At 06:32 PM 11/22/2005 -0800, Brett Cannon wrote: > > Hmm, it would be nice to give a function a module > > name (like from an import statement) and have Python resolve it using > > the normal sys.path iteration. > > > >Yep, import path -> filename path would be cool. Zipped and frozen modules don't have filename paths, so I'd personally rather see fewer stdlib modules making the assumption that modules are files. Instead, extensions to the PEP 302 loader … [View More]

2 3

a Python interface for the AST (WAS: DRAFT: python-dev...)
by Steven Bethard Nov. 24, 2005

Nov. 24, 2005

I wrote (in the summary): > While there is no interface to the AST yet, one is > intended for the not-so-distant future. Simon Burton wrote: > who is doing this ? I am mad keen to get this happening. Brett Cannon wrote: > No one yet. Some ideas have been tossed around (read the thread for > details), but no one has sat down to hammer out the details. Might > happen at PyCon. Simon Burton wrote: > Yes, i've been reading the threads but I don't see anything > about a … [View More]

4 5

DRAFT: python-dev Summary for 2005-10-16 to 2005-10-31
by Tony Meyer Nov. 23, 2005

Nov. 23, 2005

And this one brings us up-to-date (apart from the fortnight ending yesterday). Again, if you have the time, please send any comments/ corrections to us. Once again thanks to Steve for covering me and getting this all out on his own. ============= Announcements ============= -------------- AST for Python -------------- As of October 21st, Python's compiler now uses a real Abstract Syntax Tree (AST)! This should make experimenting with new syntax much easier, as well as allowing … [View More]

3 3

ast status, memory leaks, etc
by Amaury Forgeot d'Arc Nov. 22, 2005

Nov. 22, 2005

Hello, Purify is not so difficult to use: just run and learn to read the output ;-) My config: Win2k using VC6sp5, and only 512Mb RAM. I downloaded the snapshot dated 2005/11/21 05:01, commented out #define WITH_PYMALLOC, built in debug mode, modified the rt.bat file to use purify, and ran "rt -d". Here are the most important results so far : 1 - Memory error in test_coding, while importing bad_coding.py : IPR: Invalid pointer read in tok_nextc {1 occurrence} Reading 1 byte from … [View More]

2 1

s/hotshot/lsprof
by Armin Rigo Nov. 22, 2005

Nov. 22, 2005

Hi! The current Python profilers situation is a mess. 'profile.Profile' is the ages-old pure Python profiler. At the end of a run, it builds a dict that is inspected by 'pstats.Stats'. It has some recent support for profiling C calls, which however make it crash in some cases [1]. And of course it's slow (makes a run take about 10x longer). 'hotshot', new from 2.2, is quite faster (reportedly, only 30% added overhead). The log file is then loaded and turned into an instance of the same '… [View More]pstats.Stats'. This loading takes ages. The reason is that the log file only records events, and loading is done by instantiating a 'profile.Profile' and sending it all the events. In other words, it takes exactly as long as the time it spared in the first place! Moreover, for some reasons, the results given by hotshot seem sometimes quite wrong. (I don't understand why, but I've seen it myself, and it's been reported by various people, e.g. [2].) 'hotshot' doesn't know about C calls, but it can log line events, although this information is lost(!) in the final conversion to a 'pstats.Stats'. 'lsprof' is a third profiler by Brett Rosen and Ted Czotter, posted on SF in June [2]. Michael Hudson and me did some minor clean-ups and improvements on it, and it seems to be quite useful. It is, for example, the only of the three profilers that managed to give sensible information about the PyPy translation process without crashing, allowing us to accelerate it from over 30 to under 20 minutes. The SF patch contains a more detailed account on the reasons for writing 'lsprof'. The current version [3] does not support C calls nor line events. It has its own simple interface, which is not compatible with any of the other two profilers. However, unlike the other two profilers, it can record detailed stats about children, which I found quite useful (e.g. how much take is spent in a function when it is called by another specific function). Therefore, I think it would be a great idea to add 'lsprof' to the standard library. Unless there are objections, it seems that the best plan is to keep 'profile.py' as a pure Python implementation and replace 'hotshot' with 'lsprof'. Indeed, I don't see any obvious advantage that 'hotshot' has over 'lsprof', and I certainly see more than one downside. Maybe someone has a use for (and undocumented ways to fish for) line events generated by hotshot. Well, there is a script [4] to convert hotshot log files to some format that a KDE tool [5] can display. (It even looks like hotshot files were designed with this in mind.) Given that the people doing that can still compile 'hotshot' as a separate extension module, it doesn't strike me as a particularly good reason to keep Yet Another Profiler in the standard library. So here is my plan: Unify a bit more the interfaces of the pure Python and the C profilers. This also means that 'lsprof' should be made to use a pstats-compatible log format. The 'pstats' documentation specifically says that the file format can change: that would give 'lsprof' a place to store its detailed children stats. Then we can provide a dummy 'hotshot.py' for compatibility, remove its documentation, and provide documentation for 'lsprof'. If anyone feels like this is a bad idea, please speak up. A bientot, Armin [1] https://sourceforge.net/tracker/?group_id=5470&atid=105470&func=detail&aid=… [2] http://sourceforge.net/tracker/?group_id=5470&atid=305470&func=detail&aid=1… [3] http://codespeak.net/svn/user/arigo/hack/misc/lsprof (Subversion) [4] http://mail.python.org/pipermail/python-list/2003-September/183887.html [5] http://kcachegrind.sourceforge.net/cgi-bin/show.cgi [View Less]

15 37

test_cmd_line on Windows
by A.B., Khalid Nov. 22, 2005

Nov. 22, 2005

Currently test_directories of test_cmd_line fails on the latest Python 2.4.2 from svn branch and from the svn head. The reason it seems is that the test assumes that the local language of Windows is English and so tries to find the string " denied" in the returned system error messages of the commands ("python .") and ("python < ."). But while it is true that the first command ("python .") does return an English string error message even on so-called non-English versions of Windows, … [View More]

2 1

svn diff -r {2001-01-01}
by "Martin v. Löwis" Nov. 22, 2005

Nov. 22, 2005

Greg Stein points out that because of the way the subversion conversion was done, by-date revision specifications won't work. Subversion assumes that time is monotonically increasing over revions numbers - it does a binary search to find out the revision that immediately precedes(?) the specified date. Yet, as the conversion was done project-by-project (toplevel svn dirs), commit time sometimes goes backward along with increasing revision numbers; this breaks the algorithm svn uses. There are … [View More]

2 1

s/hotshot/lsprof
by Jim Jewett Nov. 21, 2005

Nov. 21, 2005

Jeremy Hylton jeremy at alum.mit.edu > Should lsprof be added to the standard distribution? > Should hotshot be removed from the standard distribution? > These two aren't at all related, unless you believe that two is the > maximum number of profiles allowed per Python distribution. One is a better number. ("There should be one-- and preferably only one --obvious way to do it.") Adding a second (let alone third) module to the stdlib to do the same thing just makes the … [View More]

2 1

DRAFT: python-dev Summary for 2005-11-01 through 2005-11-15
by Tony Meyer Nov. 21, 2005

Nov. 21, 2005

Surprise! It's November, and here's a November summary <wink>. Thanks to all those that proofread the triple summary hit last week; if anyone can spare some time to take a look over these in the next couple of days, that would be great. As always, corrections and suggestions to tony.meyer(a)gmail.com or steven.bethard(a)gmail.com. A couple of largish threads were skipped: one continuing discussion about freezing (I couldn't come up with a summary longer than "the heated debate continued"),… [View More]

1 0