Mailman 3 March 2002 - Python-Dev

Re: [Python-Dev] A Hygienic Macro System in Python?
by Michael Hudson 19 Mar '02

19 Mar '02

David Ascher <DavidA(a)ActiveState.com> writes: > My vote on the notion is that I want to know what good macros people > would write, and then implement them in a non-macro fashion. My favourite example of this is distutils.core.setup() 1) looks declarative 2) isn't Very Pythonic, IMHO. Cheers, M. -- Ya, ya, ya, except ... if I were built out of KSR chips, I'd be running at 25 or 50 MHz, and would be wrong about ALMOST EVERYTHING almost ALL THE TIME just due to being a computer! -- Tim Peters, 30 Apr 97

1 0

pymalloc on Windows
by Mark Hammond 19 Mar '02

19 Mar '02

It appears that the _sre and _winreg modules fail to build with WITH_PYMALLOC defined. Both are extension modules, and both use the macro versions of the memory allocator functions - ie, PyObject_NEW etc. Both fail at link time with, eg: _winreg.obj : error LNK2001: unresolved external symbol __PyCore_ObjectMalloc I assume from previous discussions here that _PyCore_ObjectMalloc is indeed supposed to be private and not exposed from the main module/dll. I have patches for both these modules. However, I wonder if a more pragmatic option is to expose this core function. It would allow PyObject_NEW to continue to work as before. I also just saw Martin say that some of these macros may define themselves back to the function anyway making all this moot :) So - I guess all I am saying is - at the moment, it appears there are at least a couple fo problems building the core with pymalloc under windows. I have patches, but in a short while they may not be necessary :) FWIW, with 3 PyObject_NEW changes in the Win32 extensions, the COM tests pass and pythonwin starts up :) Mark.

8 10

compiler-sig project for Python 2.3: new AST
by Jeremy Hylton 19 Mar '02

19 Mar '02

These are notes copied from a Wiki at http://www.zope.org/Members/jeremy/CurrentAndFutureProjects/PythonAST They are a bit rough, but they set out some of the work I'd like to do for Python 2.3. Some of the work belongs in a PEP, but I'm not ready to write it yet. Please reply to compiler-sig(a)python.org. Jeremy A New AST for Python -------------------- I have proposed some namespace optimizations for Python 2.3. These optimizations require more analysis by the compiler, which is quite difficult given the current parse tree representation. As a first step towards those optimizations, I plan to introduce a new AST that makes analysis easier. This step is a major undertaking by itself. The primary goal of the new AST is to provide better intermediate representation(s) so that the compiler is easier to maintain and extend. One benefit is to enable optimizations by providing better tools for analyzing programs during bytecode compiling. I expect the new AST will be based largely on the one in the compiler package (part of the std library in 2.2), which was originally done by Greg Stein and Bill Tutt. Rough plan of action -------------------- -- Define the AST. Settle on C and Python (and Java?) implementations of AST. I like the look of the C code generated by asdlGen, but haven't had a chance to get the tool working. See Dan Wang's DSL 97 paper: The Zephyr Abstract Syntax Description Language. Could always write new asdlGen tool for targetted languages -- Write converter from current parse tree to AST. Basic functionality of compiler/transformer.py. Replace parser module. WingIDE? folks have implemented something like this in C. parsetools -- Expose AST to Python programs Replace parser module and parts of compiler package. asdlGen has a notion of AST pickles. Would that be sufficient? -- An AST visitor in C? The visitor pattern is very useful for manipulating ASTs? in Python. Could it be translated to C somehow? -- Reimplement compile.c with new AST Break it up into several passes: future stmts, doc strings, symbol table, code gen. Not sure how much of the code in the compiler package may be useful here. The codegen part is relatively good, the pyassem part is pretty ugly. Testing ------- I overhauled the compiler for Python 2.1 to implement nested scopes. I spent a lot of time debugging the various compiler passes, usually because I forgot to handle a particular kind of grammar production. The new AST should make this a lot easier, but I would also like to see a thorough test suite for the new code. I don't know anything about best practices for testing compilers. I imagine that a system to generate sample inputs from the language grammar would be helpful for debugging the C code that walks the AST.

1 0

Idea - place to put string functions and consts
by Greg Ewing 19 Mar '02

19 Mar '02

I had an idea last night concerning what to do with functions like string.join that don't quite belong as methods of a string, plus constants like string.uppercase etc. Someone suggested making the os.* routines class methods of 'file'. So how about making these things class methods of 'str'? e.g. mystring = str.join(["spam", "eggs"], ',') reads quite nicely. Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg(a)cosc.canterbury.ac.nz +--------------------------------------+

5 10

RE: [Python-Dev] Re: Activating pymalloc
by Vladimir Marangozov 19 Mar '02

19 Mar '02

Hi, [Tim] > > [me] > > 2. I agree that the macro chain (especially on the pymalloc side) > > is not so useful at the end, so maybe all PyCore_ macros can be > > removed. The function names of the allocator can be cast in > > stone and then _THIS_xxx in obmalloc.c replaced with them. > > Have you looked at Neil's patch? I had a quick look. I'm not happy with it for two reasons: - it removes all those useful comments in pymem.h and objimpl.h - the overall picture of the memory APIs in less clear than before (and there are no docs trying to clarify the issue) So let's step back for a while, assume we start from scratch with the APIs, put forward the main memory concepts again, agree on all APIs then see what are the diffs with the current state. Here is a 1st shot of it which makes succinctly the round around the clock on the issue. Hopefully you'll be able to work out the details on python-dev. ====================================================================== Prelude ------- We want to introduce a Python-specific allocator that can operate on heaps managed by the Python core, and only by the Python core. We would like to differentiate two types of Python heaps: raw memory heap and object memory heap (argumentation is left aside). Accessing these heaps is done through different memory APIs. Naming conventions ------------------ The proposal is to use the following prefixes for the memory APIs: - PyMem_xxx for raw memory - PyObject_xxx for object memory (PyMemObject_xxx would be another suggestion for object memory but PyObject_ is enough) Raw memory ---------- For raw memory allocation, the proposal is to use the following two APIs: - PyMem_{MALLOC, REALLOC, FREE} - raw malloc - PyMem_{NEW, _DEL} - type-oriented raw malloc and their function counterparts for extension modules. These are defined in pymem.h Object memory ------------- For object memory allocation, the proposal is to use the following API for allocating storage from the object heap: - PyObject_{MALLOC, REALLOC, FREE} - object malloc and their function counterparts for extension modules. For creating an arbitrary (typed) Python object, which is not subject to GC, the proposal is to use the following API: - PyObject_{New, NewVar, Del} Note that creating an object is a 2-step process: a) storage allocation + b) initialization (Py_NewReference). For creating a Python object subject to GC, the proposal is to use the following API: - PyObject_GC_{New, NewVar, Del} These APIs are defined in objimpl.h Switching allocators -------------------- Without a python-specific malloc, the libc standard malloc is used. For the raw allocator, the macros PyMem_{MALLOC, REALLOC, FREE} expand to {malloc, realloc, free} respectively (in pymem.h). For the object allocator, the macros PyObject_{MALLOC, REALLOC, FREE} expand to {malloc, realloc, free} respectively (in objimpl.h). Assuming we would like to use a specific allocator, we can make the two sets of macros above expand to this allocator independently. Changing mallocs automatically should be doable with a configure option, so that the macro expansion above is automated. Python allocator ---------------- A Python allocator is a specialized allocator that tries to optimize memory management according to Python's specific memory needs and allocation patterns. Such allocator would be called only by the Python core via the APIs described above: PyMem_ for raw memory, and PyObject_ for object memory. Therefore, the proposal for this allocator is to export the following API: - PyCore_{Malloc, Realloc, Free} (typically this means that pymalloc's functions are named this way). Controversial issues -------------------- Practically, releasing memory is done the same way with all APIs, so PyMem_FREE equals PyMem_DEL. Similarly, there will be implementation equivalences for the object APIs. It is believed that these are implementation details for each memory API. Therefore, the rule of thumb is to always use the same API for a given memory block. The examples section in the docs tries to illustrate this rule. API Summary ----------- Raw malloc API: PyMem_{MALLOC, REALLOC, FREE} PyMem_{NEW, DEL} PyMem_{Malloc, Realloc, Free} PyMem_{New, Del} Object malloc API: PyObject_{MALLOC, REALLOC, FREE} PyObject_{Malloc, Realloc, Free} PyObject_{New, NewVar, Del} PyObject_GC_{New, NewVar, Del} Python's internal malloc API: PyCore_{Malloc, Realloc, Free} ====================================================================== All in all, this is pretty close to what we have now and future comments & patches should be made according to the global picture like the one above, and preferably well documented in the source in addition to the docs. About tuning pymalloc: I would suggest reviving the profiler which sleeps in the patches list. You'll find it invaluable for gathering the stats directly from Python. After some profiling, you'll have good reasons to change a parameter or two (like the small block threshold). Spacing the size classes by 4 bytes is not good (well, *was* not good at the time). Spacing them by 16 might be good now. I don;t know -- just try it. Changing the pool size might be good as well. But if the performace deltas you get are relatively small, this would be a platform-specific effect, so just leave things as they are. Vladimir

3 2

RELEASED: Python 2.2.1c1
by Michael Hudson 18 Mar '02

18 Mar '02

We've released a release candidate for the next bugfix release of Python, 2.2.1. Get the scoop (and the files) here: http://www.python.org/2.2.1/ In order to make 2.2.1 a solid release, please help by + Building the release, and running the test suite on your platform. + Building your extension modules and applications against this release, and running *their* test suites. + Reporting any problems to the bug tracker at sf: http://sourceforge.net/bugs/?group_id=5470 This being a bugfix release, there are no exciting new features -- we just fixed a lot of bugs. For a moderately complete list, please see: http://http://sourceforge.net/project/shownotes.php?release_id=80208 Depending on how many problems are found in this release, 2.2.1 final or a second release candidate will follow next week. Extra note for keenies: One of the bugs addressed in this release relates to the behaviour of floating point numbers with respect to under- and over-flow. This has traditionally been an obscure platform-dependent mess. To make sure that this is still the case, I would ask those of you with access to the more unusual platforms (i.e. not Linux/x86, Win32/x86) to run test_math in verbose mode and *email me* the results, including details of platform and OS tested. Thanks. Cheers, The Python development team.

1 0

[development doc updates]
by Fred L. Drake 18 Mar '02

18 Mar '02

The development version of the documentation has been updated: http://python.sourceforge.net/maint-docs/ Documentation for 2.2.1 release candidate 1.

1 0

2.2.1c1 is go!
by Michael Hudson 18 Mar '02

18 Mar '02

I believe the next step according to PEP 102 is for Fred to build the docs and Tim to build the Windows installer. Please, no one check anything in to the release22-maint branch without checking with me, Tim or Fred, as appropriate (unless you are Tim or Fred, of course). When Tim and Fred are done, I should be ready to do the remaining steps (praise be for PEP 102!), but I have a tutorial to give now... Cheers, M. -- 81. In computing, turning the obvious into the useful is a living definition of the word "frustration". -- Alan Perlis, http://www.cs.yale.edu/homes/perlis-alan/quotes.html

2 1

Re: [Python-Dev] 2.2.1 release schedule
by Michael Hudson 18 Mar '02

18 Mar '02

Guido van Rossum <guido(a)python.org> writes: > [Guido] > > > I have one concern: an awful large number of patches went into 2.2.1 > > > in a very short time, and I worry a bit that one release candidate may > > > not be sufficient to make sure that we really didn't introduce any new > > > bugs. > > [Michael] > > I think the time argument may be a red herring; I'm not sure there are > > so many people checking the branch out that it really makes any > > difference. > > I was more thinking that you've been working so hard that your error > rate might have gone up. :-) Quite possibly. > > But I agree there have been a lot of changes, and some pretty subtle > > ones. > > That's the real reason to be cautious. > > > > Perhaps we should consider to issue a second release candidate, > > > or at least have a waiting time longer than 1 week between rc and > > > final. (I'd be happy with 2 weeks.) > > > > How about releasing 2.2.1c1, waiting two weeks and then deciding > > whether we need a c2? In an ideal world, changing the candidate > > release into a final release would just be a matter of changing > > version numbers. > > Sounds good, but that means two full weeks of willpower exercises. I think the effort of cleaning things up for rc1 has given me sufficient grouchiness for this. > Or did you think you wouldn't get any pressure to add more > last-minute fixes to rc2? :-) > > > Two weeks gets us pretty near Easter; I may not be around so much for > > the Easter weekend. > > I don't mind about the exact timing, and assuming there are no > disasters, not much would need to be done. OK, so my plan is: get 2.2.1c1 out; then wait and see what happens. Cheers, M. -- Java sucks. [...] Java on TV set top boxes will suck so hard it might well inhale people from off their sofa until their heads get wedged in the card slots. --- Jon Rabone, ucam.chat

3 2

Re: [Python-Dev] 2.2.1 release schedule
by Michael Hudson 18 Mar '02

18 Mar '02

Guido van Rossum <guido(a)python.org> writes: > > Things are looking as if this might actually happen. > > I have one concern: an awful large number of patches went into 2.2.1 > in a very short time, and I worry a bit that one release candidate may > not be sufficient to make sure that we really didn't introduce any new > bugs. I think the time argument may be a red herring; I'm not sure there are so many people checking the branch out that it really makes any difference. But I agree there have been a lot of changes, and some pretty subtle ones. > Perhaps we should consider to issue a second release candidate, > or at least have a waiting time longer than 1 week between rc and > final. (I'd be happy with 2 weeks.) How about releasing 2.2.1c1, waiting two weeks and then deciding whether we need a c2? In an ideal world, changing the candidate release into a final release would just be a matter of changing version numbers. Two weeks gets us pretty near Easter; I may not be around so much for the Easter weekend. Cheers, M. -- I've reinvented the idea of variables and types as in a programming language, something I do on every project. -- Greg Ward, September 1998

4 3