Python-Dev February 2000

python-dev@python.org

41 participants
73 discussions

by Fred L. Drake, Jr.

I've just updated the page at python.org/patches/ to ask that documentation patches be sent to python-docs(a)python.org, with specific bug reports to be submitted through the bugs database (which I monitor). Hopefully this will clarify what should happen for documentation issues. It also means that the published documentation on the Web site isn't out of date! -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> Corporation for National Research Initiatives

21 years, 1 month

Malloc interface (was: [Patches] Re: Garbage collection patches for Python (fwd)

by Vladimir Marangozov

I'm bringing the issue on python-dev, because it may change your coding habits in the future. If you have any comments or something... ---------------------------------------------------------------------- Subject: Re: [Patches] Re: Garbage collection patches for Python To: nascheme(a)enme.ucalgary.ca Date: Wed, 9 Feb 2000 15:59:48 +0100 (CET) Cc: Vladimir.Marangozov(a)inrialpes.fr, toby(a)puckish.demon.co.uk, pygc(a)freemail.hu, patches(a)python.org (Python Patches), guido(a)cnri.reston.va.us (Guido van Rossum) In-Reply-To: <20000205092822.A2204(a)acs.ucalgary.ca> from "nascheme(a)enme.ucalgary.ca" at Feb 05, 2000 09:28:23 AM Reply-To: Vladimir.Marangozov(a)inrialpes.fr X-Mailer: ELM [version 2.5 PL1] Content-Length: 6224 nascheme(a)enme.ucalgary.ca wrote: > > On Sat, Feb 05, 2000 at 03:00:59PM +0100, Vladimir Marangozov wrote: > > Neil, could you please resubmit your 1st GC patch with the disclaimer, > > so that we can focus on this 1st step, test the patch and fold it into > > CVS? > > No problem. I've included my Boehm-Demers-Weiser garbage > collector patch too as I have improved it to use autoconfig. The > gc patch is small compared to the malloc cleanup. Thanks. After a closer look at the patch, I see a couple of problems with it. In particular, it mixes different API families. Let me start from the start: The goal is to remove Python's dependency on the standard POSIX interface (malloc/realloc/free) so that we can cleanly and easily plug in the future a "proprietary" mem manager, other than the one in the C library. For this purpose, the Python core should be patched and "cleaned" to use one or more of the following APIs: 1) PyMem_MALLOC 2) PyMem_NEW PyMem_REALLOC ==> PyMem_RESIZE PyMem_FREE PyMem_DEL PyMem_XDEL The proposed Guido's augmented version of 1) which is raw mem interface. more "Pythonic" and contains some safety additions, like _PyMem_EXTRA). This one (in mymalloc.h) should be defined in terms of 1). 3) Py_Malloc 4) PyMem_Malloc 5) _PyObject_New Py_Realloc PyMem_Realloc PyObject_NEW Py_Free PyMem_Free PyObject_NEW_VAR, ... These are implemented using 1) and/or 2) It seems to me that presently nobody uses the wrappers 3) and 4), (except one single occurence of Py_Malloc, lost in _localemodule.c), because these wrappers cause an additional func call overhead... All these APIs are certainly redundant and the added value of each of them has to be weighted one again, but for now, let's assume that we have them all (without more argumentation - we could throw away some of them later). <EXTRABOLD> The rule of thumb in this situation is: Every chunk of memory must be manupulated via the same malloc family. </EXTRABOLD> That is, if one gets some piece of mem through PyMem_MALLOC (1), s/he must release that memory with PyMem_FREE (1). Accordingly, if one gets a chunk via PyMem_MALLOC (1), that chunk *should not* be released with PyMem_DEL (2). (which is what Neil's patch does, not to mention that (2) is not defined in terms of (1). So we must be careful here. In particular, when patching a file, we must figure out which family (-ies) is (are) used in order to perform the "malloc cleanup" the right way. That is, we have to inspect every file one by one, once again. ----------- With the above theory in mind, I see that there are a couple of "problematic" files in the current snapshot. I remember that I have identified them with my allocator (pymalloc) either. (I'll use some excerpts of Neil's patch to illustrate the problems I'm talking about) 1) pypcre.c This one is "buggy" for sure. The PCRE code allocates memory through the proprietary functions "pcre_malloc/pcre_free" (which default to malloc/free), so I really don't see why there's code inside mentioning "free" and not "pcre_free". a) Would this code be dropped with the inclusion of /F's re engine in 1.6? b) If the answer to a) is negative, I hope AMK & the String-SIG will help us on this (it's a hairy code :-). > diff -cr Python-cvs/Modules/pypcre.c Python-gc/Modules/pypcre.c > *** Python-cvs/Modules/pypcre.c Sat Jul 17 12:13:10 1999 > --- Python-gc/Modules/pypcre.c Sat Feb 5 09:00:47 2000 > *************** > *** 3057,3068 **** > static int free_stack(match_data *md) > { > /* Free any stack space that was allocated by the call to match(). */ > ! if (md->off_num) free(md->off_num); > ! if (md->offset_top) free(md->offset_top); > ! if (md->r1) free(md->r1); > ! if (md->r2) free(md->r2); > ! if (md->eptr) free((char *)md->eptr); > ! if (md->ecode) free((char *)md->ecode); > return 0; > } > > --- 3057,3068 ---- > static int free_stack(match_data *md) > { > /* Free any stack space that was allocated by the call to match(). */ > ! if (md->off_num) PyMem_DEL(md->off_num); > ! if (md->offset_top) PyMem_DEL(md->offset_top); > ! if (md->r1) PyMem_DEL(md->r1); > ! if (md->r2) PyMem_DEL(md->r2); > ! if (md->eptr) PyMem_DEL((char *)md->eptr); > ! if (md->ecode) PyMem_DEL((char *)md->ecode); > return 0; > } 2) _tkinter.c I remember from my experiments that this one was really a mess from a malloc point of view because it caused lots of mixed API calls (once the core allocator was changed). I have to look at it carefully once again... 3) readline.c Neil, what's this? Could you elaborate on this one? > > diff -cr Python-cvs/Modules/readline.c Python-gc/Modules/readline.c > *** Python-cvs/Modules/readline.c Sat Feb 5 09:00:14 2000 > --- Python-gc/Modules/readline.c Sat Feb 5 09:00:47 2000 > *************** > *** 41,46 **** > --- 41,63 ---- > extern int (*PyOS_InputHook)(); > extern char *(*PyOS_ReadlineFunctionPointer) Py_PROTO((char *)); > > + /* convert a string allocated by malloc to one allocated by PyMem_MALLOC */ > + static char * > + malloc2PyMem(char *p) > + { > + char *py; > + int n; > + if (p == NULL) { > + return p; > + } else { > + n = strlen(p); > + py = PyMem_MALLOC(n); > + strncpy(py, p, n+1); > + free(p); > + return py; > + } > + } > + All in all, this patch isn't acceptable for me and I suggest that we prepare another one which respects "the rule", after demystifying some of the issues mentioned above. Note that introducing PyMem_MALLOC would constitute an additional constraint for C Python coders, who're used to malloc/free. In order to pave the way for alternatives to libc's malloc, we must propose a clean solution. Hence it would be good to collect as much opinions/reactions on the subject as possible and settle on an interface which would be convenient for everybody. -- Vladimir MARANGOZOV | Vladimir.Marangozov(a)inrialpes.fr http://sirac.inrialpes.fr/~marangoz | tel:(+33-4)76615277 fax:76615252

21 years, 1 month

FW: [Patches] Complaints new patches list.

by David Ascher

Greg: > On Fri, 11 Feb 2000, Gerrit Holl wrote: > > More than a week ago, I sent a patch for some scripts in the Tools/ > > directory. > [...] > There is *NO* guaranteed response time. If you believe so, then go rethink > things. [...] While I agree with Greg's reaction mostly (Gerrit was being childish, which maybe is not so unappropriate for a 14-year old =), I do think that the patches mailing list mechanism is only appropriate in the short-term. In the longer term, a database backend with automatic notification of interested parties when something 'happens' to a patch is needed. Distributed INBOX management just doesn't work (witness python-help). That said, I'm not able to contribute to the development of such a database anymore than anyone else currently, so the last thing I want to do is complain about the status quo. I just wanted to point out that Gerrit's complaint, while poorly phrased, is real, and is likely to be felt by others in the future. I know I've felt similar frustration sending things to the Scriptics folks and seeing patches dissapear into a black hole. I've just learned diplomacy over the years =). --david

21 years, 1 month

Perl 5.6 beta feature list

by David Ascher

This is from ActiveState, but I believe that all of the new features in Perl are not ActivePerl-specific. FYI. -----Original Message----- Greetings All, ActiveState is pleased to announce the first beta of ActivePerl 5.6. ActivePerl, including PPM, now supports the following platforms: - Windows - Linux - Solaris This beta release includes all the major features that have been discussed for Perl 5.6. - Globalization and Unicode - Concurrent Interpreters - granularity of warnings - New regex construct - Lvalue subroutines - Subroutine attributes - 64-bit platforms - Large File Systems - Additional documentation and tutorials. The following additional new features are available on the Windows platform: - Unicode system calls - Windows Installer - CPAN extensions - fork() For further information see the release notes or readme file. ActiveState will also shortly make available beta versions of the upcoming new releases of the Perl Dev Kit and PerlEx, which will take advantage of the new features in ActivePerl 5.6. ActiveState cautions that the ActivePerl 5.6 beta release may contain certain incomplete features and known bugs. It is made available for testing purposes only. Download a copy of the ActivePerl Beta from: "http://www.ActiveState.com/cgibin/ActivePerl/download.pl" You can submit bug reports at: "http://bugs.ActiveState.com/ActivePerl" You can send feedback on the new features to: "ActivePerl-Beta(a)ActiveState.com" Problems downloading the beta? Send mail to: "WebMaster(a)ActiveState.com" -- The Activators

21 years, 2 months

FW: 64-bit port of Python

by Tim Peters

Ack -- meant to cc PythonDev on this but forgot. -----Original Message----- From: python-list-admin(a)python.org [mailto:python-list-admin@python.org] On Behalf Of Tim Peters Sent: Tuesday, February 08, 2000 11:26 PM To: Trent Mick; python-list(a)python.org; Mark Hammond Cc: Andrew Kuchling Subject: RE: 64-bit port of Python [posted & mailed] [Trent and Tim talk about 64-bit porting issues in the Python code. Trent also sneakily changes email addresses.] [Tim] > AFAIK, core Python code *never* casts a pointer to any sort of > int, or vice versa, either explicitly or implicitly. [Trent Mick [mailto:trentm@ActiveState.com]] > A couple of example where I think the Python core does just that: Good eye, Trent! Thank you. I'm also sending this to Mark Hammond, since ActiveState now pays him to worry about Python's Windows story -- and perhaps pays you too <wink>. > "Modules/arraymodule.c::728": > > static PyObject * > array_buffer_info(self, args) > arrayobject *self; > PyObject *args; > { > return Py_BuildValue("ll", > (long)(self->ob_item), (long)(self->ob_size)); > } > > where 'ob_item' is a pointer. Yes, the author of the new buffer interface code is being shot for many reasons <wink>. > "Python/bltinmodule.c::899": > > static PyObject * > builtin_id(self, args) > PyObject *self; > PyObject *args; > { > PyObject *v; > > if (!PyArg_ParseTuple(args, "O:id", &v)) > return NULL; > return PyInt_FromLong((long)v); > } Oh yes. Been there forever, and won't work at all (while nothing promises that id returns an address, it's crucial that "id(x) == id(y)" iff "x is y" in Python). > Python sort of relies on C's 'long' to be the largest native integer. Don't forget that Python was written pre-ANSI, and this was a common (universal?) assumption in the fuzzier K&R flavor of C. ANSI C went on to guarantee the existence of *some* integral type such that a pointer could be cast to that type and back again without loss of info -- but one committee member told me that at least he was surprised as all heck when it was pointed out that the std neglected to say that must be a *standard* integral type. The notion that "long isn't long enough" is a loophole in the std, and I'm not sure it was an intentional one. Nevertheless, it's an official one now, so that's that. > This is evidenced by all the use of PyInt_FromLong() above. There are > no format specifiers in the PyArg_Parse*() and Py_BuildValue() > functions for converting a pointer. This was fine when 'long' would > do. On Win64 sizeof(long)==4 and size(void*)==8. > > I think this also brings up some wider issues in the Python source. > For instance, the python integer type uses the C 'long' type. Was it > the implicit intention that this be the system's largest native > integral type, i.e. 32-bits on a 32 sys and 64-bits on a 64-bit > system? More the explicit intention that it be the longest standard integral type, back in the days that was believed to "mean something non-trivial". It's been a darned good bet for a decade <wink>. The advertised semantics at the Python level promise only that it's at least 32 bits. > If so, then the representation of the Python integer type will have > to change (i.e. the use of 'long' cannot be relied upon). One should > then carry through and change (or obselete) the *_AsLong(), > *_FromLong() Python/C API functions to becomesomething like > AsLargestNativeInt(), *_FromLargestNativeInt() (or some > less bulky name). > > Alternatively, if the python integer type continues to use the C > 'long' type for 64-bit systems then the following ugly thing > happens: > - A Python integer on a 64-bit Intel chip compiled with MSVC is > 32-bits wide. > - A Python integer on a 64-bit Intel chip compiled with gcc is > 64-bits wide. > That cannot be good. Two things work against all that: 1. In 1.5.2, and more in the current CVS tree, there's already grudging support for "longer than long" via the config LONG_LONG macro (e.g., under MS Windows that's already #defined as __int64). That may spread more, although it's ugly so will be resisted (Guido hates #ifdef'ing code, and platform #ifdef'ed macros aren't exactly liked -- each one is that much more for new ports to wrestle with, and everyone to trip over forever after). 2. It's already not good that int size can matter across platforms with grosser differences than the above. For that & other reasons, the sharp Python-level distinction between (bounded) ints and (unbounded) longs is slated for (backward compatible) death. Andrew Kuchling already has much of the work for that in hand, but unclear whether it will make it into 1.6 (it's not a high priority now, although I expect you just boosted it a bit ...). Once it's in, "id" can return million-bit ints as easily as it returns C longs now. or-if-activestate-solves-this-for-perl-first-we'll-just-rewrite-python- in-that<wink>-ly y'rs - ti -- http://www.python.org/mailman/listinfo/python-list

21 years, 2 months

fyi: tcl/tk 8.3 released

by Greg Stein

http://core.freshmeat.net/news/2000/02/11/950278798.html It may have an impact on the Tkinter stuff... Cheers, -g p.s. the first beta of Perl 5.6 was released a couple days ago, too. but we have no dependencies there :-) -- Greg Stein, http://www.lyra.org/

21 years, 2 months

RE: [Python-Dev] Python -t

by Asbahr, Jason

FYI, cross platform coding using MSDev on Windows and Emacs on Unix does seem to turn up more problems with tabs and spaces. It's a standard here to use a 2 space indent level for tabs (even if it's a tab, MSDev presents it as a two-space indent), not 4 or 8 space indents. Users new to the language have reacted amazingly negatively to errors caused by mix tab/space problems. So we've standardized on spaces and every developer has his or her editor set to generate two actual spaces when the tab key is pressed. Tab-free code. ;-) The tab/space issue is a subtle problem, and as Paul mentiond, we don't need errors caused by it to draw the fire of the anti-whitespace camp. Jason Asbahr Origin Systems, Inc. jasbahr(a)origin.ea.com -----Original Message----- From: Paul Prescod [mailto:paul@prescod.net] Sent: Thursday, February 10, 2000 1:50 PM To: python-dev(a)python.org Subject: Re: [Python-Dev] Python -t Skip Montanaro wrote: > > Actually, I suspect that many people are in the same boat I'm in. I rarely > need to move code I write off of Unix systems. I had to execute "python > --help" yesterday to learn what -t means. Modules that I made public ages > ago with no consideration of the tab devil have elicited nary a peep about > indentation problems from anyone. (Of course, maybe nobody uses them and > I'm simply deluding myself thinking they might be of interest to > someone... ;-) Mixing tabs and spaces is not all that likely to cause problems, I admit. It's mostly in the hands of newbies starting from scratch (and anti-whitespace zealots) that it will be a problem. > Still, if all tabs or all spaces is the way to go and we can be reasonably > sure that most/all people will have an indentation-friendly editor at their > disposal, then perhaps after a period of time -t should be the default. Well everyone in the world has an editor that either does tabs or spaces. Even "modern" (and I use the term VERY LOOSELY) versions of DOS EDIT.COM seem to preserve spaces. It didn't before. Can you imagine that there is still someone at Microsoft tweaking that code? -- Paul Prescod - ISOGEN Consulting Engineer speaking for himself "The calculus and the rich body of mathematical analysis to which it gave rise made modern science possible, but it was the algorithm that made possible the modern world." - from "Advent of the Algorithm" David Berlinski http://www.opengroup.com/mabooks/015/0151003386.shtml _______________________________________________ Python-Dev maillist - Python-Dev(a)python.org http://www.python.org/mailman/listinfo/python-dev

21 years, 2 months

Weekly posting of FAQ index?

by Skip Montanaro

Apologies in advance. This isn't really the right place to bring this up, but I didn't want to open up discussion to the entire c.l.py without a little feedback first. I volunteer to take this off-list (maybe there's a more appropriate list already set up meta-sig?) and come back with a recommendation to the assembled masses here or elsewhere. Many newsgroups post FAQs on a regular basis. I wonder if it wouldn't be such a bad idea to try and figure out how to do that with the Python FAQ. A few issues come to mind: 1. Availability in plain text. Is it? I only every use the FAQ wizard any more, so I don't know. 2. How to keep it from percolating to the mailing list? I doubt people are going to want weekly or monthly mailings of the form "Python FAQ part 1/13" on a regular basis. I suspect the mail/news gateway could be trained to recognize this special message, perhaps if it was cross-posted to one other special newsgroup (misc.answers or whatever catches all the FAQs in the known universe). 3. Maybe a completely different form is needed for periodic posting - a Mini-FAQ - which answers the most egregious questions like "why whitespace indentation?" and "how do I run this thing I just downloaded?", then refers to the real FAQ URL for everything else. Comments? Skip Montanaro | http://www.mojam.com/ skip(a)mojam.com | http://www.musi-cal.com/ "Languages that change by catering to the tastes of non-users tend not to do so well." - Doug Landauer

21 years, 2 months

Python -t

by Paul Prescod

What are the chances of making python -t the default in Python 1.6? It isn't pythonic to silently allow people to do something that almost everyone agrees is wrong. -- Paul Prescod - ISOGEN Consulting Engineer speaking for himself "The calculus and the rich body of mathematical analysis to which it gave rise made modern science possible, but it was the algorithm that made possible the modern world." - from "Advent of the Algorithm" David Berlinski http://www.opengroup.com/mabooks/015/0151003386.shtml

21 years, 2 months

Python publishing

by gvwilson＠nevex.com

Hi, folks. A former colleague of mine is now editing a magazine devoted to scientific computing, and is looking for articles. If you're doing something scientific with Python, and want to tell the world about it, please give me a shout, and I'll forward more information. Thanks, Greg

21 years, 2 months

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

1999

Python-Dev February 2000